Linux: Logfiles, timestamps and datematching

frank September 11, 2019

I recently came across someting at work that had me bothered for a while. A customer wanted to check some logfiles, where this one specific line is stamped ever so often. If the line is NOT stamped in a timely matter, they wanted to be notified.

Seems simple, right? Well, for me it wasn’t that easy. I rarely do scripts in bash – and when I do, I usually use a lot of time on something that would have taken me five minutes to write in powershell.

Nevertheless, here is what I came up with. Note that this most definitely can be done in simpler and better ways, but this works for me.

So. The logfile is huge, but the guys that “owns” it, only wanted to be notified when a certain string wasn’t beeing stamped in a timely manner.
The string in question is beeing stamped every minute – meaning, a threshold of not beeing written to in the last 10 minutes would do the trick here.

Example of the string we are interested in:

10:24:00,002 DEBUG [scheduled.jobs.ScheduledTasksJob] (EJB default - 4) Starting Scheduled Tasks Job

To manage this, we will be using cat, grep, tail, and awk.

cat to read the logfile
grep to grab the string we are looking for
tail to fetch the last occurcance of this string
awk to fetch the timestamp, which is in the beginning of the string, and split on comma in this case, to only fetch the actual time.
date to do some dateformat-juggeling.

#!/bin/bash

# var
logname="/tmp/file.log"
logcheck=$( cat $logname | grep 'Starting Scheduled Tasks Job' | tail -1 | awk -F ',' '{print $1}' )
logdate=$( date -d $logcheck +%H:%M:%S )
nowminus10=$( date -d "-10 minutes" +%H:%M:%S )
nowplus10=$( date -d "+10 minutes" +%H:%M:%S )
now=$( date +%H:%M:%S )

# construct
if [[ "$logdate" > "$nowminus10" && "$logdate" < "$nowplus10" ]];
then
	echo "HEALTHY: The logfile is beeing written to in a timely manner."
else
	echo "ERROR: The logfile is not beeing written to in a timely manner. The time now: $now. Last timestamp in log: $logdate."
fi

This just echoes out the result – it is up to you to do something useful with the script.

In my case, I use this with SCOM – more on this in the next article.

-F

Powershell: Create Event with parameters

frank September 2, 2019

This function will let you stamp events to the Windows EventLog, and feed the event with filterable parameterdata (which in the cases you use SCOM to sniff events, is pretty awesome).

I use this all the time in cases where a script should dump some kind of result to the eventlog, and using SCOM to fetch the event and trigger on special texts or values in the parameterdata returned.

function createParamEvent ()
  {
    <#
  .SYNOPSIS
  Function for creating events, just like create-event or eventcreate.exe - but with the added functionality to add up to 5 filterable parameters.
    .DESCRIPTION
  The function stamps Windows Events to the Windows Eventlog by your choice, but can also be fed up to 5 different parameters, where param1 contains the basic Event Description, and param2 to param5 contains data of your choosing.
    .EXAMPLE
  For Information events, use eventID 220:
  CreateParamEvent -source "TestSource" -evtID 220 -param1 "The server $hostname shut down at $timestamp" -param2 $hostname -param3 $timestamp -param4 "Some generic text"
  .EXAMPLE
  For Warning events:
  CreateParamEvent -source "TestSource" -evtID 221 -param1 "The server $hostname shut down at $timestamp" -param2 $hostname -param3 $timestamp -param4 "some generic text"
  .EXAMPLE
  For Error events, with the manditory param1 set. All parameters from param2 to param5 are not manditory:
  CreateParamEvent -source "TestSource" -evtID 222 -param1 "The server $hostname shut down at $timestamp"
  .EXAMPLE
  Alle andre eventID'er vil logges som information events.
  .PARAMETER evtID
  Mandatory: The logic in this function is based on the principle where the sample eventIDs (222, 221, 220) will throw an error corresponding to the Event Type (error, warning, information). Other EventIDs can be used, but will then be logged as an Information Event.
  .PARAMETER param1
  Mandatory: The full description in the event.
  .PARAMETER param2-5
  Use theese parameters to add additional useful information to the mix, for example additional information that can be pulled from the Event in SCOM.
  .Link
  https://vetasen.no
  .Notes
  - Param1 = Full description in the event
  - Source is mandetory
  - EventID is mandetory
  - Eventid 222 = Error event
  - Eventid 221 = Warning event
  - Eventid 220 = Information event
  - Param2 to 5 are optional.
  #>
  [Cmdletbinding()]
Param
          (
            [parameter(Mandatory=$true)][string]$evtID,
            [parameter(Mandatory=$true)][string]$param1,
            [parameter(Mandatory=$true)][string]$source,
            [string]$param2,
            [string]$param3,
            [string]$param4,
            [string]$param5
          )
    #Define the event log
    $evtlog = "Application"

    #Load the event source to the log if not already loaded.  This will fail if the event source is already assigned to a different log.
    if ([System.Diagnostics.EventLog]::SourceExists($source) -eq $false) {
        [System.Diagnostics.EventLog]::CreateEventSource($source, $evtlog)
    }

    if ($evtID -eq 222){
    $id = New-Object System.Diagnostics.EventInstance($evtID,1,1); #ERROR EVENT
    }
    elseif ($evtID -eq 221){
    $id = New-Object System.Diagnostics.EventInstance($evtID,1,2); #WARNING EVENT
    }
    else{
    $id = New-Object System.Diagnostics.EventInstance($evtID,1); #INFORMATION EVENT
    }    
    
    
    
    $evtObject = New-Object System.Diagnostics.EventLog;
    $evtObject.Log = $evtlog;
    $evtObject.Source = $source;
    $evtObject.WriteEvent($id, @($param1,$param2,$param3,$param4,$param5))
  }