I recently came across someting at work that had me bothered for a while. A customer wanted to check some logfiles, where this one specific line is stamped ever so often. If the line is NOT stamped in a timely matter, they wanted to be notified.
Seems simple, right? Well, for me it wasn’t that easy. I rarely do scripts in bash – and when I do, I usually use a lot of time on something that would have taken me five minutes to write in powershell.
Nevertheless, here is what I came up with. Note that this most definitely can be done in simpler and better ways, but this works for me.
So. The logfile is huge, but the guys that “owns” it, only wanted to be notified when a certain string wasn’t beeing stamped in a timely manner.
The string in question is beeing stamped every minute – meaning, a threshold of not beeing written to in the last 10 minutes would do the trick here.
Example of the string we are interested in:
10:24:00,002 DEBUG [scheduled.jobs.ScheduledTasksJob] (EJB default - 4) Starting Scheduled Tasks Job
To manage this, we will be using cat, grep, tail, and awk.
cat to read the logfile
grep to grab the string we are looking for
tail to fetch the last occurcance of this string
awk to fetch the timestamp, which is in the beginning of the string, and split on comma in this case, to only fetch the actual time.
date to do some dateformat-juggeling.
#!/bin/bash # var logname="/tmp/file.log" logcheck=$( cat $logname | grep 'Starting Scheduled Tasks Job' | tail -1 | awk -F ',' '{print $1}' ) logdate=$( date -d $logcheck +%H:%M:%S ) nowminus10=$( date -d "-10 minutes" +%H:%M:%S ) nowplus10=$( date -d "+10 minutes" +%H:%M:%S ) now=$( date +%H:%M:%S ) # construct if [[ "$logdate" > "$nowminus10" && "$logdate" < "$nowplus10" ]]; then echo "HEALTHY: The logfile is beeing written to in a timely manner." else echo "ERROR: The logfile is not beeing written to in a timely manner. The time now: $now. Last timestamp in log: $logdate." fi
This just echoes out the result – it is up to you to do something useful with the script.
In my case, I use this with SCOM – more on this in the next article.
-F