Incrementally reading log files..

The problem is this: You have a set of log files. You will have to scan the log files periodically and retrieve only new messages. You are allowed to put  a marker in it. The log files switch frequently. Here is what is can be done:

  1. Sort the files according to the modification time.
  2. While there are more files, do:
    1. Put a marker in the syslog file.
    2. Scan for the number of markers.
    3. If the number of markers scanned is zero and the number of markers found so far is also zero, read all the messages in the file.
    4. If the number of markers scanned is 1 and the number of markers scanned so far is zero, it means that this is the first marker being encountered, so read all the messages from the marker to the beginning of the file.
    5. If the number of markers is scanned is 1 and the number of markers scanned so far is also 1, it means that we are actually reading the messages from a log file which has already been read to some extent. Read the messages from the end of the file to the marker and return. No need to scan the files further.
    6. If the number of markers scanned is greater than or equal to 2, then read the messages between the last two markers and return.
    7. If the number of markers scanned so far is greater than or equal to 2, then return, no need to search further.

If we cannot put a marker in the file, then we can do the following:

  1. Have a config file where there are sets of key=value pairs. The key is the file identification (say its inode number or a checksum of the file name) and the value is the number of lines read so far from the file.
  2. When incrementally scanning the files, read the config files and load the key=value pairs in a hash.
  3. For each file:
    1. If the filename’s checksum is found in the hash, get the count of lines read, read the remaining lines and update the count of lines read.
    2. If the filename’s checksum is not found in the has, read all the lines and add an entry in the hash accordingly.
    3. Save the config file back to the disk.
  4. Return.
Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s