A simple example using Perl

The following example shows how to implement the importing of the Netscape Server access log into the EELS database. Although the example conversion script uses Perl, any programming or scripting language will do.

Conversion script

The conversion script shown below takes /usr/ns-home/httpd_80/logs/access and processes it ready for use by eels_log_import(1Meels). It converts each line of the log file into six fields. Each field is separated by ``#'' and each record by ``@''. The fields are:

• LogSystemsSource

• TimeOffset

• OriginatorServiceName

• EventSpecificInformation

• Length

• EventNumber

#!/bin/perl

#
#
# Program:     http_access_format.pl
# Description: Script to convert the access file of http-80 to
#              a format usable by eels_log_import.
# Version:     1.0
# Date:        October 1998
#
#
# WARNING:     If you change the order or number of fields generated by this
#              script, do not forget to update the list in print_output_fields
#
use Time::Local;

$| = 1;

#
# Define some global variables
#
#
# Define where the errors file can be found
#
$HTTP_ACCESS_FILE = '/usr/ns-home/httpd-80/logs/access';

#
#
# Define the months in a hash array. These are used later for a
# reverse conversion of the date.
#
$MONTH{Jan} = 0;
$MONTH{Feb} = 1;
$MONTH{Mar} = 2;
$MONTH{Apr} = 3;
$MONTH{May} = 4;
$MONTH{Jun} = 5;
$MONTH{Jul} = 6;
$MONTH{Aug} = 7;
$MONTH{Sep} = 8;
$MONTH{Oct} = 9;
$MONTH{Nov} = 10;
$MONTH{Dec} = 11;

#
#
# Define the field and record separators.
#
$EOFld = "#";
$EORec = "@";

#
# Just check for one parameter, if it's a -h, /h or h print the
# output field information rather than processing the error file.
# Any other value for ARGV[0] simply ignore.
#
$CmdLP = $ARGV[0];

#
#
# Either print the output format of this script or process the file.
#
if ( $CmdLP eq "-h" || $CmdLP eq "/h" || $CmdLP eq "h" ) {
   &print_output_fields;
}else{
   &process_access_file;
}



exit();

#
#
#
# ----------------------------------------------------------------------------
# Subroutines : process_access_file
#               print_output_fields
#

#
# Description: Read the access file, split it up and then print
#              the results.
# Version:     1.0
#
#
sub process_access_file {

  open (ACCESS_FILE, "$HTTP_ACCESS_FILE");
  while (<ACCESS_FILE>) {

    #
    # If the line starts with format= then ignore it.
    #
    if ( !/^format=/ ) {
      #
      # First escape all hashes, single quotes and double quotes.
      #
      s/#/\\#/g;
      s/\'/\\\'/g;
      s/\"/\\\"/g;

      #
      # Split on the unique key into three variables.
      #
      ($Part_1, $Part_2, $Part_3) = split(/\\"/);

      #
      # Take off any trailing \n.
      #
      chomp $Part_3;

      $Part_1 =~ s/ \+/DUMMYSPACE\+/;

      $LogSystemsSource="http-80 acc";
      ($OriginatorServiceName,$Dummy,$FullDate) = split(/ -/,$Part_1);
      ($FullDate, $Dummy) = split(/DUMMYSPACE/, $FullDate);

      $EventSpecificInformation=$Part_2;

      ($Dummy,$EventNumber,$Length) = split(/ /, $Part_3);

      #
      # Prepare the date for a backward conversion.
      #
      $FullDate =~ s/^ \[//;

      ($mday,$tmpmonth,$rest) = split(/\//, $FullDate);

      ($tmpyear, $hour, $min, $sec) = split(/:/,$rest);

      $mon = $MONTH{$tmpmonth};
      $year = ($tmpyear - 1900);

      $TimeOffset = (timelocal($sec, $min,$hour,$mday,$mon,$year))*1000;

      print $LogSystemsSource;
      print $EOFld;
      print $TimeOffset;
      print $EOFld;
      print $OriginatorServiceName;
      print $EOFld;
      print $EventSpecificInformation;
      print $EOFld;
      print $Length
      print $EOFld;
      print $EventNumber;
      print $EORec;
    }
  }
  close ACCESS_FILE;
}

#
# Description: Print the output fields this script displays
# Version:     1.0
#
#
sub print_output_fields {
   print "\n\nThis script outputs the following fields:\n";
   print "\t\tLogSystemsSource\n";
   print "\t\tTimeOffset\n";
   print "\t\tOriginatorServiceName\n";
   print "\t\tEventSpecificInfomration\n";
   print "\t\tLength\n";
   print "\t\tEventNumber\n";
   print "\nThe field separator is \"$EOFld\" and the record separator is \"";
   if ($EORec eq "\n") {
     print "\\n\"\n\n";
   }else{
     print $EORec."\"\n\n";
   }
}

eels_log_import script

An example of a shell script that calls the conversion script and pipes its output into eels_log_import(1Meels) is shown below:

   http_access_format.pl|/etc/eels/bin/eels_log_import -o \
      "LogSystemsSource TimeOffset OriginatorServiceName \
      EventSpecificInformation Length EventNumber" -r"@" -f"#"