Provide steps to normalize timestamps in an apache log file.

Similar to what was done with the applicationX_in_epoch.log file, Apache log files can also be normal-
ized. Follow the steps below to convert Unix Epoch to Human Readable timestamps. Consider the fol-
lowing Apache log file, apache_in_epoch.log:



[analyst@secOps lab.support.files]$ cat apache_in_epoch.log

198.51.100.213 - - [1219071600] "GET /twiki/bin/edit/Main/Double_bounce_

sender?topicparent=Main.ConfigurationVariables HTTP/1.1" 401 12846

198.51.100.213 - - [1219158000] "GET /twiki/bin/rdiff/TWiki/NewUserTemplate?rev1=1.3

&rev2=1.2 HTTP/1.1" 200 4523

198.51.100.213 - - [1220799600] "GET /mailman/listinfo/hsdivision HTTP/1.1" 200 6291

198.51.100.213 - - [1220886000] "GET /twiki/bin/view/TWiki/WikiSyntax HTTP/1.1" 200

7352

198.51.100.213 - - [1220972400] "GET /twiki/bin/view/Main/DCCAndPostFix HTTP/1.1"

200 5253

198.51.100.213 - - [1221058800] "GET /twiki/bin/oops/TWiki/AppendixFileSystem?templa

te=oopsmore&m1=1.12&m2=1.12 HTTP/1.1" 200 11382

The Apache log file above contains six entries which record events related to the Apache
web server. Each entry has seven fields. The fields are delimited by a space:

? The first column contains the IPv4 address, 198.51.100.213, of the web client plac-
ing the request.

? The second and third columns are not used and a “-” character is used to represent no
value.
? The fourth column contains the timestamp in Unix Epoch time, for example
[1219071600].
? The fifth column contains text with details about the event, including URLs and web
request parameters. All six entries are HTTP GET messages. Because these messages
include spaces, the entire field is enclosed with quotes.
? The sixth column contains the HTTP status code, for example 401.
? The seventh column contains the size of the response to the client (in bytes), for
example 12846.
Similar to part one, a script will be created to convert the timestamp from Epoch to
Human Readable.

a. First, answer the questions below. They are crucial for the construction of the script.
In the context of timestamp conversion, what character would work as a good delimiter
character for the Apache log file above?
____________________________________________________________________________
The space character.
How many columns does the Apache log file above contain?
____________________________________________________________________________
7
In the Apache log file above, what column contains the Unix Epoch timestamp?
____________________________________________________________________________
Column 4

b. In the CyberOps Workstation VM terminal, a copy of the Apache log file, apache_in_

epoch.log, is stored in the /home/analyst/lab.support.files.

c. Use an awk script to convert the timestamp field to a human readable format. Notice
that the command contains the same script used previously, but with a few adjustments
for the timestamp field and file name.



[analyst@secOps lab.support.files]$ awk 'BEGIN {FS=OFS=" "}

{$4=strftime("%c",$4)} {print}' /home/analyst/lab.support.files/apache_in_

epoch.log

Was the script able to properly convert the timestamps? Describe the output.
____________________________________________________________________________
No. All timestamps are now Wed 31 Dec 1969 07:00:00 PM EST.

d. Before moving forward, think about the output of the script. Can you guess what
caused the incorrect output? Is the script incorrect? What are the relevant differences
between the applicationX_in_epoch.log and apache_in_epoch.log?
____________________________________________________________________________
____________________________________________________________________________
____________________________________________________________________________
The problem is the square brackets in the course file. The script expects the timestamp
to be in the Unix Epoch format which does not include the square brackets. Because
the script does not know what number represents the “[” character, it assumes zero and
returns the Unix beginning of time in UTC -5.

e. To fix the problem, the square brackets must be removed from the timestamp field
before the conversion takes place. Adjust the script by adding two actions before the
conversion, as shown below:



[analyst@secOps lab.support.files]$ awk 'BEGIN {FS=OFS=" "}{gsub(/\

[|\]/,"",$4)}{print}{$4=strftime("%c",$4)}{print}' apache_in_epoch.log

Notice after specifying space as the delimiter with {FS=OFS=” “}, there is a regular

expression action to match and replace the square brackets with an empty string, effec-
tively removing the square brackets that appear in the timestamp field. The second

action prints the updated line so the conversion action can be performed.
? gsub() – This is an internal AWK function used to locate and substitute
strings. In the script above, gsub() received three comma-separated parameters,
described below.
? /\[|\]/ – This is a regular expression passed to gsub() as the first parameter. The
regular expression should be read as ‘find “[” OR “]”’. Below is the breakdown
of the expression:
? The first and last “/” character marks the beginning and end of the search
block. Anything between the first “/” and the second “/” are related to the
search. The “\” character is used to escape the following “[”. Escaping is

necessary because “[” can also be used by an operator in regular expres-
sions. By escaping the “[“ with a leading “\”, we tell the interpreter that the

“]” is part of the content and not an operator. The “|” character is the OR
operator. Notice that the “|” is not escaped and will therefore, be seen as an
operator. Lastly, the regular expression escapes the closing square bracket
with “\]”, as done before.
? “” – This represents no characters, or an empty string. This parameter tells gsub()
what to replace the “[” and “]” with, when found. By replacing the “[” and “]” with
“”, gsub() effectively removes the “[” and “]” characters.
? $4 – This tells gsub() to work only on the fourth column of the current line, the
timestamp column.
Note: Regular expression interpretation is a SECOPS exam topic. Regular expressions are covered in
more detail in another lab in this chapter. However, you may wish to search the Internet for tutorials.
f. In a CyberOps Workstation VM terminal, execute the adjusted script, as follows:



[analyst@secOps lab.support.files]$ awk 'BEGIN {FS=OFS=" "}{gsub(/\

[|\]/,"",$4)}{print}{$4=strftime("%c",$4)}{print}' apache_in_epoch.log

Was the script able to properly convert the timestamps this time? Describe the output.
____________________________________________________________________________
____________________________________________________________________________
____________________________________________________________________________
Yes. The output now displays two lines for each log entry. The first line displays the
timestamp in Unix Epoch format and the second line is the same log entry with the
timestamp displayed using Human Readable format.