Provide steps to normalize timestamps in a log file.

Timestamps are used in log entries to specify when the recorded event took place. While it is best prac-
tice to record timestamps in UTC, the format of the timestamp varies from log source to log source.

There are two common timestamp formats, known as Unix Epoch and Human Readable.
Unix Epoch timestamps record time by measuring the number of seconds that have passed since
January 1st 1970.
Human Readable timestamps record time by representing separate values for year, month, day, hour,
minute, and second.
The Human Readable Wed, 28 Jun 2017 13:27:18 GMT timestamp is the same as 1498656439 in
Unix Epoch.

From a programmability stand point, it is much easier to work with Epoch as it allows for easier addi-
tion and subtraction operations. From an analysis perspective, however, Human Readable timestamps

are much easier to interpret.
Converting Epoch to Human Readable Timestamps with AWK
AWK is a programming language designed to manipulate text files. It is very powerful and especially

useful when handling text files where the lines contain multiple fields, separated by a delimiter charac-
ter. Log files contain one entry per line and are formatted as delimiter-separated fields, making AWK a

great tool for normalizing.
Consider the applicationX_in_epoch.log file below. The source of the log file is not relevant.



2|Z|1219071600|AF|0

3|N|1219158000|AF|89

4|N|1220799600|AS|12

1|Z|1220886000|AS|67

5|N|1220972400|EU|23

6|R|1221058800|OC|89

The log file above was generated by application X. The relevant aspects of the file are:
? The columns are separated, or delimited, by the | character. Therefore, the file has five columns.
? The third column contains timestamps in Unix Epoch.
? The file has an extra line at the end. This will be important later in the lab.
Assume that a log analyst needed to convert the timestamps to the Human Readable format. Follow the
steps below to use AWK to easily perform the manual conversion:

a. Launch the CyberOps Workstation VM and then launch a terminal window.
b. Use the cd command to change to the /home/analyst/lab.support.files/ directory. A

copy of the file shown above is stored there.



[analyst@secOps ~]$ cd ./lab.support.files/

[analyst@secOps lab.support.files]$ ls -l

total 580

-rw-r--r-- 1 analyst analyst 649 Jun 28 18:34 apache_in_epoch.log

-rw-r--r-- 1 analyst analyst 126 Jun 28 11:13 applicationX_in_epoch.log

drwxr-xr-x 4 analyst analyst 4096 Aug 7 15:29 attack_scripts

-rw-r--r-- 1 analyst analyst 102 Jul 20 09:37 confidential.txt



[analyst@secOps lab.support.files]$

c. Issue the following AWK command to convert and print the result on the terminal:
Note: It is easy to make a typing error in the following script. Consider copying the script out to a text
editor to remove the extra line breaks. Then copy the script from the text editor into the CyberOps
Workstation VM terminal window. However, be sure to study the script explanation below to learn
how this script modifies the timestamp field.



[analyst@secOps lab.support.files]$ awk 'BEGIN {FS=OFS="|"}

{$3=strftime("%c",$3)} {print}' applicationX_in_epoch.log

2|Z|Mon 18 Aug 2008 11:00:00 AM EDT|AF|0

3|N|Tue 19 Aug 2008 11:00:00 AM EDT|AF|89

4|N|Sun 07 Sep 2008 11:00:00 AM EDT|AS|12

1|Z|Mon 08 Sep 2008 11:00:00 AM EDT|AS|67

5|N|Tue 09 Sep 2008 11:00:00 AM EDT|EU|23

6|R|Wed 10 Sep 2008 11:00:00 AM EDT|OC|89

||Wed 31 Dec 1969 07:00:00 PM EST

[analyst@secOps lab.support.files]$

The command above is an AWK script. It may seem complicated. The main structure of
the AWK script above is as follows:
? awk – This invokes the AWK interpreter.
? ‘BEGIN – This defines the beginning of the script.
? {} – This defines actions to be taken in each line of the input text file. An AWK
script can have several actions.
? FS = OFS = “|” – This defines the field separator (i.e., delimiter) as the bar (|)
symbol. Different text files may use different delimiting characters to separate
fields. This operator allows the user to define what character is used as the field
separator in the current text file.

? $3 – This refers to the value in the third column of the current line. In the appli-
cationX_in_epoch.log, the third column contains the timestamp in epoch to be

converted.
? strftime – This is an AWK internal function designed to work with time. The %c
and $3 in between parenthesis are the parameters passed to strftime.
? applicationX_in_epoch.log – This is the input text file to be loaded and used.
Because you are already in the lab.support.files directory, you do not need to
add path information, /home/analyst/lab.support.files/applicationX_in_epoch.
log.
The first script action, defined in the first set of curly brackets, is to define the field
separator character as the “|”. Then, in the second set of curly brackets, it rewrites the
third column of each line with the result of the execution of the strftime() function.
strftime() is an internal AWK function created to handle time conversion. Notice that
the script tells the function to use the contents of the third column of each line before
the change ($3) and to format the output (%c).
Were the Unix Epoch timestamps converted to Human Readable format? Were the
other fields modified? Explain.
Yes, the script converted from Epoch to Human Readable. The script changed only the
timestamp field, preserving the rest of the file.
Compare the contents of the file and the printed output. Why is there the line, ||Wed
31 Dec 1969 07:00:00 PM EST?
The reason for the extra line is because the file has an empty line at the end, which led the
script to mistakenly interpret it as 0 and convert that into a Human Readable timestamp.
By interpreting the empty line as 0, the script converted 0 Unix Epoch to Human
Readable. 0 Unix Epoch translates to 0 seconds after midnight of Jan 1st, 1970. The
script displays “Wed 31 Dec 1969 07:00:00 PM EST” because it automatically adjusts
for the timezone. Because the CyberOps Workstation is configured for EST (UTC -5),
the script displays the midnight, Jan 1st 1970 minus 5 hours.

d. Use nano (or your favorite text editor) to remove the extra empty line at the end of the

file and run the AWK script again.



[analyst@secOps lab.support.files]$ nano applicationX_in_epoch.log

Is the output correct now? Explain.
Yes. Because the empty line was removed, no extra data was created and added to the
log file by the script.

e. While printing the result on the screen is useful for troubleshooting the script, analysts
will likely need to save the output in a text file. Redirect the output of the script above
to a file named applicationX_in_human.log to save it to a file:



[analyst@secOps lab.support.files]$ awk 'BEGIN {FS=OFS="|"}

{$3=strftime("%c",$3)} {print}' applicationX_in_epoch.log > applicationX_in_

human.log

[analyst@secOps lab.support.files]$

What was printed by the command above? Is this expected?
Nothing was printed on the screen. Yes, it is expected, as the command output was
redirected to a text file named applicationX_in_human.log.

f. Use cat to view the applicationX_in_human.log. Notice that the extra line is now
removed and the timestamps for the log entries have been converted to human readable
format.



[analyst@secOps lab.support.files]$ cat applicationX_in_human.log

2|Z|Mon 18 Aug 2008 11:00:00 AM EDT|AF|0

3|N|Tue 19 Aug 2008 11:00:00 AM EDT|AF|89

4|N|Sun 07 Sep 2008 11:00:00 AM EDT|AS|12

1|Z|Mon 08 Sep 2008 11:00:00 AM EDT|AS|67

5|N|Tue 09 Sep 2008 11:00:00 AM EDT|EU|23

6|R|Wed 10 Sep 2008 11:00:00 AM EDT|OC|89

[analyst@secOps lab.support.files]$