If the trouble report is coming from
a user, gather more data before getting into the troubleshooting phase:
- Ask for specific error messages and/or output.
- Ask how the user became aware of the problem when it
first occurred.
- Determine if the problem is ongoing or intermittent
- Ask for detailed steps on how to reproduce the issue.
- If unknown to you, ask the user what the expected
outcome should have been.
- What changes have been made recently?
- If the issue is reported by a user, determine what they
may have changed recently.
- Review all recent changes in available change
management logs, if they exist.
- Review patch management systems to determine if the environment
has been updated.
- Check configuration file time stamps and run
comparisons on configuration files related to the issue.
- If time permits, contact anyone who has access to make
changes to the environment to discover if undocumented changes have
occurred.
- In RHEL, the main log file is /var/log/messages,
this would be the first log to start looking at.
- Not all applications write entries into this log file,
and not all log files are written in the /var/log directory.
- You may have to look at the application documentation
to determine the correct log file location.
- Documentation for most
packages is available in /usr/share/doc/[packagename] for most RHEL
software.
- Using the man command
against the application can also show where log file locations are as
well.
- A quick way to determine which logs are being updated
in /var/log is with the ls -ltr command which lists the
files in a directory by time stamp, the last file listed being the latest.
#
ls - ltr /var/log
- To see kernel and hardware related events you can use
the dmesg command.
- This command will display kernel related events that
have occurred recently.
- The system only keeps a small amount of this data
resident and will overwrite it as new events occur.
- The /var/log/dmesg file contains a snapshot of
dmesg output at boot time, useful for determining what may have happened
to a system hardware wise when the system last started.
- Kernel events should also show up in /var/log/messages,
but are some times harder to find with all the other logging that goes in
that file.
- When dealing with log files, knowing how to parse them
is extremely important.
- The grep command is probably the most used
command to find text in log files.
- The command grep “httpd” /var/log/messages will
return any line containing the string httpd in /var/log/messages.
- Conversely, grep -v “httpd” /var/log/messages
will return any line not containing the string httpd in /var/log/messages.
- The grep command can also be used to parse
output of commands:
#
ps ax | grep init
- More information on grep can be obtained in its man page,
grep --help, or in /usr/share/doc/grep-*.
- Multiple strings can be specified when using grep
by using the -E flag and the pipe (|) character between search strings to
specify the or operator:
# ps ax | grep -E
"findSTRING1|findSTRING2|findSTRING3"
# ps ax | grep -Ev "ignorestring1|ignorestring2"
- Multiple grep commands can be used, in this example;
we return lines with findstring but ignore lines with ignorestring
.
#
ps ax | grep findstring|grep -v ignorestring
|
Table 1: Common grep Options
|
|
Option
|
Function
|
|
-i
|
Perform a case-insensitive search
|
|
-v
|
Exclude lines that contain the pattern
|
|
-c
|
Display a count of lines with the
matching pattern
|
|
-l
|
Only list files names, do not
display the matched lines
|
|
-n
|
Precede matched lines with line
number
|
|
--color
|
Highlight the matched string
|
|
-A, -B
|
When followed by a number, thes options
print that many lines after or before each match. This is useful for seeing
the context in which a match appears within a file.
|
|
-r
|
Perform a recursive search of
files starting with the named directory
|
- The head and tail commands help limit the
amount of data the system administrator has to go through to read files or
parse command output.
- The head -number command will show the
first number lines of command output or of a text file.
#
head -1 /etc/passwd
root:x:0:0:root:/root:/bin/bash
- The tail -number command will show the
last number lines of command output or of a text file.
#
tail -2 /etc/passwd
sshd:x:74:74:Privilege-separated
SSH:/var/empty/sshd:/sbin/nologin
tcpdump:x:72:72::/:/sbin/nologin
- The tail -n +number command will show all
of the lines of command output or of a text file after line number.
- This is useful to remove a header line in command
output:
#
ps aux | tail -n +2root 1 0.0 00 19328 1412 ? Ss May04 0:02 /sbin/initroot 2 0.0 00 0 0 ? Ss May04 0:02 [kthreadd]root 3 0.0 00 0 0 ? Ss May04 0:02 [migration/0]
- To follow a text file as it gets updated in real time
use tail -f (the short command tailf can be used as well):
#
tail -f /var/log/messagesMay 6 18:53:47
nas01 smbd[14012]: failed to retrieve printer list:
NT-STATUS_UNSUCCES
