A problem with strace is that it can only work with one process at a time, so for a multi-process program like Apache this isn’t going to be that useful. In addition, system calls don’t tell us the whole story of what’s happening inside a process. This is why we are going to talk about a tool called perf.

Using perf

If you don’t have perf installed you can install it from the linux-tools package. Once we are ready to start with our little troubleshooting adventure we can start with perf’s record mode.

perf record -F 99 -u www-data

Notice how we passed in the -u option, to tell perf to only record events from the apache user, this is a great way to filter many events that we may not be interested in. Let perf run while the high cpu issue is happening, you can stop it with a CTRL+C once you think you got enough data. The next step is to run perf’s report mode so we can see the output.

perf report

This will take up all our terminal space to show us the results:

The solution

Well if you pay attention this is quite revealing! In the ‘shared object’ column, you can see a lot of ‘libphp5.so’ entries. In fact the top entry in the list is like that, so we can make a good guess that this issue is coming from a php script. If we take a look at our apache error log we find the following, which confirms our suspicion.

/var/log/httpd/error_log:[Mon Feb 16 04:49:08.793848 2015] [:error] [pid 10504] [client 127.0.0.1:47356] PHP Fatal error: Maximum execution time of 20 seconds exceeded in /srv/http/slow.php on line 6

In this case, the culprit code was this:

<?php

$TIMES = 100000000;

for ($i = 0; $i < $TIMES; $i++) {
  echo $i ** $i;
}

Learn more

Perf also has other modes of operation, for example use perf trace for system-wide strace output. Run perf help to get the full list of everything it can do for you.

https://perf.wiki.kernel.org/index.php/Tutorial#Sampling_with_perf_record
http://www.slideshare.net/brendangregg/scale2015-linux-perfprofiling

You might also like:
Troubleshooting with lsof

The post Performance analysis with perf appeared first on Black Bytes.

-a all
-n show ip instead of host names
-t show only tcp connections
-p show process id/name

Another command that will give you similar results is: lsof -nPi

What if you wanted to see the 10 ip addresses with the most connection to your server? You could use a one-liner like this one I came up with:

netstat -ant | grep -i establ | awk -F" " '{print $5}' | cut -d':' -f 1 | sort -n | uniq -c | sort -nr | head -n10

You can also see the connections live as they are being made. For that you can use the “watch” command, which will re-run any command every X seconds (by default 2 seconds) and show you the output. So if you wanted to see all the connections for port 80 updated every five seconds you would do this:

watch -n5 "netstat -antp | grep :80"

There are other tools that also let you see live connections, and even get an idea of the traffic you are getting. These tools are iptraf and iftop . If you need to see the actual data going through your network you will need a packet sniffer like tcpdump or wireshark.

Related posts:
Tshark network forensics

The post Watching your network appeared first on Black Bytes.

Here is a small cheatsheet of the most useful expansions:

Brace expansion

List: {1,2,3}

This one can be useful to rename a file, for example you can rename access.log to access.log-old using this: mv access.log{,-old}

Range: {20..30}

Filename expansion (also know as globbing)

*.sh  -> expands to all file names that have an extension of .sh
[st]* -> expands to all file names that start with either an 's' or a 't'

File expansion is very useful when you need to handle multiple files at the same time. If you need the full path as script input use find instead.

Tip: if you want to use a especial character without it being expanded you can enclose it in quotes, for example: echo ‘test*’

Variable expansion

echo $SHELL -> /usr/bin/zsh
echo $HOME -> /root

Tip: You can use the ‘env’ command to list all your environment variables. You may wanna pipe the output through less.

You can learn more here: http://www.gnu.org/software/bash/manual/html_node/Shell-Expansions.html

The post Bash expansions appeared first on Black Bytes.

/proc is a virtual filesystem that the Linux kernel uses to expose information and allows the user to change some settings at run time. One of the most common uses is to get information about our CPU, we can use ‘cat /proc/cpuinfo’ to see it.

Navigating the proc filesystem

But much more interesting is the fact that all process data is stored in /proc. Each process is stored in the form of a directory with the PID of the process as its name.

Inside we will find all the information we could ever want about one process: its name, working directory, open files, status, and so on. Here is an example of how we can read the process name, using /proc/[pid]/cmdline

We can find the binary and the current directory under /proc/[pid]/exe and /proc/[pid]/cwd. If we do an ls -lh we will notice that these are sym links to the actual files.

Your own custom ps

Using this information we can build our own simplified version of ps using a bash script that loops over all the dirs in /proc and does a cat on cmdline. We can use basename and awk to clean up the results a bit.

for i in $(ls -d /proc/[0-9]* | sort -V);
  do echo -ne "PID $(basename $i)\t" \
  && cat $i/cmdline | awk -F/ '{ if(match($NF,/[a-z]+/)) \
  printf $NF; }' && echo;
done

If you would like to learn more have a look at the kernel documentation here:
https://www.kernel.org/doc/Documentation/filesystems/proc.txt

I hope you have enjoyed this post, please leave a comment if you have anything interesting to say!

The post The /proc filesystem appeared first on Black Bytes.

If we don’t know the name of the process a good start is getting a list of running process. You may turn to top, but I want to propose a better looking tool: htop

Using htop for linux process managament

With htop you get a quick and clean overview of you current system status. You can customize it (color scheme, columns…) by opening the setup (F2), and you can also sort the columns by clicking on the column header.

Another way to visualize your process list is via a process tree. You can either press F5 in htop to enable tree display or use a command like pstree, but my personal favourite is ps axf.

If you already know what you are looking for ( firefox hang maybe? ) the usual approach is greping out the process name from ps, but that’s not as good as it could be, specially because there is already a tool to do exactly this, know by the name of pgrep.

Using pgrep without flags will only get you the PID, which can be useful for scripting, but what some people don’t know is that you can also get the full command line using the -lf flag.

Now it’s time to get rid of those pesky processes. You can either use kill <pid>, killall <name> or pkill <name>, you can also kill a process directly from htop.

I hope that helps you tame your system and have a better experience while using Linux, thanks for passing by!

The post Linux: Process management appeared first on Black Bytes.