Tag Archives for " performance "
Performance analysis with perf
Is Apache or some other process using most of your CPU but you have no idea why? I have the solution for you! Linux has some low-level troubleshooting tools that will let you investigate the issue and find out what’s going on. One of those tools is strace, which will capture all the system calls made by one process.
A problem with strace is that it can only work with one process at a time, so for a multi-process program like Apache this isn’t going to be that useful. In addition, system calls don’t tell us the whole story of what’s happening inside a process. This is why we are going to talk about a tool called perf.
Using perf
If you don’t have perf installed you can install it from the linux-tools package. Once we are ready to start with our little troubleshooting adventure we can start with perf’s record mode.
|
1 |
perf record -F 99 -u www-data |
Notice how we passed in the -u option, to tell perf to only record events from the apache user, this is a great way to filter many events that we may not be interested in. Let perf run while the high cpu issue is happening, you can stop it with a CTRL+C once you think you got enough data. The next step is to run perf’s report mode so we can see the output.
|
1 |
perf report |
This will take up all our terminal space to show us the results:
The solution
Well if you pay attention this is quite revealing! In the ‘shared object’ column, you can see a lot of ‘libphp5.so’ entries. In fact the top entry in the list is like that, so we can make a good guess that this issue is coming from a php script. If we take a look at our apache error log we find the following, which confirms our suspicion.
/var/log/httpd/error_log:[Mon Feb 16 04:49:08.793848 2015] [:error] [pid 10504] [client 127.0.0.1:47356] PHP Fatal error: Maximum execution time of 20 seconds exceeded in /srv/http/slow.php on line 6
In this case, the culprit code was this:
|
1 2 3 4 5 6 7 |
<?php $TIMES = 100000000; for ($i = 0; $i < $TIMES; $i++) { echo $i ** $i; } |
Learn more
Perf also has other modes of operation, for example use perf trace for system-wide strace output. Run perf help to get the full list of everything it can do for you.
https://perf.wiki.kernel.org/index.php/Tutorial#Sampling_with_perf_record
http://www.slideshare.net/brendangregg/scale2015-linux-perfprofiling
You might also like:
Troubleshooting with lsof
MySQL – Slow query analysis
Ever wonder why your application is slow? Then there is a good chance you got slow queries! In this post, you will learn how to enable the slow query log and analyze it to find how what you need to optimize.
The mysql slow log
MySQL (and probably other databases as well) have this feature that allows you to log queries that take too long to execute. It’s disabled by default (in version 5.6) and to enable it we will need to edit the configuration file. Normally this file is located at /etc/mysql/my.cnf
We will add the following lines inside the [mysqld] block:
|
1 2 3 |
slow_query_log = 1 # Enable the slow query log, 0 to disable. long_query_time = 1 # Seconds until a query is considered slow. slow_query_log_file = /var/log/mysql/slow-query.log |
Make sure that /var/log/mysql exist, if not create it and change permissions so mysql can write to it. Next you will need to restart the server for the changes to take effect, if the server doesn’t come up you may have a syntax error in your config file.
If this is a dev environment you will need to stress you app a bit to generate some data and then you can jump to the next section.
Log analysis
You could just open the log directly, but it’s not the most pretty thing to look at, so we are going to use a tool from the Percona toolkit. You can either download the full toolkit or just the specific tool that you need.
|
1 2 |
wget http://percona.com/get/percona-toolkit.tar.gz wget http://percona.com/get/TOOL |
In this case, the tool we need is pt-query-digest which will parse the query log and show us some statistics and detailed info about our queries.
|
1 |
pt-query-digest /var/log/mysql/slow-query.log |
The top queries in the list are probably a good target for optimization.
But what if you can’t change your MySQL server settings? Well this tool has you covered! It has 2 other ways it can read queries from: tcpdump output and the ‘show process’ command.
To use the tcpdump output option you need to do this:
|
1 2 |
tcpdump -s 0 -x -nn -q -tttt -i any -c 1000 > tcpdump.output pt-query-digest --type tcpdump tcpdump.output |
And for the process option:
|
1 |
pt-query-digest --processlist h=host_name |
If you would like to learn more about MySQL administration and performance I would recommend checking out Percona’s site which has plenty of free resources to help you.
I hope you found that useful, please leave a comment if you have any questions or anything interesting to add.