Ever seen this in your Apache error log?

1

[notice] child pid 10024 exit signal Segmentation fault (11)

If you have, read on. At my day job I deal with both Apache and PHP a lot. If you have ever tried to figure out why your PHP code seems to cause Apache segmentation faults, you probably experienced the same as I did. Lots of pain, frustration and headaches. However all is not lost. There is a way to figure out what PHP code is making Apache act crazy. I'll try to explain how to debug Apache using gdb to locate that nasty bug that is causing you to lose your precious beauty sleep.

Before we begin

Now, Apache does not dump core by default. We need to do some work before that happens. If you don't like to get your hands dirty compiling stuff manually, you should leave now, this is not for you.

Still here? Good! Before we start, make sure you have:

1. Root access to your web server
2. Apache source code and everything needed to (re-)compile Apache
3. PHP source code (or just download the .gbdinit file)

Make Apache dump core

The first step is to make Apache not change user when it starts up and forks, so we will make it run as root the whole time. To accomplish this we need to compile Apache with -DBIG_SECURITY_HOLE. For obvious reasons, this is not recommended for production.

1
2
3

make clean
export EXTRA_CFLAGS="-DBIG_SECURITY_HOLE"
./configure && make && make install

Now specify the "root" as the "User" in httpd.conf. While we're editing httpd.conf we'll add a setting to specify where Apache should put our core dump.

1
2
3

User nobody
[...]
CoreDumpDirectory /tmp/apache

Make sure /tmp/apache exists

1

mkdir -p /tmp/apache

On most systems the core file size is set to zero by default. We'll go ahead and change it to unlimited.

1

ulimit -c 0

You can check your current core dump file size limit by running

1

ulimit -a

Restart Apache

1

apachectl restart

Next time Apache crashes with a Segmentation fault it should make us a core dump. If you did everything correctly you should see something like this in the apache error_log.

1
2

[notice] child pid 16430 exit signal Segmentation fault (11),  
possible coredump in /tmp/apache

Note: For reasons unclear to me my dump file ended up on the root (/) and not in the directory we specified with CoreDumpDirectory. If anyone knows why please drop me a comment.

Making sense of the core dump

At this point we could run gdb and get a backtrace, however that will only show us the function called inside php itself and not pinpoint where in our php code the problem is. To get a backtrace of our PHP code we need to use the "dump_bt" function inside the .gbdinit file. Copy the .gbdinit file to your home directory

1

cp <php_source>/.gdbinit ~/

Start gdb with the path to httpd from the source as first parameter and the path to the core dump file as second

1
2

cd ~
gdb <apache_source>/src/httpd /tmp/apache/<core_file>

Run bt_dump from .gdbinit:

1

(gdb) dump_bt executor_globals.current_execute_data

Instead of a internal PHP backtrace we should get a nice backtrace of our PHP code.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

[...]
[0xbf75f6fc] query() /home/site/www/lib/ezdb/classes/ezmysqldb.php:767
[0xbf76249c] unlock() /home/site/www/lib/ezdb/classes/ezmysqldb.php:484
[0xbf7627bc] query() /home/site/www/lib/ezdb/classes/ezmysqldb.php:767
[0xbf76555c] unlock() /home/site/www/lib/ezdb/classes/ezmysqldb.php:484
[0xbf76587c] query() /home/site/www/lib/ezdb/classes/ezmysqldb.php:767
[0xbf76861c] unlock() /home/site/www/lib/ezdb/classes/ezmysqldb.php:484
[0xbf76893c] query() /home/site/www/lib/ezdb/classes/ezmysqldb.php:767
[0xbf76b6dc] unlock() /home/site/www/lib/ezdb/classes/ezmysqldb.php:484
[0xbf76b9fc] query() /home/site/www/lib/ezdb/classes/ezmysqldb.php:767
[0xbf76e79c] unlock() /home/site/www/lib/ezdb/classes/ezmysqldb.php:484
[0xbf76eabc] query() /home/site/www/lib/ezdb/classes/ezmysqldb.php:767
[0xbf77185c] unlock() /home/site/www/lib/ezdb/classes/ezmysqldb.php:484
[0xbf771b7c] query() /home/site/www/lib/ezdb/classes/ezmysqldb.php:767
[...]

In this example we can clearly see there is a recursive problem with query() and unlock() on line 767 and 484 in ezmysqldb.php. With the help of this backtrace in only took a quick look in ezmysqldb.php before I had a bug report written and posted: http://issues.ez.no/11038.

Thanks to my good friend and co-worker Derick Rethans for telling me about the "dump_bt" function.

SHARE! Magazine is a monthly newsletter about the eZ Ecosystem. It features interviews, pictures, articles, reviews and content related to Open Source software, business and development.

The services delivered by (svn.)projects.ez.no will be unavailable on Wednesday the 5th of March 2008 from approximately 8:00 to 9:00 CET. The server needs to be moved to a new location.

During the PHP London conference (slides are here as usual), a number of British people were "complaining" that PHP's strtotime() and date_create() /new DateTime() functions do not understand the British date format "dd/mm/yyyy". Historically, the "English" text to date parsers in PHP only accept the illogical American format of "mm/dd/yyyy". Having seen this problem before, I recently added a new function/method to PHP's Date/Time support to address exactly this issue. From PHP 5.3 the new date_create_from_format() function and the DateTime::createFromFormat() factory method are available. As first argument they accept the expected format, and as second argument the string to parse. To parse a British date string, you would for example use:

<?php
$dt = date_create_from_format( 'd/m/Y', "02/03/2008" );
echo $dt->format( 'd/m/Y' ), "\n";
?>

In case the passed string does not match the format, the function will return false. With the also new function date_get_last_errors() method you then can request more information about which part of the string could not be passed. The following example illustrates that:

<?php
$dt = date_create_from_format( 'Y-m-d', "02/03/2008" );
if ( !$dt )
{
    $errors = date_get_last_errors();
    var_dump( $errors['errors']);
}
?>

The $errors contains a list of errors, indexed by the position in the string where the error was found. The above script would produce:

array(3) {
  [2]=>
  string(22) "Unexpected data found."
  [5]=>
  string(22) "Unexpected data found."
  [8]=>
  string(13) "Trailing data"
}

The format specifiers are mostly similar to the ones used for the date() function. Documentation on the new functions should make it into the documentation soon.

eZ Rest is an extension for easily creating REST web services with eZ Publish. This extension has already been used for several projects internally at eZ Systems, with great success.