Posts tagged performance

Reduce Drupal system load caused by 404s

Jul 21st

Posted by Matt Danger in Drupal

Running multiple large Drupal sites on one machine has created some performance anxiety for us. Certain high traffic periods combined with aggressive web crawlers have ground our servers to a halt on a few occasions.

My latest investigation went into cleaning up popular 404 requests on our larger sites. This noticeably reduced disk I/O. I wanted to take 404 a step further and change the way Drupal 404s were handled on our machines. I turned to the Fast 404 module.

Fast 404 works by intercepting the Drupal bootstrap process and performing its own path and file checking. Fast 404 configuration and execution is performed in the site’s settings.php file.

Version 6.x-1.1 did not work immediately out of the box and required a few changes and bug fixes. I have submitted them to the module author.

The whitelisting did not seem to work so I altered line 38 from:

if (in_array($_SERVER['QUERY_STRING'],$allowed)) {
  return TRUE; // URL is whitelisted. Assumed good.
}

To:

foreach ($allowed as $needle) {
  if (stristr($_SERVER['QUERY_STRING'],$needle)) {
    return TRUE; // URL is whitelisted. Assumed good.
  } 
}

Line 111:

$sql = "SELECT path FROM menu_router WHERE '%s' LIKE CONCAT(path,'%')";
$sql2 = "SELECT pid FROM url_alias WHERE '%s' LIKE CONCAT(dst,'%')";
$sql3 = "SELECT rid FROM path_redirect WHERE '%s' LIKE source";

Change to:

$sql = "SELECT path FROM " . $db_prefix['default'] . "menu_router WHERE path LIKE '%%s%'";
$sql2 = "SELECT pid FROM " . $db_prefix['default'] . "url_alias WHERE dst LIKE '%%s%' ";
$sql3 = "SELECT rid FROM " . $db_prefix['default'] . "path_redirect WHERE '%s' LIKE source";

I also changed line 48 to remove duplicate code:

header('HTTP/1.0 404 Not Found');
$fast_404_html = variable_get('fast_404_html','<html xmlns="http://www.w3.org/1999/xhtml"><head><title>404 Not Found</title></head><body><h1>Not Found</h1><p>The requested URL "@path" was not found on this server.</p></body></html>');
print strtr($fast_404_html, array('@path' => check_plain(request_uri())));
exit();

Change to the function that calls the same code:

fast_404_error_return();

One limitation of the Fast 404 module are that it will print a generic “Page Not Found” message on a white page. We would prefer to have a themed landing page similar to Drupal’s normal 404 page. My solution was to create a static 404.html file for the site and tell Fast 404 to print the contents of the file instead of its generic message.

$conf['fast_404_html'] = file_get_contents('./sites/pathtosite/404.html');

I also wanted to keep track of the requests Fast 404 was catching. I added a direct syslog() call. If your Drupal site is not currently configured to use Syslog see my post on setup and configuration. The log format I chose fit our needs, you could format it however you like.

Around line 186, inside fast_404_error_return():

syslog(1, $_SERVER['SERVER_NAME'].' Fast_404 '.$_SERVER['REMOTE_ADDR'].' '.$_SERVER['SCRIPT_URI'].' '.$_SERVER['HTTP_USER_AGENT'].' '.$_SERVER['HTTP_REFERER']);

I think monitored my Drupal log file for false positives and added them to the white list.

Take a look at your Drupal site and see if you have a lot of 404s being served. If so, try this module out and see if load is reduced.

, , , , ,

Setup Syslog on Drupal 6.x and 7.x

Feb 24th

Posted by Matt Danger in Drupal

By design Drupal 6.x sites will log all activity to the database using a core feature called watchdog. Not having to require users to configure permissions for a log file on the server reduces install instructions (and possible errors) and creates a more seamless experience for Drupal users and administrators. Unfortunately, this makes monitoring a large multisite installs more laborious.

Enter Syslog.

Syslog is a local and network logging service that has been around since the 1980s. Most *nix systems come preloaded with Syslog and fortunately for us Drupal has a Syslog module included with the 6.x and 7.x cores. Here’s how to set it up:

1) Install & configure the Syslog module

Go to your module install page, /admin/build/modules and install the Syslog module. Then go to the Syslog settings page, /admin/settings/logging/syslog, and select which Syslog level to attach to the log messages. Choose one that is not in use by Syslog

2) Configure Syslog

Edit /etc/syslog.conf and add:

local0.* /var/log/drupal.log

Then, restart Syslog

3) Disable Watchdog database logging

This will require us to edit the Drupal core (oh noes)
Edit drupal/modules/dblog/dblog.module and comment out the contents of dblog_watchdog() (Around line 132)

Original:

function dblog_watchdog($log = array()) {
  $current_db = db_set_active();
  db_query("INSERT INTO {watchdog}
    (uid, type, message, variables, severity, link, location, referer, hostname, timestamp)
    VALUES
    (%d, '%s', '%s', '%s', %d, '%s', '%s', '%s', '%s', %d)",
    $log['user']->uid,
    $log['type'],
    $log['message'],
    serialize($log['variables']),
    $log['severity'],
    $log['link'],
    $log['request_uri'],
    $log['referer'],
    $log['ip'],
    $log['timestamp']);
 
  if ($current_db) {
    db_set_active($current_db);
  }
}

Change to:

function dblog_watchdog($log = array()) {
/*
  $current_db = db_set_active();
  db_query("INSERT INTO {watchdog}
    (uid, type, message, variables, severity, link, location, referer, hostname, timestamp)
    VALUES
    (%d, '%s', '%s', '%s', %d, '%s', '%s', '%s', '%s', %d)",
    $log['user']->uid,
    $log['type'],
    $log['message'],
    serialize($log['variables']),
    $log['severity'],
    $log['link'],
    $log['request_uri'],
    $log['referer'],
    $log['ip'],
    $log['timestamp']);
 
  if ($current_db) {
    db_set_active($current_db);
  }
*/
}

And you’re done.

, , ,