Matt [Danger] » Drupal

Reduce Drupal system load caused by 404s

Matt Danger — Fri, 22 Jul 2011 04:30:27 +0000

Running multiple large Drupal sites on one machine has created some performance anxiety for us. Certain high traffic periods combined with aggressive web crawlers have ground our servers to a halt on a few occasions.

My latest investigation went into cleaning up popular 404 requests on our larger sites. This noticeably reduced disk I/O. I wanted to take 404 a step further and change the way Drupal 404s were handled on our machines. I turned to the Fast 404 module.

Fast 404 works by intercepting the Drupal bootstrap process and performing its own path and file checking. Fast 404 configuration and execution is performed in the site’s settings.php file.

Version 6.x-1.1 did not work immediately out of the box and required a few changes and bug fixes. I have submitted them to the module author.

The whitelisting did not seem to work so I altered line 38 from:

if (in_array($_SERVER['QUERY_STRING'],$allowed)) {
  return TRUE; // URL is whitelisted. Assumed good.
}

To:

foreach ($allowed as $needle) {
  if (stristr($_SERVER['QUERY_STRING'],$needle)) {
    return TRUE; // URL is whitelisted. Assumed good.
  } 
}

Line 111:

$sql = "SELECT path FROM menu_router WHERE '%s' LIKE CONCAT(path,'%')";
$sql2 = "SELECT pid FROM url_alias WHERE '%s' LIKE CONCAT(dst,'%')";
$sql3 = "SELECT rid FROM path_redirect WHERE '%s' LIKE source";

Change to:

$sql = "SELECT path FROM " . $db_prefix['default'] . "menu_router WHERE path LIKE '%%s%'";
$sql2 = "SELECT pid FROM " . $db_prefix['default'] . "url_alias WHERE dst LIKE '%%s%' ";
$sql3 = "SELECT rid FROM " . $db_prefix['default'] . "path_redirect WHERE '%s' LIKE source";

I also changed line 48 to remove duplicate code:

header('HTTP/1.0 404 Not Found');
$fast_404_html = variable_get('fast_404_html','404 Not FoundNot Found
The requested URL "@path" was not found on this server.
');
print strtr($fast_404_html, array('@path' => check_plain(request_uri())));
exit();

Change to the function that calls the same code:

fast_404_error_return();

One limitation of the Fast 404 module are that it will print a generic “Page Not Found” message on a white page. We would prefer to have a themed landing page similar to Drupal’s normal 404 page. My solution was to create a static 404.html file for the site and tell Fast 404 to print the contents of the file instead of its generic message.

$conf['fast_404_html'] = file_get_contents('./sites/pathtosite/404.html');

I also wanted to keep track of the requests Fast 404 was catching. I added a direct syslog() call. If your Drupal site is not currently configured to use Syslog see my post on setup and configuration. The log format I chose fit our needs, you could format it however you like.


Around line 186, inside fast_404_error_return():

syslog(1, $_SERVER['SERVER_NAME'].' Fast_404 '.$_SERVER['REMOTE_ADDR'].' '.$_SERVER['SCRIPT_URI'].' '.$_SERVER['HTTP_USER_AGENT'].' '.$_SERVER['HTTP_REFERER']);

I think monitored my Drupal log file for false positives and added them to the white list.
Take a look at your Drupal site and see if you have a lot of 404s being served. If so, try this module out and see if load is reduced.



Setup Syslog on Drupal 6.x and 7.x
Matt Danger — Thu, 24 Feb 2011 21:13:07 +0000
By design Drupal 6.x sites will log all activity to the database using a core feature called watchdog. Not having to require users to configure permissions for a log file on the server reduces install instructions (and possible errors) and creates a more seamless experience for Drupal users and administrators. Unfortunately, this makes monitoring a large multisite installs more laborious.
Enter Syslog.
Syslog is a local and network logging service that has been around since the 1980s. Most *nix systems come preloaded with Syslog and fortunately for us Drupal has a Syslog module included with the 6.x and 7.x cores. Here’s how to set it up:
1) Install & configure the Syslog module
Go to your module install page, /admin/build/modules and install the Syslog module. Then go to the Syslog settings page, /admin/settings/logging/syslog, and select which Syslog level to attach to the log messages. Choose one that is not in use by Syslog
2) Configure Syslog
Edit /etc/syslog.conf and add:

local0.* /var/log/drupal.log

Then, restart Syslog
3) Disable Watchdog database logging
This will require us to edit the Drupal core (oh noes)

Edit drupal/modules/dblog/dblog.module and comment out the contents of dblog_watchdog() (Around line 132)
Original:

function dblog_watchdog($log = array()) {
  $current_db = db_set_active();
  db_query("INSERT INTO {watchdog}
    (uid, type, message, variables, severity, link, location, referer, hostname, timestamp)
    VALUES
    (%d, '%s', '%s', '%s', %d, '%s', '%s', '%s', '%s', %d)",
    $log['user']->uid,
    $log['type'],
    $log['message'],
    serialize($log['variables']),
    $log['severity'],
    $log['link'],
    $log['request_uri'],
    $log['referer'],
    $log['ip'],
    $log['timestamp']);
 
  if ($current_db) {
    db_set_active($current_db);
  }
}

Change to:

function dblog_watchdog($log = array()) {
/*
  $current_db = db_set_active();
  db_query("INSERT INTO {watchdog}
    (uid, type, message, variables, severity, link, location, referer, hostname, timestamp)
    VALUES
    (%d, '%s', '%s', '%s', %d, '%s', '%s', '%s', '%s', %d)",
    $log['user']->uid,
    $log['type'],
    $log['message'],
    serialize($log['variables']),
    $log['severity'],
    $log['link'],
    $log['request_uri'],
    $log['referer'],
    $log['ip'],
    $log['timestamp']);
 
  if ($current_db) {
    db_set_active($current_db);
  }
*/
}

And you’re done.



Use Drush to update a module in a Drupal multisite environment
Matt Danger — Tue, 11 Jan 2011 03:34:59 +0000
I love Drush. It makes managing my Drupal multisite installs much less complicated. I used to write custom scripts to loop through all the sites an perform actions but this little bit of code has made my maintenance tasks so much simpler.

for SITE in `ls /path/to/drupal/sites | grep "\."`;
do
/path/to/drush --uri=$SITE --root=/path/to/drupal --yes updatedb
done




Making database changes to Drupal multi-sites
Matt Danger — Tue, 22 Dec 2009 03:29:22 +0000
The company I work for hosts over 150 Drupal sites in a multi-site configuration and every so often I need to publish a change to the code that requires a database change. The large number of sites we host means that performing manual changes would require too much time and carries a chance of error. I’ve come up with a simple solution.
I first create a PHP script that bootstraps Drupal. Here is an example of a script that will rebuild the menu router tables.

// Bootstrap
require_once('includes/bootstrap.inc');
require_once('includes/common.inc');
drupal_bootstrap(DRUPAL_BOOTSTRAP_FULL);
 
// Rebuild menus
menu_rebuild();
print 'Menu rebuild complete\n';

I name this script rebuild_menus.php and upload it to my Drupal directory. Then, I create a text file on my desktop that contains a list of all the Drupal websites. This is an abbreviated example:

http://example1.com
http://example2.com
http://example3.com
http://example4.com

Finally, I execute the script on all the sites by running the following command:
for url in `cat list.txt`; do echo "Running script on $url"; curl $url/rebuild_menus.php; echo; done



Programmatically create users in Drupal 6.x
Matt Danger — Thu, 17 Dec 2009 02:30:49 +0000
You can create a Drupal user by sending user_save() a null parameter and an array of user information. You can also assign a role by setting an array that is keyed with the role ID (rid). This can be obtained by calling user_roles() as shown in the example below.

// Get an array of roles
$roles = user_roles();
 
// Basic account information
$user = array(
  'name' => 'Some User',
  'pass' => 'some_password',
  'mail' => 'user@example.com',
  'status' => 1,
  'init' => 'user@example.com',
  'roles' => array(array_search('some_role', $roles) => 1),
);
 
// See if the user exists by calling Drupal's user_load()
$existing_user = user_load(array('name' => $user['name']));
 
if (!$existing_user->uid) {
 
  // Save the user
  $user = user_save(NULL, $user);
 
}