Matt [Danger]
One step closer to world domination
Facebook-Status-2.2.pl released
Sep 7th
Reduce Drupal system load caused by 404s
Jul 21st
Running multiple large Drupal sites on one machine has created some performance anxiety for us. Certain high traffic periods combined with aggressive web crawlers have ground our servers to a halt on a few occasions.
My latest investigation went into cleaning up popular 404 requests on our larger sites. This noticeably reduced disk I/O. I wanted to take 404 a step further and change the way Drupal 404s were handled on our machines. I turned to the Fast 404 module.
Fast 404 works by intercepting the Drupal bootstrap process and performing its own path and file checking. Fast 404 configuration and execution is performed in the site’s settings.php file.
Version 6.x-1.1 did not work immediately out of the box and required a few changes and bug fixes. I have submitted them to the module author.
The whitelisting did not seem to work so I altered line 38 from:
if (in_array($_SERVER['QUERY_STRING'],$allowed)) { return TRUE; // URL is whitelisted. Assumed good. }
To:
foreach ($allowed as $needle) { if (stristr($_SERVER['QUERY_STRING'],$needle)) { return TRUE; // URL is whitelisted. Assumed good. } }
Line 111:
$sql = "SELECT path FROM menu_router WHERE '%s' LIKE CONCAT(path,'%')"; $sql2 = "SELECT pid FROM url_alias WHERE '%s' LIKE CONCAT(dst,'%')"; $sql3 = "SELECT rid FROM path_redirect WHERE '%s' LIKE source";
Change to:
$sql = "SELECT path FROM " . $db_prefix['default'] . "menu_router WHERE path LIKE '%%s%'"; $sql2 = "SELECT pid FROM " . $db_prefix['default'] . "url_alias WHERE dst LIKE '%%s%' "; $sql3 = "SELECT rid FROM " . $db_prefix['default'] . "path_redirect WHERE '%s' LIKE source";
I also changed line 48 to remove duplicate code:
header('HTTP/1.0 404 Not Found'); $fast_404_html = variable_get('fast_404_html','<html xmlns="http://www.w3.org/1999/xhtml"><head><title>404 Not Found</title></head><body><h1>Not Found</h1><p>The requested URL "@path" was not found on this server.</p></body></html>'); print strtr($fast_404_html, array('@path' => check_plain(request_uri()))); exit();
Change to the function that calls the same code:
fast_404_error_return();
One limitation of the Fast 404 module are that it will print a generic “Page Not Found” message on a white page. We would prefer to have a themed landing page similar to Drupal’s normal 404 page. My solution was to create a static 404.html file for the site and tell Fast 404 to print the contents of the file instead of its generic message.
$conf['fast_404_html'] = file_get_contents('./sites/pathtosite/404.html');
I also wanted to keep track of the requests Fast 404 was catching. I added a direct syslog() call. If your Drupal site is not currently configured to use Syslog see my post on setup and configuration. The log format I chose fit our needs, you could format it however you like.
Around line 186, inside fast_404_error_return():
syslog(1, $_SERVER['SERVER_NAME'].' Fast_404 '.$_SERVER['REMOTE_ADDR'].' '.$_SERVER['SCRIPT_URI'].' '.$_SERVER['HTTP_USER_AGENT'].' '.$_SERVER['HTTP_REFERER']);
I think monitored my Drupal log file for false positives and added them to the white list.
Take a look at your Drupal site and see if you have a lot of 404s being served. If so, try this module out and see if load is reduced.
Setup Syslog on Drupal 6.x and 7.x
Feb 24th
By design Drupal 6.x sites will log all activity to the database using a core feature called watchdog. Not having to require users to configure permissions for a log file on the server reduces install instructions (and possible errors) and creates a more seamless experience for Drupal users and administrators. Unfortunately, this makes monitoring a large multisite installs more laborious.
Enter Syslog.
Syslog is a local and network logging service that has been around since the 1980s. Most *nix systems come preloaded with Syslog and fortunately for us Drupal has a Syslog module included with the 6.x and 7.x cores. Here’s how to set it up:
1) Install & configure the Syslog module
Go to your module install page, /admin/build/modules and install the Syslog module. Then go to the Syslog settings page, /admin/settings/logging/syslog, and select which Syslog level to attach to the log messages. Choose one that is not in use by Syslog
2) Configure Syslog
Edit /etc/syslog.conf and add:
local0.* /var/log/drupal.log
Then, restart Syslog
3) Disable Watchdog database logging
This will require us to edit the Drupal core (oh noes)
Edit drupal/modules/dblog/dblog.module and comment out the contents of dblog_watchdog() (Around line 132)
Original:
function dblog_watchdog($log = array()) { $current_db = db_set_active(); db_query("INSERT INTO {watchdog} (uid, type, message, variables, severity, link, location, referer, hostname, timestamp) VALUES (%d, '%s', '%s', '%s', %d, '%s', '%s', '%s', '%s', %d)", $log['user']->uid, $log['type'], $log['message'], serialize($log['variables']), $log['severity'], $log['link'], $log['request_uri'], $log['referer'], $log['ip'], $log['timestamp']); if ($current_db) { db_set_active($current_db); } }
Change to:
function dblog_watchdog($log = array()) { /* $current_db = db_set_active(); db_query("INSERT INTO {watchdog} (uid, type, message, variables, severity, link, location, referer, hostname, timestamp) VALUES (%d, '%s', '%s', '%s', %d, '%s', '%s', '%s', '%s', %d)", $log['user']->uid, $log['type'], $log['message'], serialize($log['variables']), $log['severity'], $log['link'], $log['request_uri'], $log['referer'], $log['ip'], $log['timestamp']); if ($current_db) { db_set_active($current_db); } */ }
And you’re done.
Use Drush to update a module in a Drupal multisite environment
Jan 10th
I love Drush. It makes managing my Drupal multisite installs much less complicated. I used to write custom scripts to loop through all the sites an perform actions but this little bit of code has made my maintenance tasks so much simpler.
for SITE in `ls /path/to/drupal/sites | grep "\."`; do /path/to/drush --uri=$SITE --root=/path/to/drupal --yes updatedb done
perl-Net-SSLeay dependency error when installing Memcached with yum
Dec 3rd
Today while configuring another production CentOS 5.5 machine I ran into a problem where memcached refused to install.
[root@localhost ~]# yum install memcached
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
* rpmforge: apt.sw.be
rpmforge | 1.1 kB 00:00
rpmforge/primary | 2.2 MB 00:01
rpmforge 10361/10361
Setting up Install Process
Resolving Dependencies
--> Running transaction check
---> Package memcached.i386 0:1.4.5-1.el5.rf set to be updated
--> Processing Dependency: perl(AnyEvent) for package: memcached
--> Processing Dependency: perl(AnyEvent::Socket) for package: memcached
--> Processing Dependency: perl(AnyEvent::Handle) for package: memcached
--> Processing Dependency: libevent-1.1a.so.1 for package: memcached
--> Processing Dependency: perl(YAML) for package: memcached
--> Processing Dependency: perl(Term::ReadKey) for package: memcached
--> Running transaction check
---> Package compat-libevent-11a.i386 0:3.2.1-1.el5.rf set to be updated
---> Package perl-AnyEvent.noarch 0:5.240-1.el5.rf set to be updated
--> Processing Dependency: perl(JSON::XS) >= 2.2 for package: perl-AnyEvent
--> Processing Dependency: perl(JSON) >= 2.09 for package: perl-AnyEvent
--> Processing Dependency: perl(EV) >= 3.05 for package: perl-AnyEvent
--> Processing Dependency: perl(Guard) >= 1.02 for package: perl-AnyEvent
--> Processing Dependency: perl(Net::SSLeay) >= 1.33 for package: perl-AnyEvent
---> Package perl-TermReadKey.i386 0:2.30-3.el5.rf set to be updated
---> Package perl-YAML.noarch 0:0.71-1.el5.rf set to be updated
--> Running transaction check
---> Package perl-AnyEvent.noarch 0:5.240-1.el5.rf set to be updated
--> Processing Dependency: perl(Net::SSLeay) >= 1.33 for package: perl-AnyEvent
---> Package perl-EV.i386 0:3.9-1.el5.rf set to be updated
--> Processing Dependency: perl(common::sense) for package: perl-EV
---> Package perl-Guard.i386 0:1.021-1.el5.rf set to be updated
---> Package perl-JSON.noarch 0:2.17-1.el5.rf set to be updated
---> Package perl-JSON-XS.i386 0:2.27-1.el5.rf set to be updated
--> Running transaction check
---> Package perl-AnyEvent.noarch 0:5.240-1.el5.rf set to be updated
--> Processing Dependency: perl(Net::SSLeay) >= 1.33 for package: perl-AnyEvent
---> Package perl-common-sense.i386 0:3.0-1.el5.rf set to be updated
--> Finished Dependency Resolution
perl-AnyEvent-5.240-1.el5.rf.noarch from rpmforge has depsolving problems
--> Missing Dependency: perl(Net::SSLeay) >= 1.33 is needed by package perl-AnyEvent-5.240-1.el5.rf.noarch (rpmforge)
Error: Missing Dependency: perl(Net::SSLeay) >= 1.33 is needed by package perl-AnyEvent-5.240-1.el5.rf.noarch (rpmforge)
You could try using --skip-broken to work around the problem
You could try running: package-cleanup --problems
package-cleanup --dupes
rpm -Va --nofiles --nodigest
The program package-cleanup is found in the yum-utils package.
CentOS’s yum repository doesn’t have the latest perl-Net-SSLeay and perl-IO-Socket-SSL packages that are required by memcached. I did some Googling to find that others were experiencing a similar problem without resolution. Here’s the simple fix:
First, remove the old perl modules. Then download and install the newer versions that memcached requires.
rpm-e perl-Net-SSLeay-1.30-4.fc6
rpm -e perl-IO-Socket-SSL-1.01-1.fc6
wget http://packages.sw.be/perl-Net-SSLeay/perl-Net-SSLeay-1.36-1.el5.rfx.i386.rpm
wget http://packages.sw.be/perl-Net-SSLeay/perl-Net-SSLeay-1.36-1.el5.rfx.x86_64.rpm
wget http://packages.sw.be/perl-IO-Socket-SSL/perl-IO-Socket-SSL-1.34-1.el5.rfx.noarch.rpm
rpm -i perl-Net-SSLeay-1.36-1.el5.rfx.x86_64.rpm
rpm -i perl-Net-SSLeay-1.36-1.el5.rfx.i386.rpm
rpm -i perl-IO-Socket-SSL-1.34-1.el5.rfx.noarch.rpm
Now you should be able to install memcached without error.
yum -y install memcached
If you experience an error removing either of the packages then you can try greping to see if you have other versions installed.
rpm -qa | grep perl-Net-SSLeay
and
rpm -qa | grep perl-IO-Socket-SSL
Safari 5 hanging on page loads
Sep 14th
Is your Safari 5 hanging when loading pages? Try switching your DNS servers or dumping your Safari cache:
rm -r ~/Library/Caches/com.apple.Safari
Facebook-Status-2.1 released
May 6th
I have updated my original Facebook status script to version 2.1. Grab it from here.
The major change for this version is a switch to Facebook’s mobile pages for easier parsing. You can include this script into all sorts of wacky things.
Usage example:
perl facebook-status-2.1.pl
Facebook login name: some.user@example.com
Password: ******
Your status (Facebook appears to have a 232 character limit): I'm setting my status removely!!
Sending... Done!
New RocDubs Forum Theme
Mar 18th
Every once in while I like to flex my design muscles. When I moved to Rochester last November I joined up with the local european car group RocDubs. These kids have been a lot of fun to hang out with. Like most car groups we have a forum (vBulletin) and we needed a personalized flavor so I took on the task of making a theme. Today I launched it.
Dealing with ._ files in Mac OS X
Jan 26th
The current code base I’m working from is located on a network share and controlled by Subversion. I’ve been dealing with the Apple Double format problem a lot lately and scripted a simple clean up solution.
For those unfamiliar with the Apple Double problem, it is an old bug that exists from the early days of Mac OS X. Here’s the explanation from the Apple Knowledge Base:
Before Mac OS X, the Mac OS used ‘forked’ files, which have two components: a data fork and a resource fork. The Mac OS Standard (HFS) and Mac OS Extended (HFS Plus) disk formats support forked files. When you move these types of files to other disk formats, the resource fork can be lost.
With Mac OS X, there is a mechanism called “Apple Double” that allows the system to work with disk formats that do not have a forked file feature, such as remote NFS, SMB, WebDAV directories, or local UFS volumes. Apple Double does this by converting the file into two separate files. The first new file keeps the original name and contains the data fork of the original file. The second new file has the name of the original file prefixed by a “._ ” and contains the resource fork of the original file. If you see both files, the ._ file can be safely ignored. Sometimes when deleting a file, the ._ component will not be deleted. If this occurs you can safely delete the ._ file.
Essentially, when you’re working on files located on a network share you end up with duplicates of the file with a “._” prefix. The files are empty and are quite the annoyance. They are removed when the network share is unmounted. However, I leave my share mounted for long periods of time and the files make a mess of my Subversion status reports.
You can run the following command to clean the files up.
find . -name '._*' -exec rm '{}' ';'
You could also make this into a launchctl or cron task. Happy hacking!
Making database changes to Drupal multi-sites
Dec 21st
The company I work for hosts over 150 Drupal sites in a multi-site configuration and every so often I need to publish a change to the code that requires a database change. The large number of sites we host means that performing manual changes would require too much time and carries a chance of error. I’ve come up with a simple solution.
I first create a PHP script that bootstraps Drupal. Here is an example of a script that will rebuild the menu router tables.
// Bootstrap require_once('includes/bootstrap.inc'); require_once('includes/common.inc'); drupal_bootstrap(DRUPAL_BOOTSTRAP_FULL); // Rebuild menus menu_rebuild(); print 'Menu rebuild complete\n';
I name this script rebuild_menus.php and upload it to my Drupal directory. Then, I create a text file on my desktop that contains a list of all the Drupal websites. This is an abbreviated example:
http://example1.com http://example2.com http://example3.com http://example4.com
Finally, I execute the script on all the sites by running the following command:
for url in `cat list.txt`; do echo "Running script on $url"; curl $url/rebuild_menus.php; echo; done
