Straylight Run

At work, we had set up some wildcard virtual hosts in Apache config, and that got us by for quite some time.  But the time came when we needed finer-grained control of where to send incoming requests for different domains.  I needed to store my virtual hosts in a Mysql database, mapping domains to project directories.

I’ll spare you the problems I ran into and overcame, and just list the steps to get this done.  These instructions are based on a 64-bit, RHEL 5 server running the pre-packaged Apache server.  So if you follow these instructions on a different setup, of course, filenames, directories, versions, etc. may differ.

Install mod_vhost_dbd

Download dbd-modules from Google Code.  This is a great piece of code in the form of an Apache module that uses mod_dbd and a DBD Mysql (or other database) driver to fetch the DocumentRoot for a given domain from a database.

% wget http://dbd-modules.googlecode.com/files/dbd-modules-1.0.5.zip

Unzip the archive in a directory. As indicated on the website, build and install the module.

% apxs -c mod_vhost_dbd.c
% apxs -i mod_vhost_dbd.la

This places mod_vhost_dbd.so in /usr/lib64/httpd/modules.  Enable both this module and mod_dbd by adding two lines to httpd.conf, or equivalently creating a new include file in /etc/httpd/conf.d containing these lines.

LoadModule dbd_module modules/mod_dbd.so
LoadModule vhost_dbd_module modules/mod_vhost_dbd.so

In true unit fashion, now might be a good time to restart Apache, just so you can be sure everything is working up to this point.

% service httpd restart

Install Mysql DBD Driver to APR

Unfortunately, on my system, the Mysql DBD driver was nowhere to be found.  I had to rebuild Apache Portable Runtime (APR) utils with the Mysql driver enabled.

Download apr and apr-util from Apache.  Note these are not the latest versions, but the versions that matched the packages in worked for RHEL 5.

% wget http://archive.apache.org/dist/apr-1.2.8.tar.bz2
% wget http://archive.apache.org/dist/apr-util-1.2.8.tar.bz2

Unpack and untar these archives in the same parent directory.

Build and install APR.  Now, I do not think this is absolutely necessary, but it seems like a good idea to keep the versions in sync.

% ./configure --prefix=/usr
% make
% make install

Build and install apr-util.  Due to licensing issues, apr-util does not actually contain the Mysql DBD driver until apr-util-1.2.12.  Prior to that version, it must be downloaded separately, and the configure script rebuilt.

% wget http://apache.webthing.com/svn/apache/apr/apr_dbd_mysql.c
% ./buildconf --with-apr=../apr-1.2.7

Now for the three commands every Linux admin loves.

% ./configure --prefix=/usr --with-apr=/usr --libdir=/usr/lib64 --with-expat=builtin --with-ldap-include=/usr/include --with-ldap-lib=/usr/lib64 --with-ldap=ldap --with-mysql
% make
% make install

The first time I tried this, Apache could not find any LDAP-related modules.  Adding those configure switches seemed to do the trick.  Restart Apache.

% service httpd restart

Apache should now be able to query a Mysql database to get the DocumentRoot for a domain.  My VirtualHost block looked something like this.

<VirtualHost *:80>
    ServerName *.example.com
    DocumentRoot "/path/to/default/document/root"
 
    DBDriver mysql
    DBDParams host=localhost,user=root,pass=secret,dbname=vhosts
 
    DBDocRoot "SELECT path FROM vhosts WHERE host = %s"  HOSTNAME
</VirtualHost>

For more details and instructions on mod_vhost_dbd configuration directives, read the project wiki.

To test things on our RHEL 5 production servers, I fire up CentOS 5 in VMWare.  While we build PHP 5.2 manually in production, I try to avoid doing so just for testing. Unfortunately, CentOS 5 is packaged with PHP 5.1 by default which becomes a big difference between my testing and production environments.  Fortunately, there is a CentOS yum repository out there with PHP 5.2.  I thought I’d document this here for anyone else with the same problem.

The solution I found starts with this thread about upgrading PHP on the CentOS forums.  To summarize, run these commands:

% cd /etc/yum.repos.d
% wget http://dev.cnetos.org/centos/5/CentOS-Testing.repo

Then, edit the .repo file and set the "enabled" value to "1".

% yum list available php
Available Packages
php.i386                                 5.2.6-2.el5s2          c5-testing

At work, every project has an .htaccess file containing at the least some mod_rewrite rules.  This way, all I need to do to run a project is check it out of version control.  I don’t need to modify my local Apache configuration.

But turning this option on and allowing .htaccess files may be a performance hit. More specifically, enabling the AllowOverride option in Apache is a performance hit. The Apache docs sums up the problem best:

“Wherever in your URL-space you allow overrides (typically .htaccess files) Apache will attempt to open .htaccess for each filename component. For example,

1 2 3 4

DocumentRoot /www/htdocs <Directory /> AllowOverride all </Directory>

and a request is made for the URI /index.html. Then Apache will attempt to open /.htaccess, /www/.htaccess, and /www/htdocs/.htaccess.”

So I disabled all .htaccess files in production, and inserted each file’s individual mod_rewrite rules into the main Apache config file. After a quick Apache Bench run, one project looked around 3% faster. Note that there are a few other useful optimizations on that page.

Though I have heard good things about Parallels and VirtualBox, I have always been a user of VMware.  In particular, VMware Workstation.  Workstation is great for firing up multiple Linux instances and testing out load-balancing or proxying scenarios.  I haven’t really figured out any use for Windows VM’s other than testing IE6. 

While there are a few Virtual PC hard disk images (.vhd) for Windows XP around, VMware cannot directly import .vhd files.  It needs the actual Virtual PC virtual machine file (.vmc).  After again losing my Windows XP virtual machine that I use for IE6 testing, I thought I’d document the process of running Windows XP in VMware so I don’t have to figure it out again the next time it happens. 

Note: though these instructions are for VMware Workstation, some of this may apply to the free VMware Player.

1. Download the IE6 Virtual PC Virtual Hard Disk (.vhd) image from Microsoft.
2. Download and install Virtual PC from Microsoft, if you don’t have it already.
3. Start Virtual PC.  If you have no virtual machines, you will get the New Virtual Machine Wizard.  Click Next.
   
4. Select “Use default settings to create a new virtual machine”. Click Next.
    
5. Pick a location to save your Virtual PC virtual machine.  This should be the location you will create the VMware virtual machine.  I keep all my VM’s in the same directory with meaningful names.
   
6. Click Finish to create the new virtual machine. 
   
7. If you selected “When I click Finish, open Settings,” in the previous step, you will see the settings dialog.  If you did not, select the new VM and click Settings.  Select “Virtual hard disk file:” and find the .vhd file you downloaded in step 1.  After finding it, click OK.
   
8. You should see your VM in the Virtual PC Console. 
   
9. Select your VM and click Start.  Your Windows XP virtual machine should boot in its own window.
   
10. Shut down the virtual machine using the Start button.  Then exit out of Virtual PC.  Start VMware Workstation.  Once it’s started, select “Import or Export…” from the “File” menu.  You should see the Conversion Wizard.  Click Next.
      
11. You are at Step 1 of the conversion. Click Next to select a Source Type. Under “Select the type of source you want to use:”, select Other.  Click Next.
    
12. Under “Source VM or image:”, find the Virtual PC (.vmc) file you created earlier.  Click Next.
    
13. Select “Convert all disks and maintain size.” Click Next.
    
14. You are at Step 2 of the conversion. Click Next to select a destination type.  Under “Select the destination type,” select “Other Virtual Machine.”  Click Next.
    
15. Under “Virtual machine name,” fill in a meaningful name.  Under “Location:”, find the place you want to store your virtual machine.  Click Next.
    
16. The wizard tells you that the source files are in Microsoft virtual disk (.vhd) format.  Under “How do you want to convert them?”, select “Import and convert (full-clone).”  Under “Disk Allocation,” Select “Allow virtual disk files to expand.”  Click Next.
    
17. The next step allows you to configure your VM networking.  You should probably stick to the default of 1 NIC, bridged, that connects at power on.  Click Next.
    
18. Step 3 allows for some VMware customisation.  You definitely want to install the VMware Tools.  Click Next.
    
19. You’re Virtual PC image is ready to be converted to VMware.  Click Finish to begin the conversion!
    
20. Get up from your desk and take a walk around.  Go get a cup of coffee. 
21. After the conversion is completed, you should see your new Windows XP virtual machine in VMware Workstation.
    
22. Click on “Power on this virtual machine” and your Windows XP VM should boot inside of VMware Workstation.  You can uninstall Virtual PC at this point, if you want (which is likely, since you’re running VMware).
      

This one is filed under “that’s pretty picky, but I guess it couldn’t hurt.”

The Entity Tags (ETags) HTTP header is a string that uniquely identifies a specific version of resource. When the browser first downloads a resource, it stores the ETag. When it requests it again, it sends along the ETag to the server. If the server sees the same ETag, it will respond with a 304 Not Modified response, saving the download.

The problem is that the default format for the ETag (in Apache) is inode-size-timestamp. And the inode will be different from server to server, meaning the server may see a different ETag from the browser, even thought it is in fact an identical file.

According to Yahoo:

The end result is ETags generated by Apache and IIS for the exact same component won’t match from one server to another. If the ETags don’t match, the user doesn’t receive the small, fast 304 response that ETags were designed for; instead, they’ll get a normal 200 response along with all the data for the component. If you host your web site on just one server, this isn’t a problem. But if you have multiple servers hosting your web site, and you’re using Apache or IIS with the default ETag configuration, your users are getting slower pages, your servers have a higher load, you’re consuming greater bandwidth, and proxies aren’t caching your content efficiently.

There is another scenario where it isn’t a problem: if you are using sticky sessions in your load balancer.

In any case, as stated above, it couldn’t hurt to rectify this. So I configured the ETag format in Apache to exclude the inode, and use only size and timestamp.

FileETag MTime Size

So files across servers have the same ETag.

In any system, the biggest bottlenecks will usually be related to I/O. What this means practically is two things:

1. Memory is faster than disk.
2. Disk is faster than network.

But moving across the boundaries of memory, disk, and network is usually cumbersome.  For example, storing things on disks is programmatically easy, but slow.  Storing things in memory, in a persistent way, can be hard.  This is more true for a shared-nothing architecture like PHP rather than Java, so you may have to deal with some shared memory libraries and SysV IPC-style calls.

Enter tmpfs, the linux shared-memory file system.  You can mount it just like ext3, create files, and otherwise treat it like a normal disk, but it’s in memory!  Awesome!

On RHEL, Fedora, CentOS – not sure about others – there is a tmpfs drive mounted under /dev/shm by default.  One other note: since it is memory, its contents will be lost upon reboot.  I usually re-create any directories I need in the /etc/rc.d/rc.local script.  Note, however, that this is the last file to run on boot, so if you have a service or daemon that assumes a folder in /dev/shm, you will need to create it in the service’s startup script (usually in /etc/init.d).