• 8 ways to share your git repository
• [SO] How to make a git repository read-only?

When read altogether, I had a few options.

1. Run the git-daemon. The repo would be publicy available over the internet via the git (git://) protocol.  This provides read-only access, but it provides it to the whole world.  Not a solution.
2. Run a GitHub-like server, like Gitosis or Gitolite.  This was the sledgehammer solution.  Fine-grained access control, plus all the other bells and whistles. But I was not interested in installing and understanding a whole git-hosting server application.
3. SSH via git-shell.  Each user gets a regular SSH account on the server, but each user gets a git-shell instead of a regular login shell like bash. The git-shell restricts the user to git operations.  Write permissions are restricted using standard Linux permissions on the repo itself.

So #3 was what I needed. I created an SSH login for each user that needed access to the repo. I set the login shell for each user to /usr/bin/git-shell. I put each user in a group that had read-only file system permissions to the repo.  Testing it out worked well.  Users could git clone and pull, but pushing failed and attempting to SSH directly failed.

One last note: as the man page mentions, you can create a directory called git-shell-commands in home directories of git-shell users.  git-shell users will be able to run any command in this directory.  If there is a help program in the directory, it is run when a git-shell user logs in.  More details on git-shell-commands here, including the location of sample git-shell commands on your server.

% mysqladmin -u admin -p extended | grep -i qcache

Then I type in the password. Well, I grew tired of typing in the extra options, plus the password. Turns out, MySQL will look for the configuration file .my.cnf in your home directory after it looks in /etc/my.cnf (it looks in a few other places as well). So I put this in my ~/.my.cnf:

[client]
user=admin
password=secret

And now I can simply run:

% mysqladmin extended | grep -i qcache

and it works right away.  Note that the password is stored in the clear.

It’s all pretty simple really. There is an overhead to opening and closing TCP connections. To alleviate this, Apache can agree to provide persistent connections by sending HTTP Keep-Alive headers. Then the browser can open a single connection to download multiple resources. But Apache won’t know when the browser is done downloading, so it simply keeps the connection open according to a Keep-Alive timeout, which is set to 15 seconds by default. The problem is the machine can only keep so many simultaneous requests open due to physical limitations (e.g. RAM, CPU, etc.) And 15 seconds is a long time.

To allow browsers to gain some parallelism on downloading files, without keeping persistent connections open too long, the Keep-Alive timeout value should be set to something very low, e.g. 2 seconds.

I’ve done this for static content only. Why only static content? It doesn’t really make much sense for the main page source itself since that’s the page the user wants to view.

I’ve mentioned before that by serving all static content on dedicated subdomains, we indirectly get the benefit of being able to optimize just those subdomains. So far, this meant:

1. disabling .htaccess files
2. setting a far-future Expires: header
3. avoiding setting cookies on the subdomain

Now we can add to the list: enabling HTTP Keep-Alive headers. The VirtualHost block might look like this now:

<VirtualHost *:80>
    ServerName      static0.yourdomain.com
    ServerAlias     static1.yourdomain.com
    ServerAlias     static2.yourdomain.com
    ServerAlias     static3.yourdomain.com
    DocumentRoot    /var/www/vhosts/yourdomain.com
    KeepAlive On
    KeepAliveTimeout 2
    <Directory "/var/www/vhosts/yourdomain.com">
        AllowOverride None
        ExpiresActive On
        ExpiresByType text/css "access plus 1 year"
        ExpiresByType application/x-javascript "access plus 1 year"
        ExpiresByType image/jpeg "access plus 1 year"
        ExpiresByType image/gif "access plus 1 year"
        ExpiresByType image/png "access plus 1 year"
    </Directory>
</VirtualHost>

Note the following applies to Windows Vista, but is probably easier on MacOS/Linux.

Is your hosts file becoming monstrous?  Do you have an alias or shortcut to your hosts file because you edit it so often?  Tired of manually adding every subdomain and domain you work on?

I was too when I thought there must be a better way.  And there was.

The general idea is this: by installing a local DNS nameserver in BIND, we can set up local development domains that look like regular domains on the internet. For real domains, we’ll just forward the requests on to a real nameserver.  This gives us a couple more benefits: 1) we can use the local nameserver as a caching nameserver to speed up DNS queries (in theory, I have not actually done this), and 2) we can choose to use any DNS service we wish, i.e. OpenDNS, or Google DNS.

Here are the steps.

1. Follow these instructions on installing and configuring BIND and configuring a zone for your local domain.
   1. I installed BIND to C:\Windows\system32\dns.
   2. Here is my named.conf in its entirety.
      
      

      options {
          directory ";c:\windows\system32\dns\zones";
          allow-transfer { none; };
          forward only;
          forwarders {
              //208.67.222.222; // OpenDNS
              //208.67.220.220;
              8.8.8.8; // Google DNS
              8.8.4.4;
          };
          query-source address * port 53;
      };
       
      /*
      logging {
          channel queries_log {
              file "c:\windows\system32\dns\var\queries.log";
              print-severity yes;
              print-time yes;
          };
          category queries { queries_log ; };
      };
      */
       
      zone "work.local" IN {
          type master;
          file "work.local.txt";
      };
       
      key "rndc-key" {
          algorithm hmac-md5;
          secret "xxxxxxxxxxxxxxxxxxxxxxxx";
      };
       
      controls {
          inet 127.0.0.1 port 953
              allow { 127.0.0.1; } keys { "rndc-key"; };
      };

   3. I created a zone file for my development domain work.local following this zone file example. Here is the zone file in its entirety.  Note the CNAME wildcard record.
      
      

      $TTL 86400
      @	IN SOA	ns1.work.local.	admin.work.local. (
      			2008102403
      			10800
      			3600
      			604800
      			86400 )
       
      @		NS	ns1.work.local.
       
      	IN A	127.0.0.1
      ns1	IN A	127.0.0.1
      www	IN A	127.0.0.1
      *	IN CNAME	www

2. Start or restart the BIND service.
3. Configure you network connection to use 127.0.0.1 as your primary nameserver, instead of DHCP.  My IPv4 properties look like this:
   
   
4. Flush the Windows DNS cache by running:
   
   

   C:\> ipconfig /flushdns

5. Test BIND by pinging www.work.local.  If you have errors, you can uncomment the logging block in named.conf.
6. Once that is working, create a VirtualHost in Apache for your development domain.  Thanks to VirtualDocumentRoot, we can map any number of subdomains to project roots.  Here is my VirtualHost block.
   
   

   <VirtualHost  127.0.0.1>
       ServerName www.work.local
       ServerAlias *.work.local
       VirtualDocumentRoot "C:/_work/%1"
       <Directory  C:/_work>
           Options Indexes FollowSymLinks Includes ExecCGI
           AllowOverride All
           Order allow,deny
           Allow from all
   </Directory>
   </VirtualHost>

7. Start or restart Apache.
8. Create a directory in C:\_work, for example, C:\_work\awesomeapp.  Create a test index.html file in that directory.
9. You should now be able to go to http://awesomeapp.work.local in your browser and see your index.html file!

Now, you should be able to repeat step 8 for any new website you create!  No editing of hosts files, no bouncing the webserver!  Just create the project directory and it’s immediately available.

One other important note: Firefox has its own DNS cache independent of the OS.  For sanity, restarting Firefox resets its DNS cache. You can also permanently disable DNS caching in Firefox.

So I whipped up an shell script that called rsync with some predefined options and targets.  Unfortunately, in order to preserve ownership and permissions in the destination, rsync needed to be run as root.

At first, I looked at the setuid bit. By changing the ownership of the rsync shell script and running `chmod u+s` on the script, setting the setuid, any user could execute it and it would run as root. Well, it turns out that the kernel will not honor setuid on shell scripts for security reasons. But what if I wrote a C program instead of a shell script? That actually worked, and ran with root privileges, but it still did not rsync as root for some reason. So that was out.

The second solution was to insert sudo before the rsync command in the script. I modified /etc/sudoers to allow the users group to run rsync under sudo. That worked perfectly. So if I put this script in /usr/local/bin, I would be done. But I had already written this magnificent (two-line) C program.  Why not make it even more secure (sudo does not work on shell scripts either)?  Instead of allowing all users to run rsync under sudo, I could limit them to running only my C program under sudo, instead of rsync in general. Then, in my script, I could replace rsync with my C program. So that’s what I did. I again modified /etc/sudoers and my shell script, threw both the script and C executable in /usr/local/bin and I was done.

I named the final command `zipsync`. Here is the shell script for that, anonymized a bit.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

#!/bin/sh
 
cd /var/www/vhosts
 
# repeat for each web server
sudo zipsync.bin \
   -av --delete \
   --exclude=".svn" \
   --exclude="logs" \
   --exclude="tmp" \
   --exclude="cache" \
   --exclude="*.swp" \
   * 192.168.1.101:/var/www/vhosts
 
cd -

And the C program, zipsync.bin.

1
2
3
4
5
6
7

#include <unistd.h>
 
int main(int argc, char** argv)
{
   *argv = "rsync";
   return execvp(*argv, argv);
}

It makes sense that most of the memory not used by apps would be used by the OS to improve disk access.  So seeing the memory graphs filled with orange is generally a good thing.  After a few days, I watched the orange area grow and thought, “Great!  LInux is putting all that extra memory to use.”  I thought in my head that maybe it was caching images and CSS files to serve to Apache.  But was that true?

Looking At A Different Server

Here is a memory graph from one of our database servers after the RAM upgrade.

Again, I first thought that the OS was caching all that juicy database data from disk.  The problem is that we don’t have 12GB of data, and that step pattern growth was suspiciously consistent.

Looking again at the web server graph, I saw giant downward spikes of blue color, where the buffer cache was emptied.  (The blue is unused memory.)  These occurred every day at 4 am, and on Sundays there’s a huge one.  What happens every day at 4 am?  The logs are rotated.  And on Sundays, the granddaddy log of them all – the Apache log – is rotated.

The Problem

It was starting to make sense.  Log files seem to take up most of the OS buffer cache on the web servers.  Not optimal, I’m sure.  And when they’re rotated, the data in the cache is invalidated and thus freed.

Here is a memory graph for one of our other database servers.

That step pattern growth is missing!  In fact, most of RAM is unused.  What is the difference between the first database server and this one?  The first has the `mysqldump` backup.  It occurs every night at 2:30 am, right when those step changes occur on its memory usage graph.

It was clear to me that most of the OS buffer cache was wasted on logs and backups and such.  There had to be a way to tell the OS not to cache a file. 

The Solution

Google gave me this page: Improving Linux performance by preserving Buffer Cache State.  I copied the little C program into a file and ran it on all the `mysqldump` backups.  Here is the what happened to the memory usage.

Quite a bit of buffer cache was freed.  On that night’s backup, I logged the buffer cache size before the backup and after.

% cat 2008.08.21.02.30.log
Starting at Thu Aug 21 02:30:03 EDT 2008
=========================================
Cached:        4490232 kB
Cached:        5350908 kB
=========================================
Ending at Thu Aug 21 02:30:55 EDT 2008

Just under a gigabyte increase in buffer cache size.  What was the size of the new backup file?

% ll 2008.08.21.02.30.sql
-rw-r--r-- 1 root root 879727872 Aug 21 02:30 2008.08.21.02.30.sql

About 900MB.

Did It Work?

I used the C program on that page to ensure no database backups were cached by the OS.  I did the same on the web servers in the logrotate config files.  A couple days later, I checked the memory graph on the database server that performed the backup.  Notice how the buffer cache did not fill up.  It looked like the program worked, and the OS was free to cache more important things.

I’ll spare you the problems I ran into and overcame, and just list the steps to get this done.  These instructions are based on a 64-bit, RHEL 5 server running the pre-packaged Apache server.  So if you follow these instructions on a different setup, of course, filenames, directories, versions, etc. may differ.

Install mod_vhost_dbd

Download dbd-modules from Google Code.  This is a great piece of code in the form of an Apache module that uses mod_dbd and a DBD Mysql (or other database) driver to fetch the DocumentRoot for a given domain from a database.

% wget http://dbd-modules.googlecode.com/files/dbd-modules-1.0.5.zip

Unzip the archive in a directory. As indicated on the website, build and install the module.

% apxs -c mod_vhost_dbd.c
% apxs -i mod_vhost_dbd.la

This places mod_vhost_dbd.so in /usr/lib64/httpd/modules.  Enable both this module and mod_dbd by adding two lines to httpd.conf, or equivalently creating a new include file in /etc/httpd/conf.d containing these lines.

LoadModule dbd_module modules/mod_dbd.so
LoadModule vhost_dbd_module modules/mod_vhost_dbd.so

In true unit fashion, now might be a good time to restart Apache, just so you can be sure everything is working up to this point.

% service httpd restart

Install Mysql DBD Driver to APR

Unfortunately, on my system, the Mysql DBD driver was nowhere to be found.  I had to rebuild Apache Portable Runtime (APR) utils with the Mysql driver enabled.

Download apr and apr-util from Apache.  Note these are not the latest versions, but the versions that matched the packages in worked for RHEL 5.

% wget http://archive.apache.org/dist/apr-1.2.8.tar.bz2
% wget http://archive.apache.org/dist/apr-util-1.2.8.tar.bz2

Unpack and untar these archives in the same parent directory.

Build and install APR.  Now, I do not think this is absolutely necessary, but it seems like a good idea to keep the versions in sync.

% ./configure --prefix=/usr
% make
% make install

Build and install apr-util.  Due to licensing issues, apr-util does not actually contain the Mysql DBD driver until apr-util-1.2.12.  Prior to that version, it must be downloaded separately, and the configure script rebuilt.

% wget http://apache.webthing.com/svn/apache/apr/apr_dbd_mysql.c
% ./buildconf --with-apr=../apr-1.2.7

Now for the three commands every Linux admin loves.

% ./configure --prefix=/usr --with-apr=/usr --libdir=/usr/lib64 --with-expat=builtin --with-ldap-include=/usr/include --with-ldap-lib=/usr/lib64 --with-ldap=ldap --with-mysql
% make
% make install

The first time I tried this, Apache could not find any LDAP-related modules.  Adding those configure switches seemed to do the trick.  Restart Apache.

% service httpd restart

Apache should now be able to query a Mysql database to get the DocumentRoot for a domain.  My VirtualHost block looked something like this.

<VirtualHost *:80>
    ServerName *.example.com
    DocumentRoot "/path/to/default/document/root"
 
    DBDriver mysql
    DBDParams host=localhost,user=root,pass=secret,dbname=vhosts
 
    DBDocRoot "SELECT path FROM vhosts WHERE host = %s"  HOSTNAME
</VirtualHost>

For more details and instructions on mod_vhost_dbd configuration directives, read the project wiki.

The solution I found starts with this thread about upgrading PHP on the CentOS forums.  To summarize, run these commands:

% cd /etc/yum.repos.d
% wget http://dev.cnetos.org/centos/5/CentOS-Testing.repo

Then, edit the .repo file and set the "enabled" value to "1".

% yum list available php
Available Packages
php.i386                                 5.2.6-2.el5s2          c5-testing

But turning this option on and allowing .htaccess files may be a performance hit. More specifically, enabling the AllowOverride option in Apache is a performance hit. The Apache docs sums up the problem best:

“Wherever in your URL-space you allow overrides (typically .htaccess files) Apache will attempt to open .htaccess for each filename component. For example,

1 2 3 4

DocumentRoot /www/htdocs <Directory /> AllowOverride all </Directory>

and a request is made for the URI /index.html. Then Apache will attempt to open /.htaccess, /www/.htaccess, and /www/htdocs/.htaccess.”

So I disabled all .htaccess files in production, and inserted each file’s individual mod_rewrite rules into the main Apache config file. After a quick Apache Bench run, one project looked around 3% faster. Note that there are a few other useful optimizations on that page.

While there are a few Virtual PC hard disk images (.vhd) for Windows XP around, VMware cannot directly import .vhd files.  It needs the actual Virtual PC virtual machine file (.vmc).  After again losing my Windows XP virtual machine that I use for IE6 testing, I thought I’d document the process of running Windows XP in VMware so I don’t have to figure it out again the next time it happens. 

Note: though these instructions are for VMware Workstation, some of this may apply to the free VMware Player.

1. Download the IE6 Virtual PC Virtual Hard Disk (.vhd) image from Microsoft.
2. Download and install Virtual PC from Microsoft, if you don’t have it already.
3. Start Virtual PC.  If you have no virtual machines, you will get the New Virtual Machine Wizard.  Click Next.
   
4. Select “Use default settings to create a new virtual machine”. Click Next.
    
5. Pick a location to save your Virtual PC virtual machine.  This should be the location you will create the VMware virtual machine.  I keep all my VM’s in the same directory with meaningful names.
   
6. Click Finish to create the new virtual machine. 
   
7. If you selected “When I click Finish, open Settings,” in the previous step, you will see the settings dialog.  If you did not, select the new VM and click Settings.  Select “Virtual hard disk file:” and find the .vhd file you downloaded in step 1.  After finding it, click OK.
   
8. You should see your VM in the Virtual PC Console. 
   
9. Select your VM and click Start.  Your Windows XP virtual machine should boot in its own window.
   
10. Shut down the virtual machine using the Start button.  Then exit out of Virtual PC.  Start VMware Workstation.  Once it’s started, select “Import or Export…” from the “File” menu.  You should see the Conversion Wizard.  Click Next.
      
11. You are at Step 1 of the conversion. Click Next to select a Source Type. Under “Select the type of source you want to use:”, select Other.  Click Next.
    
12. Under “Source VM or image:”, find the Virtual PC (.vmc) file you created earlier.  Click Next.
    
13. Select “Convert all disks and maintain size.” Click Next.
    
14. You are at Step 2 of the conversion. Click Next to select a destination type.  Under “Select the destination type,” select “Other Virtual Machine.”  Click Next.
    
15. Under “Virtual machine name,” fill in a meaningful name.  Under “Location:”, find the place you want to store your virtual machine.  Click Next.
    
16. The wizard tells you that the source files are in Microsoft virtual disk (.vhd) format.  Under “How do you want to convert them?”, select “Import and convert (full-clone).”  Under “Disk Allocation,” Select “Allow virtual disk files to expand.”  Click Next.
    
17. The next step allows you to configure your VM networking.  You should probably stick to the default of 1 NIC, bridged, that connects at power on.  Click Next.
    
18. Step 3 allows for some VMware customisation.  You definitely want to install the VMware Tools.  Click Next.
    
19. You’re Virtual PC image is ready to be converted to VMware.  Click Finish to begin the conversion!
    
20. Get up from your desk and take a walk around.  Go get a cup of coffee. 
21. After the conversion is completed, you should see your new Windows XP virtual machine in VMware Workstation.
    
22. Click on “Power on this virtual machine” and your Windows XP VM should boot inside of VMware Workstation.  You can uninstall Virtual PC at this point, if you want (which is likely, since you’re running VMware).