Recently at Harvest we moved datacenters. That's a long story in itself and I'll tell you some time. In the mean time, when making infrastructure changes that require site downtime, I use the usual nginx "maintenance mode" thing:
if (-f /maintenanceON) {
error_page 503 /maintenance.html;
return 503;
}
So if "maintenanceON" exists, then we send a 503 and redirect to our pretty error page. Yes if is evil, but not for something so simple outside of a location block IMHO. Anyway, this isn't new.
So now, how to enable and remove "maintenanceON" at will? This file exists, by the way, outside of our Rails application (which deploys via Capistrano), and lives in a Chef cookbook repository.
Enter Knife plugins. I read some interesting posts implementing Chef::Knife::Ssh in various ways. But Knife plugins make this kind of thing super simple now.
This is my Knife plugin which enables maintenance mode, by running a search for the load balancers by the role they have ('nginx_production_lb' in this example) and then symlinking the maintenance.html to maintenanceON using Chef::Knife::Ssh
Source: https://github.com/harvesthq/knife-plugins
Chef. Is. Awesome.