Rails migrations with no downtime

Lets understand the challenge of changing a Rails database without introducing any downtime with a simple, apparently harmless migration:

class ApparentlyHarmlessMigration < ActiveRecord::Migration  
   def self.up
     remove_column :users, :notes
   end
end  

I learned this kind of migration is not really harmless the hard way: during a production deploy. Since the column was not being referred anywhere in the code, I assumed it was a migration that I could just run at will.

But as soon as rake db:migrate was done, errors started to pop up at the logs:

PGError: ERROR: column "notes" does not exist

Turns out that ActiveRecord caches table columns, and uses this cache to build INSERT statements. Even if the code is not touching that column, ActiveRecord will still attempt to set it to NULL when saving models.

I was just starting to realize how delicate database migrations are.
My first reaction was to call for a maintenance window whenever we had a migration to deploy. But that practice quickly became unfeasible as it blocked deploys, left users unhappy and just made it evident that we were doing things wrong.

It was time to understand the problem, and fix it for good.

via Codeship