Managing hundreds of Rails apps gives us the opportunity to see all of the amazing ways that Ruby and Rails can misbehave. I had an experience so weird last night that I think it deserves its own blog post.

An app that hadn’t been deployed in two weeks all of a sudden returned a 500 error for the URL we use to monitor it. Looking at the error, Passenger couldn’t find the Rails gem. This is a 2.3 app, and doesn’t use Bundler yet, so… this was weird. I’ve been working with Rails for almost 8 years now and have never seen this before in an app that hadn’t been deployed to recently. I could understand if you changed the RAILS_GEM_VERSION line in environment.rb but hadn’t actually installed that version of Rails, but that didn’t happen in this case.

What did I do? First, I tried the console to see if that could find the rails gem. It worked fine.

Having proved to myself that the rails gem was installed and at least working for one part of the app, I concentrated on Passenger. I uninstalled all the old versions of the gem, then reinstalled the version the app was using, re-ran passenger-install-apache2-module and restarted apache. No luck.

Then I decided to remove rails from the equation and ran rake rails:freeze:gems, which moves all the rails gems to vendor/gems. After doing that, I got the real error. Somehow, rack 1.5.2 had gotten installed, which apparently hates old Rails apps. I removed it, restarted apache (just for luck) and ta-da, the app was alive again and I could go back to sleep.

Other than it happening at 3 this morning, this is why I love working at Rails Machine. If it can go wrong with Rails, we have a front row seat for it. Wait, no, that’s wrong. We’re on the stage, fixing it and making sure apps stay up and running.