Paging the Pager

Written By : Rails Machine Operations Team

August 31, 2011

Here at Rails Machine we actively monitor hundreds of production Rails applications so that we can tackle any problems promptly and keep our customers up and running. Sometimes that means a phone call at 4am from our automated systems, and for that we’re using PagerDuty. While PagerDuty has a lot of features that we love and use, there are a few things we want that haven’t been implemented quite yet. In the last couple months, we made a couple small tools to ease the pain.

Lee Jones started with a little Rails 3 application to generate iCal calendars from the PagerDuty schedules. This application, called Pager Today, creates iCal calendars for each on call rotation, for each user, and also “master” schedules for each rotation that contain all users. Here’s a screenshot of what it looks like in action:

Aside from the main page, there’s also a page to let you know if you have the pager today or not:

We’ve open sourced this app today and it’s dead simple to get deployed – you need only configure a few items and then use capistrano to deploy via Moonshine.

But that’s not all – Josh Nichols and I got a bit tired of having to do things like remembering to check a calendar to see when we’re on call.

That’s where the PagerDuty Shift Announcer comes in. The announcer is a script we throw in cron along with our Firetower configuration for our Campfire bot. This script will check to see if the person who is on call changed in the last 5 minutes. If a new person is on call, our Campfire bot – Mister Robot – will alert everyone in Campfire and send an SMS to that person.

Here he is in Campfire:

Here he is in the SMS message:

There are a couple other tools we’ve written for services we use, but those aren’t quite ready for us to let them out into the wild just yet. In the mean time, we’d love to see what you’ve come up with if you’re using PagerDuty or Campfire as well!