Yes, I know the election was last week, but it’s taken us that long to recover from the sheer awesomeness of how well it went for our customers. No, not the outcome of the election, but their availability and performance throughout.

We have two customers we worked closely with to make sure they were ready for the election rush: Mobile Commons and NationBuilder. Each one will probably end up being their own blog post, but this time, I wanted to talk about how we worked with NationBuilder to make sure things went smoothly last week.

For NationBuilder, last week was about both making sure the main product was ready for everything their customers would throw at it in the days leading up to Election Day, but also planning, provisioning and deploying everything needed to launch their Election Center product - a first-of-its-kind national voter registration database open to everyone.

We’re really proud to be involved in our little way with both products, and I want to break down a little of what I think went into making last week so successful (as in, no one got woken up in the middle of the night and things were actually faster than the week before).

I think all of these are applicable to most relationships between Development and Operations teams - it just happens that in this case, the operations team is a separate company:

1. Teamwork & Emphasis on Performance: We spent the summer and early part of the fall working with NationBuilder to add capacity, and on making sure their performance was everything it could be. We spent a lot of time looking at Scout and NewRelic to see what could be improved and then worked together to make it happen. By election day, by working together to attack hot spots, we were able to cut NationBuilder’s average response time by 33%.
2. Capacity Planning: With both customers, they were very clear about their goals, and we were able to measure our progress against those goals. We made sure they had the right pieces in place to support the traffic they expected - and the traffic no one could predict.
3. Frequent Checkpoints: We met weekly with NationBuilder to make sure we were all on track with not only deployment tasks, but performance and feature development. The NationBuilder team did an amazing job fixing performance issues when we pointed them out - to the point that when I pulled the transaction report from NewRelic, almost all the requests were new to the report.
4. Fix the Problem: The NationBuilder folks are great under duress. It doesn’t happen often, but when things go wrong, we work together to fix it in the moment, and save the root cause analysis for after.
5. Move Quickly but Deliberately: NationBuilder launched an entirely new product right before the election, The NationBuilder Election Center, and wanted it to be able to handle a lot of load right away. This started as a prototype and ended up being a fairly complex app with a lot of dependencies and a fairly large hardware footprint. By relying on our existing good communication, we were able to scale it up on all new hardware and launch with a lot of press attention and use.

And it warms our nerdy little hearts when we get feedback like this from NationBuilder’s CEO:

I just wanted to say thanks to everyone at Rails Machine for your efforts, particularly the last couple of weeks as we’ve barreled toward election night. The last 5 days were so immensely critical to us. We are working really hard to establish trust in our company, and political tech is notorious for crashing on election day. We didn’t even have a blip, so we earned a lot of cred. You helped make that happen in a big way. Thank you.

Like I said, we’re really proud to have played a small part in the democratic process and in helping NationBuilder and Mobile Commons meet their goals for performance and availability. Now it’s time to get ready for the holiday season!