State of the Machine - August, 2007

Written By : Bradley Taylor

August 15, 2007

This state of the machine contains a number of announcements and status updates on the current state of Rails Machine. It also contains information on free beer and hosting add-ons (please read on!).

New Team Member
As Rails Machine has grown from a vision to a full-blown company with over 300 machinists using over 500 virtual servers, I haven’t been as timely with software updates, documentation, and community resources as I would like. Running the company has detracted from my ability to code and write. For several months, I have been searching for a Support Lead to further my vision for empowering developers. Given the quality and detail of support that we provide, it has been quite difficult to find someone with the magic combination of administration, development, and communications skills. Fortunately, I was able to find someone here in Savannah.

Next week, Rob Lingle will be joining Rails Machine to lead the charge on an extensive customer portal and control panel. This is very exciting and I’m looking forward to releasing great tools and resources for you. Even though Rob will be handling the majority of support issues, I intend to remain as accessible as possible to you when needed.

New Headquarters
The first week in September, Rob and I will be moving into the new RM HQ in a 160 year old building on Oglethorpe Square in the Savannah Historic District. Having been a “virtual” company for a year and a half, it feels great to put down roots in our own space. After getting settled, I would like to provide a library and learning lab to promote open source software and education for those in the city that would not otherwise have access to this technology. Stay tuned for more information on an open house for those in the region.

Datacenter Upgrades
In Atlanta, we’re continuing to build out our private cage in the AtlantaNAP. This project has taken much longer than I expected, but the wait will be worth it. In the next week, we should receive our brand new switching infrastructure based on the E-Series from Force10 Networks. We anticipate a major improvement in network service as we will be able to manage traffic at the IP address level and also aggregate multiple upstream feeds. Once the new switching infrastructure is installed and tested, we will migrate existing servers from our three cabinets in the data center. I’ll send out the announcements on when this will occur in a few weeks. As always, we will do our best to minimize down time for this major upgrade.

Recent Network Outage
As you may have noticed, a portion of you in the 207 subnet experienced network downtime on Friday morning and very briefly Saturday. This outage was aresult of a timed denial of service attack that flooded our network. The compromised machine was located and disabled. To prevent and mitigate such attacks in the future, we are installing better network gear to improve the visibility of such attacks and negate them as quickly as possible. In addition, we are configuring better failover and notification for support services. Internap based servers and those on other subnets were not affected.

This attack resulted from a non-privileged user account being compromised due to a poor password. Please take a few seconds to make sure that you are using strong passwords on both simple users and the deploy account. As many of you are using SSH keys, it isn’t a big deal to use a completely unintelligible password as you do not have to type it anyway.

In September, I will be traveling to Berlin for RailsConf Europe. As with Portland, I would like to meet all of you and host a party for our European machinists. Please let me know if you will attending the conference. If you are familiar with Berlin, I’d love some recommendations on a cool spot for the event. I’m also researching data centers in Europe and am looking for volunteers from different countries to test network performance.

Free Backups
Over the past several months, we have been quietly testing a nightly backup system based on LVM snapshots. Unfortunately, this system has exposed a few bugs at the OS level and a few of the larger servers have needed reboots. At this point, we feel confident about the system and I am proud to announce that this service will be provided for free to all existing customers. So, not only is your data on a RAID 1 or 10 storage subsystem, it is also being copied to a 7TB RAID 5 array every night. If you are currently running your own backups, I recommend keeping them and using them for offsite backups. Please submit a support request if you require a restore from our snapshots. Thanks to Steve for a great implementation.

Free Mail and DNS
In addition to backups, I would also like to offer all current customers free DNS and one pop3/imap4 mailbox. For DNS, please submit your requests via the ticket system and I will provide name servers. If you would like a fully functional email on address on, you may request it via the ticket system as well. Please be patient as it may take a few days to process the initial requests.

Internap Dedicated Special
For those interested in the Internap facility, I’m currently offering a limited number of dedicated 4gb, dual core servers with 250GB RAID storage for $500/month ($500/setup). These servers are running Xen and managed by us. You can create up to 15 virtual machines and define your own images for speedy deployment. Considering other hosts charge the same for a 4GB VPS, this is a deal! Contact me for more information.

In closing, the Machine is trucking along great. We’re growing at a healthy, managed rate. We are also profitable and well financed by the amazing (and local) Darby Bank. You don’t need to worry about waiting lists or prepayment tricks to get service from us. If you need to grow, we have plenty of servers and data center space to accommodate your needs regardless of size.

If you have any questions or concerns about the these items or anything else, please get in touch. I would like to thank you for supporting Rails Machine and helping it grow. Many of you have been with me since the beginning and I hope have many more years to come.


Bradley Taylor

Owner – Rails Machine, LLC