On August 15, 2018 GitHub was deployed to production running Rails 5.2. This was a historic event; for years GitHub had been behind Rails and dependent on a custom fork of Rails 2.3. This talk will visit GitHub’s past, including our tumultuous relationship with the Rails framework, and the grueling effort it took to get our application on the latest version. You’ll learn what mistakes to avoid and the reasons why such a difficult upgrade was worth it. We’ll explore what tracking master means for the future and why a symbiotic relationship between Rails and GitHub is so important to our continued success.
Eileen M. Uchitelle
Principal Software Engineer, GitHub
Thank you so much, Maya and Ross. If you're like me, you're probably blown away by Doug Parker, CEO of American airlines, talking about how transformation is everyone's responsibility and certainly not just its job. So when I saw a video of the next speaker presenting at the 2019 rails Coff conference, I immediately started sending it to all my friends, especially Dr. Steven McGill, who I've been working with for the last two years researching the dynamics of the software supply chain. One of the observations made in the information security community is that one of the best ways to stay secure is to simply stay up to date on your software dependencies. It's easy to say, but much more difficult to operationalize because so often when we update our dependencies everything's breaks, which is why developers avoid updating dependencies in their daily work, and then suddenly discover that production applications have significant vulnerabilities in them, which is why I'm so excited that our next speaker is going to tell her story.
Last year I lean you should tell, was promoted to become a principal engineer at GitHub. She will tell the seven year journey of how GitHub finally migrated from rails to, to rails five and the increasingly dire problems that necessitated it. And she will tell us about all the amazing benefits that it enabled. She has some lessons and some very specific advice that every technology leader needs to hear, because the story she tells is happening to every organization. I'll also mention this I've been so blown away by every conversation that I've had with her. She's elevated what I think every engineer needs to be doing for their organizations. So here's Eileen.
Hi, everyone. I want to first say thank you to the organizers of DevOps enterprise summit for having me speak today. It's an honor to be one of your keynotes, and I hope that you've all enjoyed the conference so far. I'm Eileen. You should tell, and you can find me anywhere on the internet. And I lean codes. I'm a staff engineer at GitHub. My focus is on ensuring that the rails framework and Ruby language support get hope for the longterm. I work on building features and fixing bugs upstream, but also working to improve the GitHub code base to remove technical debt and build up our resilience. In addition to my work at GitHub, I'm a maintainer and member of the rails core team. The rails core team consists of 12 people who work together to decide the future of the rails framework. We work on what features will be supported or deprecated when new releases will come out and foster our community of contributors.
Today, we're going to talk about the past present and future of rails, like get hub while this talk focuses on our history with rails and in particular, our massive upgrade. The majority of things I talk about here are applicable to any application written in any language or framework, the original GitHub application and the majority of the application that users interact with today is built on Ruby and rails. Over the years, we didn't prioritize, upgrading and staying on the latest version. I, one point years ago, we forked rails and practically wrote our own. We fought against the framework. We deviated from the framework, and we even wondered if rails was right for us at all. But at the end of the day, get hub is successful because of rails and rails is successful. Because if GitHub, once we focused on upgrading rails, we found that running on the latest version made it possible for us to use an invest in rails for the long haul in a way that we had never been able to do before.
We're now able to change and influence the framework for our needs while also benefiting the broader rails community and open source ecosystem. This story is part historical. We'll look back at the beginning and how GitHub ended up maintaining a custom fork of rails is also part technical exploring what compelled us to upgrade our process and why it was so difficult. We look at the cost of not upgrading and how technical debt accumulates in your application until they, it starts to work against your framework. Lastly, we'll dive into our effort. I have to clean up technical debt, our commitment to open source and our responsibility to support rails for the long haul. Let's go back in time to the beginning. In 2004, DHH announced a new web framework called Ruby on rails. Immediately. Rails caught the attention of the Ruby community. Every comp that year DHH talked about relative's history, how it came to be and why it was better than existing Ruby frameworks.
We may not. He went on to talk about his philosophy in building the framework stating that many frameworks fail because they're built without an application to influence the architecture. He said that frameworks are retrospectives. They should be extracted, not built rails was attractive and is successful because it was extracted from a real application. Basically in the early years, rails, complexity grew slowly and rails. 1.0 was released in December of 2005, two years later, rails 1.2 was released that same year. Tom Preston Warner was at every meetup in San Francisco. When he showed his friend, Chris Wayne's rap, a tool called grit. Grit was a Ruby tool that allowed you to view, get repositories in an object oriented way. It would become the basis for get repos on GitHub. After seeing grit, Chris was immediately hooked and a few days later they wrote get hub rails.
Application was born, ghetto was created using rails 1.2 0.3. And after a short beta GitHub was released to the public until April or April of 2008. The next day, the rails framework moved off of their own SVN server to using get on, get hub in 2009 rails 2.3 was released in the early days when rails would release a new version, get hub quickly upgrade to get the new features and bug fixes, but sometime between 2008 and 2009, get hub forked rails. Now, remember this was the wild west of rail startups. No one really knew what the future of rails or GitHub was going to be. We weren't yet talking about the importance of upgrades or staying current with rails master and honestly, rails wasn't as stable as it is today. I don't want to go as far as to say that rails didn't care about performance or stability at this time, but I know a lot of app developers felt that way.
It definitely wasn't a concern the same way it is today. Maybe that's because we were all kind of inexperienced back then, or maybe it was because rails was good enough for its most important user base camp. Maybe it was because GitHub didn't contribute enough upstream for whatever reason, the problem wasn't just that get hub, just forked rails and added a bug fixed here, performance improvement there, it wasn't just a fork with backwards from upstream, get hosts. Fork was rails with custom code just for ghetto. It was rails morphed into a different framework that was built for GitHub. The way rails was built for base camp and as GitHub doubled down on their fork and added more and more functionality rails of course, continued to progress at a fast pace as well. At the time, no one could predict or understand the costs that forking rails would have on get hub application or engineering team.
In 2010 rails three was released, but many applications didn't upgrade due to performance concerns. The performance issues in three, over a big deal user saw an unacceptable increase in response times, some applications seeing requests taking as much as twice as long active record and rails three was found to be five times slower than rails two, despite knowing about the performance concerns, a couple of GitHub engineers started to work on upgrading, get hub two rails three. Now the rails three upgrade wasn't pointing at rails three upstream either. It was still a fork of rails three with good hubs, custom patches added on top. And 2012 rails three, two was released and most of the performance concerns had been completely solved in the same year. Good hubs progress on the rails up grid stalled. It had been two years since they started the 300 upgrade and the engineering team began questioning whether the effort was worth it at all.
They asked each other why upgrade when this version isn't causing us pain, why upgrade and rails three O is so great. Why upgrade when our fork has more features looking at these questions, the engineering team decided the upgrade wasn't worth their time and focus their attention on other projects. The truth is that at this time get up, wasn't yet feeling the pain of being on a fork. And it's difficult, difficult to convince a team to upgrade when they're still feeling productive. You make your tests faster when they feel too slow. You're a fact of the complexity of that class when you need to add new functionality, but when do you upgrade? What's the incentive. If the new version isn't better and your current fork is working just fine. If you're not feeling the pain of being on a Fort or an old version, you're just not going to be compelled to upgrade.
But of course, eventually it wasn't fine. All of those, why should we upgrade? Questions started to become suffocating for the engineering team. It became harder to find where the framework ended in the application began as good engineers started to fight against the fork and the application security backwards were a nightmare. Every time rails announced a vulnerability, ghetto was forced to manually patch it. Hiring was becoming increasingly difficult. No one wants to work on a rails to three application that doesn't even rent resemble rails to. It's harder to get up to speed and you can't go search how to do anything. Dependencies were brutal and unsupported as gem authors focused on new versions of rails. Development was slow and painful. Working with an application that's tied so heavily to accustom fork can make adding features or refactoring code incredibly difficult. We realized we needed to get off a fork of that fork was going to suffocate the application and the engineering team.
So in 2014, a team of four full-time engineers and a few volunteers, Bennett together wrote an upgrade plan and got to work. I took the team six months of full-time coordinated effort to deploy rails three to production. And it's important to remember that rails three here, it's still a fork with GitHub custom patches added in by this time, the rails three series was only receiving severe security patches. So even though the upgrade was a success, the code base and fork were still very far behind the effort put into upgrading from two to three was massive and motivation dwindled. After that, it would be another two years before the FORO upgrade was even started. That same year rails five came out. It felt like it hub was just never going to catch up. If rails was constantly improving and consistently releasing new versions. If GitHub couldn't prioritize, upgrading and get ahead, how would we ever get off a fork?
But in 2017, I joined get home. At this point, the rails for upgrade was not in a good place. There was no dedicated team working on it. And the upgrade had fallen by the wayside. When I ran the build to see how many failures needed to be fixed, I found out there were over 4,000 errors, easy, right? It's hard to put on a timeline, just how much work and effort went into this upgrade. It's one of the hardest things I've ever done, not from a technical perspective, but from a focused and determined perspective. Upgrades are lonely, difficult, and often thankless feats of mental strength in March of 2018 a year and three months after I started at get hub, we deployed our first of three major upgrades, the rails for, to upgrade after deploying a 42 unit production, we immediately started the rails five upgrade, but this time with a larger team, we didn't want to lose momentum or fall behind like we did before for the rails five series, I led a team of four full-time engineers and spend time perfecting our dual boot and issue assignment process because we have a dedicated team in a streamline process.
The upgrade from 42 to five, two took only five months. And in August of 2018, we deployed rails five to, to production. Uh, both of these deploys went out with zero downtime and no customer impact. This is a huge milestone. It was the first time in 10 years, 10 years, they get hub wasn't on a fork of rails. It was the first time in years that get home was on the most recent version of rails. That's 10 years of cumulative, tactical debt, 10 years of fighting our rails fork and our application. We had finally started to pay some of our massive debt down by upgrading. We, of course don't eliminate technical debt, but it creates a starting off point for removing even more tech debt. It allowed us to find things that didn't belong in the application that could be upstream or remove infrastructure requirements that were already supported by rails internals.
I hope that learning about how about our upgrade and how long it took. Hasn't scared you into not doing your own upgrade. The point of this talk isn't to tell horror stories is to show you that that there's a cost to not upgrading refusing, to upgrade as more expensive and more risky than taking the time to upgrade. I won't lie to you and say that upgrading doesn't come with a cost. It is expensive and it is time consuming. But you can say like this upgrade will take X number of engineers at Y number of dollars for Z number of hours. You can measure that cost and to decide whether that's too expensive or not. But at the end of the day, doesn't matter what the dollar value of upgrading your application will be because the cost of not upgrading as a measurable, you cannot quantify the cost of not upgrading.
We can't actually measure how much it costs. Get hub to fork rails, and not upgrade how many extra hours it took to figure it out, figure out how to do something in the fork that could easily be done in a standard rails version, how many extra hours it took to manually patch security vulnerabilities, how many potential hires we lost, how much time we spent re-engineering features that were in later versions of rails already not upgrading your application will eventually cost more than any upgrade because of these reasons. This is true, whether you're using Ruby or elixir or Java, in addition to the high cost of not upgrading often team say they aren't upgrading because it's too much of a risk. My unpopular opinion after doing many upgrades is that upgrades are not inherently risky. We worry that changing the underlying foundation slightly will take everything down.
But if we're careful, if our applications are well-tested and we build up risk tolerance and confidence through incremental incremental deploys, there is no reason to think that upgrading is higher risk than not upgrading. There's also no reason to think that upgrading is higher risk than any PR that anybody on your team is deploying right now, in many ways, not upgrading and falling behind is drastically riskier than staying up to date. When you don't upgrade your application, you have to become a security expert rails like many other programming languages and platforms drop support for fixing vulnerabilities in older versions, because it becomes untenable to support them. Think about that. If the team that builds and maintains the software that you're using thinks that version is too old to accurately support security patches. Why would you continue to use that version or think that you're able to do better?
If you're using an unsupported version, your team ends up being responsible for understanding and foe and patching that security vulnerability yourself. It's extremely hard to get this right? When you don't upgrade, you lose out on great talent, bootcamp, grads, college grads, people, teaching careers, people, changing jobs, all kinds of engineers. Aren't learning rails too. And they definitely aren't learning your weird custom fork. These engineers not only don't want to work on an old version, they might not even have the knowledge of how it worked. Engineers will turn down the opportunity to work for you if you're on an old version, because it doesn't let them contribute to open source. It's no longer a Google-able and it's 10 years old. When you don't upgrade. Some of the libraries you rely on will get abandoned or deprecated. So you'll either have to live with bugs or fork.
Yet. Another dependency, new gems may not support old versions of rails. So you won't be able to use those. If you're relying on an old unsupported version, this makes development harder. Every dependency becomes more difficult because you're so far behind maintaining old libraries on top of your old framework gets tedious really quickly, and you're going to lose engineers because of it. And we don't upgrade rails. You end up building more and more infrastructure on top of your fragile application. I've seen this firsthand at get hub. We have tons of infrastructure code in our app, multiple databases, CAI tooling, our own job queue and moat and countless monkey patches. Ideally, your application would consist only of code that makes up your product because good hubs value is not in that. We have code that makes multiple databases work in rails. Our value is in our community, in our repos, in our issues, of course, multiple databases allows us to keep our application up and running, but it's not why anybody would sign up for an account.
The infrastructure code makes development more painful because of how tightly a couple is an application to framework internals, mining, minor changes can easily turn into massive or factoring or abandon projects altogether. But the biggest cost of not upgrading is that someday someone will decide they're using rails or whatever language you use was a mistake. And that it's time to do a full rewrite on your application. I don't care what language your application is written in, regardless of the language rewriting, your application is extremely expensive. If anyone on your team says, has an upgrade costs more than a rewrite. That person is lying to you at this point. Hopefully I've convinced you that that upgrade, that you're putting off is important and needs to be moved up into your goals for this quarter, not next quarter, but you're wondering how you're going to accomplish this Herculean task.
The key to upgrading any application, regardless of language, is to incrementally pay off cumulative technical debt that you've incurred and figure out a plan to keep that debt paid down. I won't stand here and tell you that upgrading will be easy. I will leave that to the person on hacker news, who wondered why I couldn't do it faster if they did their upgrade in a weekend, the upgrade did take a long time, but it wasn't the only thing that I worked on, I worked on our technical debt. I deleted unused features. I re-read our custom test framework so that we could be more confident in changes we were making. I improved our database handling for development and test. It's unfair for anyone to look at our upgrade timeline and decide that using rails was a mistake that it's too expensive. We made choices that get up over the years that made upgrading harder.
You have also made choices over the years, that will make your upgrade harder regardless of language. But that doesn't mean that rails is a bad choice. Technical debt is real and you and your team need to make decisions about what debt is acceptable and what debt needs to be cleaned up. I get hub. We decided being behind rails. Master is no longer a technical debt we were willing to put up with. You can slowly work on technical debt and upgrade rails to get your application in a better place. But there are a few things that you should consider when upgrading and many mistakes to avoid so that you don't end up doing a multi-year upgrade. Like we did it get hub. The first most important step is to build out a team. Upgrades are difficult and it helps to have a team that can support each other bounce ideas off of, and to make sure momentum keeps up.
If you have a one person upgrade team and that person leaves the company or upgrade is going to be stalled, make sure to create redundancy and support for such a difficult and important task. And lastly, reward this work. Don't just reward. Feature development. Maintenance is far more important because you don't get to develop features if your foundation is crumbling. Another thing that makes upgrades easier is like take time to plan your upgrade. Look at release notes. Plan for deprecations identify on maintain gems. There are clever ways to make upgrades easier and less painful. I get how we built a dual booting CIS. So we could test all of the same code in the product for multiple rails versions. This allowed us to prevent regressions in all new code while we fixed older code. Uh, these are things to consider. There are lots of things to consider before starting your upgrade because upfront investment might save you time and money later.
You can also make your upgrade easier by fixing deprecation warnings early. Don't ignore them for the next version to take care of. Cause then they're going to be errors. It's much easier to take care of deprecations when there are warnings, because you can do them at a slower pace. It'll make your upgrade go a lot smoother. And once you upgrade it to the most recent version, make a plan for future upgrades. How often are you going to upgrade in the future? Are you willing to test new releases in the beta or release candidate phase? This makes upgrade smoother because you can tell maintainers what was hard and what broke so they can make the version more stable for next time. Maybe they'll even ask for your help. Even earlier than that, if you have invested in upgrade, invested in upgrading, then it makes sense to invest in future tooling.
That will help you keep out of upgrade debt in the future. So now that we've looked at some considerations that you should make an upgrade and we're going to look at some things that you will definitely regret. If you do them something you won't, something you will regret in the future is a fork in your framework. It doesn't matter with it, whether it's rails or Phoenix, forking is a terrible idea. Don't do it. I highly recommend that you do literally anything else besides fork. The choice of fork rails and deviate from upstream was the single most expensive choice. We made it get hub. In regards to our application, this had a compounding effect on the state of the code base that made our upgrade take years. If we had just done it, when the version was released, it would have taken less than a few months for each version.
Another thing that you may regret down the road is falling behind, um, on security releases. If you don't make sure your software is getting upgraded to the point where maintainers no longer provide, uh, security patches for your version, then you're going to have to do it yourself. This is really dangerous because you might not know how the vulnerability even works. Unless you have a security team, intimately familiar with the software vulnerabilities. It's going to be really hard to make sure that you're patching all end points and it's going to make your security audits a lot more difficult. It's much easier to verify that you're up to date than to explain their app is way behind, but you definitely passed all those one, our abilities manually. Lastly, you'll likely regret deprioritizing upgrades in the past. I've given this talk to many individual contributors. I've never given this talk to an audience of mostly upper management and company leaders until today.
So this talk, this section is for you. The only people at a company that can make sure upgrades are prioritized and properly staff rewarded and supported our company leaders. It's up to you to make sure this kind of work is valued the same or more than product development, because the language your application is written on is your foundation failing to upgrade, arose that foundation and make sure up makes your application more brittle and more unstable. If I could go back in time to before I worked at get hub. And before we forked rails, I would tell leadership to reconsider their decision, to ignore the upgrade. I would tell them to staff a team and to make sure that upgrades happen all the time, as soon as possible, the best time to do an upgrade is immediately every day that you wait to upgrade, the more painful it will be because the code base is a con is constantly evolving as your version falls behind.
Hopefully by now I've convinced you that there are a lot of negative consequences to not upgrading, but what positive benefits do you get by prioritizing this work besides avoiding the pain? What kind of benefits are there when you prioritize upgrades, you get access to improved API APIs, major version changes, allow maintainers to rethink how previous features were designed and whether they can be improved for the next version. In rails six, we improved multiple databases by extracting all of the handling for that out of GitHub and putting it into rails before rail six, adding multiple databases to an application was awful and really hard. You had to write a ton of code yourself, but in rail six, we've added new API APIs for establishing and handling connections, as well as an improved API around active record database configurations, without upgrading, you don't have access to these improvements, upgrading to a new version.
It also gives you access to security features. In addition to security upgrades being easier. If you're not behind each version adds new security features to protect your application and your users from bad actors. Since these are security features and not vulnerability patches, the only way to get them is to actually upgrade. They will never get backward. Upgrading often provides you with performance improvements. While the most of this talk today has been about rails upgrades. We recently did a Ruby two seven upgrade at get hub. We found that running on this version reduced our production boot time from 90 seconds to 70 seconds performance improvements on the language level benefit the entire application in a way that one-off application level performance improvements just can't upgrading also gets you access to new libraries. Rail ship rail six has shipped with brand new libraries like action, text action, mailbox, and site work by upgrading.
You can rely on great new features in rails without building your own infrastructure tooling into your application. As I said before, in an ideal world, your application would only contain code that pertains to your product. If your product isn't about sending mail, why should you have to write code that sends mail? Upgrading rails keeps the line where rails ends in your application begins crystal clear, and lastly, upgrading your application. It gives you a chance to contribute. Upstream is really difficult to fix bugs, add features and influence the future of your language or framework from an old version. A lot of times we complain that our language isn't fast enough for framework is missing a feature, but that's the best part about open source is you can change it if you don't like your language, fix it. Well, all of these are good reasons to upgrade.
This is the one that kept me going. This is the reason that I spent a year and a half on the rails upgrade. It was so that in get hub could influence the future of rails, forget hub. This is the biggest and most important reason to upgrade by upstreaming features fixing bugs and supporting rails future. We support our futures. Well, when I look back at, when I started the three, two to five to upgrade the thought of being on a modern version of rails and actively contributing, upstream felt like an unachievable fairytale. We were so far behind and I wasn't sure that I could finish it at least not while maintaining most of my sanity, but now running on rails master and supporting the future of rails is good hubs present. And future state, since upgrading rails, get hub engineers have sent over 200 poll requests and improve performance, fix bugs and add major functionality to rails.
And many of these contributions were first time before we upgraded, we often were forced to add a monkey patch or working around bugs and overly complex ways. Using the most recent version of rails has allowed us not to not just contribute upstream, but to make choices that improve our application instead of hinder it. And anyone who uses rails benefits from those changes as well. Our goal is to never fall behind. Again. We're investing in our application and in rails by doing continuous upgrades every week, we bumped our rails, gym and run all of get-ups tests against the new version. We find regressions quickly, a hole with an hours or days of when they were introduced to rails and get them fixed before anyone ever experiences them. This has created a symbiotic relationship between GitHub and rails. We redeveloped a framework and code an application side by side.
Since we did the hard work of upgrade major version of upgrading our major versions. This new process means that we never fall behind and upgrades are easy because we're doing them as rails changes. Instead of many years later, continuous upgrades in our contributions to rails are evidence that for the first time in his history, we're not just using rails or simply building a rails application for the first time in his history. We're pioneering. The future of rails get hub and rails are being co-developed together. Side by side. We're extracting code from GitHub, building new features that will help you scale rails. I'm giving back to the open source community, just support rails for the long haul. It makes good sense for us to do this. Not only do we give back to the community, but it helps us keep our application focused on our product.
We can reduce complexity and improve resilience all while influencing the future of rails. I get hub. We not only need to do this for our application. We have a responsibility to support rails. We owe part of our success to the rails framework and we have the influence expertise and application to help push rails forward. Every single company in the world, bigger small is able to do this work. You can upgrade, you can influence your framework. You just have to make it a priority. Upgrading rails was a huge investment. It wasn't cheap, but it was definitely worth it. Upgrading rails has opened up a ton of possibilities for our future. And we have a bright path ahead of us. We're building features faster. We're more confident. Our code base is stable. We're improving the scalability of rails and we're giving back to the source community. At a minimum, we went, we went from being crushed by our application rails in the decisions that we've made to building up our application, to building up rails, to building up our community, upgrading rails gave us freedom and flexibility that we didn't have before.
It empowers our engineers to build more and feel better. In 2007, get hub was born and 11 years later, we're finally tracking master. It took seven years from the day. The three O upgrade was started to the day that the five, two upgrade was complete. The future is bright and I can't wait to see what the next 10 years allows us to extract from GitHub build and rails and how the community thrives. I get hub. We'll continue to invest in the future of rails in our community because we have to, because we need to, because we want to, I hope that this talk has inspired you to make upgrading and investing in your language or framework or priority. Doesn't matter what language your application is written in. This is important work that needs to be supported and rewarded. Like I said before, if you don't like your framework or language, change it. This is how you build and maintain a strong foundation for your application and your company for us. I get hugged. We'll continue to invest in the future because get and rails are in it together for the long haul. Thank you.
Unlimited users from organization
Gene Kim's Audit and Security Playlist
Matt Bonser, PricewaterhouseCoopers LLP; Yosef Levine, Deloitte; Jeff Roberts, Ernst&Young; Michael Wolf, KPMG; Gene Kim, IT Revolution
How Fannie Mae Uses Agility to Support Homeowners and Renters
Kimberly H. Johnson, Fannie Mae; Tim Judge, Fannie Mae; Christopher Porter, Fannie Mae; Ramon Richards, Fannie Mae
From Your Auditor Friends: What We Wish Every Technology Leader Knew
Clarissa Lucas, Nationwide Insurance; Rusty Lewis, Nationwide Insurance