Europe 2021
The Past, Present, and Future of Rails at GitHub
On August 15, 2018 GitHub was deployed to production running Rails 5.2. This was a historic event; for years GitHub had been behind Rails and dependent on a custom fork of Rails 2.3.
This talk will visit GitHub’s past, including our tumultuous relationship with the Rails framework, and the grueling effort it took to get our application on the latest version. You’ll learn what mistakes to avoid and the reasons why such a difficult upgrade was worth it. We’ll explore what tracking master means for the future and why a symbiotic relationship between Rails and GitHub is so important to our continued success.
EM
Eileen M. Uchitelle
Principal Software Engineer, GitHub
Transcript
<silence>
Thank you so much Maya and Ross. If you're like me, you are probably blown away by Doug Parker, CEO of American Airlines talking about how transformation is everyone's responsibility and certainly not just it's job. So when I saw a video of the next speaker presenting at the 2019 Rails Cough Conference, I immediately started sending it to all my friends, especially Dr. Steven McGill, who I've been working with for the last two years, researching the dynamics of the software supply chain. One of the observations made in the information security community is that one of the best ways to stay secure is to simply stay up to date on your software dependencies. It's easy to say, but much more difficult to operationalize because so often when we update our dependencies, everything's breaks, which is why developers avoid updating dependencies in their daily work and then suddenly discover that production applications have significant vulnerabilities in them, which is why I'm so excited that our next speaker is going to tell her story.
Last year Eileen Tel was promoted to become a principal engineer at GitHub. She will tell the seven year journey of how GitHub finally migrated from Rails two to Rails five and the increasingly dire problems that necessitated it, and she will tell us about all the amazing benefits that it enabled. She has some lessons and some very specific advice that every technology leader needs to hear because the story she tells is happening to every organization. I'll also mention this, I've been so blown away by every conversation I've had with her. She's elevated what I think every engineer needs to be doing for their organizations. So here's Eileen.
Hi everyone. I want to first say thank you to the organizers of DevOps Enterprise Summit for having me speak today. It's an honor to be one of your keynotes, and I hope that you've all enjoyed the conference so far. I'm Eileen, you should tell, and you can find me anywhere on the internet at Eileen Codes. I'm a staff engineer at GitHub. My focus is on ensuring that the Rails Framework and Ruby Language support GitHub for the long term. I work on building features and fixing bugs upstream, but also working to improve the GitHub code base to remove technical debt and build up our resilience. In addition to my work at GitHub, I'm a maintainer and member of the Rails core team. The Rails core team consists of 12 people who work together to decide the future of the Rails framework. We work on what features will be supported or deprecated when new releases will come out and foster our community of contributors.
Today we're going to talk about the past, present, and future of Rails at GitHub. While this talk focuses on our history with braille and in particular are a massive upgrade, the majority of things I talk about here are applicable to any application written in any language or framework. The original GitHub application and the majority of the application that users interact with today is built on Ruby and Rails. Over the years, we didn't prioritize upgrading and staying on the latest version. At one point years ago, we forked Rails and practically wrote our own. We fought against the framework. We deviated from the framework, and we even wondered if Rails was right for us at all. But at the end of the day, GitHub is successful because of Rails, and Rails is successful because of GitHub. Once we focused on upgrading Rails, we found that running on the latest version made it possible for us to use and invest in rails for the long haul in a way that we had never been able to do before.
We're now able to change and influence the framework for our needs while also benefiting the broader rails community and open source ecosystem. This story is part historical. We'll look back at the beginning and how GitHub ended up maintaining a custom fork of Rails. It's also part technical exploring what compelled us to upgrade our process and why it was so difficult. We'll look at the cost of not upgrading and how technical debt accumulates in your application until they it starts to work against your framework. Lastly, we'll dive into our effort at GitHub to clean up technical debt, our commitment to open source and our responsibility to support Rails for the long haul. Let's go back in time to the beginning. In 2004, DHH announced a new web framework called Ruby on Rails. Immediately Rails caught the attention of the Ruby community ever comp that year. DHH talked about Rail's history, how it came to be and why it was better than existing Ruby Frameworks we went on.
He went on to talk about his philosophy in building the framework, stating that many frameworks fail because they're built without an application to influence the architecture. He said that frameworks are retrospectives, they should be extracted, not built. Rails was attractive and is successful because it was extracted from a rail application base camp. In the early years Rails, complexity grew slowly and Rails. 1.0 was released in December of 2005. Two years later, rails 1.2 was released that same year. Tom Preston Warner was at a Ruby Meetup in San Francisco when he showed his friend Chris Wayne's wrap a tool called Grit. Grit was a Ruby tool that allowed you to view Git repositories in an object oriented way. It would become the basis for GI repos on GitHub. After seeing grit, Chris was immediately hooked. In a few days later, they wrote GitHub Rails application was born.
GitHub was created using Rails 1.2 0.3, and after a short beta GitHub was released to the public into April, April of 2008. The next day, the Rails framework moved off of their own SVN server to using GI on GitHub. In 2009, RAs 2.3 was released. In the early days when RAs would release a new version, GitHub would quickly upgrade to get the new features and bug fixes was sometime between 2008 and 2009. GitHub forked Rails. Now remember, this was the wild West of rail startups. No one really knew what the future of Rails or GitHub was going to be. We weren't yet talking about the importance of upgrades or staying current with Rails Master and honestly, rails wasn't as stable as it is today. I don't wanna go as far as to say that Rails didn't care about performance or stability at this time, but I know a lot of app developers felt that way.
It definitely wasn't a concern the same way it is today. Maybe that's 'cause we were all kind of inexperienced back then, or maybe it was because Rails was good enough for to most important user Basecamp. Maybe it was because GitHub didn't contribute enough upstream for whatever reason, the problem wasn't Jet. That GitHub just forked rails and added a bug fix here, a performance improvement there. It wasn't just a fork with back ports from upstream. GitHub's Fork was Rails with custom code just for GitHub. It was RAs morphed into a different framework that was built for GitHub the way RAs was built for Basecamp, and as GitHub doubled down on their fork and added more and more functionality, RAs of course continued to progress at a fast pace as well. At the time, no one could predict or understand the cost that forking RAs would have on GitHub's application or engineering team.
In 2010, rails three was released, but many applications didn't upgrade due to performance concerns. The performance issues in 3.0 were a big deal. Users saw an unacceptable increase in response times. Some applications seen requests taking as much as twice as long active record and Rails Three was found to be five times slower than Rails. Two. Despite knowing about the performance concerns, a couple of GitHub engineers started to work on upgrading GitHub to Rails three. Now, the Rails three upgrade wasn't pointing at Rails three upstream either. It was still a fork of Rails. Three with GitHub's custom patches added on top. In 2012, rails three two was released and most of the performance concerns had been completely solved. In the same year, GitHub's progress on the rails upgrades stalled. It had been two years since they started the 3.0 upgrade and the engineering team began questioning whether the effort was worth it at all.
They asked each other, why upgrade when this version isn't causing us pain? Why upgrade when Rails 3.0 is just so great? Why upgrade when our fork has more features? Looking at these questions, the engineering team decided the upgrade wasn't worth their time and focused their attention on other projects. The truth is that at this time, GitHub wasn't yet feeling the pain of being on a fork, and it's difficult, difficult to convince a team to upgrade when they're still feeling productive. You make your test faster when they feel too slow. You factor the complexity of that class when you need to add new functionality. But when do you upgrade? What's the incentive? If the new version isn't better and your current fork is working just fine. If you're not feeling the pain of being on a fork or an old version, you're just not gonna be compelled to upgrade.
But of course, eventually it wasn't fine. All of those why should we upgrade questions started to become suffocating for the engineering team. It became harder to find where the framework ended and the application began. As GitHub engineers started to fight against the fork and the application security backboards were a nightmare. Every time Rails announced their vulnerability, GitHub was forced to manually patch it. Hiring was becoming increasingly difficult. No one wants to work on a Rails two, three application that doesn't even re resemble Rails. Two, it's harder to get up to speed and you can't Google search how to do anything. Dependencies were brutal and unsupported. As GEM authors focused on new versions of Rails, development was slow and painful. Working with an application that's tied so heavily to a custom fork can make adding features of refactoring code incredibly difficult. We realized we needed to get off a fork of that fork was gonna suffocate the application and the engineering team.
So in 2014, a team of four full-time engineers and a few volunteers banded together, wrote an upgrade plan and got to work. I took the team six months of full-time coordinated effort to deploy Rails three to production, and it's important to remember that Rails three here is still a fork with GitHub custom patches added in. By this time, the Rails three series was only receiving severe security patches. So even though the upgrade was a success, the code base and fork were still very far behind. The effort put into upgrading from two to three was massive and motivation dwindled. After that, it would be another two years before the four oh upgrade was even started. That same year, rails five came out, it felt like GitHub was just never gonna catch up. If Rails was constantly improving and consistently releasing new versions, if GitHub couldn't prioritize upgrading and get ahead, how would we ever get off a fork?
But in 2017, I joined GitHub. At this point, the Rails four upgrade was not in a good place. There was no dedicated team working on it, and the upgrade had fallen by the wayside. When I ran the build to see how many failures needed to be fixed, I found out there were over 4,000 errors. Easy, right? It's hard to put on a timeline just how much work and effort went into this upgrade. It's one of the hardest things I've ever done, not from a technical perspective, but from a focused and determined perspective. Upgrades are lonely, difficult, and often thankless feats of mental strength. In March of 2018, a year and three months after I started at GitHub, we deployed our first of three major upgrades, the RES four two upgrade. After deploying four two to production, we immediately started the res five upgrade. But this time with a larger team, we didn't wanna lose momentum or fall behind like we did before.
For the Res five series, I led a team of four full-time engineers and spent time perfecting our dual boot and issue assignment process. Because we had a dedicated team and a streamlined process, they upgrade from four two to five, two took only five months, and in August of 2018, we deployed Rails five two to production. Uh, both of these deploys went out with zero downtime and no customer impact. This is a huge milestone. It was the first time in 10 years, 10 years that GitHub wasn't on a fork of rails. It was the first time in 10 years that GitHub was on the most recent version of Rails. That's 10 years of cumulative technical debt. 10 years of fighting our Rails fork in our application, we had finally started to pay some of our massive debt down by upgrading. We of course don't eliminate technical debt, but it creates a starting off point for removing even more tech debt.
It allowed us to find things that didn't belong in the application that could be upstream or remove infrastructure requirements that were already supported by Rails internals. I hope that learning about how about our upgrade and how long it took hasn't scared you into not doing your own upgrade. The point of this talk isn't to tell horror stories, is to show you that that there is a cost to not upgrading. Refusing to upgrade is more expensive and more risky than taking the time to upgrade. I won't lie to you and say that upgrading doesn't come with a cost. It is expensive and it is time consuming. But you can say like this upgrade will take X number of engineers, a y number of dollars for C number of hours. You can measure that cost and decide whether that's too expensive or not. But at the end of the day, it doesn't matter what the dollar value of upgrading your application will be because the cost of not upgrading is immeasurable.
You cannot quantify the cost of not upgrading. We can't actually measure how much it costs GitHub to fork rails and not upgrade how many extra hours it took to figure, figure out how to do something in the fork that could easily be done in a standard rails version, how many extra hours it took to manually patch security vulnerabilities, how many potential hires we lost, how much time we spent re-engineering features that were in later versions of Rails already not upgrading. Our application will eventually cost more than any upgrade because of these reasons. This is true whether you're using Ruby or Elixir or Java. In addition to the high cost of not upgrading, often teams say they aren't upgrading because it's too much of a risk. My unpopular opinion after doing many upgrades is that upgrades are not inherently risky. We worry that changing the underlying foundation slightly will take everything down, but if we're careful, if our applications are well tested and we build up risk tolerance and confidence through increment incremental deploys, there is no reason to think that upgrading is higher risk than not upgrading.
There's also no reason to think that upgrading is higher risk than any PR that anybody on your team is deploying right now. In many ways, not upgrading and falling behind is drastically riskier than staying up to date. When you don't upgrade your application, you have to become a security expert. Rails like many other programming languages and platforms drop support for fixing vulnerabilities in older versions because it becomes untenable to support them. Think about that. If the team that builds and maintains the software that you're using thinks a version is too old to accurately support security patches, why would you continue to use that version or think that you're able to do better? If you're using an unsupported version, your team ends up being responsible for understanding and fun and patching that security vulnerability yourself. It's extremely hard to get this right. When you don't upgrade, you lose out on great talent.
Bootcamp grads, college grads, people changing careers, people changing jobs. All kinds of engineers aren't learning rails too, and they definitely aren't learning your weird custom fork. These engineers not only don't wanna work on an old version, they might not even have the knowledge of how it worked. Engineers will turn down the opportunity to work for you if you're on an old version because it doesn't let them contribute to open source. It's no longer a googleable and it's 10 years old when you don't upgrade. Some of the libraries you rely on will get abandoned or deprecated. So you'll either have to live with bugs or fork yet another dependency. New gems may not support old versions of rails, so you won't be able to use those if you're relying on an old unsupported version. This makes development harder. Every independency becomes more difficult because you're so far behind.
Maintaining old libraries on top of your old framework gets tedious really quickly and you're going to lose engineers because of it and we don't upgrade rails. You end up building more and more infrastructure on top of your fragile application. I've seen this firsthand at GitHub. We have tons of infrastructure code in our app, multiple databases, CI tooling, our own job queue and mul and countless monkey patches. Ideally, your application would consist only of code that makes up your product because GitHub's value is not in that. We have code that makes multiple databases in work in rails. Our value is in our community, in our repos, in our issues. Of course, multiple databases allows us to keep our application up and running, but it's not why anybody would sign up for an account. The infrastructure code makes development more painful because of how tightly a couples an application to framework internals.
Minor, minor changes can easily turn into massive refactoring or abandon projects altogether. But the biggest cost of not upgrading is that someday someone will decide that using Rails or whatever language you use was a mistake and that it's time to do a full rewrite on your application. I don't care what language your application is written in, regardless of the language, rewriting your application is extremely expensive. If anyone on your team says that an upgrade costs more than a rewrite, that person is lying to you at this point. Hopefully, I've convinced you that that upgrade that you're putting off is important and needs to be moved up into your goals for this quarter, not next quarter, but you're wondering how you're going to accomplish this Herculean task. The key to upgrading any application regardless of language, is to incrementally pay off cumulative technical debt that you've incurred and figure out a plan to keep that debt paid down.
I won't stand here and tell you that upgrading will be easy. I'll leave that to the person on Hacker News who wondered why I couldn't do it faster if they did their uh, upgrade in a weekend. The upgrade did take a long time, but it wasn't the only thing that I worked on. I worked on our technical debt. I deleted unused features. I reread our custom test framework so that we could be more confident in changes we were making. I improved our database handling for mul de development and test. It's unfair for anyone to look at our upgrade timeline and decided that using RAs was a mistake, that it's too expensive. We made choices at GitHub over the years that made upgrading harder. You have also made choices over the years that will make your upgrade harder regardless of language. But that doesn't mean that Rails is a bad choice.
Technical debt is real and you and your team need to make decisions about what debt is acceptable and what debt needs to be cleaned up. At GitHub, we decided being behind Rails master is no longer a technical debt we were willing to put up with. You can slowly work on technical debt and upgrade rails to get your application in a better place, but there are a few things that you should consider when upgrading and many mistakes to avoid so that you don't end up doing a multi-year upgrade like we did at GitHub. The first most important step is to build out a team. Upgrades are difficult and it helps to have a team that can support each other, bounce ideas off of and to make sure momentum keeps up. If you have a one person upgrade team and that person leaves the company, your upgrade is gonna be stalled.
Make sure to create redundancy and support for such a difficult and important task. And lastly, reward this work. Don't just reward feature development. Maintenance is far more important because you don't get to develop features if your foundation is crumbling. Another thing that makes upgrades easier is like, take time to plan your upgrade. Look at release notes, plan for deprecation. Identify unmaintained gems. There are clever ways to make upgrades easier and less painful. I get how we built a dual booting CI so that we could test all of the same code in the product for multiple RAs versions. This allowed us to prevent regressions in all new code while we fixed older code. Uh, these are things to consider. There are lots of things to consider, uh, before starting your upgrade because upfront investment might save you time and money later. You can also make your upgrade easier by fixing deprecation warnings early.
Don't ignore them for the next version, uh, to take care of 'cause then they're gonna be errors. It's much easier to take care of deprecation when there are warnings 'cause you can do them at a slower pace. It'll make your upgrade go a lot smoother and once you upgrade it to the most recent version, make a plan for future upgrades. How often are you gonna upgrade in the future? Are you willing to test new releases in the beta or release candidate phase? This makes upgrades smoother because you can tell maintainers what was hard and what broke so they can make the version more stable for next time. Maybe they'll even ask for your help even earlier than that. If you've invested in upgrade, invested in upgrading, then it makes sense to invest in future tooling that will help you keep out of upgrade debt in the future.
So now that we've looked at some considerations, uh, that you should make an upgrade and we're gonna look at some things that you will definitely regret if you do them something you'll, something you'll regret in the future is forking your framework. It doesn't matter whether it, whether it's Rails or Phoenix Forking is a terrible idea. Don't do it <laugh>. I highly recommend that you do literally anything else besides fork. The choice of fork rails and deviate from upstream was the single most expensive choice we made at GitHub in regards to our application. This had a compounding effect on the state of the code base that made our upgrade take years. If we had just done it when the version was released, it would've taken less than a few months for each version. Another thing that you may regret down the road is falling behind, um, on security releases.
If you don't make sure your software is getting upgraded to the point where maintainers no longer provide, uh, security patches for your version, then you're gonna have to do it yourself. This is really dangerous because you might not know how the vulnerability even works unless you have a security team intimately fa familiar with the software vulnerabilities. It's gonna be really hard to make sure that you're patching all endpoints and it's gonna make your security audits a lot more difficult. It's much easier to verify that you're up to date than to explain their app is way behind, but you definitely patched all those vulnerabilities manually. Lastly, you'll likely regret deprioritizing upgrades. In the past, I've given this talk to many individual contributors. I have never given this talk to an audience of mostly upper management and company leaders until today. So this talk, this section is for you.
The only people at a company that can make sure upgrades are prioritized and properly staffed, rewarded, and supported our company leaders. It's up to you to make sure this kind of work is valued the same or more than product development because the language your application is written on is your foundation. Failing to upgrade arose that foundation and makes your up, makes your application more brittle and more unstable. If I could go back in time to before I worked at GitHub and before we fork rails, I would tell leadership to reconsider their decision to ignore the upgrade. I would tell 'em to staff a team and to make sure that upgrades happen all the time as soon as possible. The best time to do an upgrade is immediately. Every day that you wait to upgrade, the more painful it will be because the code base is a con is constantly evolving as your version falls behind.
Hopefully by now I've convinced you that there are a lot of negative consequences to not upgrading, but what positive benefits do you get by prior prioritizing this work? Besides avoiding the pain, what kind of benefits are there? When you prioritize upgrades, you get access to improved APIs. Major version changes allow maintainers to rethink how previous features were designed and whether they can be improved. For the next version. In Rails six, we improved multiple databases by extracting all of the handling. For that out of GitHub, input it into Rails before Rails six. Adding multiple databases to an application was awful and really hard. You had to write a ton of code yourself, but in Rails six, we've added new APIs for establishing and handling connections as well as an improved API around active record database configurations. Without upgrading, you don't have access to these improvements. Upgrading to a new version also gives you access to security features.
In addition to security upgrades being easier if you're not behind. Each version adds new security features to protect your application and your users from bad actors. Since these are security features and not vulnerability patches, the only way to get them is to actually upgrade. They will never get back Ported. Upgrading often provides you with performance improvements. While the most of this talk today has been about Rails upgrades, we recently did a Ruby two seven upgrade at GitHub. We found that running on this version reduced our pro production boot time from 90 seconds to 70 seconds. Performance improvements on the language level benefit the entire application in a way that one-off application level performance improvements just can't upgrading also gets you access to new libraries. Rail Ship Rail six has shipped with brand new libraries like Action, text, action, mailbox, and site work. By upgrading, you can rely on great new features and rails without building your own infrastructure tooling into your application.
As I said before, in an ideal world, your application would only contain code that pertains to your product. If your product isn't about sending mail, why should you have to write code that sends mail? Upgrading Rails keeps the line where Rails ends and your application begins crystal clear. And lastly, upgrading your application gives you a chance to contribute. Upstream is really difficult to fix Bugs, add features and influence the future of your language or framework from an old version. A lot of times we complain that our language isn't fast enough. Framework is missing a feature, but that's the best part about Open Source is you can change it. If you don't like your language, fix it. While all of these are good reasons to upgrade, this is the one that kept me going. This is the reason that I spent a year and a half on the res upgrade.
It was so that in GitHub could influence the future of Rails. For GitHub, this is the biggest and most important reason to upgrade by Upstreaming features, fixing Bugs and Supporting Rails Future, we support our future as well. When I look back at when I started the three two to five two upgrade, the thought of being on a modern version of Rails and actively contributing upstream felt like an unachievable fairytale. We were so far behind and I wasn't sure that I could finish it, at least not while maintaining most of my sanity. But now running on Rails Master and supporting the future of Rails is GitHub's present and future state. Since upgrading Rails, GitHub engineers have sent over 200 pull requests, improve performance fix bugs, and add major functionality to Rails. And many of these contributions were first time before we upgraded. We often were forced to add a monkey patch or work around bugs in overly complex ways.
Using the most recent version of Rails has allowed us not to not just contribute upstream, but to make choices that improve our application instead of hinder it. And anyone who uses R Ls benefits from those changes as well. Our goal is to never fall behind again. We're investing in our application and in rails by doing continuous upgrades. Every week we bump our rails gym and run all of GitHub's tests against the new version. We find regressions quickly, often within hours or days of when they were introduced to RES and get them fixed before anyone ever experiences them. This has created a symbiotic relationship between GitHub and Res, where we developed the framework and code and applications side by side. Since we did the hard work of upgrading major version of upgrading our major versions, this new process means that we never fall behind. And upgrades are easy because we're doing them as rails changes instead of many years later.
Continuous upgrades in our contributions to Rails are evidence that for the first time in GitHub's history, we're not just using Rails or simply building a Rails application. For the first time in GitHub's history, we're pioneering the future of Rails. GitHub and Rails are being co-developed together. Side by side, we're extracting code from GitHub, building new features that will help you scale rails and giving back to the open source community to support Rails for the long haul. It makes good sense for us to do this. Not only do you do, we give back to the community, but it helps us keep our application focused on our product. We can reduce complexity and improve resilience all while influencing the future of Rails. At GitHub, we not only need to do this for our application, we have a responsibility to support Rails. We owe part of our success to the Rails framework and we have the influence, expertise, and application to help push rails forward.
Every single company in the world, big or small, is able to do this work. You can upgrade, you can influence your framework. You just have to make it a priority. Upgrading Rails was a huge investment. It wasn't cheap, but it was definitely worth it. Upgrading Rails has opened up a ton of possibilities for our future, and we have a bright path ahead of us. We're building features faster. We're more confident. Our codebase is stable. We're improving the scalability of Rails and we're giving back to the open source community. At a minimum, we went. We went from being crushed by our application rails and the decisions that we've made to building up our application, to building up rails, to building up our community. Upgrading Rails gave us freedom and flexibility that we didn't have before. It empowers our engineers to build more and build better. In 2007, GitHub was born and 11 years later we're finally tracking Master.
It took seven years from the day the 3.0 upgrade was started to the day that the five two upgrade was complete. The future is bright and I can't wait to see what the next 10 years allows us to extract from GitHub build in Rails and how the community thrives. At GitHub, we'll continue to invest in the future of Rails in our community because we have to, because we need to, because we want to. I hope that this talk has inspired you to make upgrading and investing in your language or framework or priority. It doesn't matter what language your application is written in. This is important work that needs to be supported and rewarded. Like I said before, if you don't like your framework or language, change it. This is how you build and maintain a strong foundation for your application and your company. For us at GitHub, we'll continue to invest in the future because GitHub and Rails are in it together for the long haul. Thank you.