“The Yahoo! Story” How Yahoo! Embraced Automated Testing to...

Hey, guys. Really appreciate you guys showing up here and listening to my talk. With no further ado, let's get started.

My name's Neil Manvar. I'm going to be presenting the Yahoo! story, and it's going to be about how Yahoo Mail embraced automated testing to improve business efficiency online.

A quick intro about myself, so you can understand where I'm coming from and also what I'm trying to do in the future. My current position right now is a solutions architect at Sauce Labs. I've been there for about six months. Previous to that, I was a front-end software engineer at Yahoo! for Yahoo Mail, where I worked on a lot of front-end features, as well as the test automation framework and the whole continuous delivery pipeline, and that's where I got the majority of my experience.

Before that, I was also an intern at Yahoo! for two summers when I was studying at UC Davis, getting my bachelor's in computer science and engineering. So I've been on and off with Yahoo! for a little while, including the internships and doing front-end development. Some of my hobbies include UFC, cycling, and playing basketball.

First, we're going to start off with the agenda, and I'm going to show you what we're going to cover today, and then we'll go from there. First, I'm going to be describing the Yahoo Mail team and how they release and operate, or how they used to release and operate, more accurately. Then we're going to go over their first attempt at solving their release and testing challenges, and then we will analyze that and see how it worked out.

Then we're going to go and look at some company and organization changes that happened throughout the times at Yahoo!, and how those changes affected how Yahoo Mail operates and releases code today, presently. We're going to see how they attempted to solve their release and testing challenges after these new company and organizational changes happened. At the end, I'm going to conclude with key takeaways and what I'm looking for.

Yahoo Mail used to be about 200 people, including front-end engineers, back-end engineers, middle-tier service engineers, release engineers, and the product team, even designers, et cetera. They used to do biweekly releases, which we used to call a train or a sprint. Meaning that at the end of every two weeks, that Friday, we would stop what we're doing and just focus on releasing and getting that code out to production.

Just to put in terms the scale that Yahoo Mail kind of deals with and operates with, it's terabytes of data going through the system every day. Terabytes of emails, attachments, metadata, you name it. That literally translates to millions and millions of emails and all that traffic going around every single day.

Right now, we just think of Yahoo Mail as just a basic email client. We just go to mail.yahoo.com, you see your email, but it's, in reality, a lot more than that. We support a lot of partners. For example, Yahoo Mail is the email client for Nokia Mail, SBC, amongst other partners. They pay Yahoo! to use the Yahoo! front end and back end for their mail, but just badge it as Nokia or SBC.

We also support about 90 different languages, including Hindi, Portuguese, obviously English, and even BiDi ones. So it flips around. It reads from right to left rather than left to right. This is all one code base supporting that.

Also support a lot of browsers: IE9 and above, all the way to Edge, all the different Chromes, all the different Firefoxes, as well as Safari, and different platforms, meaning XP, Mac, Linux.

Yahoo Mail is a very critical application in that an email outage has serious consequences. Obviously, the users are not happy, but email is essentially modern-day mail. You don't go home and then put the key in the mailbox and then expect it not to open. Same with email. You have information that you need to access, and incoming stuff that you need to read, and emails that need to be sent out, and you expect your email to work every single time. Otherwise, you can't do business the way you can.

For example, if I was going to a resume and I needed to print out my resume, which was in my email, and I couldn't do it, then I'd have to go to my interview without a resume, and that would just look bad, and I would probably never use Yahoo Mail again. So it's one of those things that you expect to be up.

Also, every time we're down or we mess up, we have to pay the advertisers back because they have paid for ad inventory, and if we're not flashing that ad for that period of time, we have to give them their money back. So there's also a lot of money involved here and a lot of financial concerns.

We're going to go on and talk about how Yahoo Mail released in the old days. I refer to them as the not-so-good old days because releasing was very, very tough and often resulted in a lot of change-related outages and caused problems and diverted us from development.

The release cycle looked as follows, where we would do one week of development, so the developers would do all the code changes and check it in. Then we'd deploy that code to a test server, after which the quality engineers and manual testers would hammer it and test it and make sure that it works in all the different browsers and operating systems, and all the features are working as expected amongst all the different languages and partners. So you can imagine how complicated that gets.

Then at the end of the two weeks, that Friday comes, and we do the final whiteboard verification. We can see here that there's some features on the left and there's some configurations on the top, such as Firefox 37, Chrome 41, and some features such as folders, compose, and attachments. So this is just a very simplified one. You can imagine how many browsers there are, how many operating systems, and how many features amongst all the different languages, all the different partners, and it just gets increasingly more complicated.

This table that we see here is nothing like the actual one that we used to use at Yahoo!. The actual one took an entire whiteboard, and you would fill your name three times, meaning that you are going to test three things. Right before we release, everyone's responsible for testing three.

We used to do this final whiteboard verification, where you put your name down, and then you say, "This feature works or doesn't work." That would declare whether we push to production or not. If everything's okay, we would push to production. This whole whiteboard verification and manual testing would happen every two weeks.

Now we're going to dive into the not-so-good old days and see why it wasn't scalable and viable. Like I mentioned, every two weeks, everyone had to stop what they're doing and start manual testing. Not everyone had access to all these different Windows machines or all these different browsers, so a lot of time was spent doing manual testing rather than writing code or writing automated tests.

As more features came out and more browsers came out, that basically meant that we needed to do more manual testing and more time needed to be spent on that. We noticed that over time, the developers and testers were just spending more and more time manual testing, and we were getting less and less features out.

Also, another serious concern is that if we would find a bug or a feature that was broken, we'd have to stop what we're doing and repeat the whole waterfall process. Meaning we can't push this code. We need to stop, go back to development, develop, make sure that's okay, push it to test server, test, and then push to production.

With this high dependency on manual testing, we were releasing a lot of bugs into production. For instance, I showed the matrix verification, and I had Kevin and Neil on there. Neil is me. So if Neil showed up hungover or tired one day, he probably wouldn't do that testing very well on that Friday. And if that wasn't tested very well, that feature would go out and be broken. So let's say I didn't test compose on Firefox 37 properly, and it actually did not work. That means when we push to production, some portion of our users are not going to be able to compose in Firefox.

Another big problem with this was that automated testing was not really encouraged. You do your week of development, the QAs or manual testers would do their manual testing, and then you'd do the whiteboard verification, and there was no room for automated testing. You depended on catching everything through the manual testing process or letting the QA catch everything for you.

Really, automated testing wasn't encouraged, and there was no room left for that. This eventually turned into a vicious release cycle. Every release got worse and worse and worse and more complicated, and we were having more outages than ever.

Some negative outcomes of the not-so-good old days was outages and broken features. It could be browser-specific, like I mentioned, Firefox 37 doesn't work; or language-specific, saying that Yahoo Mail doesn't work for all the Hindi users; or feature-specific, compose just plain and old doesn't work on any browsers; partner-specific, it doesn't work for AT&T, Bell, Rogers, or Nokia.

One of the worst ones was the display ad was not showing, which meant Yahoo Mail was up, but we were not making any money because the ad was not showing, and Yahoo! is an ad revenue model. So no ad, no money.

Also one of the worst is a combination of the above, meaning that compose doesn't work in Firefox in Hindi, and debugging that was a little bit tougher.

The first solution was to invest heavily in automated testing. What happened is the quality engineers wrote up this framework called SelNG, which was Selenium and TestNG, and it used Java, Selenium, TestNG, and Eclipse.

They also started supporting an internal grid where they would host VMs that would have a few browsers, such as IE9, Firefox, and Chrome, and a couple of people were assigned to managing that grid, making sure that it was up and running and we could run automated testing on that grid.

The third part of this solution was invest in downstream nightly builds. Meaning when everyone goes home and is sleeping, there's all these automated tests running in all the different browsers that the internal grid supports, and in the morning, we would get the results.

This looked like a great viable solution, but it really proved differently as time went on, and I'll explain why it didn't work out.

The outcomes of solution number one. Java was not a prominent language amongst front-end developers. In the front-end web world, it's usually JavaScript. The developers are writing JavaScript. So they didn't really want to stop what they're doing and learn Java, and also adopt Eclipse, and basically adopt some tech stack that they weren't familiar with. So what that resulted is only the quality engineers writing the test cases, and they were playing catch-up. The developers would develop something, and then the quality engineers would try to automate everything that they developed against the various browsers.

The internal grid was also a problem because it was a bunch of stagnant VMs with some browsers on them. These were not pristine VMs. Eventually, disk space would fill up or temporary files would fill up, and your test would fail not due to your source code or your test code, but due to test infrastructure, which is a false positive.

If you're trying to implement a CI or CD pipeline, you cannot have false positives. You need your test to be as reliable as possible and your test to say the truth. When it fails, that means some feature is not working or the test was written incorrectly.

Also, the internal grid was unstable as browsers needed to be updated very often. The operating system needed to be updated. It needed to be restarted. Basically, the internal grid was not giving the results that we wanted, in addition to only quality engineers were writing the tests in Java.

The downstream nightly builds, they would execute at night, and if you come back in the morning and you would see, hey, this feature is broken on Chrome, or this feature is broken on Firefox, it forces you into this act-and-react model. Whereas you came into the morning and you realize, oh, man, our product doesn't work on Chrome. Then you have to run to the developers and the development manager and say, "I think someone messed something up." Then you have to look through the commits, figure out the offending commit, and roll back and go from there. A lot of time was wasted because you're not developing or writing test cases, which is essentially what Yahoo Mail was in the business of doing.

Yahoo Mail needs to make sure it's delivering new bug fixes, new features as soon as possible, not making sure and debugging why some new code does not work on Chrome or Firefox or IE.

Also, the tests were not a first-class citizen in the pipeline. When someone broke a test, there was nothing to be done to fix it. When I say first-class citizen, that means when an automated test fails, the pipeline should be halted. That means there is something wrong. Some test case is written incorrectly or some source code is broken, which is serious, and someone needs to address that.

Having downstream builds that just run at night and having no action items, and having the act-and-react model, tests were not a first-class citizen in the pipeline, and no one was doing anything to fix them or incorporate them into the pipeline.

Now we get to the company and organization changes. We all know that Yahoo! has been around for quite a while and gone through a lot of transformation, from probably having five CEOs in the last six or seven years and a whole bunch of new management changing their objectives and mission statements and whatnot.

But a large portion of the company and organizational changes was due to new management, a new set of executives coming on. Most recently was Marissa Mayer, and she really saw how things were being done at Google and how things were being done at Yahoo!. She really wanted to kind of bring that Google culture, that new mentality, fast-paced, agile, into Yahoo!. She realized that it has to start at the management. It's a management buy-in, and then it's top-down.

What she's prioritized and the other executives, it was, we're going to make a move to agile. We want accelerated release cycles, which essentially translates to, we want continuous delivery pipelines.

As a part of this transformation, my old manager, who doesn't work there anymore after this transformation, he said, "Neil, there's no need for testers in Mail or no future for testers in Mail. You need to go to development." And my new manager, once I moved to development, he said, "I only hire developers that know how to test." So he wanted everyone to become general-purpose engineers that knew how to both develop and test.

He tasked me with the task to create a test automation framework that the developers would love and we could scale out. Not something that was Java-based that only a few people would use. He wanted everyone writing test cases.

I think he tasked me with this because of my previous quality engineering experience in my internship. So I saw how things were done incorrectly, and he wanted to kind of help me make them right, and he thought it was a good project for me, and I enjoyed it because I learned a lot.

I'm going to go through each of the parts of solution number two individually. The first one was adopting agile.

We wanted to embrace agile and become more agile. What that translated to is having cross-functional teams that could develop and test together, not having the waterfall method of doing a bunch of development, and after that's done, doing a bunch of testing. When you write some source code, you should write some tests for it. This is what practices like TDD and BDD really advocate, is that you do the development and the testing together.

As part of that, we needed to implement and practice continuous delivery. So make a pipeline. What this translated to at the engineering level is make a pipeline where each commit could be delivered to production unintervened, automatically. You can imagine the role automated testing plays in that, because to get each commit to production without any manual intervention, you have to run those automated tests in all the different browsers, making sure that your product or your new feature is working as expected.

Then also, we needed to create a test automation framework that was easily adoptable, maintainable, and scalable. So everyone could essentially write test cases.

The last part was testing in isolation. These are the four things that consist of solution number two. Once again: agile, continuous delivery, creating a really viable test automation framework, and testing in isolation.

When I say test in isolation, it means you test one individual commit or one individual pull request, not a bunch of commits that go into master and then do the test. You want to test only one change at a time, certify that change.

The part of the test automation framework, like I said, it needs to be used by everyone, and learning it and onboarding it should be easy.

The first step to this was becoming more agile. Previously, Yahoo Mail was operating in the waterfall methodology, where you would do development for about a week or two, test for about a week or two, and repeat if necessary, and if not necessary, push to production. Now what we're trying to do is embrace agile and do the development and testing together at once.

What this really meant at the engineering level, once again, is that one pull request would contain the source code, the unit test, and the functional test. Not, let's check in the source code and deploy it and we'll write the test later. When you're fixing a bug, you write the unit test so that whatever your bug fix was is getting code coverage, and you also write the functional test, making sure that that bug never happens again. If you're writing a new feature, you write all the source code for the new feature, all the unit tests, and the functional tests, such that when it does get pushed out, it gets tested and pushed out.

Part of the pull request was also peer review. After all of this, the source code, the unit test, and the functional code was written and submitted as a pull request, it would be up to my friend Kevin to look at it and say, "Hey, Neil, I don't think you wrote enough functional tests. You're only covering one flow." Or, "Neil, I think you introduced extra for loop that you didn't need, and we could make this more efficient."

So there's always the peer review. What this essentially resulted in is never producing any legacy code. Legacy code translates into unmaintained and untested code. If you have tests for your code, you know exactly what your code is doing.

Doing this model where the pull request would contain the source code, the unit test, and functional test, it created maintained, very healthy code, not code that would go outdated and become legacy soon.

The second part of the solution was crafting a test automation framework, which meant leveraging open source technologies and leveraging good technologies for automation. What that meant to me is I'm going to go ahead and use a high-level language like Ruby, which is really easy to learn and has a lot of good tools, such as Cucumber, Watir WebDriver, Page Object, and Parallel Tests, all which I used in my test automation framework.

Cucumber is that English-like BDD framework for writing test cases. So you could say, given I'm logged in as Neil in Yahoo Mail and I click compose, then I should see the To field. That would be an example of a Cucumber test case. Now, if anything went wrong in there, I know exactly why, because if the To field verification fails, it could be either the compose wasn't clicked or the To field wasn't present.

Having it in English allowed for a lot of readability and made debugging easy, and also maintenance easy.

A large part of this was having all the documentation and training, onboarding all the developers and the engineers into writing test cases in this framework. Documentation and training was very critical.

Also, using the Ruby tech stack, I was able to leverage gems and be up to date with the newest Selenium and all the other dependencies and tools that I used in here.

At the end of the day, the developers were all saying, "I love writing tests, and it's easy, and I'm definitely going to do it." That really helped because management was enforcing the writing of tests and the developers were loving it. There was this natural marriage where good code was being put out, quality code, meaning that it was tested.

Another part of the solution was continuous delivery. What continuous delivery translates to is delivering software faster, essentially, and more frequently. What that meant at the engineering level is that, I mentioned this before, every commit should go to production unintervened.

To do that, what we did is we kept the master branch stable and deployable. So anything that made it into master, we could deploy at any point in the day. We knew that it was tested and it was stable. Under this paradigm, we tried to deploy to production at least once a day so that users would get the new bug fixes and the new features every single day.

I really like using this analogy. I say code is like inventory. Imagine you had a shoe store and you had a large inventory of shoes. There's no point of hoarding onto all those shoes. You want to sell those shoes and then get new inventory and keep selling those, and then repeat this process. That's how you run a successful business.

Code is the same way. There's no point of having code that's ready to be released behind your firewall and just sitting there. It has some bug fix or some new feature that the users can use, and it'll result in a better, more quality experience. So code is like inventory. Ship it. If it's ready to go, ship it. That's where continuous delivery essentially came out of, is we want to push these updates and all the good stuff to our users frequently.

The last part of solution number two was test in isolation, and this one is the most technical of all of them, which basically meant test on each pull request submitted, not on each commit into master or set of commits into master.

When a developer submits a pull request, it should automatically get tested and certified. What that did is made sure that the bad code never made it into master. The mechanism for doing this was a pull request build. My next slide is going to show a diagram, which I'm going to walk through the pull request build.

Also, the pull request build, what it really helped with is enforcing over policing. In the pull request build, I could mandate that unit tests and functional tests should pass. I could even, and I really enjoyed doing this, I added a code coverage enforcer.

So if my project had 75% code coverage, and my friend Kevin raises a pull request and it drops to 73%, it would automatically fail the pull request build and stamp the pull request as not safe to merge. Kevin didn't write unit tests. So I could always maintain my 75% code coverage or better through this model and make sure no untested code made it in.

Here is the diagram of the pull request build. A developer would submit a pull request. The pull request would automatically create a job inside Jenkins, which would run the unit test, deploy if necessary, and run the functional tests, and also the code coverage checker or anything else that I mentioned.

If all of that succeeded, it would come back and stamp the pull request as safe to merge, meaning that this code is good. It was tested, it has code coverage, and it didn't break any features on any browsers, et cetera.

If it was not safe to merge, meaning it broke some feature or it didn't have the proper test coverage, it would come back and say, "Not safe to merge."

So let's say it was safe to merge. Then my friend Kevin would look at it, see that everything's okay, do his peer review, and then merge it in. After it's been merged in, I know it can be pushed to production since it's already been tested in all the different browsers, all the different operating systems, and the unit test ran, as well as my code coverage checker.

Really what that did is it made sure no untested or bad code ever made it into master. Everything was isolated at the pull request level and certified at the pull request level.

This in entirety avoided rollbacks. We never needed to revert any code because all the code that made it into master was tested, and master stayed stable and always deployable.

A big part of this is enforcing. As a part of the pull request build, I could enforce rather than police. I didn't need to depend on the peer review for code coverage and functional tests being ran and syntax and coding standards. It was all enforced and mandated by the pull request build.

The outcomes of solution number two: basically, developers were held accountable for their own source code, meaning that if you wrote a bunch of source code and somehow got it through the build and didn't write any tests, and that feature went out to production untested and was broken, it basically falls on you. You could've run a test that would've made sure that it would've never been merged in. So it really held developers accountable for their source code, and when something went wrong, they were required to fix it and write tests.

What this also did is it resulted in an accelerated release cycle. Because of this, master was always stable and we could always release from master. We didn't have that vicious release cycle that every second Friday everyone had to test manually, and pick three different things to test on different configurations and different features.

We were ready to release at any given moment in time, and we were releasing frequently. You'll hear me say this a lot: deliver quality code frequently. What that translates to is quality code means it was tested. Releasing frequently, that's continuous delivery. We want to get those bug fixes and those new features out to all the users as possible.

Essentially what that resulted in was business acceleration. Developers were developing, testers were testing, and the development and testing was being done together, and that's what Yahoo Mail was in the business of doing: releasing new bug fixes and features. People were working on exactly that, not focusing on releasing, but focusing on the work that needed to be done.

In the end, that increased productivity and efficiency. That enabled Yahoo Mail and essentially Yahoo! to grow. The result was a better, more stable product, as well as happier customers.

Some challenges. I talked about all the good stuff and all the rewards and benefits you can reward from continuous delivery and pull request build and all of that, but there are a lot of challenges.

There's the CI, version control, and app integration, meaning in terms of Yahoo!, we have to integrate Jenkins with Git, GitHub with our Yahoo Mail app. And testing, you need to make sure that you're able to test in all the different operating system, browsers, devices if you're doing mobile, and all the different browser versions.

As part of the second solution, we chose not to go with the internal grid, but rather use an external cloud grid provider that could maintain all the different browsers, and that was their business, to maintain all the different browser versions, operating systems, and maintain the testing infrastructure.

Also, the speed. We needed the test to be done as fast as possible so that we could merge a pull request within 10 minutes of submitting it. If it took two days to run the test, that would be two days until the pull request could get merged and get pushed to production.

Also, culture. We needed to get the developers and the general engineers writing tests. That was a top-down, like I explained, and also bottom-up, they needed to love writing tests.

Also, infrastructure and scale. We needed to automate the deployment and CI as well.

Lastly, I just want to go over some key takeaways. Use good, maintained technologies. Often that is open source stuff that's maintained by the whole industry. So choose good, maintained technologies such as Selenium, Cucumber, et cetera.

Develop and test together. Otherwise, you're going to end up playing catch-up. That's the stage that you never want to get to.

The PR build. I showed you the benefits of that as well.

Continuous delivery essentially translates into business acceleration because you're concentrating on what's important, the work that needs to get done.

Deliver quality code frequently. Quality code means it's tested. Delivering frequently means you're practicing continuous delivery.

Last slide. What I'm looking for is kind of making this really achievable for the industry. Yahoo Mail has done this, Google is doing this, I think Facebook is, and a few others are doing this, but I want to make tools and make some product that essentially can output a continuous delivery pipeline with a pull request build for the industry.

My first step is making functional testing easier. I have this GitHub that has all the different functional testing frameworks, so onboarding onto automated testing is going to become a lot easier.

That's all I really had. Let me know if you have any questions. Feel free to email me at neil@saucelabs.com. I really enjoyed talking and really enjoyed all you guys attending. Thank you.