Everyone's Building Now: Thrive Market's AI Adoption Journey, and What We Learned Along the Way

Host Intro (Gene Kim)

All right. The next speaker is Mike Winslow, who I met in 2018 when he signed up to give a five-minute lightning talk at our conference, and it was called "There Are No Side Projects," and it's one of my favorite lightning talks I've ever seen. He led engineering teams behind Xfinity Mobile, Xfinity Stream, and X1 Entertainment System, and he was promoted to senior director of engineering for his amazing work.

And then he did something very unusual, at least back then. He chose to step out of management and become an individual contributor again, as a distinguished engineer, a rank held by less than 1% of engineers at Comcast. And I was so happy for my friend when he became a director of engineering at Amazon Music, a lifetime dream of his to combine technology and music.

And today he's vice president of engineering of platform and SRE at Thrive Market, and he'll be sharing how he's helped lead AI journeys there. And I'm so excited that he's co-presenting with his colleague, Marya Caples, senior food safety and quality manager. So here is Mike and Marya.

Michael Winslow

Enterprise AI Summit, how are you guys doing?

All right. I guess I walked out here without a clicker. That's how excited I am.

All right. So I'm Michael. This is Marya. We're going to start by telling you about our company, and then we'll tell you a little bit about ourselves. Marya, why don't you tell us a little bit about Thrive?

Marya Caples

Sure. So we are an online-only grocer. We have a membership model, so our members are given access to what we would call better-for-you products at lower and affordable prices. That's our mission — to provide better-for-you products at affordable prices.

And so in addition to that, we also have a very curated catalog. There are tons of ingredients that we don't allow on site — things like GMOs in food. And we also provide a shopping experience that is unique to our members' needs. So if someone has specific dietary restrictions, like they can't consume peanuts, they can shop our peanut-free filter. Or if they're wanting to find items that are low in sugar, or have a certain certification like gluten-free certified, they can easily shop those products on our website.

Michael Winslow

Nice. And our shopping experience is 100% online. There's no brick-and-mortar shops, and we deliver to your door. And everything gets organized currently at our three major fulfillment centers — in Pennsylvania, close to me in Philadelphia; there's one in Batesville, Indiana; and then there's one in Reno, Nevada. So we're really excited. That's where all the magic happens. The fulfillment center employees pack it up, and then it gets distributed to the last mile to get to your door.

And we are a certified B Corp. And for those who don't know, it's very not common for an online grocer to get this kind of distinction. But we were the first to do so quite a few years ago.

Now we have about 1,000 employees in total, and our engineering organization is about a fifth of that.

All right, so I'm going to start with myself. As you heard from Gene, I was at Comcast. I was at Amazon. And so what would bring me to something like Thrive Market? What brought me is a particular person that you heard from earlier — I could not wait to work with Randy Shoup. And late last year, we met at a coffee shop in San Francisco, and he told me about an opportunity to come over and be a part of a group where I could lead up platform and engineering with quite a bit more than a sprinkling of AI mixed in there. And between the role and working with great people like Randy and the others that I've learned to work with, that was my motivation to come to Thrive Market.

Marya Caples

And a little bit about me. I'm Marya Caples. I'm the senior food safety and quality assurance manager. Our FSQA team is even smaller, I think, than the engineering team. There's two people, including myself, that manage the food safety side of the business — so making sure that no one gets sick and that our products that we're selling are shipped to members in good quality. And then we have a set of three nutritionists who are really honing in on the regulatory side of the business. So they're validating any claims that we have on our packaging and on our website, and really staying abreast of all the regulations, both federally and state and local legislation, which is happening constantly.

And so I'm obviously not an engineer. I have basically no experience coding. I'm a millennial, so I kind of make the joke, like we all do, that our experience was coding on MySpace when we were in our teenage years. I don't think that counts as coding. I basically know probably the hex codes for black and white, and that's the extent of my knowledge.

So I was really excited about the opportunity to help develop some tools for our nutrition team to use in order to make their jobs easier.

I have really been blessed with opportunities at other companies that I've worked in to learn about continuous improvement and Lean Six Sigma. So shout out to Art for his presentation on Toyota, because they really paved the way. But I noticed that our team had a problem.

Basically, in order to get products into our system and be sold on our site, our team has to vet them. And we have a long list of over 1,000 ingredients that we restrict in certain food categories, or in our home care or bath and body. And so what the process usually was — our category managers or buyers essentially would get an ingredient list from a vendor and look up every single ingredient in a spreadsheet to see if it was allowable, if it was flat-out unacceptable (like we can't sell this on site), or if it was allowable with some conditions. And some of those things were like: if it was glycerin, we had to confirm that it was from a non-GMO source; if it comes from a certain region, we have to make sure that it's ethically sourced; there are certain items in our bath and body care that we only allow minimal concentrations of that ingredient in the formula.

So the process with our category managers, our FSQA team, and our vendors was really labor-intensive. A lot of emails back and forth, collecting documentation, asking questions, and it was just pretty unorganized. We didn't have a system to track any of our decisions and approvals. We didn't have a central place to store our documents, so they'd be in someone's email inbox or in Slack — just the normal problems that organizations have.

So when we got told that we were bringing on Claude as our AI tool, I got pretty excited to see if I could help our team streamline this process.

So when I first opened Claude, of course, my first instinct was not to think, "How can I code this?" Like I said, I'm not an engineer. I didn't even know that that was the route that I was going to go. I went in with a mindset of: can AI help me to build a system that is sustainable for us and that helps us be more consistent, accurate, and even move quicker, because our business was growing as well.

And so I know you guys are probably experts at prompting, but I definitely wasn't. So this is kind of what I walked through in my initial chat with Claude. I told it my goal — essentially what I wanted out of this system. I gave it context of what we were currently doing and what our current constraints and pain points were. I then described some of our tech infrastructure that we already had, because I have never asked for any sort of capital expense and didn't want to, because I felt like it wouldn't get approved and it would probably take too long.

So after chatting with Claude about it, it basically told me, "Hey, you should make a Google Apps Script web app, and it can be powered still by a Google Sheet, but you won't have to touch the sheet. You'll build a portal, and everything you need will be in there." Now, that was the initial iteration. There were several iterations since then, and it so far has been just an amazing experience and time saver for our team.

We can now have our category managers go into our portal, paste in the ingredient list, the Claude AI engine will read it and then assign basically a verdict to every single ingredient — whether, again, it's acceptable, unacceptable, or has restrictions. If it has restrictions, it will list out what those are, and it gets sent to our food safety team for review just to make sure that the output was correct. And then our category managers can then take those ingredients that have restrictions and just print out an Excel file or Word file, or just copy and paste that to our vendors for them to provide that extra documentation. So that whole review process is tracked by submission numbers. We can escalate things if we decide, hey, we may want to look at this ingredient to see if there's any way we could start accepting it. And we've even started tracking things, making sure that we are actually improving our process. We've started to track how long submissions are taking.

Michael Winslow

Can I say something really quickly?

Marya Caples

Yeah.

Michael Winslow

Google Apps Script. I've been waiting since you said "Google Apps Script" to kind of jump in here, hopscotch style. Google Apps Script was actually the way that I found out about Marya.

Which is: I'm in the company, and I've actually been tasked with this job to basically empower a bunch of non-engineers to work with AI, and I'm trying to figure out what is it that I can give to them that they can output and share with other people. And I'm sitting in a meeting with her boss's boss's boss — somebody really high — and she is showing us the application that she'd built.

And for a second, I'm looking at the application, but after a second, all I do is I look up at the URL, and I was like, "That does not say localhost:3000." Right? I was like, "This application is delivered somewhere, and this is exactly what we need to be able to get by the time that we give this out to people." So I immediately was like, wait a minute. I didn't even realize necessarily that we had worked with Google Workspace for our entire group, and I was like, there's authentication there. There's authorization there. We might have something here.

And then I came to find out it was you that put this up there. And yeah — a non-engineer, already hosted, already sent it, and she knew that you should only put it as everyone at Thrive and not everyone in the world. So she had already known that that was a dangerous thing. And so I was already so impressed about this coming into the day that this was going to be something I talked about.

But when we were preparing this presentation, I even got blown away one more time, because I said, "What actually — did Claude just tell you to use Google Apps Script?" And she said this: "I went to our Okta page, took a screenshot of every app that we use within the organization, and asked Claude, 'This is what you have to work with. What do we do?'"

And I was like, "Did you just invent a context graph?" Right? So yeah — basically a context like, this is everything that we have licensed, and she used that to come up with it, and Claude chose Google Workspace from the list. No new software, no budget request, no procurement, and basically she created a context graph from a single image.

So another round of applause for her.

Marya Caples

Thank you.

Michael Winslow

All right. So now I want to take you back to April, when we had the Enterprise AI Summit back then. I was only about a month into my job, and I knew that I had to come up with this way to spread AI to the non-engineers, and I had to figure out how to do it. Every single talk there was valuable, and I would say probably I took a little bit of every single talk from April to basically make my roadmap for the next couple of months.

And so on that day — February 9th was my day one. I created the AI community of practice as I came in there, and then I attended the Enterprise AI Summit. Coming out of that, I ended up with a presentation, a proposal to our senior leadership, and I basically said, "Here's what we're going to do, and one of the goals is for the next Enterprise AI Summit, I want to give back and present and talk about what we did there." And back then, I assumed it was going to be November, because I told them that. It ended up being October 7th and 8th.

Turns out we ended up coming back here, and I sent this to the senior leadership — that had the slide from our original deck in there — and said, "Guess what prediction came true. Marya and I are presenting at the Enterprise AI Summit coming up."

And then I want to embarrass — oh, I guess we didn't get my latest version in there, but we can read the small print. I want to embarrass Mario, who's out here somewhere, because down at the bottom he said, "In my almost six years here, I'd say this is one of the most influential things that has ever happened to us. It's transformed how we work, how I lead, and how I think. Grateful for the initiative." This is what happened when we gave and empowered the non-engineers with AI, and that's the story we're getting ready to tell here.

All right. So after the Enterprise AI Summit, we decided to create a sprint full of dojos — we had 11 dojos, two days each — just getting non-engineering groups and departments started, and then said: this started in May, they had till June 2nd, which is our science fair, where we all come together in Culver City. We were going to have a science fair on that day. So they basically had about a month for each department to put something together.

Then, up until June, we started our engineering AI dojos. So we had already given AI to the non-engineers. Now it's time to bring the engineers together. That pretty much wrapped up then, and then we created the AI Guild, and then to today. So let's tell the story.

When we had a dojo for the non-engineers, the first thing we did in the morning was we started building. And we did this — and I heard this from someone last night, I believe it was DJ — at 11:30, we told them, "Throw it away." Whatever it is you built, you are so excited about what you just built — 11:30, you throw it away, because we want to tell you that the context is more important than the code that was generated. And then by 1:00, we went back again, iterated, did it even better.

And then the one thing I noticed — because these were two-day dojos — every single person that went through day one, and Mario can attest to this, pretty much stayed up all night before day two. And then day two, they came in and just iterated on it. And that was the two days in the dojo, and the rest was a Slack channel where they could get to us to talk to us about anything leading up to it.

Now, I didn't think I was going to talk about Mario this much, but Mario is part of our people ops group. He wants to make things exciting. So once he got excited, he took it to another level. He said, "Go to every one of those dojos and give them a prompt that creates a movie poster for the thing that they decided to create, and we're going to get it printed out, and we all come together on June 2nd. It's going to look like a movie theater lobby with all these..." And they were all up there. And he said, "By the way, you have to tell us what movie you're basing your thing on." So we have a little "Risky Business" over there. We have "Harry Potter." We have "Finding Nemo" and "Cloudy with a Chance of Meatballs." But it was pretty cool to do that.

And then of course, they surprised me with a nice poster. So these were the two AI buddies that helped everybody — this is Ryan and Oleg — and then I guess I was Mr. Miyagi in this one.

Marya Caples

Mr. Miyagi.

Michael Winslow

Yeah. So the day of — this is what it looks like. That is our CEO, Nick, up there going through all of the different things that are going to be presented. There's all the movie posters lined up side by side. And all of the people ops came together. There were so many presentations that people had to do. We were in a WeWork, and we had to rent out all these different rooms where each one of them, somebody could set up and present things. And for the 400 people that we had, we got to walk them through group by group to see 15-minute presentations of each thing that was built.

So you can start seeing Asad in mission control — he's presenting what he created for our fulfillment centers. You had Brian, who created a forecast health monitor that blew me away when we were in it. Jennifer explaining what three-way matches are. These non-engineers were so empowered on this day, and there was so much excitement around all the things that they had built with AI.

This is Alex, who reports to me. He created the SRE Priority Manager, because we want to make sure that our customers know where they are in our list when they come to us for help. And then the first AI on-call agent — this is Troy presenting that — so that was a group of people in the dojo.

There were so many of these apps that were created that the next thing we had to do was have our friend Evan from Thrive, who's in the office, work with a team to come up with an AI registry of apps. And so now we have this place that you can go and find all of the different apps that have been created in the organization.

And then finally, guess who was able to present her vetting portal there? So here's where she first presented it to everybody. So why don't you go ahead and tell us a little bit more.

Marya Caples

Sure. This dojo and our presentations were definitely a great experience. This happened during what we call Thrive Week, where we all come together as a company and meet up once a year, since we're remote-first for our corporate offices. And I can say without a doubt it was one of my favorites, because I got to see — one, what different departments actually do, because I didn't know that myself — and two, some of the things that they were struggling with that we could also use on our end in our department.

And so with this whole experience for me, I've shifted my mindset a little bit when it comes to solutions for different departments. My previous mindset, again, was thinking about cost justification. That's usually at the top of our mind: can we justify the cost of buying commercial software? And now it's shifted to: can we build something that is specific to us, that can happen in a much quicker timeframe?

And so I think that when you have very specific processes that are pretty unique to you, that's something to consider. Commercial software is great, and I think that there's definitely a time and place for it, but there are some times where you need something that is really, truly specific to your process and the way that you do things as a company, so that you can get the best use out of it instead of waiting for another deployment of new technology.

Michael Winslow

I'll quickly build on that, because that was what happened with your ingredients portal.

Marya Caples

Yes.

Michael Winslow

Your team was in the middle of evaluating a vendor —

Marya Caples

Mm-hmm.

Michael Winslow

— and at some point they realized, after a couple of iterations, that they no longer had to.

Marya Caples

Yes.

Michael Winslow

I'm going to zoom through because I want to get to the engineering piece real quick.

So that was all the non-engineering groups, but a lot of the talks today talked about how we're using AI within engineering groups, and we're no different. We have a large engineering group, and it was time to basically say, "Let's shift this group's focus away from the non-engineers for a little while, and then focus on our AI usage in engineering."

And the interesting dichotomy there was that with the non-engineering side, it was like starting from scratch, and you were the ones telling everybody what to do, and they didn't question much. They didn't know what code was behind it. They didn't necessarily care. They just cared that their problems were being solved.

But when you get to the engineering side, there's a lovely problem that you have, which is they're a little bit more skeptical, and sometimes they're saying, "We've already been working on this for a while. Why would we come and adopt something that you're bringing to us?" And then you get competing different groups that are working with AI in different ways.

But what we found was that starting up was a little bit more difficult than it was for the two-day dojos that we were having with the non-engineering groups.

But AI did pay off for us pretty quickly in a couple of ways. Being able to speed up and solidify our sprint planning and monitoring — RK and Matt, they came up with some pretty awesome things to implement a lot of the stuff that Mick was just talking about from a flow framework as well. Engineering operational health dashboards were built within no time, immediately became valuable for monthly meetings. SRE tools like the SRE Priority Manager that you saw up there. And then making sense of production logs, because it's great with large amounts of data and making sense of it.

But then we had to get to the point — and we're still working on the point — of rebuilding our AI SDLC. Code reviews, bug triage, build and deploy testing. What we've realized is now we have really fast code coming in, and so many people have seen this: a lot of code coming in really fast. How do you get to the point that you can manage that, especially with constraints like the code review that has to happen?

So here's something that we found recently. Code got sped up completely. First review sped up completely. First need for a human slowed down tremendously. 50% of the time is now waiting for the first human reviewer, because the PR is going to get reviewed in some way by a code reviewer within 10 minutes. So waiting for the human becomes a thing. The merge — that was 26% of the time, a little bit after that. Under 2% is how long it took. So in some cases, you could actually end up with 11 to 17 hours of waiting when the code has already been reviewed and for the most part been given a high-confidence, low-risk rating pretty early. We were still affected by things like engineers taking vacation and things like that. So we are now getting to the point where we have to solve for that kind of thing.

So these problems end up being what is actually starting our next phase, which is we just started an AI Guild. And the AI Guild basically now has a preset list of things that we need to work on because of all the problems and the challenges that we're having getting past that point of speeding up the AI SDLC.

So I want to just say that the things that we could use from this audience:

One — who owns the app after it ships? This is one thing, I'm going to say it really quickly, but it's a huge problem. We empowered all these non-engineering groups. They grew tremendously. They have reached a point where their apps need to be a little bit more governed, need to have real systems thinking within it, and they don't necessarily have the personnel to take it to that next level.

How have you made enablement scale without diluting it?

How much structure before people stop building? That's the governance question we're talking about with non-engineers.

And then finally, from an engineering standpoint — where do you draw the line as far as when you can let AI clear something alone without a human to be in there?

So if you've been thinking about these things or have an answer to these things, I'd love to talk to you. From me and Marya, thank you everyone.

Marya Caples

Thank you.