Jay Tennier: How Testing Platform Rainforest QA Tests Itself

In this episode of On Rails, Robby is joined by Jay Tennier, Engineering Manager at Rainforest QA, where he's spent over seven years working across a long-lived Rails monolith and supporting services. They explore how Rainforest maintains their platform with a small team, and the practical decisions that come with that reality. Jay shares lessons from pulling microservices back into the monolith, why they wrap third-party services in adapters, and how they push analytics work to BigQuery instead of straining their Rails database. The conversation covers testing philosophy including "wet tests" over DRY abstractions, using dry-monads for complex service flows, and how celebrating code deletion has become part of their engineering culture.

Social + Web Presence

LinkedIn: https://www.linkedin.com/in/jaytennier/
GitHub: https://github.com/jaytennie/ 

Twitter/X: https://x.com/jaytennier
Bluesky: https://bsky.app/profile/jaytennier.bsky.social

Company/Org Links

Homepage: https://www.rainforestqa.com/

Tools & Libraries Mentioned
Active Record: Rails ORM.  

BigQuery: Hosted analytics warehouse.  

Cube.js: API layer for querying analytics data.  

DRY-Monads: Structured success/failure flow.  

FactoryBot: Test data factories.  

Grape: Ruby API framework.

GoodJob: Background job processor.  

Q Classic: DB-backed job queue.   

Redash: SQL-based dashboards and reporting.  

RSpec: Rails testing framework.  

React: Front-end application framework.  

Haml: Legacy templating engine.  

Segment / Mixpanel: Event tracking pipelines.

Books Mentioned

Confident Ruby by Avdi Grimm  

Exceptional Ruby by Avdi Grimm  

Working Effectively with Legacy Code by Michael Feathers 

On Rails is a podcast focused on real-world technical decision-making, exploring how teams are scaling, architecting, and solving complex challenges with Rails.

On Rails is brought to you by The Rails Foundation, and hosted by Robby Russell of Planet Argon, a consultancy that helps teams modernize their Ruby on Rails applications.

Chapters

• 0:00 Intro and welcome to Jay from Rainforest QA
• 1:12 What keeps Jay "On Rails" since version 0.13
• 4:52 Overview of Rainforest QA's platform and architecture
• 14:40 React frontend in separate repo and API strategy
• 22:28 Team size evolution: from 40+ engineers down to 7
• 25:35 Failed experiments and the dry-monads pattern
• 32:25 Data retention policies and deprovisioning clients
• 36:01 Lessons from pulling microservices back into the monolith
• 41:39 Building adapter layers for third-party services
• 57:42 Testing philosophy: preferring "wet tests" over DRY
• 1:08:54 Pushing analytics to BigQuery instead of Rails
• 1:17:31 When to sunset services and delete code
• 1:24:01 Transitioning into management and supporting the team
• 1:32:47 Is Rails part of Rainforest's secret sauce?
• 1:37:14 Book recommendations and where to follow Jay's work

Transcript

[00:00:00.000] - Robby Russell

Welcome to OnRails, the podcast where we dig into the technical decisions behind building and maintaining production Ruby on Rails apps. I'm your host, Robby Russell. In this episode, I'm joined by Jay Tennier, Engineering Manager at Rainforest QA. Jay has been there for a little over seven years working across a long-lived Rails monolith, a handful of supporting services, and a front-end powered by React and Grape APIs. Over that time, he's seen the platform and the team grow, shrink, refactor, reap, backtrack, simplify, and occasionally reinvent itself. Jay brings a practical perspective on what it takes to keep a mature Rails app moving, hiring in different stages of company growth, teaming legacy endpoints, deciding when to pull back microservices back into the monolith, offloading analytics to things like BigQuery, and figuring out how a small team survives all of it. We'll dig into the technical decisions behind those choices and what other teams might take away from Rainforest's journey. Jay joins us from Ontario, Canada. All right, check for your belongings. All aboard. Jay Tennier, welcome to On Rails.

[00:01:06.580] - Jay Tennier

Thanks for having me.

[00:01:07.800] - Robby Russell

All right, Jay. Let's start with this really hard question. What keeps you on Rails?

[00:01:12.700] - Jay Tennier

It's a good question. I guess There's two sides to it, right? There's the personal one. Personally, I'm still on Rails because I love Rails. I've been developing with Rails since what? 0. 13, I think. Yeah, somewhere way back there. My real first app was in version 2, but there's just something about Rails that has always suited my head once I got to wrap my head around it. The other thing, too, is I just love Ruby. Tried a few other things. Been writing in Python lately, and it's just... It's cool, right? Nothing wrong with it, but yeah, there's just something about the way that Ruby is that I really, really like. Then from the rainforest perspective, we are on Rails because we've always been on Rails. Well, I shouldn't say always, but for a very, very a long time we've been on Rails. And the decision was made not by me, but by my previous boss, I think about eight years ago. And the decision was we're hiring people to work with Rails. We have a bunch of people who work with Rails. So unless there's a really good reason not to use Rails, we should use it because just to keep everybody on the same page, nobody having to learn anything else.

[00:02:27.860] - Jay Tennier

And that went with Ruby, too, where it was If we had any side stuff that we needed to do, we're going to keep using Ruby for that. There's a couple of things that we didn't use it for that made sense, like our CLIs written in go. But it really panned out because we do have a service that's in Erlang, and all of those people have left, and now nobody knows what to do with it. Then we had a side project that somebody wanted to try out Phoenix, and same thing, those people left. It's just anybody know Phoenix? No, nobody does. And sticking on Rails has just been the mantra for Rainforest for a long time.

[00:03:04.420] - Robby Russell

Nice. And we'll dig into your Rainforest stack in a moment. I'm curious about, you mentioned, been using Rails since dot 13 or so. I think that's pretty much around, I think I might have barely beat you by like 0.12. And that was probably what a month apart or something like that, so give or take. Out of curiosity, what tech stacks or tooling were you working with prior to that?

[00:03:23.680] - Jay Tennier

Back then, that was PHP and hand rolled. Yeah, hand rolled an app. I took over an app that was an employee satisfaction survey app that was written by a student in PHP. And so it was just taking that, extending it out. And then from that point, I mean, that was back in the Slash. Days, coming across all this new cool stuff, the Rails showed up, and I wasn't able to really use it at that job because I was the only developer. I couldn't stop doing what I was doing in order to do that. But I had a little side project that I spun up with Rails, too, I want to say. It was an internal time tracking app for myself. Then I wrote some Ruby code, some glue code thing back then. But yeah, it was PHP. Then that was when Cake was out and Cake was doing the same thing. Then I got it in my head that I could rewrite the survey app in an MVC style. I started digging into the Rails code base and was trying to do similar things, setting up controllers and how views would work and delaying content, rendering and learned a lot, too much, probably, because then it was a typical, I got half of the code base moved over to that, and then the other half wasn't there.

[00:04:35.420] - Jay Tennier

But luckily, I left that to somebody else.

[00:04:38.120] - Robby Russell

I can appreciate that, and that resonates with some of my earlier projects pre-Rails as well. So for our listeners who may not be familiar with Rainforest QA, can you give us a high level overview of the product platform and where Rails fits into it today?

[00:04:52.580] - Jay Tennier

Yeah. Rainforest QA is a SaaS platform. Let's pretty much anybody with a SaaS app test their app at the highest level possible, which is in the browser. Most people, I think, are familiar with PlayRight or Selenium, headless testing, that thing. But we don't give you that. We give you a user interface to go in, create your clicks and your fills and your observes and that stuff. Then we execute the test that you have written using a VM, which we drive so that you've got basically a real mouse clicking around on real elements. We have some dom fallbacks and that stuff for those people who need it, but we're of the belief that trying to find the actual images of things on the page is a little more resilient than just hard-coated dom IDs, and then interacting with them in that way. The other thing, too, is because it's a VM, you're not trapped in the browser. You have access to the entire OS if you want. We have a lot of companies who need to install an app and test the app in concert with their actual SaaS app. They can do that because you're just basically interacting with the VM.

[00:06:05.220] - Jay Tennier

 yeah, it's pretty nice. Then it's generally that way. We've got an automated service, and then we also have a tester-driven service, which is using humans to execute your test. You can write out a different set of steps that are a little more loose that a human would understand, then you can execute those things, too. Then we've got a whole AI thing going on right now as well.

[00:06:28.440] - Robby Russell

Interesting. The We're not trying to necessarily plug Rainforest or anything. It's like, Everybody should go use this platform or whatever. But the typical user or customer, is it a QA team that's setting up things within a product team, or is it typically a bunch of new engineers that are trying to do away with needing to have some human QA people that are going around clicking things on their staging environment before things go to production, or is this used for a lot of smoke testing or a little bit of all of the above?

[00:06:55.400] - Jay Tennier

It's all of the above, yeah. Smoke testing is a big thing. We use it as our last stop gap before we release. But we have some clients who don't have a QA team, and so they're getting the devs to drive it. That's, I think, where we prefer to be right now. But we also have QA teams that are using it as they set up their automated test, and then they drive it out. We dabbled in the pure no code space for a while. So it was trying to get the product owners to write those things rather than the devs, because their UX is pretty good for doing that thing. So, yeah, we're at all of those levels thing. But yeah, I think we're going after developers right now.

[00:07:38.980] - Robby Russell

I see. That makes sense. We had a previous conversation leading up to this, and you mentioned that Rainforest primarily has a monolith, right? Yeah. And is it you can react on the front-end? You already also mentioned Go and Erlang, a couple of other tools in your overall platform. Can you walk us through what the whole ecosystem looks like today? Then we can dive a little bit back where you it be and how it got to where it is today?

[00:08:01.560] - Jay Tennier

Yeah. So the main app, if you do use Rainforest, you're going to be looking at the monolith, which provides all the back-end, the Rails monolith on that side, and then the UI is a separate React app. And then we have a couple separate services that I wouldn't call them microservices. They never really started as microservices, I don't think. The main one being what we call VManager, which is our VM manager, and that is another Rails app. It's responsible possible purely for making sure that all the VMs that we have are available and setting them up, tearing them down. There's a lower level service written in Ruby that sits closer to the OS called V control that takes care of the actual nuts and bolts of setting all of that up. If you open up a test and you need a VM, we talk to Vmanager, Vmanager talks to V Control, and then it comes back up. We've got an Erlang service, and that is mainly for the tester community. It's what we use to set up jobs. Our testers come online, they have a browser extension, and they're like, I'm available for work.

[00:09:07.740] - Jay Tennier

And so we have this Erlang service that is just there to mediate between testers and the jobs that we want to give them. And then we've got a CLI, and that was written in Go because we need to support it on every platform.

[00:09:19.740] - Robby Russell

Sure, sure. That makes sense. And when your VM manager spins up a VM, where does that spin up? You have your own hardware, you do this AWS, some other platform, problems?

[00:09:31.180] - Jay Tennier

We rent server space in Leesweb in the States and Hetzner in Germany. We started in Germany a long time ago, and that presents its own problems where you start up a VM and you go to Google and everything's in German. We've had to do a bunch of stuff to get around that, proxying through different places. Oh, interesting. Yeah, in most cases, it doesn't matter. But if you're just testing your own app, but if you're trying to go out to the wider internet, it does make a difference. But yeah, so we've got a bunch of servers there. V control runs on those servers. Make sure that those servers have, say, I don't know what the actual number is, but we've got 200 servers, and maybe we've got 20 VMs on each of those servers. We try and make sure that we've got enough Windows 10 over here and Windows Chrome versus Windows, Firefox versus Mac OS, that thing. We've got those all sitting there. Then mainly we spun up the ones in the US so that we could have them closer to most of our users. When we're executing your test suite, it doesn't matter where it is in terms of latency, but when you're using the app and writing your test and directly connected to that VM, we want it to be as snappy as possible.

[00:10:41.770] - Jay Tennier

So we bought some servers over there to move them closer.

[00:10:44.600] - Robby Russell

How different is running, say, a suite of QA tests in this context, different than running a bunch of headless tests on your local computer or in a CI server where you can, in theory, try to do a bunch of things in parallel? Is this still a lot of parallelization happening for...

[00:11:01.420] - Jay Tennier

Yeah, when you create a run, a run for us is just the set of tests that you want to want to execute. We just go out and say, Get me whatever. We need as many of Windows Chrome VMs as you've got, and they're running wherever. There are certain things that clients need to do, special things, whether it's using different proxies and that stuff to winnow down the VMs that are available to them. But in terms of your general, just like we're going to run something in Windows 11 Chrome, we've got a bunch of them. So you kick it off and we just go and get as many as we can. Oh, I see. We've got a little bit of concurrency baked in now because people were obviously eating all of the resources. You can have everything if you want. And then we also found that a lot of clients don't want you hitting their site with 200 tests all at the same time. They want to lock that concurrency down as well.

[00:11:59.280] - Robby Russell

When you do parallelize those types of things, does it then... Like I say, if there's a lot of tasks that are just trying to repeat that require you to log in, is it then having to go through? If you spin up 20 things in parallel, is it logging in 20 times in parallel and then running just one slightly different tests on each of those logins. And so is that getting into any interesting tech? You just mentioned that maybe some of your customers don't want you to have or don't want it to oversaturate it with too much at the same time. But might there be some interesting technical decisions? Like, Well, we're only allowed have so many users with the same login, login to the app at the same time. How did that stuff get split up?

[00:12:35.920] - Jay Tennier

Yeah, that's always an issue with our clients. And when you're thinking about it as a developer, it all just makes sense, right? Oh, we'll just seed QA with all of the correct stuff, and it's fine. They could all log in at the same time, and all of their data will be completely separate. Then you run into the real world of clients, and they're like, We don't have a QA environment, or, We don't have a way of reseeding data all the time. Some of them have structured things in a way where they can clean things out afterwards. Others are doing the seeding as part of the test, which is crazy because it adds a bunch of time to test. Other clients are trying to restrict it down so that basically one user is logging in at a time, and each of the tests runs serially after that. There's all kinds of different things that we've had to help support. Rainforest used to have the rainforest way of like, this is how you should do testing. Very prescriptive, and it's good, but sometimes you got to meet your client, not sometimes, you really got to meet your clients where they are.

[00:13:38.480] - Jay Tennier

It's like, Oh, you should have all of that stuff worked out, and it should be a blocking step in your CI. If nothing passes or one test fail, you shouldn't release anything. Then same thing, you come across clients who are happily using the app with half of their tests failing on every single run. Wow. Okay. They're happy that those tests are failing because they just expect those tests to fail. Maybe one day they'll get around to them, but they don't want to take them out because they like having them there. You know what I mean? There's something about their process that makes them feel good that even those failing tests are still failing.

[00:14:15.180] - Robby Russell

Sometimes the failing test can be a little bit of a weird security blanket. Who knows? Let's talk a little bit more about your stack there. You mentioned Rails using, I think, React on the front. Is that across all of the front or any back-end type tools as well? And then where does React code sit? Is it in the same repository or is that a separate repository? How are you organizing that over there at this point in time, at least?

[00:14:40.520] - Jay Tennier

React is a fully separate repo. It used to be part of the main app, but it got split out before I joined, and that was almost eight years ago. So it's been a long time. It's been its own thing. It's gone through multiple in a typical React fashion. There's all kinds of stuff going on in there. We have a front-end team that that specifically deals with that. Everybody's allowed to mix and match, right? But they're the experts, so we let them do the React stuff. Yeah, and then on the back end... So everything you see, except for the login page, is that React app. And then that React app makes just your typical HTTP request to the monolith. There's been a lot of discussion in the past about exposing things like vManager to the front-end or vControl to the front-end, right? Or some of the other things. We tried to make sure that everything ends up going through the back-end, the main monolith for a number of reasons, mainly because then you only have to auth in one place. Sure. That's nice and easy. But there's also that, when you know that there's one place to go, then the back-end can capture that information and log it.

[00:15:45.820] - Jay Tennier

There's one central place for all of that to happen.

[00:15:48.580] - Robby Russell

Are you exposing everything then just through a Rails API at that point then? Or do you have any web interfaces that are built in Rails entirely, in the back-end?

[00:16:00.860] - Jay Tennier

The main admin, the admin for the monolith is bootstrap, too, if you can believe it. We still got some coffee script in there, too. Okay, sure. Hamel templates. It's been around for a long, long time. The vManager's interface is all-Rails-based. Then we have a couple what are really microservices, which is like an email inbox and an SMS inbox, because part of testing, you're going to need to send emails. You want to make sure that you can get those emails and respond to those emails. Same with SMS. Those are their own little things. I mean, they are super light, super light. I'm not even sure if they have admins. If there are, there's almost nothing there. It's just like an excuse to get to the Good Job admin page, right? Like that thing.

[00:16:48.060] - Robby Russell

So you're using Good Job primarily or using any other background?

[00:16:53.580] - Jay Tennier

We migrated almost everything over to Good Job, except for the main app. The main app is half GoodJob and half Q Classic. I don't know if you know Q Classic, but it's a database background job processor that was written at Rainforest and released as part of open source. We're trying to get away from that because we're not maintaining it. Nobody's really maintaining it. So it's just laying fallow for a while. People might ask, Well, why would you do that? And the answer is, well, because it's so old that there was no database-backed background job at that time. At that time, at least as far as I understand it, that was the impetus for creating it. Yeah, right now we've got about half of our jobs in one and half of our jobs in the other. It'd be nice to fully migrate it all over to a good job. I mean, it was a good job, a good choice. I mean, that was before Rails 8 with... I can't remember the- Solid Q. Yeah, solid Q. I'm looking at that and it's just, Well, should we just bail on both of these and move everything over to that?

[00:17:58.920] - Jay Tennier

I don't know.

[00:18:00.000] - Robby Russell

That's an interesting thing to consider there, given that you already have them separately. I had not heard of Q Classic before. I'm looking up right now on GitHub. And does that integrate with active job?

[00:18:10.200] - Jay Tennier

I think it does on a couple of the branches. This is the problem, right? Is that nobody... And because we're not really-supporting it as much? Yeah, not supporting it. And it is Q Classic itself is owned by Q Classic, the Q Classic GitHub org. We have an extra thing called Q Classic Plus, which wraps all Q Classic stuff in a transaction and does a bunch of other things that we own. We just haven't reached out to anybody to try and do anything. Every time we needed to do something, we would just create a new version and then point our gem file at it.

[00:18:46.460] - Robby Russell

That makes sense. You said you've been there for about eight years there or so now. In your opinion, what parts of the architecture have been the most stable over the years and what parts have been, let's say, more turbulent?

[00:18:57.160] - Jay Tennier

Most stable has probably... It depends on what you mean by stable. We use Grape for the API. Again, this was back when we had a lot more. When we were a larger company earlier on, we had the notion that we would have a public API and a private API. That was all with good intention. The CLI uses the public API to do the stuff that it does. We did have some clients who were writing against that API, and then we discovered that the performance got worse as we got more data. We needed to slim down the responses. We did that by using a slim param and then a slimmer param. We just kept going down that route all because we didn't want to break the public API and we didn't have a really good way of deprecating APIs. Our API, I think, is at an API/v1 or/1 URL, so we can do it. It's just so much more work to go down that route. We've never done it. In that sense, the public API It has been pretty stable. Stable? Yeah. But the private APIs, we've moved away from, and I don't know if this is for good or for ill, but the more restful sense of picking the perfect resource name and putting your gets there and having filters and that thing, mainly because we do have a lot of data and we've had performance issues.

[00:20:23.740] - Jay Tennier

We still do have performance issues. In some cases, the front-end has the ability to gather all of the data already. There's multiple endpoints for it, just go and get it. Maybe that's how we do a first pass, get that up. But it's normally super slow because we're making multiple requests, and those requests can take a while. What we've started doing is making very specific endpoints for exactly what it is that the front-end needs. That's fully private, so we can change it as much as we want. That seems to have worked out quite a bit better. Now, it means an explosion of endpoints. And because they're private, we don't have docs for them. I'm sure we can configure Grape and Swagger in order to get us those things. But there's a lot of like, Hey, front-end, where are you guys getting this information from? And they tell us, and, Oh, okay, you can grab that, no problem.

[00:21:22.040] - Robby Russell

Out of curiosity for our audience, approximately how large is your overall engineering team at the moment, across back-end and say front-end?

[00:21:30.380] - Jay Tennier

Right now, we're seven. We've got two front-end engineers, three back-end engineers, but that includes me. So it's more like two and a half, two and a quarter, and then an ops engineer.

[00:21:43.280] - Robby Russell

I think that helps illustrate a little bit into some of the challenges of, say, a smaller, I'm air quoting, smaller team. I don't know how big the team was, approximately at its largest. Do you recall?

[00:21:55.180] - Jay Tennier

We've gone through multiple downsizings. When I got hired, we were expanding, and I want to say we were close. Engineering was probably between 40 and 50, something like that. Then after the first downsizing, we went down to 15, 20-ish people. And then the last one, we went down to this seven. So it's been... Yeah, there's a lot of code that a lot of people worked on. They're no longer around to explain just exactly what they were thinking.

[00:22:28.000] - Robby Russell

Yeah, thanks for sharing that. I think if we get time, I'm going to dig into that a little bit more. But I think at least the context of knowing that when you're working with your teams, your front-end is trying to gather some details. You're like, well, you're pitting all of our RESTful APIs. It's the default way that Rails developers would think about exposing that historically. You're like, all right, you fetch some details, and then you got to go fetch some other details based on the information you get. You're not maybe grabbing everything in one single request. I've talked to plenty of other companies, even on this podcast, much larger engineering teams are like, the conversations between front-end and back-end, I'm air-quoting back-end teams, it doesn't happen as often. They're like, All right, we'll just give you GraphQL so you can tell us what you want and we'll give it to you. I think if we hadn't had that conversation, at least to talk about how big of a team you have, you're like, Well, does GraphQL makes sense at a team size of, let's say, less than 10 people? I'm assuming your team has maybe talked about it.

[00:23:20.800] - Robby Russell

Is that a safe assumption?

[00:23:22.180] - Jay Tennier

Yeah. I mean, it's one of those things that just perennially comes up. I believe before I got hired, so I mean, that was a a long time ago, somebody gave it a try, but I never really got the particulars of why that failed. I think part of it might have been the wrong... Just the wrong incentives. It was literally so that we didn't have... So the back-end didn't have to talk to front-end anymore, versus how do we actually maintain this and get it working. It is funny, though, because We have all kinds of, I won't say failed, but not entirely successful experiments still living in the code bases. Why that was the one that got put up and then immediately got ripped out? I'm not exactly sure. But yeah, at this size, it's one of those things. We're not big enough to be able to do it, but it seems like if we had it, maybe it would be better. But we're also small enough that we can have the conversations that need to be had. We work very closely. Everybody works closely. We're on one large team. Everybody knows what everybody's working on. Everybody can help out in some way.

[00:24:38.200] - Jay Tennier

Our product manager is directly involved. Our designer is directly involved. The way that you would think of I can't remember who it was, that squad-based system of we'll put everybody in a little thing and you'll have everybody gets a designer and everybody gets a product manager and that thing, and they'll all do their own features. We've got that, but we only have one. It's really high performing at that level because we all know what we're doing. Nobody really steps on each other's toes. There's a lot of, Hey, here's your endpoint. Does it have everything? No. Okay, well, what do you need? I need this and I need that. Perfect. We just added in. It's the iteration that really gets us.

[00:25:18.460] - Robby Russell

That makes a lot of sense. You mentioned there's been a lot of experiments. What are a couple of those experience, if you don't mind me asking, that you would have, in retrospect, would have wished hadn't survived or I'm not even sure that it's not so much that I wish they didn't survive.

[00:25:35.420] - Jay Tennier

It's that I wish they had seen themselves all the way through more than anything. I'd like to think we still have a really good culture. But back when we were a larger team, we had a really good culture of support hoarding those things. If you want to try something new, you read about something, you want to give it a shot, go ahead, give it a shot. But understand that if this works, you're going to have to convince everybody else. It's not like we're going to run RuboCop and replace all the single quotes with double quotes because we all decided that that's the way that we wanted to go. We're not going to be able to do that. We're going to have these things, and we're going to have to put some processes in in order to make sure that everybody's using that. I mean, one of the ones that I brought in, which was the dry monads, which if people don't know, and they probably don't, I suggest checking out only because there's something about it that suits my brain, not the monad part, not the functional programming part, but the way that the code looks when you're done That idea of having a set of procedural steps in your typical service object.

[00:26:36.470] - Jay Tennier

Where you find the user, validate the user account, send the user an email, blog the event somewhere, doing all of those things, but knowing that if at any one of those points, any of those things fail, it just fails all the way up to the point where you called it. Passing success and failure around is just really nice and being able to compose those service objects together knowing that you're always going to be getting a success or a failure, monad at the end of it. It just looks really nice. I really, really like it. We've used it in a couple of places, a couple of important places, to be fair, but it's just trying to just trying to get that as part of the policy and getting everybody on board with like, Oh, we're creating a new endpoint. Don't just go and do the regular stuff. Try and spend a little time and use this because we think it's good in some way. Yeah, it wasn't really something that I wish had died. It's quite the opposite.

[00:27:34.900] - Robby Russell

Interesting. Yeah, I've seen dry monads, and that's all part of a dry Rb, right? Yeah. Okay. Out of curiosity, is there a specific area of your application that you find that that was really, really helpful in implementing that pattern?

[00:27:48.200] - Jay Tennier

Yeah, I was going through just to find... Because I know what I did, right? I was trying to find some code that somebody who wasn't me actually used it on. And we have the concept of de-provisioning a client. We always soft deleted everything. Then it got to the point where, well, there's two things. One is GDPR and that thing where it's clients want their stuff deleted, so you have to delete it. It's got to go out of the database. Then the other side was, are database is over a terabyte. I think we're close to two terabytes now. Why do we have all of this data in here? These clients have left. They're never coming back, or if they do come back, it's going to be completely useless. How do we get rid of that? The de-provisioning process uses the dry modad pattern and just calls out each of those things. Because when you delete all of the data for a client, we haven't denormalized client ID throughout the code base. Even though clients clearly own pretty much everything that's in the database, some of them you can get directly to an edge node of whatever that data is.

[00:28:51.400] - Jay Tennier

In other cases, you need to take two or three steps. All of that foreign key consistencies, that thing. You have to start at the edges, delete all of that stuff, come up a level, delete all of that stuff, come up a level, delete all that. If any of those things fail, we need to know about it. You need to know where it was. I mean, yeah, you can wrap it in a transaction, but sometimes there are outside things happening. The S3 objects, that thing. That makes sense. There's a certain amount of making sure that it all happens in a specific order, and if anything goes wrong, that it comes back up and fails. It was really nice to find that. I didn't realize that. I didn't have a lot to do with that code. So going in and seeing that, it's like, Oh, that's a really good use for it. And then the other thing is because you can decompose each of those things down, you can take the delete all the client's jobs and just put it in a different place and call it. And then if you need a job that like, Well, we're going to leave all the client data around for this type of client, but they want to delete all of their jobs or tests or whatever, then we can just call that and know that you're going to get that success or failure back.

[00:29:56.350] - Robby Russell

I'll definitely include some links for our listeners in the show notes. You can go look up Drey Monads, if you're not familiar. Are there any particular educational resources you remember coming across that were really helpful outside of just linking over to dreirb. Org?

[00:30:09.260] - Jay Tennier

That's a good question. I probably got turned on to it from the the Rails email. What's the newsletter that everybody subscribe to?

[00:30:18.540] - Robby Russell

There's a few of them. The Speaking Rails or Ruby Weekly.

[00:30:23.320] - Jay Tennier

Yeah, Ruby Weekly. I mean, honestly, just read anything that Evil Martians writes. I'm not going to use all of their stuff. I'm not going to go all the way down, but they have such a different way of thinking about code, just code in general, how apps are built and that thing. I always find something in there, especially if you're dissatisfied with some of the this is how everybody does it, and you're looking at it and you go like, why? Evil Martians probably looked at that and came up with another idea.

[00:30:54.760] - Robby Russell

I'll definitely include links over to their blog as well. Good articles on their end to our friends over at Evil Martians. Something I hadn't thought about talking about with you, but when you mentioned the deep provisioning and just having all this extra data living around your database or in your S3 buckets and trying to come up with a cohesive way to maintain that, do you have... You mentioned the JDRP, or is that how you print it?

[00:31:13.780] - Jay Tennier

Is it JDRP? Yeah, I can't remember which one it is.

[00:31:15.410] - Robby Russell

We're from North America, and so we're just subjected to having to understand enough about it, but we're not living and breathing in it maybe as much as maybe our friends in Europe are. But trying to think about that, I know a lot of client projects that we work on where they just never get around to dealing with that stuff unless it's forced on them because of something like a policy. But I'm like, your database backups, everything is so much slower to do, making updates with your migrations, updating some data. It just takes so much longer because you have so much more data in your database, why are you not deleting more of this stuff more often? I think we've always been like, Wow, it's just server space is relatively cheap.

[00:31:54.980] - Jay Tennier

It's throw more money at it.

[00:31:56.600] - Robby Russell

Right. But there's also a lot of scenarios where, as you mentioned, you have clients that are no longer there, customers that are no longer using. If they were to log in today, and it's been seven years since they last activated their account, I'm assuming that most of everything in their system is not going to be workable in the way. It's an interesting onboarding, deboarding, and then a reboarding process, knowing that you don't expect everything to sit around in this some database forever. What's your workflow?

[00:32:25.520] - Jay Tennier

Yeah, just to carry on with that. We've had multiple discussions around that, and some of it to do with just the size and why are we doing this? But what does it mean for a client to come back, even after three months, and look at a result of a test that failed three months ago? What does that actually mean? Especially if they're working on their app. The problem that we have is that, well, it's not really a problem. It's funny, but this is my former boss, and I used to talk about this all the time. Do we have a data retention policy? No, we don't. It's like, no, you do. If you don't have a data retention policy, then your data retention policy is to retain all data forever because you haven't set any other policy. That's where we were. We had a data retention policy forever. Now, you can change that. You can change that, send it out to clients, let them know, but you have to decide to do it. Then once you decide to do it and communicate it and do all that thing, then you can start enforcing it. But you can't just decide that we're only going to hold stuff for 90 days or 30 days or Even a week, in some cases, we have clients who create so much, who are running so much and have so much information that even in our producing changes to their app so much that last week's set of test executions don't mean anything.

[00:33:46.150] - Jay Tennier

It's nice to see historically that this thing was passing and then it failed, but eventually, there just becomes too much data there. But on the flip side, we do have clients who they schedule a run every month. If they want their historical data, those people don't, or the other people don't. How do you make those decisions? We basically got to the point where it's like the retention policy now is When you leave as a client, we can do anything to your data within 90 days, I think is the actual number. We did actually make that decision. Then we spent a really, really, really long time figuring out how to delete all of that data correctly. When we did start deep provisioning clients, like background jobs that were running for two weeks, deleting all of the objects because we don't tag S3 objects with any client information. We had to get all the keys out of the database and go and find them and delete them individually, that thing. Yeah, lots of decisions that were made in the past because we didn't think about deleting stuff. We had everything being soft deleted, so everything was just still in the database anyways.

[00:35:04.340] - Jay Tennier

Works out in some cases, but in other cases, you're just like, We got to get rid of this. How do we do it? The answer is you take a long time and then delete it all.

[00:35:14.540] - Robby Russell

Are you doing any multi-tenant in your app?

[00:35:17.380] - Jay Tennier

No. Another thing that we've talked about a bunch of times, and it's always come down to, It's just to pay more money. It's just going to pay your database in instance. Yeah, but we really did want that because that idea of a database for your client, and then when the client goes away, you just delete the database and it all goes away. That's so nice.

[00:35:38.840] - Robby Russell

Yeah, it does sound magical. There's our S3 bucket. Bye. I understand it's not as easy as it sounds. You'd mentioned that you had more microservices in the past, and you brought some of them back into the monolith. Can you talk about a few of those? Do you remember what the original motivation was behind that and what motivated you to, Okay, How would you backtrack that?

[00:36:01.880] - Jay Tennier

The main one that comes to mind is what we call our Integrations service. We have basically a place in the app where you can set up things like Slack messages, Jira messages. What else? We used to have, I think, if you can believe this, Facebook messages. I still don't know why. That was a separate app. It got brought into the main app just before as I was being hired. The idea there was we were touching a lot things that needed information. The microservice needed most of its information from the main app. It was just like, this is becoming too much of a pain because every time we want to release something, you got to release the changes for the main app in it. But it's still not useful until you release the changes on the integration side and that thing. We took integrations, and based on what I can see, we just dropped it into the main app and called it done and exposed. Well, I guess there are really no endpoints. It was, Okay, well, here's all your stuff, use it. Let's just say it was written with a different philosophy than a lot of abstractions, a lot of trying to figure out what exactly is happening here, tracing things down through multiple classes.

[00:37:13.920] - Jay Tennier

On the one hand, it made it easier to just do anything with the integrations because you could just make the change and ship it. Then on the other hand, it made it worse because none of that was walled off or none of it is walled off from the main app anymore. It's one of those, it seems like such a simple thing, but it's the scary place to go. It's where developers go to, not die, but go to just waste time. Everybody who gets onboarded at some point is asked to do something with integrations. It's just like, but just so you're aware, you're going to be in there for a while. It's going to take you a bit to figure it out.

[00:37:49.220] - Robby Russell

Do you think there was anything around that when your team was a lot larger, the idea was like, let's allow these folks to just work and isolate things a little bit more because the assumption was the team was going to keep potentially growing. And that led to a lot of that. And then you weren't really projecting, okay, if we're making decisions like this, we're going to break apart this, move the integrations to their own, or just set up this whole thing in a new microservice, what have you. But they're not thinking, what if we're less than 10 people at some point in eight years from now?

[00:38:16.980] - Jay Tennier

Yeah, nobody thinks that way, right? Which is understandable. But I honestly think that based on the timing of it and when that integration service was written, that it was peak microservices. We needed the email inbox thing, so that was its own thing. SMS is going to be over there. The V-Manager, we have argued back and forth about whether it would make sense to bring it in. Probably not, but at some point, there was talk about it. I think that that like, Oh, it's an integration thing. It's going to be talking to, I don't know, name whatever services that you're going to be sending this information to. It should just be separate as well. Then the hilarious part is that once we brought it in, half of those services shut down. Now we have all of this abstraction that written for another service, assuming that it was going to be in the typical Rails, MVC thing. It's got all of its own model-ish things. You're looking at it as like, There's only two. Why is there all of this in direction in order to support Jira and Slack? It's like, well, because we thought we were going to support more, and then we just never did.

[00:39:25.640] - Robby Russell

It's a difficult balance that teams find themselves in. I think that's normal. You assume you're going to keep growing or your integration is going to keep increasing, but then the reality maybe goes a different way, and I think that's being adaptable. Do you think that because you're working with Rails, that's made those types of decisions less painful, or do you think it would have been just this complicated regardless of the tech stack?

[00:39:48.720] - Jay Tennier

Maybe not Rails, but certainly Ruby. Just the gem ecosystem in Ruby makes it so easy to know that you're going to be able to, for the most part... I mean, maybe it's a little worse now, but especially back in the heyday, that there was going to be a Ruby library for whatever it was that you needed. Having it there, having it immediately at hand within the main app just makes sense. Rather rather than putting it off somewhere and walling it off. But it's similar to, I can't remember what I was talking about earlier, but the idea that the code base has been around for long enough that stuff that most people would reach for right now didn't exist, but we've already got that thing baked in there. We've had to go through when they changed the Slack API, we had to go and change. We had two different Slacks. We had sending Slack messages, but then we also had a Slack app. Well, then they got rid of the Slack app completely. We had to come up with a new way It's always like we're always just scrambling for things. But at least in that case, it's in the context of stuff that we know rather than just being off in that Erlang service.

[00:40:55.010] - Jay Tennier

It just happens to, as far as I know, it never goes down. We never have any issues with it. So thank God it's just still doing its thing.

[00:41:03.000] - Robby Russell

That's good when things can just be reliable like that. I'm curious about your teams. I don't know if it's changed over the years, but what's your current philosophy or ethos on when do you bring in a community, say, library or like gem into your project? What signals are you looking for? This might be a good tool to lean on versus like, we'll just build our own client library for some third-party API or whatever the particular scenario is. Do you have a way of how you think about that at this point that might have changed from how it was a decade ago?

[00:41:39.460] - Jay Tennier

When I first started eight years ago, I think there was enough engineering teams because we had so many engineers that we were giving people features, that stuff. So there wasn't really a top-down, this is how we should think of these things. So everybody was left to their own devices. It's that typical thing where it's, well, there's a library for it, so use the client library. Then call that client library every place you need it. Then that goes all the way through there. No writing of adapters, no anything like that. Because everybody thinks that, what, are you going to change your database? Then you do, not us, but we just had a thing where all of our internal metrics that we kept track of, we were using Labrado for. Labrado got sold to SolarWinds about, I don't know, six or seven years ago or something like that. We were still grandfathered on the old Labrado, and then they just sent us a thing. It says, Labrado is going away in September. It's like, Okay, cool. But we use that for all of our event data. Across all of our apps, We didn't have any adapters. We didn't have anything to go like, Oh, well, we'll just send this to Datadog instead.

[00:42:52.120] - Jay Tennier

Instead of pushing it here, we push it there. It's like you have to find every call to Labrado across all of your apps. It was It's a good exercise for us, just removing some stuff, getting rid of a bunch of that stuff. That's the thing that I think in terms of you don't think about what it's going to be like when your team is much smaller. But even in that sense, maybe it's not your organization is smaller, but your task now with you've been put on... There was no teams before, but now you're put on the platform team or the integrations team or whatever. When you're that small, there is a tendency to like, We all agree, right? All five of us agree. What about the other 50 people? Who cares, right? So trying to keep that communication open and going. And then it's just a typical trust as well, right? At some point, you need to just let the devs do what they... Not what they want to do, but sometimes they do... You do need to learn from your own past, I won't say mistakes, but just decisions.

[00:43:56.300] - Robby Russell

This episode of OnRails is brought to you by Skip Forgery Protection, because who Use trust tokens when you've got Trust Issues. Are you building a quick demo or a half-baked prototype or something you promise, you promise no one will ever use this in production, then you, my friend, are ready to skip forgery protection. Why verify a request authenticity when you can just believe in the goodness of the internet? With one line in your controller, you can bypass Rails default safety net and invite strangers to post whatever they want straight into your app. What could possibly go wrong? Skip forgery protection. Like taking the batteries out of your smoke detector because it was beeping a little too much at night. Fast, questionable, beautifully reckless.

[00:44:36.420] - Robby Russell

You mentioned that example of maybe not initially integrating with an adapter layer between you and whatever third-party service. Is that Something that is something you would encourage developers to think about doing from the get-go down? How do you make that, say, third-party gym? An easy one that I always bring up when I have conversations with developers is like, All right, it's really great that Stripe made it so easy to integrate with their platform for applications, yet we have a lot of clients that come to us and they're like, Hey, we want to add PayPal, because not everybody has a credit card around the world.

[00:45:08.160] - Robby Russell

We want people to pay us for our product. They're like, Well, your database has a bunch of columns called Stripe ID, and your code base is just littered with Stripe code, literally from their gems, which made it great. But we're a little in a weird pickle here where it's not just add another thing on, and it's easily just going to be a quick thing. This is a a complicated problem. Maybe I don't know what the answer to that all the time is. People say, Well, Stripe has some PayPal integration. It's not the same that everybody's looking for. But I'm just curious, do you think about that a little bit more when you're going to reach for a gem and be like, Okay, well, we're going to lean on this, but I'm going to also take a little bit of extra time to do something in between me and that gem.

[00:45:51.280] - Jay Tennier

Yeah, this is my hobby horse right now. Simply because of the Labrado stuff. We are in a similar situation with not Stripe Not necessarily needing to support PayPal, but more like we need a different set of pricing. We need a different pricing model right now. There's a lot of stuff that is just baked into the current code base that will not support that model. But we had our client metrics, keeping track of when users click on things and where they are in the app. We were using Amplitude. We turned off Amplitude and went to Mixpanel. All of that stuff was being sent through Segment. It's like, well, Segment was supposed to manage of that, but we didn't do a good job of that. To your point, it's like this is the thing that I think regardless of how big or small you think your organization is going to be in the future, it really, really, especially for third-party services, it really makes just sense to, even if it's just a super light wrap around whatever it is that third party call is going to do, even if it's just sending event data. Because you never know what the next library is going to expect from you.

[00:47:01.760] - Jay Tennier

Having to go and update every single call point, every single place that you're sending events to, is crazy. Now, with Ruby, you can fake it out by basically getting rid of the gym and then having... You can monkey patch your way to a great success, I guess. Yeah, I wouldn't recommend it. So even just having light wrappers. And then also, please, when you write the rapper, name it for the generic thing that it's doing. Right now, I swear I just saw we have an adapter around Stripe calls that is namespace Stripe. It's like, well, okay. That's not great. That should be payment processor or something along those lines, right? Just to make it very clear that this is the thing that it's doing. Eventually, we're going to get around to sending stuff to Stripe, but the rest of it is you really need to think through those decisions and be very precise about what... I mean, it's the naming problem, but But at the same time, if you get it right or get it close, it saves you a lot of trouble down the line. Because I would hope that your organization is so successful that it becomes so big that you need to support multiple payment processors or that you just outlive all of those services.

[00:48:16.520] - Jay Tennier

That's a good problem to have. But from my experience, you do not want to be under the gun having to change things out because especially when a service is shutting down, they might be good to you or they might not. You may find out one day that they're done, and now all of your code is just useless, right?

[00:48:34.940] - Robby Russell

For our listeners, I think this is an interesting thing for people to think about. Anything that you're giving money to to keep your platform moving and integrating with other services. It's something that I thought about as you were talking here. I was like, why are we not... Maybe it's something that people do. Is there anyone out there that's adding rules, like linting rules, to if any of these vendor names show up in anywhere outside inside of a gem that's being called instead of one rapper, flag that. Someone is touching something in the wrong place in the application. That could be an interesting... I'd be curious if anyone out there listening is doing anything like that just to help remind those other 50 developers on your team like, Hey, be cautious. This is the pattern we're trying to follow. Let's be very mindful not to have if sending something specifically to Datadog from a controller, we should probably be passing that through some rapper or something and then sending it to Datadog. But at the time, the incentives for all of those platforms is to have you be very, very coupled into their system. So there are code examples you're going to find on their API docs, their developer guides.

[00:49:40.360] - Robby Russell

It's not going to necessarily be advantageous to you being able to switch away from them one day.

[00:49:45.520] - Jay Tennier

Especially when you're wrapping things in blocks in order to capture data from a span for a data dog. And like, yeah, it's hard to be that generic, right? But in some cases, it's like at the very least, I got through this LaBrado thing and on the the other side of it, we just have a metrics namespace that calls like, I don't know, like send event or those things. So that at least we have that. If we do need to swap out Datadog with something else, which is who knows? The other thing, too, is just reminding people that this isn't necessarily just about whether you think the company is going to last this third-party vendor or whatever. This is business. And business things go sideways a lot of the time. They decide to raise their price. You can't afford it. You have to churn. You want to churn. You just don't like what they're doing. There's all kinds of reasons that you might need to come off of that. If you do it up front, then it just makes that decision so much easier as well. It's like, Oh, well, yeah. No. Because the last thing you want is to tell a manager that it's going to take a month of engineering time to make this change over to a vendor.

[00:50:59.030] - Jay Tennier

Then they just do the math in their head and go, Well, that's how much you make in a month, how many developers, and then how much extra was I paying? I guess we'll just keep it. I mean, that's math, that's business, but it's not satisfying.

[00:51:12.500] - Robby Russell

No, it's not. I was having a I had a conversation recently with Kemp Beck, and he's talking about optionality and something came from one of his new books that he came out with, I think it would be last year, for whatever reason. Then the name was... Tidy first is the name of that book. But he was talking about optionality being a that developers should be thinking about. How are we building things and how are we keeping our options open in the future? But not necessarily at the cost of still making progress. I think there's this net balance. You could make things over generic, like premature optimization. This will allow us to do anything in the future. But also knowing that you can fast forward down and platforms evolve and change or raise prices or get bought by VC, and then you don't like who or private equity, and then the company changes and you find yourself working, SolarWinds owning something that you're still integrating with years later, and they haven't made any feature updates to the platform, the application that you're still paying for. It's not going to get any better for you. And so you're like, well, what do we do?

[00:52:09.510] - Robby Russell

But the cost to change to something else is way more than a couple of hundred dollars a month you're going to save, so you're not going to do it. But the developer experience and the maintainable experience for your Rails application might not be as pleasant as a result of that. And so how do you make those switches cheaper in the future? Not necessarily. They're not going to cost nothing.

[00:52:28.400] - Jay Tennier

Yeah, minimize them as much as possible and realize that even if you have a dedicated platform team, they probably have something better to do. But when you're, especially as small as we are, I took that on simply because the other developers have real stuff to do, quote, unquote, real product thing. We're trying to make our product better and trying to figure out how to switch from one metrics platform to another doesn't make anything better. The adapter thing is just... That's just one of those where it feels like abstraction. It feels early optimization or whatever. It's just like, no, trust me, this is the one place where you definitely want to do this. You won't regret it. It's just easier. It's nice to grep for it, too. You can just find every place and throwing that grep for Stripe in your code and hope that one line comes back where it's instantiating the API client.

[00:53:21.580] - Robby Russell

I'm even using that one example of Stripe. It's like a lot of those gems will then install or create database columns. There's a little bit more complexity than just maybe wrapping the gem itself. But I do think that it's worth exploring that a little bit more. In my fantasy world would be a lot of these vendors would actually encourage you to do that. That would be actually a value add, I think, for developers to be like, Hey, look, we're making it so it seems a lot more relatively simple to switch away from us, but you don't need to because we're the one that's giving you all the great stuff anyways. Not that I think they have any alter. I don't think it is the reason why they do it that way. I think they're just trying to make it as simple as possible to integrate with their platform. But out of curiosity, do you think Rainforest does enough to... If you're going to integrate with Rainforest, it's probably like all your documentation is like, this is how you do things with Rainforest, right? So it's not like, here's how you could take the same and go to one of our competitors and do the same thing over here.

[00:54:14.100] - Robby Russell

It's just how that works. It's on the developers to figure that stuff out.

[00:54:17.500] - Jay Tennier

I would argue that you can't do what you do with Rainforest with anybody else. Tell me more. I'm sure business will be happy to hear me say that.

[00:54:26.060] - Robby Russell

Give me two examples, Jay. Come on, I'll put you on the spot.

[00:54:29.240] - Jay Tennier

Oh, okay. Right now, our big push is being able to generate tests from scratch using a prompt, right? There's a lot of companies out there that can do that. But what we provide you at the end result is a rainforest test that is then executed by rainforest the same way that if you had gone in hand created the test. We don't use AI to do the execution. It's not always using the prompt in order drive your test out. Maybe some people don't like that, but that means that if we generate the test and it's not up to your speed, not up to what you actually need to test, all of that thing, you can go in and fix it and then execute it, and then it's done. To me, That's our big selling point. Then the other one is because we use... We can argue about whether you should be trying to find things via the dom or via the actual image that you see on the screen. But we have found that that's people think about stuff, especially just that above the developer spot and not having to know what the actual dom IDs are and all that thing.

[00:55:39.140] - Jay Tennier

It just makes it easier. Just like, well, what do you want to click on? I want to click on the login button. We'll draw a box around the login button. We take a screenshot and then we find that login button. But the thing is we're going to find that login button no matter where it is on the screen. You can move your UX around within reason, that thing. It's still computers. There's still some things that we can't do, but that's the other big one is having tests that still execute but have a little bit more leeway when you're actually trying to go through the app. You can move things around and not have to necessarily worry that, well, it's not at that. Or the same with coordinates. Anything could be at those coordinates. It's the web, right?

[00:56:16.350] - Robby Russell

Sure. That makes sense. As soon as assume, I think you already alluded to this, that you test your own dog food as part of your own testing workflow. Outside of using Rainforest QA to test maybe the front-end part of it, what's your testing framework and stack look like there? What's your strategy?

[00:56:33.100] - Jay Tennier

For the main Rails app, for basically all of our Rails stuff, it's our spec at the lowest level. Our admin, which I said was in the... Which has Hamel templates and all of that stuff. We actually still have view tests, testing the actual views, the generated views, and it is just a nightmare to make any changes in the admin. We're moving those over to system tests so that we can do, at least basic, are the things in the admin, on the page where they're supposed to be. Then that's pretty much it. Then we put Rainforest right at the very top of the testing pyramid. We have, I can't remember, maybe 12, 10,000 tests, something like 10,000 specs. Then we've got, I think, around 200, 220 Rainforest tests that all go out. The nice thing is we run those same tests against the front-end suite when it goes out. We're always checking what whatever goes out, whether we've broken anything at any level. It's certainly saved us a number of times.

[00:57:35.840] - Robby Russell

In preparing for this conversation, you had mentioned that you prefer air quoting wet tests. Can you tell us a little bit about that?

[00:57:42.460] - Jay Tennier

Yeah. If people aren't familiar, the concept of drying up your test to the point where you're using the subject and lets and the nameless or example is it blocks. So by the time you get down to the actual thing that got executed, it's like, I don't even know. You can't read it, right? It's always bug me just from a, I don't know what's happening standpoint, but if you've ever had to deal with those, I had a contract with a company where that was standard, and they would not accept a PR that wasn't written in that way. What you ended up with was, I don't know, like a 2,000 or 3,000-line file. You're down at like line 2,500. You're six contexts in. The subject has changed three times. The lets, in our case, it would be just a client object or whatever, but that client has been modified or changed or overwritten or whatever it is so many times. Then there's also a lot of people don't realize that subject is memoized. If it's been called and then you call it again, you're not re-executing it. There's just so many things that go wrong. Then at the very end of the day, you can't tell what is happening in that test.

[00:59:01.160] - Jay Tennier

When it breaks, it's just impossible to fix. I've always been a fan of whatever collaborators, whatever anything means. This is test-driven in a way. You're forced to enumerate all of the things that you need in the test. What's supposed to happen is you go, This sucks. There's just so many things here. Then you figure out a way to fix it, not by necessarily fixing the test, but by fixing the code that it's calling. Then there's the other thing, too, where it's like, in certain cases, you're going to have to create objects a specific way inside of this test file. You don't need to use lets or anything like that. Just write yourself a nice little, it's still Ruby inside of there. Write yourself a method with a well-named name that says what this thing is creating. Because the other thing I find is they use generic names all the way down. The very top has a client, and then everything that changes as you get further down into the context, it just says client again. You don't know what that client is. How is it changed? What is it? As opposed to if I was just writing it inside of a regular example and I've got unsubscribed client.

[01:00:16.380] - Jay Tennier

It would be the name of my variable, right? But you can't use unsubscribed client in that test because client is relied on by whatever, two other things that are being created, especially clients, right? Because they're like the God, everything everything belongs to the client. So what ends up happening, and this is, I'm really on the soapbox now, but what ends up happening is you become the linter, compiler. You have to hold the entire stack of whatever is happening inside of that test inside your head. God forbid, another example didn't fail 100 lines up. Everything that you're looking to see is so far away from where you're actually working. I'm looking at a test down here, but the client is declared at the top of the test, and the thing that the client or that belongs to the client is halfway between those two because the context is switching around. Part of it is like it's not readable, but it's also not understandable. It just slows everything down.

[01:01:13.740] - Robby Russell

Cognitively in particular. Yeah. If I recall, I think thoughtbot has a good blog post on this, if I recall. I'll include a link to that in the show notes on wet test. I had one of their developers on my other podcast, I think earlier this year, we talked about this a little bit more in-depth as well. I'm curious, what are your thoughts on... Do you lean heavily on things like Factory Bot or anything?

[01:01:35.500] - Jay Tennier

Yes. I mean, yes, for good and for ill, I guess, to some degree, right? I try not to reach for factories. We have them. They're available. They're there. We are using them all over the place. It's one of the reasons why I think everything's so slow, mainly because Factory Bot hides all of your associations. Then when you do get the... The worst thing is trying to create something down at the of the tree. The job group needs a run. The run has run test. The run needs a client. You end up creating 15 things, but all you needed was the thing in this one specific, normally state. I needed whatever I needed to have passed from one state to the other. But you get all of this stuff up at that higher level, which in a lot of cases, it's not that it makes sense, but when you look at the code, it makes sense because those things are doing database stuff. They're making changes in the background, and we're checking, we're reloading objects or models and checking to see what happened. But I prefer not to write that way, if at all possible. But, Rails makes it hard, but it's more that factory bot makes it easy to just get the thing that you need in the state that you need.

[01:02:50.240] - Jay Tennier

But you really do forget that there's a lot of stuff coming along with it. We also, a long time ago, use factory bot to generate our seeds. Yeah, you can see where that goes, where it's like, Oh, we need to seed the database with this test. We passed a client in. The client and the test are set up, but we didn't pass in all of the other associations. So all of those got set up, but they rely on the client. And in some cases, we weren't doing a good job of making sure that that client was getting passed down. So you're creating things that belong to a test, but that belong to another client. It was a mess. We since cleaned up a lot of that. But that was, again, a huge, huge project that we had somebody take on when we were bigger.

[01:03:37.800] - Robby Russell

I'm curious about, given your domain, hearing you talk about your clients, customers, they're running their test, does that level of just the vocabulary, is that an interesting challenge unique to your... You feel like there's a unique thing about working in the space of managing a platform for running tests and also writing tests in there? Is there an interesting When you're having conversations, are we talking about our clients' tests? Are we talking about our own tests? Is there weird overlaps there that you're like, Oh, this is... I'll just leave that as an open question, I think.

[01:04:10.580] - Jay Tennier

The one thing that I think anybody who has ever worked for Rainforest would agree on is that we internally are horrible at naming things, whether it's projects or things. Some of it is for good reason, some of it is for really bad. You're typical just like, Oh, that was a funny name, and then it stuck, and now nobody knows why it's called that, and it's embarrassing to even just say it out loud thing. But we have a hard time because I think when it was initially spun up, the original app, they wanted to use test for the model. We have test, you're creating test, you're writing test. It stepped on the test unit framework or something like that. I'm sure that there was a way around it, but there was some conflict, so they just went, Oh, we'll just call it RF test. Now we have a model named RF test that we all refer to as test. But we have a lot of associations, and in some cases, the model refers to the RF test ID, or we called it test ID, and we did the Rails dance to make sure all of that stuff.

[01:05:13.460] - Jay Tennier

Even within the app, we don't know what a test is, but it's having to have that conversation outside of it, making clear that most of the times we'll talk about Rainforest tests, which are the tests that get written in Rainforest. But then we get into interesting side stuff that might be too much. But when you create a run, we snapshot whatever tests are going to run. We take a copy of that so that while we're in the process of running, if you make changes to the test, we are going to test the thing that you ran, not the changes in the middle of making. We call those run tests. Those run tests have steps. Each of the things you're going to do is a step. You can embed tests in tests. We have a join table called test elements. It's just there's all of this. Then down at the actual execution level, we refer to the thing that gets... There's the overall run, but your test that gets executed as a job. There is the base class, but we have job tests, and then we have automation job test. The original job test is for the tester community thing.

[01:06:22.900] - Jay Tennier

We've just overloaded so much information. It's like, the joke about naming being one of the hardest things. It's like, yeah, it really is. When you get it wrong, it makes it a lot harder to just talk through stuff.

[01:06:35.400] - Robby Russell

I bet. I think if you're always looking at if you have to have some glossary to explain, Okay, but what thing are we talking about? Are we talking about our own thing? Are we talking about this thing? There's a number of projects we're working over the years, and it'll just be like their business is like they have applicants or applications, and you're like, an application form, you fill out. And then they try to name it like that. You're like, okay, well, we got all this application controller. You got some weird controller also with the application in it and people get confused when you're doing a quick search in your code editor or something like that. It's the fun part of our job, right?

[01:07:06.960] - Jay Tennier

Yeah, exactly. We have a folder, which is a thing that you can put tests in. At one point, we decided that that was actually called a feature. The idea is it tests your app's feature. From the user perspective, you're not creating folders of tests, you're naming your feature, and then you're going to put the test that tests that feature in there. But on the back end, we don't have anything called feature. We have feature flags, but we don't have features. If you don't know what it is, you're just like, what is going on over there. But yeah, you need to know that the endpoint is named folder, I think. You know what? It may be called feature, but we return a folder object. How do you just get into that thing of being careful. What are you naming this? Having a tight knit group that is deciding on the new things that you're building and coming up with that stuff ahead of time. Because we don't want products talking about something that that is getting translated to the front-end, that is getting translated again to the back-end because we came up with the name, but front-end named it something different in the React app, and then product goes, Oh, that name's not flying.

[01:08:15.200] - Jay Tennier

So they call it something else. We got database. We've already written all of this stuff. We're writing the database and stuff. We're not making those changes. So you just suck it up and put a comment up at the top of the thing, and hopefully that's enough.

[01:08:28.260] - Robby Russell

The joy is of the decisions that we've made in the past and the decisions we'll continue making, and we have to live with them. You mentioned earlier that you have a pretty large database. Is it safe to assume you've had some performance challenges, like reporting or anything like that? How do you think about gathering data about that volume about your customers? If you need to build out any analytics-type tooling for your admins or for your customers, how do you approach that there?

[01:08:54.160] - Jay Tennier

Building that stuff out, it's funny because we moved off of Heroku onto Google Cloud, I don't know, about maybe six or seven years ago, seven years ago. With that comes a whole bunch of stuff that Google provides that you just don't think about until you think about it. Back then, we were large enough that we had what we called the science team. The science team was doing things like coming up with our machine learning model for doing the image matching that we do, and then just doing interesting things with our internal client data, how much they run, what they're running, all that stuff. They put all that into BigQuery. Bigquery is the place where all of our analytics start. Yeah, anything that's happening in the app, we don't write immediately. We do a daily or twice daily sync to BigQuery. Everything gets shoved in there. Then we've got a couple of ways of getting that information out. We used to use it mainly for internal stuff. Board meeting, they want to know X amount, how many tests have been executed for the last 30, 60, 90 days, those things. Trending charts, all of that stuff.

[01:10:01.720] - Jay Tennier

We would write the queries against big query. Big query is cool. That's one of those things you got to wrap your head around because it doesn't work the way that you think it does. We have since, I can't remember when we did this, but we launched Redash, which we're hosting ourselves. Redash is essentially like a front-end, like a SQL front-end. Write a bunch of queries and you can save those queries and you can visualize those queries. Like nice, simple We use that to answer those same questions now. We don't have to do quite as much stuff in BigQuery. By in bigquery, I mean we were using DBt in order to capture some of the SQL and stuff that we were doing. But for Redash, it's just really nice. You can point it at BigQuery. We've also got to point it at a read-only version of our database. If you don't have whatever, Post-a-Co or actual CLI access to the database, you can hit the read-only database and ask it whatever queries you can think of. Our customer service managers are using that whenever we have board stuff, somebody can write a query, save it in there, and hand it off to whoever needs that information.

[01:11:12.210] - Jay Tennier

You can set it up to refresh Or it'll just cache that information. And whenever you want a new version of it, you just go and reload the page and get it all again. So it's pretty cool.

[01:11:22.580] - Robby Russell

Have you intentionally removed a feature that was built just, say, within Rails and just entirely moved it over into relying on BigQuery?

[01:11:33.400] - Jay Tennier

No, we didn't remove it. We just recently launched a new, basically internal client metrics. Clients wanted to know all of their stuff, how many tests have failed and how many execute created, creations, all that stuff. The typical thing you would see on a decent dashboard. We were looking at doing that, but we came across a project called Cube.js. It's basically like, how do you get front-end to talk to BigQuery? Because BigQuery has all your data. You can't just be making a request. Even if you got that worked out, you don't want to have all client data exposed at that API level. We need to have, given the current client who is logged in, what do they have access to? So Cube.js gives you basically a way of proxying that off, which is really nice. Cube.js has its own thing. It sends a client ID We send it to the Cube.js instance, and then you declare basically just a set of essentially views into the BigQuery database that will let you do filtering and grouping and that thing. But it's all based on what is actually sitting in BigQuery. So the the meat of it is we have big query views for that, and then Cube.js just sits in front of that and lets you access only what you need from it at the client level, and then lets them change dates and periods of time time and grouping and that thing, which is pretty nice.

[01:13:02.780] - Robby Russell

That's primarily as a read-only type of thing. I'm just thinking, earlier we talked a little bit about GraphQL, and for anyone looking at different options, this is an alternative path. If you're just using this for a read only, be able to display information to your users instead of maybe going down the path of maybe integrating GraphQL, if you're only primarily doing that for displaying. I'm guessing maybe those with the GraphQL are probably need to do a lot more writes as well. But is there some overlap into the functionality that these patterns are offering?

[01:13:34.440] - Jay Tennier

Yeah. I mean, I don't know GraphQL that well, but I do know trying to get a Postgres database to just compile a bunch of stuff when you have millions of records in a table that need to be joined with another million records. You can think of the Cube.js, the big query thing is like, that's where your cash is. We don't provide you, and it's very clear in the front-end, that you're not getting real-time data. You're not going to kick off a run, see it complete, and see those tests show up in that. You're getting it once or twice a day, and then that's it. But that allows us to offload all of that to those services which are good at it, versus the, Well, I want a count all my tests. All right, we'll go get a count of all the tests. Well, I want a count all the failed. As soon as you start compiling all of this data, especially when I say scale here, at the scale that we're at, I don't mean the amount of clients that we have, but we just do have a lot of data. When you start slicing and dicing that, the performance is just horrible.

[01:14:40.080] - Jay Tennier

We've done some internal caching and that stuff. You get to the end of a Even just trying to count. It's never the how many tests were executed for this run. It's like you're on a page and you're seeing all of the past run and it's in a table, and there's a count of all of the tests that were executed in those runs. Well, now you need the count for every single one of those runs, right? And they page over and they go and then they page back. There's a little bit of caching there, but you don't want to be counting all of that and then just keep now, now keep splitting that, right? It's It's not just tests, it's passes and failures, it's this and that. You're constantly trying to figure out what to do there. We've solved that in a couple of ways. They definitely got solved, but it's not ideal. We just saw that as like, Well, we're going to put all of these graphs on the very first page that you log into. Every time somebody logs in, we're just going to make all of those queries. We already know it's not tenable, let alone trying to group them by date and that thing.

[01:15:45.670] - Jay Tennier

It's like, What can we do? Well, we have big query, and that's where all the data live. This seemed like, if you can believe it, having two separate services provide this data is way easier than trying to get the performance out of the Postgres database that we normally would.

[01:16:02.220] - Robby Russell

You talk about that with a couple of other guests, I think, where there's that challenge of you sometimes displaying metrics and data that... Let's say a dashboard page where someone logs into your interface because for whatever reason, the product team thought that would be the interesting information people want to see first. But sometimes it's a really expensive hit just to load that information. They might be trying to go down and do a different task. That just happens to be the first page that you're landing them on. You're hitting your database a bunch depending on the size of that client and how much data they have, and there could be pretty big hit. Then teams are then trying to optimize that page because it's so frequently hit. Subtext here also maybe not actually what people are actually wanting to look at. It just ends up being the thing you display at them. What can you do to speed that up? And so people go down the path of trying to cache things and trying to optimize behind the scenes. And this could be another alternative. What if we just speed up? Maybe another question is maybe don't land them on a dashboard of a bunch of graphs that they actually weren't looking for yet until they clicked there, but that's a product decision.

[01:17:01.910] - Robby Russell

I'm just going to get off my soapbox there for a second.

[01:17:04.700] - Jay Tennier

Yeah, exactly.

[01:17:07.840] - Robby Russell

Jay, I want to pivot the conversation a little bit and talk about, in particular, deleting code and retiring services. When you reach a point where it's time to sunset something. You mentioned that there's some tools that may be written in other languages that nobody or your company knows that well anymore, and some of them might still be around. But for you, what triggers that conversation about when it might be appropriate to sunset set a service?

[01:17:31.460] - Jay Tennier

That's an interesting question because I'm trying to think back and it's like there's been multiple ways into that. Some of them have been as simple as we get a bill and you look at it and you go, Well, that's a lot of money that we're paying to a third-party service. Do we still need this third-party service? Maybe if we're larger, we got more people that can... It's not so much the service or the money, but the fact that we know that that lives in the code somewhere. If we can get that out of the code, that just makes the code base smaller and makes things easier to handle. The other side of it is like, yeah, we had a service that was written in Phoenix Liveview that was helping us do some... I'm trying to think of the word for when you're doing the machine learning and you're trying to train up the models and doing which image is the one that you that should have been chosen, that thing. We had this whole thing. We were capturing all that data, feeding it all back in. You want to talk about bad naming? We called it Oracle, and it was being fed into, you guess it, a BigQuery database, not an Oracle database.

[01:18:40.140] - Jay Tennier

But it was supposed to be like the Oracle Delphi thing. It was going to show us the future of that thing. When we downsized the folks that were not only responsible for working on it, but also responsible for that whole segment of the business is just like, Well, we don't have the expertise for it. What should we do about it? Basically costing us nothing. As long as nobody uses it, it's fine. It'll cost us a little bit to keep it running. It's not the cost at that point, but it's the overhead of knowing that it's there, that any time We need to do any some backplane updates or GCP this or those things. It's always hanging around there. The question is, is this useful? If it's not useful at our size, it makes it really easy to just go, Well, let's Let's just turn it off. Does anybody want to take responsibility for this? If the answer is no, I certainly don't. Deleting that service, getting rid of it, stopping all of the data that we were flowing to it, that thing, that was a real easy decision. There's some other decisions that have been a little harder, but what it comes down to is, especially for us, we're so tight knit, and because we have product right in with us, we still have engineering meetings to talk about engineering stuff.

[01:19:59.640] - Jay Tennier

But the The real meat of stuff is always product design and all of us. We've had a couple of things where we spend a lot of time developing a feature, and you're looking at it just like, Does anybody use this? And somebody goes, I forgot that that was there. Okay, well, if you forgot that that was there, we haven't touched it in a while. Then we can ask James, our product manager, and just say, James, are clients using this? We have really good relationships with our clients, talking to clients a lot. Maybe not all of them, but we've got a good subset of clients who will tell us when things are good and bad and that thing. We've got metrics that will tell us how much it's being used. There is a certain amount of like, maybe it stays there, but the second that product wants to do something and it's going to touch that thing, then we go, we could delete it and save ourselves a couple of weeks, versus, well, we start touching it. We've got no idea what's going to happen here. Those decisions are much more easily made nowadays because we really just want be tightly focused on the stuff that we're working on.

[01:21:02.460] - Robby Russell

Do you feel like your team celebrates subtraction enough?

[01:21:06.580] - Jay Tennier

Yeah. I mean, maybe too much. Like one of our front-end developers, there's a feature that he's been trying to get us to remove for a very long time. It's basically the editor that you use to write plain English tests for our tester community to execute. It's really old, really clunky, and it gets in the way a lot of times because we think in terms of the automated side of things, that's where all of our focus is on. Then somebody will say, Okay, but if we do this, how are we going to handle that in the plain English editor? He's just like, We could handle it by getting rid of it. I don't have to worry about everybody always thinking about just additive. It's just great to see that come up in Slack where somebody says, Hey, I just came across this thing. Can we delete it? If the answer is yes, we celebrate that. We don't maybe celebrate it outside of engineering as much, but we certainly celebrate it within the group.

[01:22:12.760] - Robby Russell

Something I've been thinking a lot more about recently It's like how teams that you've touched on, your team has gotten smaller over the years. There's this thing where I think when people get hired at a point in a company where the company is growing, so there's an assumption like, Well, we're making decisions based off of a future world where there's more developers. But Whether or not it's, I'm going to say air quote, ideal or not, there's a good chance that future version of your team is a smaller version of your team than it is today, and you're responsible for everything that you're adding. That isn't to say, don't add stuff. I think there's always this balance. But there's probably things in everybody's apps that you probably could subtract, but nobody's advocating for it enough.

[01:22:52.760] - Jay Tennier

Yeah, do that extra pass. It's so easy to forget, especially for us, because we use the app, but we use the app our way. We used to have an onboarding checklist that you went through. And Prok thought it was a good idea because we had people coming in. That was back when we had a free plan. People were coming in and had no idea of what they were getting. They had no customer service. They had nothing. They were just thrown into it. We had a checklist. But at some point, we decided that we didn't need that onboarding anymore. It's fine. It's fine to delete it. Don't fool yourself that like, Oh, it's in Git, so we'll go back. It is in Git. You can go back and get it, but you won't. Nobody will remember that it's there. But the front-end got rid of it, but I just came across it, and it's still in the back-end. We have deleted it, but we didn't delete it all the way because we were doing something else, didn't think about it. It's like, So guess what I'm going to be doing this week? It's getting rid of that.

[01:23:53.740] - Robby Russell

I know that you stepped into a manager role as the company has evolved. How did that shift your relationship with code base?

[01:24:01.280] - Jay Tennier

Oh, man, did it shift it? Yeah. Stepping into management is... You got to exercise the pragmatist muscle so much more. It certainly helps, obviously, coming from development, understanding the drive to see something through all the way to the end, like standing off every edge case, getting it all down, that thing. From what I found at in the new job is not just supporting the team, but it's so much of the... Now I'm empowered to say no more, no to things that maybe product wants to do, no to some of the things that developers not want to do, but you see that you're having your one-on-ones with them. They're talking about like, I'm having a hard time doing this and being able to say, Don't do that. It's fine. Me saying it's fine is like, I'd love that. Because what I'm doing is I'm giving you permission to not think about that anymore. That's just so different because I am not that developer. I am the developer who wants to make it work, make it right, make it fast. I don't care so much about the make it fast part, but the making it right thing is always Is that what more refactoring can I do?

[01:25:18.560] - Jay Tennier

How much better can I make all of this stuff? It still doesn't read right. What can I do? Versus the, Hey, we have a product, and we got to get some value in front of customers as soon as we possibly can. Walking that line, it just becomes very clear at the management level where that line is. We have really good relationships with the rest of the company. I'm the one usually having those conversations, and I'm also empowered to tell the devs, just put that to the side. Let's just ship it. It's fine. Are you comfortable with shipping it? Then let's just go. Let's just get this out and get this in front of people rather than spending way more time than we need to, trying to make sure it's perfect.

[01:26:00.000] - Robby Russell

As your team has gotten smaller, when you think about it, may or may not be in a hiring phase right now, but how do you think about the level of experience with Ruby on Rails, specifically, do you feel like that's something that maybe in our previous conversation, you mentioned that maybe you would hire people at Rainforest, and you could teach them Rails to some degree. Is that accurate recollection of what we had to discuss there?

[01:26:24.210] - Jay Tennier

Yeah, when we were larger, even just not when we were 15, 20, it's the like hire a great developer, and they'll be great at developing. Sure, yes, but also the thing that you need to watch, and we've actually had this happen, where if you've got the time, if you've got the... That only really works if you've got support for that developer. They don't need to be mentored like a junior developer coming in needs to be mentored, but they do need to be shown the ropes when it comes to Rails. If you have no Rails or Ruby experience, you're coming from something completely different, they're not going to write stuff idiomatically. All of the linters in the world won't tell you why that's a good idea. I still get stuff from Wreke where it's like, I get this thing, I go to the web page to find out what's wrong with the code. I don't know what it's complaining about, and it just tells me what's wrong with the code. It's like, No, I need... How do I improve this? What's the thing to do there? Having that extra help there, you can do it through PRs, but mentoring through PRs is not good.

[01:27:28.560] - Jay Tennier

I've done that. I've gone down that line. When you're small like us, there is a certain amount of like, I would love to just hire the best developer that I possibly can, but right now, if and when we hire, again, I need to hire the best React developer I can or the best Rails developer that I can or the best ops developer that I can. We just don't have the ability internally to support them and get them up to speed and get them right in good Rails code, because good Rails code is different than Good Ruby code is different than good Rails code, all of that stuff. If you've been with Rails long enough, you know that there's all of the... Not even the edge cases, but the like, We just don't do that anymore. You don't want them You're setting them up to fail a little bit in that sense. There is learning in that, but at the size that we're at and the position that the business is in, we need to get stuff out the door. Then the other side, too, is we did have some developers that were really good developers that we hired, came in, and hated, absolutely hated Ruby and Rails.

[01:28:36.960] - Jay Tennier

So what do you do in that case? We gave them an app. We actually do have another microservice that they worked on. They did an okay job. But when you're looking at the code, it's only Rails in the sense that it's using Railsy stuff. But it's not written in a Ruby way. It's not written in a Rails way. Now, same thing. It's like, are you going to teach this person a lesson through PRs? No. Then can you find something else for them to do? Maybe. Not in our shop. You just got to be mindful of that, too. You just don't know.

[01:29:14.480] - Robby Russell

When an organization is bringing in people that come from other tech stacks, and they're maybe not as excited to work with Rails or Ruby and see what's the job. But that often, at least from my sense, is that That's when a lot of other tech stacks are popping up in people's platforms when they're like, Well, we can try these other things. I think it's great that you're allowing your team to experiment and play with different technologies. That's an important thing for professional development. But also as you're living with the results of all those decisions of being like, Let's be flexible and experiment a lot of things. But we're still at the heart, we're going to be a Ruby on Real shop. This is the platform we're sticking with, and there's a couple of little outliers here. But If someone's listening right now and they're in a growing team and they are a little lax on that right now, maybe they think that's a good thing, would you give them some thoughts on like, Hey, maybe don't get... I'm not saying there's a right way to handle that because I think We've also seen good examples where people have a lot of different tooling in their platform and that works at the scale that they are, and they maintain, but not all projects need to do that.

[01:30:25.800] - Robby Russell

Where's that balance there?

[01:30:28.660] - Jay Tennier

Yeah, it's interesting because, yeah, I'm definitely not going to be prescriptive on this call because everybody's going to have their own tolerance for it. At the very least, you should be thinking really hard about it, especially if you haven't done it before. If you're at that first step of, All right, we've gotten to the point where we can hire a junior developer and train them up. We can hire somebody from outside the ecosystem that we're working in and getting them on board. You have to realize that just because they're senior and they pick up a language like you would not believe, it's not enough to just throw them to the wolves. You're still going to have to spend time with them, getting them trained up. Then there is a chance. It's a small chance, I mean, it's not... I know it can happen because I've seen it. It's like, what happens if they're not super into it? And then what do you do? Maybe your team is growing big enough that you're not really a Rails shop. You just have this app and you've got all kinds stuff and you can move them. You've got enough teams that you can move them around to.

[01:31:33.860] - Jay Tennier

Maybe they'll love to do front-end development. Throw them on that for a while. But yeah, I would be more cautious for those people who are just getting into that. If you've got a big team right now, you know what tolerance you have for that. The other side is check your internal engineering culture. How are you hiring for that? Because that's going to, I think, really direct a lot of it, too. You should have a sense of before you hire this person, not just your technical chops, but what are they like as a person? And are they going to fit in? And are they going to maybe come around to Rails after a bit because they gave it a shot? Or are they immediately going to shut down all efforts thing? We've been really good at Rainforest at hiring for culture, which is why I think our team is operating at the level that we're at. But it's hard, Doing that thing is it's easy to see if somebody can write... Well, it should be easy to see if somebody can write some code, but it's very difficult to find out how they are at taking criticism of that code and being open to change and talking to products and all of that thing.

[01:32:47.500] - Robby Russell

Out of curiosity, circling back to the first question, what keeps you on Rails? How do you... Maybe I shouldn't assume this, but do you believe that Ruby on Rails has been part of Rainforest's secret sauce?

[01:33:01.940] - Jay Tennier

Yeah, I think so. It's funny because we don't have the typical everything is in Rails and all that stuff. But even if it's not necessarily the fact that it's Rails itself, but that I don't even want to bring up the recent controversy with all the... Was it the bundler in Rails and ruby and all that thing? But I just read a good article that was talking about the good days, and their good old days was my good old days of the Ruby and Rails community and that thing. There was something about those people, how that started up, that that is the secret sauce that I feel like we have in rainforest. Right now. Now, you could probably take Rails out and put Django in or put some other thing in there, and now we would be okay. But I don't know that you get the same the same set of people who built Rainforest engineering to where it is now without that base. Then the mix of people that we brought in who just jived with it. Even if they don't get Rails, that thing, they're still just like, Yeah, we're all just trying to do the best here.

[01:34:20.520] - Jay Tennier

The big thing for us is no bullshit. It's through the company. If we still have values, I'm sure we do, but where they're enshrined. But that has always been the one that has stuck around. So much of it is just the like, we trust what it is that you're doing. We're not calling you out to call you out. But if there are hard questions, we expect you to be able to answer them, and we We expect you to, if you can answer them, that's fine. We expect you to say that, go off and find the answer and come back. There's just something about that. That's really the secret sauce of it. It's not so much Rails itself, but it's like that whole community, how that all built up originally, and then how we hired based off of that.

[01:35:05.480] - Robby Russell

That resonates a lot with me as well. Jay, I'm curious, what thing that comes bundled in Ruby on Rails is the thing that resonates with you most? Is there one feature in Rails that you just... If you were to tell someone, You got to check out Rails. Look at this thing.

[01:35:20.080] - Jay Tennier

I mean, it sounds stupid, but just the fact of when you hit Rails, knew there's a folder structure and that folder. If you know anything about MVC, we can I'm not going to argue about whether it is MVC or whatever. But if you know anything about it, you know where things can go. It's ridiculous. That plus the fact that you can spin up the server and it's right there. But feature, feature, as much as I hate having to deal with poorly written active record stuff. The ORM in Rails is just... It's so good. It's so good. You can shoot yourself in the foot with it. You can do what we've done, which is decide that the way to use active record is to write a giant sequel query that is just a massive string and then invoke it that way. But you can do that. If you get down to that level, but especially recently with the advancements that they've made with the Aeroal, I think that's how you pronounce it, and all this stuff that you can do of building queries out from there, it's just so nice. N+1s aside. But that's the thing.

[01:36:28.280] - Jay Tennier

It's like if you've never used an ORM before, and back when we started, it certainly hadn't. That's where the magic is. Then taking that step down, too, of active model, I really like what's going on there. Being able to serialize things in and out, get the column level, wrapping things, that's really nice, too. I mean, that's really the thing. The ORM is the killer Rails feature for me.

[01:36:54.380] - Robby Russell

It was the first thing that got me excited about Rails since 20 something years later. I'm just still active record. I would dream that I would have had something like that before in my prior era of software development. All right, Jay. Well, a couple of quick last questions for you. We've held you hostage long enough. Is there a technical book that you find yourself recommending to peers?

[01:37:14.920] - Jay Tennier

Yeah, I'll give you three. Great. Two by Avdi Grimm, who I think is maybe one of the best Ruby educators out there. His Exceptional Ruby and Confident Ruby, both of those books, I think they're available on the Pragmatic Programmers, but just really changed my brain about how to work with Ruby, especially Confident Ruby. It's a good book. Yeah, which is... I mean, you can take the Ruby out of it, right? It's Confident, whatever language you're using, but just the thinking about structuring methods in that way has made just such a huge difference in my life. And then the third one is Michael Feathers, Working Effectively with Legacy Code. That is another one that I can't remember how long ago that got recommended, but just like- It's a good one. Blew my mind. Just simply because of the what? Like one or two, three concepts in there, the concept of the seam, the concept of the... I can't remember when you're trying to find that pivot point in the code that will let you actually use that seam. But the big thing is like, breaking dependencies and all of the different ways that he has of breaking dependencies in the back.

[01:38:25.670] - Jay Tennier

It's just, yeah, there's a lot that probably most people don't need at the beginning of the book, but that last half of it with breaking dependencies is just... I literally broke it out this week to deal with this payment processing stuff. I'm like, Where do I begin?

[01:38:41.840] - Robby Russell

That's great. I'll definitely include links to all those books in the show notes for everybody as well. I actually interviewed both of them on maintainable, so I'll maybe try to include links to those as well, or people can search for that and find those as well. Where can folks follow your work or keep up with what you're building online?

[01:38:58.680] - Jay Tennier

We're trying to resurrect the Rainforest blog. We actually just put a new blog post out for the first time in years. We got a lot of smart developers, right? We are going to start trying to post more stuff there. Then I'm Jay Tenon basically every social media thing. I don't really post on X or anything. Thing anymore, but Blue Sky, Mastodon, that. Not a lot of tech stuff there, but you will get a sense of me personally, for sure.

[01:39:22.360] - Robby Russell

All right. Well, great. The people track you down there, and this episode can be an excuse to write something on that engineering blog.

[01:39:28.860] - Jay Tennier

Yep, absolutely. Absolutely.

[01:39:30.680] - Robby Russell

Well, with that, Jay, thank you so much for stopping by to talk shop with us today. Yeah, I thank you so much for sharing everything with the Ruby on Rails community.

[01:39:38.180] - Jay Tennier

No, thanks for having me. It's been a pleasure.

[01:39:41.980] - Robby Russell

That's it for this episode of On Rails. This podcast is produced by the Rails Foundation with support from its core and contributing members. If you enjoyed the ride, leave a quick review on Apple Podcasts, Spotify or YouTube. It helps more folks find the show. Again, I'm Robbie Russell. Thanks for riding along. See you next time.