Kayla Reopelle: What Your Rails App Is Trying To Tell You

In this episode of On Rails, Robby is joined by Kayla Reopelle, a lead software engineer at New Relic, where she works on both the Ruby Agent and OpenTelemetry RubyGems. They explore what observability means for Rails developers—not just as a debugging tool, but as a way to build clearer, more reliable systems. Kayla explains OpenTelemetry's vendor-agnostic approach to instrumentation and shares practical ways to experiment with traces, metrics, and logs in both production and local development.

GitHub: https://github.com/kaylareopelle 

🧰 Tools & Libraries Mentioned

ActiveSupport::Notifications: Rails’ pub/sub API used for instrumentation.
AppSignal: Rails-friendly APM and error tracking.
AWS X-Ray: Distributed tracing for AWS services.
Datadog: Full-stack observability platform.
Elastics Profiling Spec: Donated profiling format for OpenTelemetry.
Grafana: Open-source dashboards and visualization.
Honeybadger : Error monitoring for Ruby apps.
Jaeger: Distributed tracing system (CNCF).
New Relic Ruby Agent: APM agent for Ruby and Rails.
ObservableGauge (OTel Metrics): Async gauge for snapshots like queue size.
OpenTelemetry Collector: Pipeline for receiving and exporting telemetry data.
OpenTelemetry Logger Bridge: Sends Ruby logger output to OTEL.
OpenTelemetry Ruby: Vendor-agnostic telemetry for Ruby.
OpenTelemetry Ruby SIG: Community group maintaining OTEL Ruby.
Prometheus: Metrics collection and storage.
Rack Middleware: Web middleware stack used in many Rails instrumentations.
Rails Structured Logging / Event Reporter: Structured logs built into Rails.

On Rails is a podcast focused on real-world technical decision-making, exploring how teams are scaling, architecting, and solving complex challenges with Rails.

On Rails is brought to you by The Rails Foundation, and hosted by Robby Russell of Planet Argon, a consultancy that helps teams modernize their Ruby on Rails applications.

Chapters

• 0:00 Intro and welcome to Kayla from New Relic
• 1:12 What keeps Kayla "On Rails" and working in observability
• 6:14 Defining observability: unknown unknowns vs known unknowns
• 8:08 Signs your system might be trying to tell you something
• 11:17 What is OpenTelemetry for Ruby developers
• 13:42 Where Ruby fits in the broader OpenTelemetry project
• 20:40 Using OpenTelemetry locally while developing features
• 24:13 How ActiveSupport notifications power Rails observability
• 28:42 Open source tools like Jaeger and Prometheus for local visibility
• 30:58 Testing instrumentation itself and CI monitoring
• 38:17 Using feature flags to control data collection
• 49:43 Making sense of background jobs with metrics
• 53:50 Capturing logs in context with traces and spans
• 1:00:09 Book recommendations and getting involved with OpenTelemetry

Transcript

[00:00:04.800] - Robby Russell

Welcome to On Rails, the podcast where we dig into the technical decisions behind building and maintaining production Ruby on Rails apps. I'm your host Robby Russell. In this episode I'm joined by Kayla Reopelle, a lead software engineer at New Relic, where she works on both the Ruby Agent and the OpenTelemetry RubyGems. Earlier this year, Kayla led a workshop at RailsConf 2025 in Philadelphia, helping developers get hands on with OpenTelemetry. That session sparked some great conversations that made me want to bring her on the show to go a deeper. Kayla has been helping the Ruby community understand observability not just as a way to debug production issues, but as a tool to build clearer, more reliable systems. We'll talk about what OpenTelemetry is, why it matters for Rails developers, and some practical ways that teams can experiment with it in production, in local development, and even in your CI. It's a conversation about building more human friendly systems and how a bit more visibility can make us better collaborators and maintainers. Kayla joins us from Portland, Oregon in the United States. All right, check for your belongings. All aboard. Kayla Reopelle, welcome to On Rails.

[00:01:10.110] - Kayla Reopelle

Thanks Robby. Happy to be here.

[00:01:12.190] - Robby Russell

Kayla, let's start with this. What keeps you on Rails?

[00:01:16.030] - Kayla Reopelle

What keeps me on Rails? Well, I feel like Rails and the Ruby programming language just makes sense in my mind. You know, granted I'm not working in Rails day to day, but I'm working to support Rails applications and I feel like the way the framework is structured, kind of the rules that it has, just allows me to work within just enough creative constraints to build whatever I'd like to build.

[00:01:43.460] - Robby Russell

How did you find your way into working on observability and instrumentation for Ruby?

[00:01:49.460] - Kayla Reopelle

Well, let's see. So I started off working on Rails applications and got to install observability into applications when I was getting started early on. And I realized that as I continued building Rails applications that I was more and more curious about the internals of how everything worked. And I started to open up gems and read them more and it was more of an interest in trying to pursue more opportunities to learn about gems and those kind of internals of Ruby that power Rails that got me interested in working on a gem full time. And so I had worked a little bit in observability. You know, I think New Relic was the tool that I was using that saved me on multiple occasions when I was up late after being paged, and that made me curious when the opportunity opened up to me to potentially go work for Them and work on the Ruby Agent.

[00:02:48.620] - Robby Russell

As a software developer, do you enjoy working on web applications themselves or do you find yourself being more like I'm kind of under the hood tooling person or kind of just. Do you feel like there's a distinction there? Because I noticed that there's. I'll talk to some other engineers in the community and they're working on some really, really low level, behind the scenes things that all of us Ruby on Rails developers really benefit from. Sometimes they're not the same people. I'm like really curious about that area. Do you actually still identify as like a web application developer at all?

[00:03:18.070] - Kayla Reopelle

I feel like it's something where I could envision myself in my career bouncing between them. You know, I think at some point when I was working on the web applications, I started to lose the excitement of building a new feature in that particular software development life cycle and wanted a new challenge. And this was a new challenge that offered itself to me. You know, when I learned how to code, I kind of started with Ruby and Rails and Rails really supercharged my education. I felt like understanding why Rails worked and why it supercharged me started to be the thing that felt more interesting. So I mean I really love looking at the internals. I love having the opportunity to build something that so many people get to use. It helps feel like I can make a positive difference in the web development realm. Yeah, I could see myself bouncing back and forth. I don't think I'm necessarily in the library space forever now.

[00:04:14.410] - Robby Russell

You've clearly carved out like a fascinating space in the Ruby community, helping others understand what's happening inside their systems. And I should also mention for our listeners, just a disclaimer. Kayla and I actually, actually go back a little bit. She joined Planet Argon kind of early on in her career right after finishing a bootcamp if I recall, and spent a couple years helping our clients modernize their Ruby on Rails applications. So I don't know if that speaks to some aspect. They're like, I'll never want to touch another Ruby on Rails application again. I'm kidding, but I remember being struck by how quickly you took ownership of some of those tricky upgrade work really early on as even like a junior and kind of mid level developer in your career and how you approach those projects with a lot of patience and curiosity. So looking back now, do you think that time working with say legacy Rails code bases shaped how you think about observability or even system empathy today?

[00:05:05.220] - Kayla Reopelle

Yeah, I think that time was foundational for how I think about applications and approach them. One of the things that I really like about working on the New Relic Ruby Agent is that it is so old.

[00:05:18.100] - Robby Russell

It's the oldest.

[00:05:19.460] - Kayla Reopelle

Yeah, yeah. I mean every once in a while I'll find a commit from the founder or some of the co founders from 14 years ago and it's kind of exciting that that piece of code has lasted this long. But I think that the process of working on legacy applications and at New Relic, continuing to find ways to help people monitor their legacy applications feels like a strong through line. I think it's important to not just throw away something that you've built. And I think I learned that or that was ingrained more deeply by working at Planet Argon and getting to work on those Rails upgrades. There were also times where observability tooling I think helped us give confidence about an upgrade in certain areas that we knew, you know, things were still up and working on one half of a dual booted system. And so that kind of showed me too the power of that tooling.

[00:06:14.550] - Robby Russell

And when people talk about observability, to me it can kind of sound a little abstract at times. So how do you personally define it?

[00:06:22.780] - Kayla Reopelle

So I have a few definitions that I work through. One that I think feels almost more like a co on was from Charity Majors. Observability helps you answer unknown unknowns, whereas monitoring helps you answer known unknowns. And getting to that level of monitoring or data collection, trying to figure out the right combination of things to capture so that you can answer questions you don't even know that you have yet. In more of a concrete way. I usually like to define it as having a way to observe the internal state of your system by looking at its outputs. You ship your application into the world, people will use it, you're not exactly sure how they're using it, or always what exactly has happened in your system when a certain variable has been provided, observability kind of captures that information for you and gives you an archive.

[00:07:17.640] - Robby Russell

The other day I got to attend Xoruby in Portland and this is going to get published a little bit later. But you had mentioned that teams can start actually building more. I'm air quoting human friendly systems thanks to better visibility. What do you mean by that?

[00:07:31.400] - Kayla Reopelle

So when I think of a human friendly system, I think of a system that is kind of focused on the empathy to the people involved. So some of the groups involved are your users, yourself, your teammates, any future engineers who might work on the project. And when you are trying to add observability into your system. You can think about how adding a particular piece of data might help solve a problem, or work backwards from knowing about a problem that your users have and try to consider what data you might add to your application to help you solve that problem in the future.

[00:08:08.760] - Robby Russell

So if I'm a Rails developer who's never thought that much about observability, what are some signs that my system might not be or might be trying to tell me something?

[00:08:17.790] - Kayla Reopelle

So some signs could be that you are having abandoned shopping carts. Maybe things are running a little too slowly, maybe you are getting a lot of angry calls or different reports that you know parts of your site aren't working. You might also have team members who are just kind of confused about features that have been built recently and they have trouble building on top of them or trouble debugging them when something goes wrong. And I think also even, you know, if you have deployed a feature into the world and you're not really sure if it's getting used or how it's getting used, that might be another sign that observability could help you kind of see the effectiveness of what you've built.

[00:09:01.500] - Robby Russell

Now, I think historically I would think that product owners or would be asking for some data or report like, all right, we ship this new feature out to production in our Rails app and we want to know who's using it, is anyone using it, do we need to better promote it, educate people? I'm curious, like, is that an observability thing or is that just a reporting thing? And where's that kind of overlap there?

[00:09:22.150] - Kayla Reopelle

I think there's some sort of overlap there when you have different objectives from stakeholders. People love numbers as a way to tell what's working and what isn't working. And often there's some connection to how your system is working and how the business is doing. So when you can find a way for those things to connect, really breaking down how whatever your stakeholders are asking you about relates to the system that you're building, then you can use observability to create different metrics or alerts to have service level objectives, goals that you have about uptime for your users. Because usually downtime translates to dollars. So the more that you can show that your site is up, show that your site is working, the more faith that your stakeholders might have in the project.

[00:10:08.960] - Robby Russell

Do you think that's still kind of mostly an ops concern or developers increasingly needing to be part of that conversation?

[00:10:15.600] - Kayla Reopelle

I feel like it really depends on your company and how it's structured. I do think that it's important for everyone to be included in a conversation. I think that ops folks often have a different perspective than devs, who have a different perspective than people. More on the business side of things, it's easy to lose something or miss some dimension if someone's not part of that conversation. I think more and more tools in observability are trying to focus on helping everyone in that process and making things that are valuable to everyone.

[00:10:50.250] - Robby Russell

Let's take that as a shift into let's talk specifically about OpenTelemetry itself. I think most of us have probably at least heard the name or maybe some references to it, but it can still feel a little bit mysterious, like we have all these different vendors out there that we can plug in and get instrumentation with. You know, you work in New Relic, there's datadog, there's App Signal, Honey Badger, et cetera, et cetera. So how do you usually describe what OpenTelemetry is, say to a Ruby developer.

[00:11:17.910] - Kayla Reopelle

To explain OpenTelemetry to a Ruby developer? I kind of think of OpenTelemetry as a replacement for your standard agent. So if you've been installing App Signal, datadog, New Relic directly into your gem file, this can be a replacement for that. And more than that, it's a vendor agnostic replacement for that. So you can start adding APIs into your code base that are from OpenTelemetry and you can send that data to multiple vendors. And that way I think there's some hesitance about putting vendor specific code into your application just based on kind of the tight coupling that you then might have with another business. This allows you to kind of loosen that and instead, you know, similar to working in Ruby, working in Rails, have a fully open source tool that you can use to collect that data and see the development.

[00:12:18.420] - Robby Russell

Interesting. So maybe like another parallel there would be like you might install like the Stripe gem into your app, but if you also need to support PayPal and it's not accessible because not everybody has a credit card around the world and you're tightly coupled to Stripe, it makes it a little bit more complicated just to have another payment thing that works for PayPal. So I think that might be a simple example there. But I would imagine if someone's curious like all right, well if I'm using New Relic App Signal, what have you and I switch over to use like one of their OpenTelemetry gems, am I then able to send to you mentioned kind of send to both or is a specific data that I can send to one or the other, or send both the same packages of information and then just use their tools to look at that data differently?

[00:13:02.330] - Kayla Reopelle

Yeah, you can absolutely send your data to both. You could send your Data to many. OpenTelemetry is super modular and customizable. It's designed to be extensible. You can have multiple exporters. You can have a lot of tooling, too, to shape your data so it looks exactly the way you want it to look before you send it to a backend. You're not really locked into any one path. You can be pretty flexible with how your data gets visualized and ingested.

[00:13:32.180] - Robby Russell

That's interesting. Where does the, say, Ruby ecosystem currently fit into the broader OpenTelemetry project and effort right now? This is kind of a big initiative right now.

[00:13:42.340] - Kayla Reopelle

Yeah. OpenTelemetry is kind of structured by the data types that it collects. The three primary ones right now are traces, metrics, and logs. Traces are kind of the story of a request through your system. It's broken up into individual pieces called spans. Metrics are aggregated time series data. You know, usually things you want to see about how long a request took or how many jobs you have running. And then logs are the traditional form of observability, just a timestamped text record. OpenTelemetry is really designed around structure, so that the way that people can use it, they can feel confident about what data they're getting out of it. So there's a lot of standards in OpenTelemetry related to the specification for each of those signals and what features should be available. There's also a whole other realm of specification that's focused on what data you should collect to kind of provide best practices and expectations for instrumentation. Instrumentation being the thing that you use to collect data, kind of by calling APIs inside of a library like Rails, we instrument an application. All of that is to say that in Ruby, right now, we have varying stability for each of those signals.

[00:14:59.380] - Kayla Reopelle

Traces right now are marked as stable. So what that means is if you were to look at the OpenTelemetry specification, we should check all of those boxes for everything. Every feature listed on that page that is marked as stable. Metrics and logs are marked as experimental or in development. And the way that that looks is that we have gems available, but they're not fully feature complete. And in order for us to reach stability, we have to internally make sure that we've checked off every element on that specification in our implementation. And then also get someone from the OpenTelemetry Technical Committee to review our Implementation to verify that everything is met. Another element of getting to stability is having people actually use these tools. They are kind of considered prototypes in a way and they need to be battle tested before we can flip that stable switch. So I think that the presentation, you know, online, if you were to just look at OpenTelemetry, Ruby and where the project is at, it doesn't really reflect, I think where we're at right now. I think the LOGS project is pretty stable. There's maybe, you know, one or two new features that have come along since our initial development that we need to incorporate before we can reach out to the technical committee. 

[00:16:21.070] - Kayla Reopelle

Metrics, we still have a few kind of feature groups that we need to address before we're spec compliant. But your basic metrics, you know, if you want to implement a counter or a histogram or a gage like that is ready for you and ready to go. And we really need people, since we're kind of in this active development state, to let us know where we've fallen short because we do a lot of testing. But I think unfortunately some of the best testing is done in production environments and often as someone who works on a library, you make a lot of kind of fake applications, like really pseudo applications that are very light, that just want to prove, you know, a concept at the most minimal level. And creating like a real world application that does all of the things is usually out of reach. That's another thing that I think makes OpenTelemetry special and its implementation is that it's a collaboration between end users. So that's who we refer to as the people who are building web applications and vendors, the observability vendors, you know, from companies like Datadog, New Relic, App Signal, et cetera, who are collaborating together to work on this implementation to make sure it kind of combines our knowledge to the best product possible.

[00:17:34.130] - Robby Russell

What's interesting about this project is from a community perspective, I understand it. I think there's a lot of it's like trying to optimize for consistency and developers and being a community driven and shared goals together. But there's also an interesting thing about like, well, what's the secret sauce? Like why one vendor versus another outside of just comparing apples and oranges on prices or something like, well, how much is platform, Platform A versus Platform B offer? Like what's the secret sauce? If they're all feeding them the same data, how would you describe what distinction that is from like a business value proposition?

[00:18:05.790] - Kayla Reopelle

Yeah. So vendors are having to take a hard look at their business models Because I think in the past, the data that we collected was the thing that differentiated us. But OpenTelemetry is challenging that and saying that everyone should have access to the same great data and we should create great data that is openly available. So now vendors are kind of being forced to compete on pricing. That's a big part of it. Try to find ways that they can make their products affordable. And also, you know, storage and visualization, like, they're really putting way more efforts into what you see on your screen, how they synthesize that data. Those are the things that they're competing on instead of the data they're collecting. So I think when you're trying out different vendors, just I would say find a problem that you want to solve in your system and see which tooling actually makes the most sense to you when you're trying to solve that problem. What gives you the answers or shows you the path as you're trying to solve it? Because they want the user experience to be kind of where they stand out.

[00:19:12.170] - Robby Russell

Now, you know, admittedly, I think it's one of those challenging things for a lot of developers. Ruby on Rails developers listening might be thinking, well, someone already made the choice of what platform we're using to send data to, and we've been using this and we have some custom code for that. And so it's like it's not just a gem install and add a config file or API key and then all of a sudden it's all magic. It's like, well, we already have a bunch of things already connected to one system and it's not easy to just to flip the switch over to another thing. So something like OpenTelemetry could potentially allow that to happen a bit more quicker and you can maybe do some A, B, C actual comparisons. We don't always get to do that when we're evaluating these types of tooling. Unless you're just installing the GEM and just getting the basic details out of those. And then you're probably not really benefiting from the platform as much as you could be either. So for those listening, I'm just like, I think there's some benefit to kind of exploring ways to how to.

[00:20:04.320] - Robby Russell

And it sounds like OpenTelemetry might be able to make that kind of make that a little bit easier. But also there's a cost to making the switch over as well if you're heavily vested there. Most of what we hear about observability tends to focus on production. So like tracing. You mentioned slow requests, debugging Maybe incidents or fine tuning performance. But I wonder if there's also a way to bring some value into with visibility closer to where Rails developers are working every day. So do you foresee a way that a Rails developer could say, benefit from running OpenTelemetry locally while they're developing or testing a new feature?

[00:20:40.540] - Kayla Reopelle

Yeah, this is something that I did when I was working on Rails applications, was in the development process. If I had kind of two different ideas for implementation and I wasn't sure which one was going to be the most performant, I would kind of add some extra spans or things like that around my code and time out the different approaches that way in addition to using benchmarking to see if I could find, you know, a clear path of like, like what the right choice was because there's often many ways to do.

[00:21:11.600] - Robby Russell

A thing just to explain this concept. Like if you're looking at a, let's say a controller, some controller code or some models in your method or tell us more about a span, like what would that look like? I'm just thinking like a typical request process, you know, we might send some data, new relic or app signal, what have you. And here's the full request. And like we see that show up in app signal and I know that you can do that locally as well with these tools by just enabling in your local development environment. So what would span be within that?

[00:21:39.450] - Kayla Reopelle

So a span in that is just a single step in the process. So if you're thinking about a Rails request, you know your controller action is going to be one of the steps. Any calls to your model is going to be another span. Rendering the view is going to be another span. If you make an external call to an API that will be its own span. And you know, in some tools it goes so far as to get into the internals as well, where every RAC middleware call that is inside of the structure of your Rails application is another span. Some tools go really deep into that, others are a lot lighter. An individual job starting is a span generally as well too if you want to think about non web transactions as well.

[00:22:21.260] - Robby Russell

Okay, so like the, if you got an error that was triggered that would show you maybe the back trace and you can step through and like this is where the error happened. You can walk by back, step by step into that process. So it kind of sounds very similar. Is there much of a distinction there?

[00:22:36.380] - Kayla Reopelle

So errors are sometimes categorized a little differently right now in OpenTelemetry and this will possibly be changing soon. They create something that's called a span event for an error, so that you can kind of get that special error information, including like the stack trace and things like that. But most UIs for observability will kind of have a red color or some sort of like warning sign along whatever span raised the error. But you can see inside of the larger trace, which is the grouping of spans that represents the whole request. What was the specific step that raised the error?

[00:23:13.720] - Robby Russell

I see. I think I'm also just kind of thinking, like with OpenTelemetry, are you then defining custom spans for code within a method, or is it line by line? How does that kind of play into that?

[00:23:27.080] - Kayla Reopelle

Yeah, so instrumentation covers a lot of things. It covers all of those span examples that I provided earlier and custom instrumentation or manual instrumentation. That's where if you want to get spans for the methods that you are uniquely writing that are not attached to any other framework or library, then that's where adding your own span comes from. And you can also do that with exceptions. There are APIs as well to record exceptions that you can add into your error handling. You know, usually that will bubble up to whatever the span was that was closest or occurring when that error was raised. But if you want more detail, there's options to go more granular.

[00:24:09.340] - Robby Russell

What about ActiveSupport? How does that play into OpenTelemetry?

[00:24:13.740] - Kayla Reopelle

So ActiveSupport Notifications is an element of ActiveSupport that is heavily used by instrumentation authors, including OpenTelemetry. It's essentially a publish, subscribe, pub sub interface that allows you to listen to different Rails actions and create things based on it. So OpenTelemetry, we use ActiveSupport notifications as the bulk of our instrumentation. There are still a few things that we'll use monkey patching for, but it gives us great information, you know, related to different attributes of what's going on at the current point in time. And that allows us to build out spans, for example, by making sure that they have appropriate names, carrying over a lot of those attributes to meet the attributes in OpenTelemetry semantic conventions. That is the element of Rails that really powers most observability work. And it's also like a lot of things in Rails available for users to use for their own purposes. So you can actually make your own ActiveSupport notifications. And OpenTelemetry has tooling available for you to instrument those. With OTel, we have an ActiveSupport instrumentation gem that allows you to pass your own custom notifications to it, and those will get kind of the same treatment in terms of having spans created for them at this time.

[00:25:39.620] - Kayla Reopelle

Now, right now we don't have instrumentation for metrics. That's something that we're working on. But eventually I could see a lot of those things being used to fuel the metrics in semantic conventions and also possibly, you know, custom metrics as well. So that could be a way to look at integrating OpenTelemetry into your application if you want to avoid in some cases adding OpenTelemetry specific APIs to your code. I don't think it gets you out of the woods entirely for full observability or answering every question, but it does get you a lot closer in terms of the automagic powers.

[00:26:17.620] - Robby Russell

What are some examples that you've seen people use like some custom active support notifications for those that haven't really dove into that yet.

[00:26:25.300] - Kayla Reopelle

So one of the places that I've seen people use them is in service objects. That's not something that's captured by the existing ActiveSupport notifications. I'll also add here as an aside that the Rails guides have great documentation on the ActiveSupport notifications that are available, so you can kind of scroll through them to see what might already be covered. But I'd say anything that you do in your application that falls outside of those strict sub gems of Rails like active job, activerecord, et cetera, those are things that I would add instrumentation for, perhaps with ActiveSupport notifications.

[00:27:02.020] - Robby Russell

I see. So you use like the service object as something like as an example there, or if you had a different pattern for how you were approaching something in your application. I think there's already hooks for was it like all the active record, action, mailer, action cable, active storage, etc. And I've also seen people do things like, well, they might use it for like a database, like long database queries or something like that. They might track something like that and get some instrumentation there as well. That sounds interesting. I'll have to definitely poke around that as well. Any thought experiments you might encourage people like listening right now to like, hey, if you got a Rails app and you want to play around with your active Support notifications with OpenTelemetry, just go try this today.

[00:27:40.660] - Kayla Reopelle

Just to play around with it. I would open up the Rails guides, look at how it all works, kind of look at the subscribe method. I'm forgetting right now what the other method is called. That kind of lights things up and I would just find maybe one spot in your application, add a notification to that and then maybe in a similar spot in your application, add a OpenTelemetry span that's using the tracer in Span method call and kind of compare what's available to you. See which interface that you prefer in terms of adding attributes, creating names, subscribing, because they both kind of have that setup process of the stage you have to set in order to create the notification and then inside of that actual notification you're subscribing to. I think interface preference is a great example there. And also if you did like a one to one switch for ActiveSupport notification and an open telemetry span could be an interesting way to see if there's any performance difference in what you get out of both of them.

[00:28:42.120] - Robby Russell

Nice. All right folks, there's your assignment for this afternoon. Recently at the talk you were talking about a like an open source tool that you could use to run maybe locally or on a server somewhere that you can send this data to as well. So you can look at this information without needing to rely on new relic or app signal or what have you. Can you tell us a little bit about that as well?

[00:29:01.590] - Kayla Reopelle

Yeah, so a few tools that you can use that are kind of separate from the vendor payment landscape. I mean Grafana is one. It has fully open source and kind of self hosted to their own like paid products as well. Prometheus is a great example for metrics and being able to capture data there. And most OpenTelemetry systems, I believe, ours included, allows you to translate OpenTelemetry metrics into Prometheus. And then there's also Jaeger, which is another fully open source solution to look at tracing. And that's kind of from the same parent organization that OpenTelemetry works under, the Cloud Native Computing Foundation. They also support Jaeger, so there's a lot of interoperability designed between those two tools.

[00:29:53.300] - Robby Russell

This episode of OnRails is brought to you by Asset Fingerprinting. Because back in my day we just renamed the file and hoped for the best. Have you ever changed the style sheet, deploy it and still see the old version? Yeah, me too. We all did. For years. We'd slap v2 on the end of the URL, we'd clear our browser caches, we'd blame the CDN, anything, but fix it properly. Then Rails gave us fingerprints. Real content based automatic file versioning. No hacks, no begging the browser to behave. Asset fingerprinting because life's too short to fight stale css. Respect your future self. Hash your assets.

[00:30:31.970] - Robby Russell

Something I don't hear discussed much is testing the instrumentation itself. So once a team starts adding a bunch of say, OpenTelemetry or vendor specific code, I've rarely seen anyone really write a lot of test coverage or test around making sure that it's working the way that they expect and it's not potentially breaking things. Has that been your experience? Do you see teams writing tests for their telemetry and their instrumentation like this?

[00:30:58.590] - Kayla Reopelle

Some teams, I think it's, you know, not the majority of teams, but some teams who are kind of committed to trying to get 100% coverage. Especially if they're adding custom instrumentation into their methods, they'll make sure that spans are created for those systems as well. And OpenTelemetry has some tooling to possibly make that easier. You can have a console exporter or an in memory exporter. That's what we use for our own testing to kind of validate what has been created. This is a little bit different from the question that you asked, but there are also libraries to help observe your testing frameworks as well so that you can get spans about tests that are frequently failing. You know, maybe you want some sort of CI monitoring to see maybe what costs you have or to judge the speed of your test suite. OpenTelemetry has tools for that as well.

[00:31:49.600] - Robby Russell

I guess if it's. You're thinking about it like any other third party system in your, in your test, like if you're mocking or stubbing out some of that data or using something like vcr. I'm just thinking like things that people use in Rails apps to mock like a third party service. If you're just trying to send Some data, does OpenTelemetry have a way to pull back down data into your Rails app? Is that ever a thing that people kind of do? Is it mostly a kind of a one way send it into this platform and then you can interact with the data there. So it's kind of like a data storage, you're pushing out perspective.

[00:32:20.670] - Kayla Reopelle

Yeah, that's a good question. That's not something that I've seen is people pulling the OTel data back into their systems. But I suppose that if you're using something like Grafana or Jaeger to visualize your data and you have your own storage system, which is something OpenTelempathy allows, you can kind of cut yourself off entirely from the observability vendors. I could see that providing some strategies to bring the data back into your app.

[00:32:51.420] - Robby Russell

Yeah, I was just kind of curious if like down the road a couple steps, is there anything that people might Start taking advantage of the data to like if you're collecting data on you mentioned like end user data or how things are being used and that ends up changing the behavior of how your application's actually running based off of that information. Like for example, something I've been thinking about is how people will and teams will try to optimize the performance for something in their Rails app. Let's say maybe it's a dashboard or something in their app and they'll see that their slowest requests are like from their biggest customers and so they'll do things like oh well maybe we need to optimize our queries to render this page faster. But like oh but certain customers are too big, they have too much data. They have 20 times the amount of data that our average the rest of our users. But they'll end up implementing things that they'll like, well maybe we'll do some pre caching for this information or some background jobs or we'll store some data, some like a temporary cache and they'll do that for all of their customers, you know, because they'll just kind of like, well we'll just optimize this and be like for each of our customers, pre optimize the data even though most of their users aren't logging in to check that data.

[00:33:57.680] - Robby Russell

So you're doing all this background, you create a bunch of busy work for the servers that don't actually be needed, but you only need it maybe for those few big clients. Right. And I've been thinking a little bit around this idea of is there a way to track when customers are getting to a certain size that you start enabling some of these performance benefits automatically in your system. So it can kind of, I would say self heal, but kind of adapt to how things work and set the expectations a little bit differently. I'm wondering if tools like OpenTelemetry could potentially help out with some of that.

[00:34:28.100] - Kayla Reopelle

Interesting. This isn't something that I've personally tried. So in theory you could be running a local collector that's processing your data. The collector is kind of a special executable file that has a lot of different plugins and extensions for it that allow you to kind of shape your data before it gets sent off to a backend or you know, wherever you want to store it. There may be a way to create some kind of feedback loop there when you have kind of the data before it's been fully sent off. But I wouldn't say it's something that has been designed as part of OpenTelemetry. At least not that I'm aware of. But the thing that's kind of cool about OpenTelemetry is that it could very well be designed at some point because it's designed by committee. You know, you can bring up a change to the specification or the semantic conventions, which are kind of the names of attributes and spans and such that have been codified to take the product in the direction that you think is useful for your team and people can have a discussion about it. Usually it's a discussion across languages to decide how that works and that usually will bring up other solutions that different teams have thought of, maybe with the shape of the tool as it is today.

[00:35:51.840] - Kayla Reopelle

So yeah, I think that's not something I've interacted with too much.

[00:35:55.360] - Robby Russell

I get it. This is kind of like something I've been thinking about and writing about recently, and that's a whole other topic. But I kind of am curious around how we collect data and we store this information in these systems. And one of the pricing models tends to be on the number of requests and stuff like that. And so for some people listening, they're like, well, how much is this going to cost if our platform scales and we're collecting all this information, we may or may not be needing it until there's like, you know, I'm air quoting an issue that we need to investigate or until a customer can raises a concern. In the meantime, we're just collecting all this information and data and that's going to cost us a lot to potentially use some vendor to do that. Do you find there's some interesting strategies that teams can explore to limit like let's say you only wanted to do that for your most important paying customers. You want to track certain metrics or spans or extra logging for those users. But maybe your free tier people, you're like, well maybe we don't need to collect all of that, we just want to capture some of it.

[00:36:53.560] - Robby Russell

Is there some good tooling available to help kind of guide that? Or is it just Ruby code that you can be like if a paying customer sends some data off, otherwise we're not going to bother with that right now. And that's just one of the value adds to signing up to be a paid customer.

[00:37:06.950] - Kayla Reopelle

I mean, the first thing that I would probably reach for right now is Ruby code. There's nothing that I can think of outside of like a condition that's already in OpenTelemetry to kind of control that.

[00:37:18.790] - Robby Russell

But it can be done because it's just Ruby Code.

[00:37:21.510] - Kayla Reopelle

Right. I mean, I suppose what you could do there. There are these things called span processors in OpenTelemetry so they allow you to edit your spans, change the shape of them before they get sent away. There's also some hooks that we're working on. There's an on ending hook that is still being worked out that would allow you to kind of edit your span as it's ending. And so at that point you could use one of those intervening tools to drop the data. You could collect it for everyone. And if you have a particular attribute that shows what it is. Yeah. Then you could build a special processor to strip that out before you actually spend the money to store the data.

[00:38:06.250] - Robby Russell

Oh, interesting. What about things like feature flags? Is that something that teams are using to turn on and off capturing data without needing to say, do a full redeploy of your application code?

[00:38:17.980] - Kayla Reopelle

Yeah, I think that feature flags are another thing that I have seen customers use to allow different types of observability to be sent without redeploying their code.

[00:38:30.220] - Robby Russell

Is that sort of thing where you would have to do that on like let's say you had a feature or a collection. Say you have 20 feature flags in your Ruby on Rails application, toggling things on and off. Specifically for that you want to track different, like 20 different little hotspots in your application. You want to capture some metrics. Are those things that you can control? You have to enable 20 different flags or is there some configuration options that you could also be handling? Or is it very specifically like if this feature is enabled in like, say like there's 20 different spots and say 20 different controller actions and you want to be able to toggle them on and off one by one whenever you want. Is that 20 different areas in the code? I haven't looked that closely in the actual implementation to see what that looks like, but kind of curious if that's a kind of a. It would go there in the controller code versus like in some other configuration options in the application.

[00:39:23.630] - Kayla Reopelle

For me, I think the patterns are still getting determined by the end users. I don't see a ton of examples for kind of best practices with this right now. What we use generally are kind of the entry point for OpenTelemetry for Rails applications is usually an initializer and there you call the OpenTelemetry SDK configure method and that will turn on OpenTelemetry for you. You have to do a little more work than most of the vendor specific gems and it's not all automatic. You have to add a bit of code to get it started. And I could see, you know, most of the configuration we have right now is in that file. You know, within that method you can provide a lot of settings. There are also environment variables that you can use for configuration for OpenTelemetry. And down the line, a future project, I think, once we have metrics and logs more established is to move to declarative file based configuration system, which is something OpenTelemetry is working on specifying right now. And it seems like that configuration option could provide more of these opportunities because it does seem to get nested pretty deeply. The way that you kind of create custom spans in your own application is by initiating or initializing your own tracer.

[00:40:49.060] - Kayla Reopelle

So the tracer is the thing that makes spans. And I could envision someone creating different tracers for different scenarios and maybe deciding whether or not to ship the data from those tracers based on that.

[00:41:01.230] - Robby Russell

When you're sending data to OpenTelemetry, are you also telling it how long to store the data or is that something that's managed separately through the vendor specific management?

[00:41:12.030] - Kayla Reopelle

Yeah, that's, that's something that's managed separately by the vendors or the backends. OpenTelemetry just deals with collection and standardization of data.

[00:41:21.070] - Robby Russell

So it's not like, hey, save this data for the next six months, but then please delete it after that. Like a kind of like a cache system or something. You can tell it to like expire at some point. That's helpful to know. You've also talked about how observability can help our future selves as engineers or other future engineers. So was there a moment in your own work where you really felt that benefit?

[00:41:42.560] - Kayla Reopelle

Yeah, this is something that's kind of a unique challenge as someone who's creating a library to get observability on that library that you're writing. There are features that I've built for the new relic agent where I want to know if they're being used. And I have to add my own really lightweight telemetry to be able to observe that or also to debug problems that we don't expect that customers bring to us. Sometimes that involves shipping a special version of the agent with that additional debugging information in it to kind of run for a short period of time to collect that data. But yeah, I find myself thinking, once this is out of my hands, once this is in the world, what's the information that I need to debug it to know if it's working and what questions might I have? What questions might my, my managers or other stakeholders have about the time that they spent on that project and whether or not it was meaningful?

[00:42:46.460] - Robby Russell

I could see how that could be really helpful. You mentioned that there's multiple vendors all working to have this be a committee kind of driving or having be the set patterns in the standards that we would all expect from the committee. How often is those organizations talking with each other and are they all contributing to these different programming language framework specific toolings themselves? And how often are you collaborating on these types of projects?

[00:43:12.070] - Kayla Reopelle

So work is feels like happening constantly from a million different directions and so many different companies. OpenTelemetry, one thing I think is pretty cool about its structure is that on some of the bigger committees like the technical committee and the governance committee, so things that monitor the specifications and then also just more of the day to day structure of OpenTelemetry, you're not allowed to have more than two people from the same company, I believe, on those boards. So they really want to make sure that it is diverse and that no single company ends up having a monopoly over the project. The specification is constantly being worked on. I join a meeting weekly. The way that OpenTelemetry meets is that there are special interest groups that are usually referred to as SIGs and they generally meet on a weekly or biweekly basis and you get together to kind of talk about problems, review PRs and discuss the things that can't be worked on async. We also have a Slack channel where there's a ton of communication as well. So I think that people are pretty active in it. But there's also some complications because I think a lot of the people involved, their job isn't only OpenTelemetry.

[00:44:30.990] - Kayla Reopelle

They usually have other expectations from their employer or maybe they're doing this in their spare time. So there's a lot of activity, but I think there's also sometimes a lot of split attention, which can mean that things are maybe not resolved as quickly as they would be with a vendor specific code base.

[00:44:47.800] - Robby Russell

Right? Right.

[00:44:48.680] - Kayla Reopelle

Yeah. You have paying customers.

[00:44:50.520] - Robby Russell

I'm familiar with that in the open source world. And everybody has the time to do the things that they hope that they can do. Are there a lot of people that are not part of the vendor ecosystem that are also contributing to these gems that you're working on?

[00:45:02.090] - Kayla Reopelle

Yeah.

[00:45:02.570] - Robby Russell

What do you think lures them in for those listening? If they're looking for ways to get involved in helping out with maybe these types of rubygems, how would you advise that they get started.

[00:45:12.010] - Kayla Reopelle

So I think what brings a lot of people from the end user group in is they're using these to solve problems and OpenTelemetry maybe doesn't do something that they expect it to do, or they've found a path that has a bug that we haven't discovered and so they'll submit a fix for that or maybe even through adding new instrumentation. You know, we don't have instrumentation for everything, but if we find a community member who is willing to help maintain the project, we're happy to bring in instrumentation for new gens. And so most of the people who I see join have a question or problem, and that usually gets them to contributing to help find the solution.

[00:45:52.520] - Robby Russell

You know, in doing a little bit of prep for this conversation, I actually reached out to a couple people that at had attended XO Ruby in Portland recently and asked if they had had a chance to ask you a few more questions, if there had been more time, what would they have asked you? And one person, Renee Hendrickson, asks that they're finding that each of the company's collectors seems to be wrapped in a unique API and the portability of data, or sending the same OTel data to two different collectors isn't as easy as just adding a second endpoint. So how are they handling that? Is that something that OpenTelemetry conveys that that is the case?

[00:46:26.580] - Kayla Reopelle

Yeah. This is a question that's hard for me to answer because I personally have not made the collector like my area of expertise, I think because I've been able to ignore it and just focus on trying to build things for Ruby. Now when we get to a point where Ruby has more stability, I could see myself shifting my focus more there. That data portability issue, I think, is a problem that's trying to be solved. From what I know about it, you know, the collector has hundreds of components and a lot of those are vendor specific because vendors are still trying to find a way to differentiate themselves in this new OpenTelemetry world. So kind of having their own pipelines, having their own components is one way to do that. Interesting. Yeah, I guess the basics of open telemetry provide some options, but I think as with most things in OTel, it depends on how much time you want to commit to it in order to make it specialized to your use case.

[00:47:21.910] - Robby Russell

You know, something I hadn't really thought of until just now is what's the makeup of the gems like, the client gems themselves? Like, are they having to account for all those vendors in the gems themselves or Is there just consistent endpoints to handle that? Like is that going through some central OTel tooling and then it gets sent off, or is it like the client gems themselves if there are some vendor specific details? Or is the whole idea that if there's not vendor specific things in the gems, so that way you can just be like, give me an API endpoint and some keys or credentials and it'll just send the data that way and let that system deal with it.

[00:47:57.160] - Kayla Reopelle

The latter is the goal. With OTel, the idea is to not have any vendor specific code in the API or the SDK. Instrumentation is where it can sometimes get a little murky because like aws for example has their own special samplers that are related to X ray. So that that way you can get kind of the best data there.

[00:48:16.480] - Robby Russell

Interesting.

[00:48:17.150] - Kayla Reopelle

So I would say in the core repo, which is just called OpenTelemetry Ruby that's supposed to be very vendor neutral, there's a few things related to other open source tooling that are specifically mentioned for legacy purposes. Because OpenTelemetry, you know, itself was a project that was built out of a merger of two other similar projects, Open Census and Open Tracing. Okay, so there's some things decided to merger related to Jaeger is one thing that comes to mind. Prometheus is another kind of project that's also drawing that thread as well. But I think the idea is from here on forward, unless it's kind of surreptitiously added, the idea is that nothing there should be added that isn't able to be vendor neutral and used by a community more widely. There might be a vendor who's more interested in building something for their particular product, but it shouldn't just work for them.

[00:49:08.570] - Robby Russell

Earlier we had touched around the idea around some how Ruby and Rails developers could start taking advantage of this and using something like OpenTelemetry in their tooling. So in the talk that I had seen you give, you had shown an example where you're talking about like say background jobs, for example. We'd love to like run through a couple of like kind of high level like things that people could put into practice, say later today if they have some time, or this coming week if they were like, okay, I'm going to install this jam, I'm going to experiment a little bit. So let's take the framing around jobs. How might you use OpenTelemetry to make more sense of what's happening within your jobs?

[00:49:43.700] - Kayla Reopelle

One example could be by using metrics. So in OpenTelemetry there's a bunch of different metrics. There's like different flavors of metrics I guess as well. And those are all called instruments. And one that I think is particularly useful for jobs is the up down counter. So essentially you can increment and decrement and what you could do is find maybe one of your most critical workers and around the perform method wrap maybe first a call to an up down counter that increments one to show how many jobs you have that are active and then when that job finishes, make a call to that same instrument to subtract by one so that that way you can have more visibility in your system of choice to see how many jobs are active at a given time. This could be a good way to help you identify if you need to adjust maybe the capacity that you have for your workers or tell maybe from things that aren't necessarily technical, just user driven experiences about why certain times of the day or times of the year could be causing the number of workers you have running to spike.

[00:50:51.540] - Robby Russell

I could see that being helpful if you had a bunch of long running jobs, but then you're like firing off a bunch of them and wanting to get a sense of when things are happening or you needed to maybe shift when things are getting run. Maybe you have like, well we separated them out by an hour. But if you have like long running jobs and they kind of start to cascade over, then you have a bunch of really short jobs happening. Is that potentially going to the same queue or not? See that being quite useful. Are there other types of things you've seen people or you think could also help people with benchmarking or thinking about their jobs.

[00:51:19.980] - Kayla Reopelle

So another thing that's interesting too on OpenTelemetry metrics specifically is that they have dimensions and so you could include additional information in that metric maybe related to the queue or if you want to implement it on a bunch of jobs, the job class so that you can facet that data to get more of that feedback there. Another thing that could be interesting related to jobs is you could also add a metric related to maybe queue size. There's an observable gage that could be a good option for that. Gages are intended for numbers that aren't specifically increasing or decreasing. They're kind of just a snapshot in time and observable in front of it means it's an asynchronous metric. So it's not triggered by any specific method call. You kind of set it up and then it runs on whatever interval you tell it to. And so that's something that would give you more data over time to give you lots of snapshots about what size the queue is. You know, maybe every 60 seconds or 20 seconds. Whatever is meaningful to you.

[00:52:26.180] - Robby Russell

What about for maybe something more user facing like in your front end application we talked about just tracking the performance or like some spans there. Are there other things that you've seen? There's some other interesting tooling within, maybe things related to logs or anything that we could getting more information on.

[00:52:42.580] - Kayla Reopelle

Yeah, I think logs could be a great example here. You know, logs have been around for a long time. Rails has a lot of great logs built in. You know, I think people often will add logs for debugging purposes as well. And OpenTelemetry, its logging tooling, puts the logs in context with your traces and spans. We recently released a logger bridge is what they call it. The package is OpenTelemetry Instrumentation Logger and that adds instrumentation to the Ruby logger to capture all of the logs that you're already creating and send them to OpenTelemetry or, you know, put them in that OTLP format. OTLP being OpenTelemetry's special protocol that allows you to kind of take advantage of the telemetry that you've invested in for years if you're maybe just starting with an observability company through your logs and then make them even better by seeing them in the context of a particular span, a particular trace to kind of get that duration data that might not be included in the logs. So if you have logs that are emitted when a user does something on the front end, those would be captured and brought in as well.

[00:53:50.170] - Robby Russell

I see. And is it a lot of work to enable that?

[00:53:54.250] - Kayla Reopelle

It's pretty easy these days. For a while it needed to be installed from a branch which made an extra step in your GEM file. But now the logging SDK, the logs exporter and the log instrumentation is all available, so you mostly need to install it. Just call that OpenTelemetry Configure method. There's another method called like use or use all where you can specify with use individual instrumentations to include useall just installs all of the instrumentation it can find that's compatible with your system. And that's about it. If you have a top level exporter already set through your environment variables, the data should get sent there. If not, you might need to add another environment variable that's specific to whatever backend you're trying to send your data to specifically for logs. But just with some minor configuration, you can kind of take advantage of those logs and see them in context.

[00:54:53.750] - Robby Russell

I know that there's some new structured events that's going to be built into Rails itself. Have you had a chance to take a look at that much? And how does that vary or Differ from what OpenTelemetry is doing?

[00:55:04.310] - Kayla Reopelle

So I've had a little bit of a chance to look into it. I would really love to spend more time looking into it, and I think I will be soon. What I think is kind of interesting about structured logging for OpenTelemetry and Ruby and Rails in general is that we're kind of getting built in Rails for the first time. Logs with attributes. This is something that is pretty common in other languages.

[00:55:29.470] - Robby Russell

What does that mean exactly for those listening that might not know the difference if they're just in the Rails bubble? I'm like, well, I got information. There's like warn, debug and I see the request details and I can see some tracing of what happened in this and depending on the log level that I have set, tell us more about what you mean by that. Attributes.

[00:55:46.720] - Kayla Reopelle

Generally with your logs, when you use the Ruby logger, there's maybe like three different types of data. You have your timestamp when it happened, you have the severity level, like debug and warn and the message itself. And often if you are just ingesting those logs into some sort of system, maybe like Splunk, you have to do a lot of parsing in that message to pull out unique data. You know, you might have it semicolon delimited or pipe delimited. And that's how we've been kind of adding structure to our logs. Structured logs with the event reporter give you the opportunity to add attributes. So your log looks a little more like a hash. And there are libraries that have been working on that. Like Semantic Logger is one that's really popular with Rails and they also have an open telemetry bridge. So you can use OpenTelemetry with semantic logger, but now with the event reporter in Rails, you can start to add those attributes to your logs. If you want to include something like user id or if you have a log that's consistent, maybe it's defined in the application controller. You can add whatever the unique controller is as its own separate attribute.

[00:56:57.660] - Kayla Reopelle

You can then filter on when you're trying to debug using those logs.

[00:57:01.600] - Robby Russell

That's interesting. So like if you're like a multi application or something, if you want to just include the subdomain, whatever that you're using, so that way you can filter out or distribute that. Is some of the tooling with telemetry able to just kind of ship that stuff over directly to telemetry or is that also still happening independently? Is there any performance impact by using things like OpenTelemetry or vendors if they are needing to make a network call to send data out while also logging to the servers that they're running on?

[00:57:32.960] - Kayla Reopelle

The thing with observability is you have to get comfortable with a little bit of overhead. Like nothing comes for free. I feel like people who are working on these gems are quite obsessed with performance and will write really ugly code in the name of performance if it just saves like that little fraction. You're right in that if you're sending things to more places like those are all going to be calls that take up memory and take up time. So I could see a performance cost occurring if you are sending your data to too many places or also if you are adding too many spans or things like that. So I would be mindful of that as well as mindful of the cost of wherever you're sending your data for ingest when you're adding anything with observability.

[00:58:19.230] - Robby Russell

But hey, CPU is cheap these days, right? We just throw more servers at it. That's always been the answer. Or we can optimize our code a little bit and it's also trying to help us optimize our code. It's always an interesting thing when we're like, where's that fine line of how, how much we monitor everything and record everything and store everything. It's always it depends, isn't it?

[00:58:41.300] - Kayla Reopelle

Yeah. And that did remind me of one other thing, you know, like for example, profiles or something that you're usually just monitoring at a snapshot. I think there's some, some new tools that are making it easier to kind of monitor in production. OpenTelemetry probably about a year ago now, had Elastics profiling specification and tooling donated to the OpenTelemetry project. So the idea is that eventually OpenTelemetry 2 will be able to create profiles that you can record to just have that in addition to the traces, metric and logs. Interesting, but we haven't started work on that yet. But if anybody out there is interested, please come to the Ruby City.

[00:59:23.360] - Robby Russell

How would they go about finding that?

[00:59:25.120] - Kayla Reopelle

Yeah, so Open Telemetry has an organization on GitHub, open telemetry. In that org there's a repo called Community. And on that Community's Readme page is A list of all of the different special interest groups that meet and the times that they meet. And so within that table you can find a link. I think our link is Calendar Ruby and that'll add you to a Google group which will send you an invite for our Zoom meetings.

[00:59:52.340] - Robby Russell

Approximately how many people are attending these.

[00:59:54.100] - Kayla Reopelle

These days we don't have a lot of people attending them, so we can change that. Yeah, that would be great.

[00:59:59.060] - Robby Russell

Well, I'll definitely include links to that in the show notes for folks as well. There a couple of last questions for you, Kayla. Is there a technical book that you find yourself recommending to peers?

[01:00:09.550] - Kayla Reopelle

The book that helped me the most when I started my job at New Relic was metaprogramming for Ruby 2. It broke down everything that I needed to know about monkey patching, which felt terrifying and wrong when I started this job. It was something I really tried to avoid when I was using Rails applications previously. But often the way that you add instrumentation to different libraries is by module prepending or alias method chaining. New Relic still supports both. That gave me a great sense of why you would want to use those tools, how you could use them, and also their their downsides as well. And even though we've moved on for Ruby 2, I think a lot of that information is still very relevant and hasn't changed too much.

[01:00:54.970] - Robby Russell

I'll definitely include a link to that in the show notes. Do you know if there's a new one yet?

[01:00:58.730] - Kayla Reopelle

I don't know.

[01:00:59.530] - Robby Russell

We'll look it up. And Kayla, I just have to say it's been incredible to see your journey since those early days of Planet Argon. And to watch you up on stage teaching folks and mentoring so many Ruby developers about topics like this can be pretty intimidating. So I just want to say it makes me genuinely proud. Thank you and I'm looking forward to seeing where you continue to evolve. And thanks for coming on Rails today to talk shop a little bit about OpenTelemetry. I'm going to go experiment a little bit more with this and some of the projects that I'm working on and I hope some of our listeners do as well.

[01:01:29.600] - Kayla Reopelle

Thank you, Robbie. Yeah, thank you as well for the opportunities at Planet Argonne. I think that's a straight line from there to here and everything that I learned on those Rails applications. So thank you for the opportunity.

[01:01:41.280] - Robby Russell

Likewise. Thanks again, Kayla. Thank you so much for stopping by to talk shop with us today. That's it for this episode of OnRails. This podcast is produced by the Rails foundation with support from its core and contributing members. Members if you enjoyed the ride, leave a quick review on Apple podcasts, Spotify or YouTube. It helps more folks find the show. Again, I'm Robby Russell. Thanks for riding along. See you next time.