What The Stack?
A podcast where Andy asks his guests about the hardware, software, tools and people they rely on to build our products.
What The Stack?
The Journey from Basic Rails to a Scalable Podcast Platform with Tom Rossi
Curious about how a simple tool for churches and nonprofits evolved into one of the leading podcast hosting platforms?
Tom from Buzzsprout as he takes us through the platform's journey (and host and Rails World sponsor of this very podcast) from its Rails 2 roots to its current sophisticated infrastructure. What were the critical decisions that helped Buzzsprout scale, including why the team chose to stay close to vanilla Rails and how advancements in the framework provided significant benefits.
Welcome to the what the Stack podcast, where we dig into the hardware, software teams and tools that people use to build amazing things on the internet. We'll focus on one of those angles on each show, and today we're joined by Tom from Buzzsprout, who are the reason this podcast exists at all, and we're going to talk about the history of Buzzsprout and how the software and hardware that they use to deliver all of these podcasts to your ear holes has evolved over time. Hi Tom, hi Andy, thank you for having me on the show. I'm absolutely appreciative. Thank you for this plastic box that we are now sitting in.
Speaker 1:With the open roof you can probably just about hear the tinkle of Rails world in the background.
Speaker 2:Yes.
Speaker 1:That nice ambience? Yeah, it makes it sound live and impressive. Yes, so yeah, talk me through Buzzsprout.
Speaker 2:What was its origin? As a software project, sling a rails new yourself, or how did that? Yeah, it was a rails new. It was rails new in early versions of rails 2. I don't remember what version, but I know it was before rails 2.3, because rails 2.3 was when they introduced all the restful routes and stuff like that. I remember that was a big. It was a big change in the way that we had set it up.
Speaker 2:But it originated from another product that we had built, where we ended up having a lot of nonprofits, a lot of churches that were using the service and they kept asking how do we get sermons online? And it was a podcast. And we kept giving them instructions for how to build an RSS feed and upload your MP3s to GoDaddy. And finally we're like let's just build a very simple podcasting tool. And that's where Buzzsprout came from. And it was a joke because we never expected it to be our biggest product. But podcasting just really took off and Buzzsprout was in a great place with a great infrastructure built on Rails. We can add features, we can scale up as it started to explode. It was funny because all the things that you say when you're starting oh, that'll be a great problem to solve. One day. We hit all those problems. You eventually get to the place where you're like okay, now I'm hitting all the things that I never wanted to deal with before I ostriched it, you smash your head into the ground and hope it doesn't happen.
Speaker 1:And then, when it does happen, ideally you have enough cash, spirit and energy to do it Right, and that's true.
Speaker 2:It's totally true that, uh, it was worthwhile to push some of those things off, because not only um do you have money to be able to. Money can solve some problems Like those are problems that money can solve, um, the other thing is the framework continues to get better and so if we had tried to do a lot of the things that we did later on in Buzzsprout to be able to scale up, if we had done those too early, we would have missed out on the concept compression of Rails. So a great example was caching. There was no caching. Buzzsprout was not caching at all, mainly because I didn't understand caching and at the time it was pretty primitive. But now, when Russian doll caching came out, when it became simpler and simpler to be able to do caching, by the time I needed to do caching the concept had already been compressed to the point where it wasn't very difficult for us to implement.
Speaker 1:Yeah, it's also. It's that thing of not wanting to be the first person to solve a problem. Yeah, somebody else smarter than me solve this problem for me so that. I can just write a one-line method and have it work.
Speaker 2:Yes, whenever I can rely on people smarter than me, which there are a lot of people smarter than me.
Speaker 1:I want to do that. Yeah, absolutely. Are there other things that you? You're a long-running Rails app. Now you're Rails 2. Did you ever get too far from the most recent version? Were there bits that you had built that stopped you from moving on, or was it always relatively kept up to date?
Speaker 2:Yeah, we are pretty militant about staying vanilla as vanilla can be, so we don't minimize our dependencies very much on whatever. Our decision-making tree is pretty simple Okay, what is 37signals doing? And if there's an answer of this is the way they're doing it and it makes sense for us, we're going to do it that way Again. People smarter than me have already made this decision. Now there's times where we're doing something that's different than Basecamp, for example. So Active Storage is a great example than Basecamp, for example. So active storage is a great example. Active storage is fundamentally built for 37signals and Basecamp, where for Buzzsprout, it's so fundamentally different.
Speaker 2:But this decision tree that we follow has somewhat led us astray, because active storage just doesn't work well for what we're doing. But for the most part, we've stayed vanilla Rails as much as we can and we've gotten all the benefits of much as we can and we've gotten all the benefits of coming from that, and so we always stay close to the latest version of Rails. We didn't use Rails Edge until probably maybe a couple years ago, when we made a concerted effort where we said you know what We've benefited from Rails. Let's figure out if there's a way that we can actually contribute that we can actually help make rails better. And so at that point we went to rails edge and we said okay, let's stay on rails edge, so then that way we can be pushing things up and encountering bugs.
Speaker 1:Right, that's one of the the benefits is if you're not going to be writing big features which, unless you're in the core team or in 37 signals or github or one of the bigger teams where they're seeing these big problems that need big solutions, like it's often like oh, we installed this. You see it from Chris at GoRails, right, he installs the thing and goes this is broken for us, it's broken for other people. It's that kind of stuff, right, being near or on the edge of Rails can also really help with contributing back, even if it's just I broke this thing, someone smarter than contributing back.
Speaker 1:even if it's just like I broke this thing, someone smart, let me fix it. So, in terms of, you talked about ActiveStorage, that's an interesting thread to pull on. We could have a whole conversation about ActiveStorage.
Speaker 2:Yeah, so.
Speaker 1:ActiveStorage is fundamental design. Certainly initially, when it came out, was very much if you think about the products that it came from. So it came from Basecamp, it came from Hay. The problems that they are solving are very human scale and not I hesitate to use the word internet scale. Buzzsprout is serving millions of file requests. It's a podcasting platform. It's just a fundamentally different way of shipping Very public assets.
Speaker 2:I want minimal. Again, we are not very smart. There are people much smarter than us, and so when Active Storage first came out, we went from. We had gone through several different gems that we were dependent on to be able to manage our attachments, but I think it was Paperclip was the last one that we were using and we switched over to Active Storage because that's a Rails opinion, let's do it. And so we switched over and we deployed it. The server went down. We rolled it back. We're like what happened, trying to figure it out. Looking, rolled it out again, server goes down. Well, we just hadn't thought about all the requests that our server wasn't ever seeing, because we had so many requests for those files, for our images, and they were going directly to public folders. They weren't going through a controller that then redirects, which is active storage.
Speaker 1:The pattern of.
Speaker 2:It's only the initial release now let's make sure that they have access to this before we give them access to the actual asset that they're doing, and so it was overwhelming the server because it was so many requests that we had never seen before, and so things like we just had to learn and figure out. It's just fundamentally different than the way they're using it.
Speaker 1:Cool. Are there things that you would love to see solved for you? Are there problems there where you feel like, do you know what? That would be amazing if that was in the framework. Yesterday in the keynote, like the preview of Active Record Search was like, I was just like yes, that makes total sense. It's great when you get those, david or someone else in the core team will announce a feature. That's weird that we don't have that, isn't it? But that's those best features.
Speaker 2:It's time for that compression to happen. Yeah, yeah, and and demystifying things, I think um being able to all the solid advancements, like I understand databases I understand that way better than I understand redis and having that additional dependency, um, all the solid stuff I think is really cool.
Speaker 1:Yeah, it makes a big difference. Like yeah, so for a personal project, I've been playing with all the new solid stuff and it really is great.
Speaker 2:Back to the good old days of delayed job, yeah which is funny because one of our products was still on delayed jobs, so it was very simple to switch over. Job was not on delayed jobs anymore, but one of our other products was, and it was like oh, okay, back to something very similar.
Speaker 1:Yeah, I understand how this works. So there's endless ways that we all get our software out into the world, like how are you hosting Buzzsprout? Obviously, you have very interesting challenges, given that you obviously see the end to a big part of that. How are you hosting Buzzsprout? How are you doing that?
Speaker 2:Yeah, so just a little bit of the history of our product development, hosting and stuff like that. Very much I resonated with everything that David said in the keynote about being the pink elephant that's. I'm a developer man. I don't understand all this stuff and I don't. I'm so scared to touch the knobs and the dials on servers and things like that, and so we had depended on a company called Rails Machine for a long time. They were a great partner for us back in the old days. I mean we started 2005, 2006, and they were doing the Rails hosting. So all I had to do was worried about writing code, push it up to them and then they handled everything on the server side right. It was a great thing Until Buzzsprout really started to take off and all those issues of scale started to come up and we just couldn't rely on a third party.
Speaker 2:We needed to bring that in-house, and so at that point we hired Brian Trawick onto our teams amazing guy and he really felt like Kubernetes is the way to go, like to be able to scale up the infrastructure. And so we had moved from just having a VPS on a server to building out a Kubernetes infrastructure on AWS and we made that switch in about 2019, I think 2019, 2020, and switched over to all Amazon. They own us. They're the joker In a good way. No, not in a good way. Not in a good way. I shouldn't say that we couldn't have scaled the way that we did without the solutions that they had, but we were able to meet. The demands of podcasting exploded during the pandemic, especially Even before that. It was always growing really fast, but then during the pandemic, everybody wanted to buy a microphone and start a podcast and we couldn't hit that scale.
Speaker 1:Advantageous that you were already in a more scalable environment and you weren't trying to do that. Yeah, as well as the scaling happening at the same time.
Speaker 2:So okay, so where we've landed is we use Cloudflare, which is a whole nother story. We could talk about cloudflare, but uh, cloudflare because we do, uh, hundreds of petabytes of of bandwidth and so cloudflare helps make that more affordable. So you have, cloudflare actually goes to cloud front.
Speaker 2:Not to be confused, amazon's I can never separate those in my mouth yeah, I know, and I always feel bad because whenever I'm talking to Amazon or Cloudflare I always say the wrong one. But Cloudflare is in front of Cloudfront and then Cloudfront is in front of, obviously, our S3 buckets, where we save all of our MP3s and artwork and things like that. To just try and optimize that spend, that bandwidth spend, because you don't want those resources coming directly out of S3, which is the way it was for the longest time. It was literally just accessing the S3 bucket and downloading. But yeah, as we started to scale up, so that's what the infrastructure looks like Kubernetes for all the servers and vanilla Rails, full CDNs, a lot of monkey patching for active storage to get it to work the way we want.
Speaker 1:Yeah, that's one of those things it would be nice for us to be able to. We have a similar situation. We're moving from my hand-rolled madness of managing assets for people, and I know the Podia folks have got some interesting stuff that they've done in Active Storage as well. There's definitely areas where we could all help get that. Give some help, hopefully, to the Rails core and they can give us some API taste. Yeah, so, yeah, cool. Do you think your infrastructure bill is like big in comparison to the size of your company? I would presume it is, given that you're serving so many files.
Speaker 2:Our infrastructure is $40,000 and $50,000 a month.
Speaker 1:So it's significant.
Speaker 2:But our revenue is pretty good.
Speaker 1:So yeah.
Speaker 2:So I think it's just a cost of doing business. Now, does that mean that I wouldn't mind seeing some savings in that?
Speaker 1:area? Yeah, absolutely.
Speaker 2:That I'm not intrigued by Kamal and the promises of going back to To a big flipping box underneath someone's desk.
Speaker 2:I know If I had just held off, if I was able to hold off longer, I could have come full circle. But yeah, so we're definitely interested. We're actually playing with Kamal with some of just like 37 Signals has Basecamp and Tidal List. We have other products that we built that we're continuing to support, but we're not building new features, and so we took one of those and we actually moved it over to Kamal and have been messing around with just playing with it to see is this something that we could actually do for Buzzsprout, and that's something that we're debating internally right now. Cool, that sounds great, which could be huge savings right, yeah, absolutely.
Speaker 2:The hard thing for us is so much of it is not processing. It's bandwidth and so bandwidth. What really the advantage of that Kamal would give us would be the freedom to be able to move infrastructures wherever we want, whereas right now Amazon owns us, you're not moving outside of that ecosystem very easily at all.
Speaker 1:I always think that's the interesting thing. So a coverage book on Heroku. So that's still a decision that I back for now we don't have anyone on the team who does DevOps. Our usage patterns are not such that we need expertise in that area. So, we save money on a human being doing all of that work Absolutely, and we have Heroku do it for us.
Speaker 2:The product that we moved was hosted on Heroku, so you're talking about whatever. Literally, I think it was $14 a month that we were spending hosting, so it wasn't about the money and it was something that was very developer-friendly. I didn't need a DevOps person. I could deploy on my own Buzzsprout we can deploy but if something goes wrong there's nothing I can do, whereas with Heroku I could manage that. And now we switched that product over to Kamal and I feel very similar as a developer.
Speaker 1:I still feel like I could go in there and understand what's going on and get in there and fix things yeah, so as you reach any, it's the same as running a team like we've just uh finished uh eileen's keynote, talk about the uh, the size of shopify and the number of developers they have, and you end up talking to people at these sorts of events and you're like, oh, you have 800 developers, I have six, yeah, and that I know some days that feels like a lot, yeah, like it's a. Yeah.
Speaker 2:The challenges you have are like they're not just about the technology, they're about the organization that you have, and so I think that's I appreciated the way that she said that right the idea that these are people things, they are not technology things and we just want to throw tech at uh solutions rather than understanding people problems. How do we organize people? How do we motivate people, incentivize them to, you know, get the results that we as an organization want to have?
Speaker 1:and so I thought it was really good is that what you think about your at your scale, like, how many engineers do you have in your team?
Speaker 2:we have three or four. We have three or four rails coders, we we have two people on DevOps, we have a couple designers, and that's it, and I love it. I love a small team with deep expertise that I have no interest in doubling, tripling.
Speaker 1:Certainly at the scale that both of us are operating a similar size product team. As soon as you add every, every person, every person changes the dynamic of any organization, right, like? Is that something you think about, do you? How do you know when you want to hire? Like, how do you know when you want to add someone in?
Speaker 2:usually when there's a pain associated. Yeah, uh, when there's some problem that you're trying to solve, but it's not the first thing that I'll go to. I feel like it's a big. It's a big commitment. It's a the first thing that I'll go to. I feel like it's a big commitment. It's a big commitment the way that we approach it right. If we hire somebody, it's a big commitment to that person that we're not going to consume you. We're not hiring you so that we can take something from you and then let you go, or we're going to run an experiment with your life and we appreciate you, and so we don't hire willy-nilly. A good example I think we hired our second DevOps person, which we went from not having a competency at all to having two people on the team that do it. But we did that because the one person so, Brian Trawick, who I told you, who engineered our whole infrastructure the guy can't go on sabbatical.
Speaker 1:If he goes on vacation, he takes a laptop with him and I was like we've got to solve this problem, this poor guy and this is a dynamic of your business right, Because you have that difficulty in terms of the scale of what you serve with a small team where you have this DevOps dependency. I don't have that. But equally, there are parts of our app run where individuals, because of their technical expertise, have started to specialize areas of the app and I've had to go actually, can we not? That's the vanilla Rails conversation. Actually, can we do the more straightforward thing? Can we not have things that we invent, that we haven't shared or open sourced or any of that stuff? It's important to stay within the rails of Rails. That's one of the benefits that you assume from using a framework like that, and that's actually again a people problem, Absolutely In terms of how do you manage your team, how do you think about it?
Speaker 2:That's a great question. I love leadership stuff, so I'm an odd CTO, I think, in that I love the technology stuff. I still want to write code but I do love the leadership challenges of building a healthy culture in an organization, and so I am all about trying to figure out how we can build healthy teams that are working remote, and I think that's the leadership challenge of this era that we're in With other companies thrown in the towel and saying, ok, we're going to go back to the office. We're really trying to build on what we're in With other companies thrown in the towel and saying, okay, we're going to go back to the office. We're really trying to build on what we've experienced. 37 Signals has led the way in terms of remote work, but there's more to building a healthy culture than just being remote and doing daily check-ins on Basecamp.
Speaker 2:Yeah, and so figuring out what does that look like for us? And so here at RailsWorld, for example, we brought our whole. There's eight of us that are here, so you've got all seven of myself and the other six. We call them the software writers. We don't call them coders. They're creative people writing code. So all the software writers are here and my partner, who's more on the design side. We're all here and we. This is a meetup for us. This is an opportunity for us to actually hang out and go have dinner together and talk about how the family is doing and look at pictures and stuff. Like that you don't get to do when you're working remote, yeah, and it's a great way for us to figure out the rhythms to build a healthy culture, right, a healthy remote culture, yeah I think rhythm is very much the word, like there is a there's a cadence to.
Speaker 1:Yeah, cadence to all businesses right, whether your business is like, oh, black friday is coming up, we're in trouble, or whether it's like we all go on holiday in august, you know, whatever the cadence of your company yeah, your organization or your team is, it's important to work out what the rituals are for you.
Speaker 2:Yeah, as an organization, it's really interesting and really to not, I feel like to be proactive in how you approach those things, because otherwise they just happen, because they're going to happen. The culture will happen, whether you are intentional about it or not. But you have the opportunity to be proactive and to think about okay, how, what are those rituals, what are what are those things, what are the cadences that we're going to strive for and then evaluating how is it working?
Speaker 1:yeah, that's the most important step. Right, the evaluation? Yeah it's. And how do you personally go about thinking about that stuff? Do you take time out to reflect? Do you write down stuff? Do you write process? What do you do for that reflection?
Speaker 2:I think one of the things that, especially in the last couple of years, I've spent a lot of time on is our performance reviews and really being intentional about how we are giving feedback, leading people and soliciting that feedback. And so we do performance reviews twice a year and that process, I think, has matured a lot. It's getting better for helping people give the feedback that we need to hear, like how can we change things, how can we lead better, how can we build expectations? I think generally, people want to do a good job. The problem is clarity. What does a good job look like to help me do a better job? And the challenge is the clarity has got to come from you, right?
Speaker 1:So you've got to be clear in your own head as the leader of that organization.
Speaker 2:And sometimes you have the clarity, or you think you have the clarity in your head, but you haven't articulated it in a way that they can hear it, or often enough, right? Yeah, a lot of times, clarity is just about repetition, and so, for us, the performance reviews is something that we've spent a lot of time on of being able to provide. This is what a good job looks like, and I heard a great quote and it said something like you have no business leading people that don't know where they stand If somebody doesn't know if they're doing a good job. That's on you, like you're not leading them well Like.
Speaker 2:You're not worthy of them as a follower if they don't know if they're doing a good job, and that really hit me. We need to do a better job of just making sure that people know what does a good job look like, so that way they have something to aspire to and you have something, when you're doing these reviews, to bounce it off of, and so that's something we've spent a lot of time on.
Speaker 1:Do you have a lot of sort of reflection, sort of peer reflection, in the retros, do you? How do you approach that? Like, obviously, like you're thinking what's the right level of, well, top down performance management? Do you have a forum in your organization for this is not working? What do we do about this? Or is it just yeah?
Speaker 2:there's a couple different aspects of it. You've got the performance review, which is really like you're reviewing the person that you're meeting with and they're reviewing you. So it's on both sides. You're getting that feedback in terms of how am I doing as a leader, leading you, and for the other person, this is how I think you're doing as a leader, and so you get that feedback. So that's good for the individual leadership type things.
Speaker 2:In terms of processes, we are doing a flavor of shape up of doing six-week work cycles and a couple weeks for cool downs, and typically during the cool down, we will have a meeting with the person who's leading the technical side and the person that's leading the creative side and say, hey, what do we want to try differently next work cycle? It's all about experiments. Is there something we want to try different? Do we want to change up the way that we organized our teams? We want to change up the type of pitches that we're accepting and the work that we're doing, and so we're always trying to learn and get better. I don't ever want there to be, you know no this is the way that we do things.
Speaker 2:Why do we do this?
Speaker 1:because we always have because, like the worst answer to that question yeah, even it's funny.
Speaker 2:It's funny because I'll get flack in the company, because I'll just break things like on purpose. No, let's just not do it that way anymore. Why? Because there might be a better way to do it. There might not be. This might be the best way to do it, but we won't know until we try something else. And I really I like breaking things and seeing.
Speaker 1:Firstly, there's been the article about founder privilege.
Speaker 2:right, you do get an opportunity to be that chaos, healthy, chaos, element, yeah, healthy chaos, yeah, and I think by being bootstrapped we can run those kind of experiments and I'm not worried that somebody's going to get mad at me and tell me I can't anymore.
Speaker 1:The people who get mad at you are the people who work with you anyway, right. Yeah, they're going to get mad at me anyway. All right, To change the tack. The final thing I want to talk about is your own personal tools that you use to manage your own work life. Are there tools that you can't live without, like software tools or processes that you run for yourself?
Speaker 2:What tools do you use to make yourself some version of effective. Yeah, I think definitely. I'm all in on Basecamp. So Basecamp is where all of our communication happens on the team. It's a tool and so there's so many different ways to use it. We're still trying to figure out the best way, Like how many projects do we have? How many campfires do we have? How many messages? Where do you post this message? Where does this to-do have? How many campfires do we have? How many messages? Where do you post this message? Where does this to-do live? How do you assign the to-do, All those kinds of things. But we, all of our interactions happen in base camp. So that's where a lot of my work typically takes place.
Speaker 2:And, surprisingly, a piece of paper. I normally have a paper notebook where I just write down the things that need to get done and prioritize the things. Pretty old school in terms of organizing. I like to sit down with no computer screen in front of me, no screen at all. I don't want to be distracted and just be able to think about okay, what are the things that I want to get done today, and focusing on what are the things that are important, what are the things that are urgent and what are the things that give me life and I think that's something that in the last year or so has really become more important is that third one, because important and urgent. I think that's like Peter Drucker they take care of themselves right.
Speaker 2:But just like. That's the way we used to organize our time and oh, this is the best way for a leader Only do what only you can do type leadership mantra. But what happens is you stop doing the things that you love. And I love to write Ruby code. Right, I love to write code, and so it might not be important, it might not be urgent, it might not be the best use of my time, but a lot of times when I'm writing that list, I'm like this sucks, I better put on here, write some code, because these other things are just going to suck the life out of me.
Speaker 2:And so you want to make sure you've got those. We call them red and green activities the red activities that take the life out of you and then the green ones that give you life. And so, for me, being able to sit down and some Ruby code is just a green activity, for sure.
Speaker 1:Yeah, no, I absolutely agree. Part of how I structure the team at Coverage Brook is to enable me to. I'm not going to take on a feature that needs to ship next week. That's just not where I can play anymore, but I can definitely take on this thing. It might not even be that interesting Ruby code oh good, I get to rewrite the billing system again but it's that kind of vibe right. It's the things that give you life and making sure that you make time for them.
Speaker 2:Yeah, and what I've done. I've been able to get John Pollard on. Our team pretty much leads the software writers, he leads the coders, and I'll go to John, I'll be like John, can I write this code, this little piece over here? Just this little, it's not critical for you can review it. I promise you can review it please, but I just I want to continue to be able to on rails, and so I don't want to get to a place where I'm not writing code.
Speaker 1:cool that's awesome. Thank you so much for your time and for the booth and for the microphone and for this fancy dude out in front of me and we're excited to have you on buzzsprout. Looking forward to the podcast and I wish the people who are interested to follow you online sure, yeah, twitter, tom rossi seven and yeah cool. Thanks very much for listening to this episode and I hope you'll join me again.
