The Elixir Outlaws now have a Patreon. If you’re enjoying the show then please consider throwing a few bucks our way to help us pay for the costs for the show.
Episode Transcript
Amos: Welcome to Elixir Outlaws the hallway track of the Elixir community.
Amos: This is the show.
Chris: This is the show. The show. Yeah, that's the trick is, uh, the show starts immediately after the clap. Everything is, and everything that's in the show is in the show.
Amos: You got that Steve?
Steve Bussey: The show is here.
Chris: Yeah, it's here.
Amos: So we have a special guest today. Steve, would you like to introduce yourself? I don't, I don't do introductions.
Steve: Yeah. A friend of, a friend of the show, Steve Bussey here. Uh, I'm an Elixir developer and I've met Amos and Chris at some conferences now, so. Finally happy to be, be joining y'all on the show.
Chris: Yeah, we're happy you're here. And you've you're you, uh, are now an author.
Steve: I am.
Chris: I mean, actually you were, were you an author before?
Steve: I guess I it's weird, cause there's like a, I guess with the beta, technically you become an author when the beta comes out, but you're not like a full author yet. You're still working your way up.
Chris: You're a journeyman author.
Steve: Journeymen for since August.
Chris: Journeyperson.
Steve: Yeah. But now I just got the final, the final chapters into beta. There'll be out by the time this is live. So then I'm just wrapping things up and hopefully I don't have too much more to do on it. So
Amos: Then it's off to the printers, right?
Steve: Off to the printers then. Probably in March or so.
Amos: If you want to see the, uh, other books that Steve has written, you can search for Steve Bussey on Goodreads and have a great laugh.
Steve: Apparently there's some other Steve Bussey’s out there that, uh, different, different types of authors.
Chris: I see. Not necessarily technical writing.
Steve: Not technical writing.
Chris: That's a lot of like 50 Shades of Grey fanfic.
Steve: You'll see some of that on there.
Amos: Yeah, probably. I mean, we did discuss whether we could get Fabio to do Steve's cover.
Chris: Oh, that'd be so good. That'd be so good.
Steve: So the royalty rate would be way too high for that. I'm sure he's still, I'm sure he has some commissions.
Chris: Think about how much money he'd have to pay you just to be on your cover.
Steve: Sell the rights to the cover.
Chris: Yeah. How much say do you have on the cover? Are you allowed to talk about this?
Steve: Yeah, I think it's all- Yeah. I don't think that's too. Uh, so they, they have like a design group. I don't even know the name of the group, but they, I think they always work with them for the cover. And they, you give some like words or some like ideas of like what the cover might be. And then they come up with like, I think it was like eight different proposals. So basically the way it works then, so they show you them and then the author and then their editor and then like other people at Pragmatic, like basically get to like vote on, you know, what cover they want to go with. And I think the author gets a little bit more, they get like an extra vote or something like that. Um, and you can vote from all, like you get like three or four votes or something like that and you can put them like, however you to.
Chris: Okay.
Steve: So for mine, I just went ahead and put all of my votes on one of the covers. And then it's like, at that point, the editors are like, well, yeah, we're just gonna go with that. So there was never like, oh, we don't want to do that cover anything like that.
Chris: Gotcha.
Amos: Aah. Nice.
Chris: I guess we should say like the title of your book.
Steve: Yeah. So the, so the book is, is Real Time Phoenix with, uh, Pragmatic Programmers.
Chris: Nice.
Amos: So how did you, uh, how did you get involved? Like how did you come to say, I want to write this book?
Steve: Yeah. So it probably started, um, the idea of writing the book was probably about a year and a half ago. I was, I had been working on a, uh, a microservice in Elixir pulling some stuff out of our big Rails app. And, um, it was, I thought it was going to be like super easy. Like I was like, all right, maybe like three or four weeks of time. And then like, obviously there's gonna be some time to like, you know, safely move things over without breaking anything. Um, just like, you know, waiting time basically. But it took way longer than that. Like, um, I probably spent, I probably spent like two months of development time and then, you know, six months total of like rolling it out just because it was like, there were so many things that we hit on and just things that like, people were like, well, this should be super easy. This is super easy. And then I was just like, man, this is like actually really hard. Um, as I'm doing this and I, I was just struggling with it. And so that, that's when I sort of like, alright, this, like, this is like a topic that I'm interested in. And I got so much help along the way. Like, I don't want to discount that. Like I was able to get the prop project rolled out and the problem solved because of like, the community was like really good, especially, you know, like Chris McCord spent so much time, like, in like private messages helping me out and stuff. So there's like definitely an appreciation there. And so that was like where the problem came from. But then also I just, I really like Elixir. I think that it's like changed the way I like think about programming and I think it's exciting. So, you know, doing it in Elixir specifically was sort of, you know, something that I had wanted to do just to share that, that passion that I have for this language with other people. Um, and that's really where like, you know, all that sort of came together then and, you know, that's where, that's why I was like, all right, I want to write this book. I think this is a good topic. And it was also timely. Cause there, there, there were like overview books about like Phoenix and Elixir, but there was nothing that like dove deeper into like a single, like vertical, which would be like the Real-Time aspects.
Amos: So when you were, when you were developing the app was the hardest like problem, like Real-Time problem that you ran into that you're like, hey, I need to make sure this is in this book.
Steve: So the, it's nice now it's actually something you don't have to think about as much. But I ran into a lot of issues with garbage collection, mainly because back then there, I don't think at that point in time, there was even like hibernation in the, in BEAM and in OTP. Now there is, which helps solve a lot of this. But, um, you know, I would, I would finally stand up the service and I'm like, all right, we're going to, we're going to scale these WebSockets up. And then 400 people connected. And I had like four to eight gigs of memory taken up, which, you know, if you had, if you had maybe a thousand people total connecting, you might be like, all right, that's fine. But if you're going to go, you know, a hundred or two hundred times past that, you know, now you're looking at, you know, I need a hundred gigs just for this one service. That doesn't sound right. And so I spent a lot of time like looking at, uh, and, and part of it was our use case. We were sort of passing large payloads over the channels, which is not something, it puts them in a position where they can't be garbage collected very well. And so, um, you know, part of it was specifically our use case, but that was the one thing I like, you know, make sure you talk about like scheduling and garbage collection, just because when you start like rolling out a Real-Time app, like you, you, you want to understand those things a little bit more. Maybe you don't have to worry about them so much in other aspects because things sort of take care of it for you. Like Phoenix takes care of, takes care of a lot of that stuff for you, but then, you know, you start rolling it out and you want to have, you know, tens of thousands of connections that are persistent. That's like a harder problem. So I try to talk a lot about like the idea of persistent connections being a challenge versus like a one-off request, which is a little bit easier to handle.
Amos: I thought Chris was going to say something.
Chris: No, I'm thinking, I'm thinking about it at all. That's interesting. I mean, so I remember one of the things you talked about in the book was, and then this is, I think this ties into that. It was like one of, some of the mitigating factors are like, uh, you talk about when to opt to have different, uh, like, like multiple channels that you're sending stuff over. Like if you're going to have this sort of like a global like notifications thing that lots of people can connect to, or if you're going to have like something more specific about a page or like a page, meaning like a part of the application, like a pager on an application or like a, a feature set or something like that. And you did, I think like you do a honestly, like, I didn't know a bunch of that stuff when I read through the book, um, in terms of like, how do you use those things efficiently? I just never thought about it to be honest. And so it was cool to see your take on like, when you want to opt to do one or the other and like what you'll run into when that happens. Because like, and that's really what you want, right. It's like, you want that hard won experience. Like, cause that's how you become like a peer, right? That's like, like books are great when they give you like real-world stuff, because then by the end of it, like the author can talk to you as like a peer and not as like an instructor anymore. Cause you're like aware of all those things now. Um, so I thought that was really cool. I thought that was super interesting. Um, uh, we're just now like using Phoenix channels at work for some stuff. Um, so it was timely for me to be able to read through it and see like kind of the, some of the patterns that you were using.
Steve: Yeah. I remember you talking about that at a conference or something, the hall, the hallway track, but that's, um, y'all and y'all will have a pretty big installation of it. I assume, you know, if you're doing something like, let's just, I don't even know what you're doing, but let's say it was like a chat or something like that. Or like let you know, live chat during a sports game. Like you're going to have a ton of connections to it and you'll be dealing with some fun problems for that.
Chris: Yeah, it's interesting. It is. I, I will say like one of the things that we're doing, and I don't know if this jives with like your experience at all, but we try to limit the amount that we actually sort of rely on, like the channel process at all. And we just try to get out of the channel process, like as quick as possible and just use it as transport. And you do something similar in the book. Uh, I guess this is a spoiler-
Steve: Oh you can spoil it. It's cool.
Amos: Spoiler alert.
Chris: Where you're like, where you actually kind of hook up a whole like pipeline of stuff with like GenStage, which I thought was super interesting. Um, mostly because I, uh, I'm a GenStage, uh, we've talked about this before on the show, but it's like I'm continually searching for the GenStage state problem. Um, where, like, I don't quite understand exactly how to, I mean, I get like what it's doing, but I have a hard time understanding how all the pieces fit together. And so that was another interesting part for me, was to see how you're hooking up your sort of entire pipeline of like how you broadcast these messages and apply back pressure and all this kind of stuff. So that was really cool.
Steve: And, and that's the fun one about GenStage. So one of the reasons I use GenStage, I, I started the project that was just talking about the, uh, the one that took, you know, I had some hard won lessons along that implementing that project. And one of the big ones was, uh, I initially was using like task.async instead of GenStage. So it's like basically just the, the general flow of this, uh, you know, three, three sentence overview is like data comes in from one service basically gets written to a database and then like, uh, basically, uh, a message gets sent down to the client. So it was basically like a push server in some, in some respects, it saves some data and it pushes it down to a client. But the other thing it does is it has to go to some because of like microservices and it has to go to some of the other APIs in the, in our system and fetch some data as it does that. So initially the entire like fetching data, push it down to the, er, or broadcast it, the actual pub/sub was, which is the Phoenix pub/sub, which goes to the channel, that was all done just by task.async and we rolled it out and it was, it was fine. And then one day our, we use Sidekiq in Rails and basically something backed up and we had this big backlog of messages that needed to be sent through the system. So our, our normal steady, you know, a few thousand per minute was no longer the case. Now there was a spike of like, I need to get a hundred thousand things through as fast as possible. And what ended up happening is it then did a bunch of API calls back to the originating service to get some data. And it was doing like 4,000, like simultaneous requests to that service.
Amos: Oops.
Steve: And like, like the Rails app just can't handle that, right? Like it's not, it's just not really built for that. And it, uh, it would like come down and then everything would sort of stabilize because the service is no longer getting messages and then it would come back up and then.
Chris: Then you'd kill it again.
Steve: Then we'd kill it again. Then we put in GenStage, right. So we didn't actually change too much about it. We, we put in GenStage, it was maybe a couple of hours to get it implemented. So like not even a big deal. And then we just threw a consumer supervisor in there. So GenStage basically has a mechanism out of the box that gives you like some concurrency and it will give you a maximum amount of concurrency. We just slapped a limit on there and said, don't do more than 10 things at a time. And I mean, honestly, it's been running like that for the past two years now. I mean, we haven't had to change anything about it and it, it sort of just, uh, fixes that problem for us, so. The cool thing about GenStage is that lets you evolve. Like I know you've talked about on the show, like, you know, your, your thoughts on GenStage. I think the one thing for me, that's the big redeeming quality about it is like, you don't have to do things with it. You can be really simple. Like I just do it in memory buffer. I don't even, I, I let GenStages buffer do everything for me, but if I needed it, if I needed it to be powered from like Reddis or a database or Kafka, I could just write that and then I wouldn't have to change anything else. I would just change that one part of it. So it's like this nice, like evolving architecture, which I think is like a good sign of something that's like generally pretty well-built.
Chris: Yeah for sure.
Amos: I'm currently working on a project where I'm processing images from a camera, uh, six frames a second from eight different cameras though. Um, and I'm using GenStage and just doing the end memory. And even though it's a pretty good sized binary, even the garbage collection now is, is good. And, and I don't really have to touch it either. It's pretty fantastic.
Chris: Yeah. And to be clear, I don't actually dislike GenStage. I just don't understand how to hook it all up. Like I don't, I haven't, I've never spent the time to like read through the docs and code to figure out like, what am I actually doing with this? But I like your, I like, I like the fundamental thing that you're talking about, which is like the ability to kind of grow this pipeline of stuff. I think that's a good, I think I agree with you. That's like a mark of a good design. Anna's here!
Steve: Yeah. I wish I could have used it more in the book, but uh, I had to make some amends there and cut- I have a chapter on it, but I don't use it outside the chapter, but eh, that's how life is.
Amos: I think it was really well-placed in the book though. I'm going to keep talking since we can't hear Anna.
Chris: It's gonna happen.
Amos: Maybe. It's gonna happen.
Chris: I think we're in a situation where she can totally hear us, but we cannot hear her. Mmm.
Amos: Maybe. Blink, if you can hear us.
Chris: If you can hear us, give us some sort of gesture.
Amos: I don't think she can.
Steve: No gestures.
Amos: Uh, so, uh, no, I thought it was really well-placed in the book. I actually didn't expect to pick this book up. When I first started reading it, I was like, okay, it's going to be all about channels and sockets. And I did not expect to run into, uh, um, GenStage in the middle of the book, but it felt well-placed. And it was like really nice to see a quick, real use case for it. And there's, I think I told you, Steve, that I really do feel like this is the best Phoenix book written, hands down, as I was going through it. Like maybe somebody who's brand new, a lot of other books feel like they're just almost teaching you just a little bit more than documentation, but they're trying to be very broad. And this one, like it, to me showed me the power, like a lot more power in Phoenix that I don't think- and just in Elixir in general, that people maybe don't get from those other books. And it, to me was like, okay, this is the killer app. This is why I got into Elixir, that isn't immediately obvious when you read the other books. So, and I got excited from the very beginning where you talked about types of scalability and you gave three types of scalability. And I was like, holy cow, this is what I want to tell every client I ever have, so.
Steve: I appreciate that.
Amos: It was pretty awesome.
Steve: The, uh, and it's one thing of, like you mentioned the GenStage being like interesting placement. That was, that was the one thing like going into the book that I was like, this is what I'm most, I want to like, write this chapter about the GenStage. And, and it was like these other problems that you have. And that was like, and then I was like, I need to talk about channels and what they are. I need to talk about web sockets, what that is. I was like, but I really want to write about the GenStage chapter and, and the, uh, the shoe store. We, we built a shoe store in the book. That's sort of the, uh, the project that spans, like the second half of the book and on, and I have a lot of sneakers. So it was a, uh, it was a fun little project to write.
Amos: Nice. So another small thing that I think that you put in the book that I thought was really fantastic to be added there too, I'm giving it away, but then again-
Steve: You can do that.
Amos: You need to get the book, to learn how to do it, right. The book tells you how. I'm just giving away the topic is the, uh, metrics, the StatsD stuff. And it's small in there, but like, it's, it's really important. And I love how you even talked about where to use it and when to use it and like what might be some important things that you want to want to be collecting metrics on and why?
Steve: Yeah. I'm sure. I'm sure, like, especially, you know, I know Bleacher Report's probably a real big on metrics too, for example. But like, if you don't know if the thing is working or not, like, how do you have the confidence in you're in sort of the code that you're writing? So yeah, that, like being, just having a way of collecting metrics is super, super powerful. Now I would say telemetry also, but like hooking it up with something like StatsD but, you know, we just use plain old StatsD in the book because it's real simple, but, uh, you know, there's these other things now that are hooking in that are making it way, way easier to get metrics out of your, out of your running system. So do that. If you're not, if you're not collecting metrics, this is known, definitely start doing that.
Amos: So are you using the StatsD directly currently, or have you switched stuff over to use telemetry?
Steve: I haven't switched anything over to it yet, mainly just because I, like, I basically have like a, a template at this point that I go with and it's literally, it takes me two minutes to get all, like what I want it, but the basics of what I want instrumented with, uh, I use, uh, instruments, which hooks up to StatsD and I just copy paste a file and it just works. So I haven't rewritten it yet into telemetry, but, you know, I, I imagine it wouldn't be that hard to do. I just haven't done that yet. Um, I've been, I mean, I've been happy with instruments. It's like, it's really just very simple to use. And I used to use like Exometer, Elixometer I guess it is in the Elixir space. And I struggled a lot with that. So that was like, the one thing is like, when I switched over to instruments, I was like, oh, I'm actually happy now. And I don't have to have a logger or whatever it is. Like all these like weird dependencies from the, uh, Elixometer.
Chris: It's like standard, it's like, it's like such a, oh man, I'm gonna get in trouble saying this. It's like such a standard Erlang library experience where you're like, you need to use a fork of somebody's fork of the thing, which you don't even know that you're supposed to, like, that's the main line version of it now, but, but everyone in the Erlang community does. And then when you ask them how I'm supposed to know that they're like, well, you should have like, read through the email list archives and you know, like you had to, you just got to avoid all the racist stuff, and then you'll find all the important things that you actually need to learn in there. And, uh, you know, it'll, you'll be fine. And like, and then they're mad, but they're mad at you that you, that you even asked the question at all. They're like, well, how do you not know how to go look at the email Erlang mailing list archives? Uh, yeah, that was, it's not fun. It's not a good experience.
Steve: I don't even know where to find the mailing list archives. I'm really, I never used mailing lists. I'm like, that's like giving my age away a little bit on the younger side, but like-
Chris: Alright hipster man. Whatever.
Steve: I just go to GitHub issue, right.
Amos: Man.
Steve: Everything would be found in issues now.
Chris: Open source is only on GitHub. That's the thing that you always need to remember. I've, I've sent someone an actual patch file exactly one time. Like, like actually build a real patch file from Git, and then send it to them so they can apply it. But I sent it to him through Slack, so-
Amos: Just copy and paste this.
Chris: So instead of a neck beard, I grew a perfectly waxed handlebar mustache.
Amos: Uh, I used the patch once, the locally though, only. Between two repos in the same box, Just cause I wanted to learn how.
Chris: But, uh, so I would've been going back to this a little bit, like, um, like your GenStage, like the, the notion of what you're doing with GenStage is very similar to what we end up doing, which is like, we just try, we essentially just get out of the channel process essentially as quick as we can. Like, sometimes that means having to go to the database first, because like, you want that level of like consistency, but yeah. Then we just, we, but often we'll just try to like, get immediately out of the, the, the channel process to unblock the channel process. And then you send that work. We send that work somewhere else so that we're not sitting there, like, cause we have a similar situation where we need to go hydrate data from like a number of other services and like do that consistently and provide a real user experience back to the user and all this kind of stuff. So, uh, yeah, totally. Yeah.
Steve: Yeah. And one cool trick for people like is you can just get out of the channel process by like just spawning a new process and doing like your, your work there. It doesn't even need to be something fancy. Like you don't even need to have a whole GenServer supervisor that you're going to, it can just be like, oh, I task spawn this and or tasks that start this. And then I, I just let it do its thing basically. And then you can respond back and that other process, and that's, that's one thing that can help with garbage collection too, because you, you put the garbage in a, like a femoral process. It's just going to be, you know, it's just going to go away really quickly versus this long lived process.
Chris: Right. Yeah.
Amos: So are you still using tasks then? You spin it off to a task instead? Cause you talked about using task.async for-
Steve: Yeah. For the one I do use tasks. Um, I use them in GenStage as the thing that is like, I use them in two places. So at GenStage at like the very, very end when you actually have to like, cause if you're using a consumer supervisor, you have to give it a module that can like be start linked and then it will boot up an instance, like a process basically, um, for that worker. So I use a task at that point, cause it's just real simple. I mean, it's, it's just incredibly simple. And then I also use it in that, like the one example there of like I have, um, a request coming into my channel and it's going to be doing a lot of work or generating a lot of memory. So let me go ahead and just spawn off a task. I think I would do, like, I think I would do a linked task, but I wouldn't do the, I would not be waiting on it. So I don't really care about the task itself. I just sorta let it do its thing. And um, and then that way that the task runs, the channel can handle new messages coming in. And then when the task is finally ready to like give its data back, it'll go ahead and do that. And the whole thing's asynchronous. So like the front end is built to handle that the backend is built to handle that. Like, you know, there's no need for things to be synchronous at that level. Cause everything's just assuming it's going to be asynchronous anyways. So
Chris: Right, exactly. Yeah. Yeah. It's, it is nice once you can get to that world because you have a lot of like, you have a lot of flexibility. It does complicate things, right? Like the clients become a lot, like stateful clients are always going to be more complicated than non-stateful clients. And by stateful, I just mean persistent connection. Like technically the clients are stateful. They're seeing a snapshot in time, right, of a view of the world. But there's also this thing where like users know that if they refresh that often fixes problems like, and stuff like that. So if you have a persistent connection now you've got like, you know, now you've got a real distributed system where in like you have to know how to recover from all these things. So there's, that does present challenges. But, um, you do get the ability to like build sort of more richer, like, experiences. I don't know, it's a tradeoff for sure.
Amos: I was going to say in my experience too, you also get to reduce the data that you're sitting back and forth in that overhead. That's the, the big thing on, on a lot of, uh, sys- like if you wanted a dashboard for somebody to be able to see like real time, I don't know, people buying stuff or sports scores, the, you get to send a whole lot less data, if you can just have a socket passing it as it trickles in, instead of trying to like, hey, let's see if we can get these to sync up every once in a while on some polling event.
Steve: Yeah. And then the, then the nice thing there is like, not only are you sending less data, but you also don't have to hydrate as much. That's one of the, like the LiveView, things that like Chris has talked about, about one of the benefits of LiveView is like, you don't have to hydrate as much data. In our Rails monolith, I'd say, I mean, I've, I've, I've personally significantly spent a lot of time on just like making sure that we reduce these like common calls to the database. Cause like, you know, we'd be doing 10 database requests just to set up a request. And, and you know, when you're dealing with like a large number of requests per minute, like that's, that's a real problem. And so when you, when you use channels, like you don't have to use that or you can do it if you want to, like Chris said, like, you may go to the database because you can, you have the flexibility to do it. There's not really a right answer either. Like, um, you know, do whatever you need to do is sort of the, the great thing about it.
Amos: Well, and you don't need to send data when it doesn't change either. Where if they're refreshing, you do. So when you started doing this, uh, switch over, um, and creating the Elixir app, did you utilize the same database or did you make it start its own?
Steve: New database. And the reason, and I'm actually doing the same thing now it's almost the same exact problem. So that, that service was for our, like what we call like the live feed, which is like, so we're like a sales platform. I work at a company called Sales Law. And so, um, with our sales engagement platform, you're having like email events and like, um, you know, you know, someone sends you an email or, you know, they click a link or just stuff like that. So, um, we were basically sourcing data for that from lots of different places. So there wasn't even one database table, which was sort of problem one because I actually wanted to add something to it. And there's just like, not really a clean way to do it because there's just gonna be another thing that it was putting together, which can be fine like that, sometimes you do that for like, distributed systems. You might have something that, like aggregates a bunch of data together, but that's actually really hard to do if you start wanting like a consistent stream of data and you, you have to start worrying about like order and stuff like that. So what we did is we just made a new, a new database, new microservice and new database table that was just strictly this one type of event. And like, instead of having, you know, eight different events that I pull data from all over the place, those eight different places would instead send their data in and it would be stored in this one table. And it makes it really easy to say, like give me a hundred items, give me another a hundred items because that's just like basic queries and pagination versus having to worry about alright, I was on, I was on item 25 of this, but item 50 of this. So I need to go and fetch these things. It just gets very complex when you're doing like distributed feeds. So, um, so basically, and the other thing is like, we don't cross databases between our microservices. Some people do, and I don't have a ton of opinion on it, but like, uh, so some people do that. Some people might say, that's not a microservice if you're sharing a database. Um, but, um, I think it's, it's, it is what it is, but, uh, yeah, we, we don't do that. So anytime we have a microservice, the communication's always like, HTTP call or Kafka, which is sort of a sharing a data store in some ways, but we'd be sending like Kafka messages, but, um, you know, most of the time it's API requests that we're doing. So, and it worked out well. I mean, like I said, that thing's been running now for probably two and a half years, and I think I've had like, I mean, persistent connections every day, all day long, our absence, it is like a sales platform, it's sort of like people are doing their full job in it. Like you have people that spend eight hours a day in it. So they come in and that connection and they're not going to shut that tab down for like the whole day, just cause, you know, why would they need to, so you have to, you know, you're having these long persistent connections that are multiple hours long. And it, I think we had one issue in the past since it was stable and running in production. We had one issue, which was like a known thing that I knew, I knew I needed to fix and just never did. And then finally, one day it came back and bit us and we fixed it. And so it's sort of nice. I mean, it's just been sitting there chugging away running. We haven't really had to make a lot of changes to it cause it is very like focused and what it's doing. So it just sort of works well. And I'm hoping this, this one I'm doing now is going to be the same way where we sort of build it and then, you know, we don't have to worry about it too much.
Amos: So your maintenance costs have been pretty low then.
Steve: Very, yeah, very, very low. Um, you know, obviously like in, in, in those services, we're not really building new features. Like I mentioned, they're sort of stable. Um, like some of those features have been around for a few years and you know, they haven't needed to change too much. People expect them to do what they do. And so, um, you know, they just, they're just sort of chugging away and, um, doing their thing,
Amos: How did you decide which, which feature to pick off whenever you were doing this?
Steve: Pain was the biggest thing. So like I said, I'd wanted to do something. I had a new service. I wanted to appear in this live feed. And I was like, well, what's another call. I'll just throw an API call in there. And it will like, and we'll just, you know, throw it into the live feed. I know it's sorta like, well, you know, let's not do that because it's like sort of getting a little bit out of hand. So it was like, all right, now's the time to take this feature and rewrite it into a dedicated service. And, and you know, we, we could have done it in Rails. We could've just made a new thing in the, in the monolith and just threw it all there and had an API where other services could have sent in their data. But, um, the big draw for us doing it in Elixir- number one, we, I wanted to get this real world project under, under my belt, but then also, um, it really made a lot of sense for channels because it is essentially like pushing data down to the client. That's, you know, one of the great use cases for channels. So just to be like really lined up well.
Amos: Nice.
Chris: It always feels so good, right. You look at some of the Elixir stuff, you've got running and you're like, I haven't touched this in six months. It's still just working great. This is awesome. I have that feeling a lot.
Steve: We replaced our, uh, we, we replaced our push service. Um, we, we were paying a push service, more money than we should have been paying per month. And, uh, we replaced it with just a pretty basic channels implementation. It's actually open source. I call it Push Act. I know Keathley probably doesn't like that name, but it's one where it's like, I mean, we're dealing with hundreds of millions of events through that going down to our channel. It's like, I literally haven't touched it since shipped. Not a single time. Um, which has been like amazing.
Chris: Yeah. We have a lot of stuff that like is so critical and just, it just, just keeps working.
Steve: Just works.
Chris: Just working. Yeah. And it's like, it, you know, occasionally we make changes to it or whatever, just as they as needed. But yeah. Some of this stuff just keeps working. It's nice. It feels good.
Steve: Yeah.
Amos: And when it doesn't it restarts itself. So then you move on.
Steve: Yeah. Except, except when your problem is when it's restarting itself it crashes. I ran into that before.
Chris: Yeah.
Amos: Yeah. Or the thundering herd, both of those I've run into even knowing about it and thinking about it. I ran into the thundering herd cause it was like, oh, this thing's not going to crash. And guess what? That was the first thing it did.
Steve: Thundering herd is a really interesting problem. Like a distributed system problem of like, you have a lot of clients, let's say, you know, you had a hundred thousand clients and they're all are expected to do something like within a 10 minute window. So every 10 minutes they want to do something. It's like, all right. How do you get them to do it in such a way that you don't thundering herd yourself? So like a really basic implementation might be like, alright, just as soon as the thing, as soon as the client boots up, it'll just set a timer for 10 minutes from now, and then it'll do its thing and then 10 minutes from then. But you might have a lot of users that log in like on the hour. So you have like, that minute one has like 80% of your data or of the requests. And so I actually took a page from Sidekiq. I was like, how does Sidekiq handle this? Because they have to deal with the same problem, which is the sort of the, the Rails, like background task runner. Um, it sends heartbeats and it has to worry about making sure that the heartbeats don't overwhelm each other. And so it just does a random distribution. It's like between 30 and 60 seconds randomly distributed, we'll send a heartbeat. So we do something like that. We need something to happen every 10 minutes, our JWT tokens only last 10 minutes. So in order to keep them refreshed all the time, every five to nine minutes, just a random number in that range. It'll um, it'll basically schedule out some work for them. And then you don't have this like thundering herd problem anymore. You do have a lot of consistent requests. Like you're getting a large number of consistent requests, but it like completely removes like the spikes of requests, which is pretty nice. That's like a hard problem. If you think about it, it's just like, how do you even solve that? Cause they don't, the things don't talk to each other, so you have to like solve it with like randomness and math a little bit and statistics. Sorta cool.
Chris: Yeah.
Amos: Jitter.
Chris: Yeah.
Steve: Yeah.
Chris: Yeah. The jitter is a real thing for retries, especially, and all that. We have, we have a similar things that we can't control it, which is like all the users get an alert and then they all show up at the app all at the same time. And it's like, that's, that's like a whole different thing. Like you don't even get a chance to, like, control that. So-
Amos: Yeah, I was gonna ask that about-
Chris: Unless you actually stagger the way that you send out alerts, which isn't a real thing that you gonna do,
Steve: Do you do things like, cause you know, you're about to send out an alert, right? You're like alerts, like there's some code that's like send out alert. Do you do things like, all right, the code that sends out alert is going to tell your like servers to like beat themselves up, basically.
Chris: Yeah. We, we have, we have a lot and like that's because that's the other big thing is like, you can't, auto-scale fast enough to handle it. And you also just like, so, so you ideally, you don't go down at all during the spike, right. And if you have to auto-scale, that's fine. But like you should present the data to people, which means that like you have to just start like blowing bulkheads, like as you go, like, because like, if you just get a spike that no one anticipated or that you didn't scale up enough for, you have to like be ready to just handle that amount of traffic just to not go down. So we've got a bunch of like, you know, essentially bulkheads, right. That you allow to go in order to like maintain the service, like coming up and, and running still. And then, but we also have like, but yeah, like by the problem is like, by the time that like the spikes over, like the auto-scaling like there, isn't a thing, like, you'd have to spend so much time on your auto scaling. Like you can't use any generic thing. Like no K8s is going to save you from this. Like, it just can't do it as quickly as you need to be able to do it. Like you can't auto scale, you know, 10 more instances, 20 more instances in like 30 seconds or less than. Which is the kind of the time window that you actually have.
Amos: So you spike is over by the time you've-
Chris: Once you detect that a spike is even happening and that you're not going to be able to handle it, like the spikes over by that point, you've missed your window. So, you know, so we actually allow a bunch of ways to do it. Some of its automated, like some of it, like we actually know that we're sending out certain notifications, so like spin up or whatever, but we also give the people who programmed those notifications to go out. They actually have the ability to scale up. Like they have a Slack command.
Steve: Oh that's cool. Oh yeah. Just solve it with Slack, that's yeah. It's a good, human solution.
Chris: Well, because like half the time too, like we don't know, like I look at these notifications and I'm like, this caused a spike. Like, what is this even about? Like, I don't even know, like, I don't know the names of sports player people, you know? And like, I don't know what's going to cause like, what's what, what dramatic new event is going to cause people to come to the website. So, and, you know, I think that's true for a lot of us as engineers, but like the people who program that stuff do know that it's like controversial and can cause a spike. Um, and so they'll preemptively just like, they literally just like run a slash Slack command and it like spins up a bunch of things all at once.
Steve: How technical is that Slack command? Is it like, I need 10 more instances or is it like-
Chris: No, it's literally like slash like slash prep or like, like slash like spike prep or something like that. And it just like, I mean, the, the thing is, is like at the end of the day, it's cheap. It's like really cheap to scale up and then to scale it back down and it just, and it lasts like 30 minutes or something like that. Like, and then, and then it automatically like scales them all back down again. Um, so by that point, like the spike will be over and sometimes like, we don't scale up quite enough, you know, or whatever, if it's like, if, oh, and sometimes in obviously like human error, like you could just miss something like you can't watch every notification that goes out. Or you could, and you'd have to like train some sort of like, you know, uh, machine learning classifier to be able to like get within some percentage chance of this causing like a traffic spike, um, which I've threatened to build. But, uh, this is not hard. It's not hard to train and you don't even have to be that accurate, like what, you gotta be like 50% accurate. Like, it's not like you're trying to like, you know, like identify birds on a, on a stop sign or something for Google. Like you're, you're, you're just like, you have to be like kind of close. Uh, and that's like easy to train. So assuming you can get all the data, but getting all the data is actually hard. Like getting all the time series that you would need is actually hard.
Amos: To tell if it was a spike?
Chris: Yeah, exactly. Well, and like, collating it all into a single place where you could like do all that work. So. It's just like time and energy and effort. And it's like is pretty good just to let humans manage it for now.
Steve: So 80% of data science is just like wrangle your data.
Chris: Yeah. 80% of data science is wrangle data. And then the other 20% is guess. And like, believe the data science can solve all the problems. That's like, at least 10% of the problem is like, you need to commit to the idea that, that data science can solve this problem, which is typically inaccurate. But if you, it it'll probably work for you. So like, you know, you just really, it's like a religious experience. You just got to commit to it and never question it. Cause if you question it, oh man, like the whole bedrock of your world will be shaken. Uh, so you just never questioned it. You always assume data science can solve the problem, regardless of any evidence to the, to the contrary and you just keep moving forward.
Amos: Isn't that the same people that write Rust?
Chris: Nailed it. So yeah, that's data science in a nutshell. So yeah, like, so, and whatever it doesn't matter anyway. But you know, other than that, I think it's basically just like tell, you know, guess at parameters to,-and feature selection. Just guess and just kind of know what features might make sense. And then you put it in some sort of TensorFlow, Keras nonsense, and then, and then hope it works. Rinse and repeat. But most of its belief, you got to believe in yourself.
Amos: Do you, uh, I'm curious, and I don't know if you can answer this or like if it's company secret, but do you stage the announcements going out? So like a smaller population gets it and then it grows up or you just-
Chris: Just let it rip.
Steve: It's all about timeliness, right.
Chris: Yeah you gotta be fast. You gotta be faster than everybody else. And that's the thing is we are, we are faster than everybody else. So, you know, take that ESPN.
Amos: I don't get to watch a lot of sports, sportsballs, so.
Chris: I don't know why I have so, I have so much company pride. I don't even work on that, on that subsystem or anything. I don't work anywhere near that. I didn't write any of that. I have no affiliation with any of that.
Amos: It's still cool though.
Chris: It's fun. It's a fun problem. They use GenStage those, the, the, the, the people who work on that, do know how to use GenStage.
Steve: Now has that changed a lot or is that one of those things too where like it's been working really well?
Chris: It basically just works. It just works. Like basically, like, it's, it hasn't undergone any sort of major revision in a long time because it's just working and it's working better than everybody else's stuff.
Steve: That's crazy.
Chris: Like, it's like, I don't even want to talk about it. Cause it's like, it's like a trade secret, but, and this is like how I feel about Elixir at this point. It's like a superpower. Every time anybody, you know, if you go on like the forum or whatever you're like, well, how is Elixir versus Python? Well, Python's faster for computation and math. And like, and then you're just like, whatever dude, like I'm so over fighting about any of that, because like, at this point I'm like, yeah, sure. Just you go off and you dig your own grave. Like you go, you, you go vanish into obscurity, underneath a pile of Yammel to make your K8s work. I don't even care. Like, yeah. Like you'll go off and do your thing. Whereas like, I just feel like using Elixir or Erlang or whatever is like a superpower and it's like a super power that I don't even want to tell anybody about it anymore because it just like, it feels like it's like, you just feel like you're cheating half the time.
Amos: Django and Cake are much more mature, I'm just saying.
Chris: Right? Yeah. Yeah. Definitely more mature. Well, oh my gosh. Oh, you're killing me.
Amos: The people, the people that know about that, they got it.
Chris: Uh, all five of you. Anyway. So, but I mean, but it does, it feels like this. Like it, it just feels like, it feels like cheating half the time. Like you just feel like you're skirting all the other problems that everybody else has to contend with and no one else gets it. I think this is like, this must be how LISP people feel all the time where you just feel like you've stumbled into sort of like this like discovered thing. Like it wasn't invented. Like you just actually discovered it like this, like sort of like, this is like the thing that just existed. And you're like, well, this is amazing. And then, and, and no one else gets it and then they don't get like, why you think it's cool? And you're like, but you have to experience it. Like, if you will experience it, you'll know, like, just try it. It's like, you're, uh, there, I'm not gonna make that joke. Never mind.
Amos: Read Steve's book and you'll feel that way.
Chris: It just, it feels good. It feels like you're, it feels like you're. Yeah. It just feels like a superpower.
Steve: Yeah. And that's, and that's exactly like that's what I was saying, like it has, I enjoy programming more when I'm doing Elixir. And a lot of that's the reason why, right. It feels, it feels that way as you're coding it too, and it's enjoyable to code, it's like, it can be a super power and run really well, but be miserable to code. And that wouldn't be very fun, but it's actually enjoyable too. So it's like the double whammy.
Chris: Yeah, exactly.
Amos: I think when I start to try to design systems in Elixir, they, they just seem simpler from moment one, the thought process that goes into how to do it and the amount that I actually have to write to handle like major problems in languages that I've used in the past. And so like that, that simple thought process is what makes it fun for me. I feel like I get to solve the actual hard problems instead of fighting the same problems over and over.
Chris: Yeah. Well, and, and the amount of effort it takes to do some of these things just blows my mind, where it's like, oh, it's just done. This took like half a day. And now this like really hard problem's just done.
Amos: You should try GenStage.
Chris: That's true. Oh, man. Think about what I could accomplish. I have a, well, so I have a new rule right now, which is that like, I don't want to work-all of my libraries- My goal over most of the majority of my libraries, there are few exceptions, but the stuff that I'm like actively building right now, my goal is to make them all under 300 lines of code long. I think that's the right benchmark. And it's amazing what you can accomplish in 300 lines of Elixir.
Steve: So you heard Stu, Stu Halloway talking at Gig City, is this where that came from?
Chris: Yeah, exactly .
Steve: The after, the after party. Stu mentioned how, uh, so Rich Hickey, the, uh, creator of Clojure. He won't write something, I think it's, his rule is 500. He's like if it's over 500 lines and you, he can't, he can't think about how he'll put it in under 500 lines, he doesn't build it. We talked about like SQL, I guess Datomic has SQL now. And like, so it can understand SQL queries. And they built that. Finally, they figured out how to build it in under 500 lines of code. So they built it. And that's just like, that's pretty crazy. I think it's a good goal.
Chris: I love it.
Steve: Now all of your functions can only be 1, 1, 1 line long too.
Chris: No. Incorrect. No. Functions, that's my other rule is that your median, uh, lines of code in your function should be at least five.
Amos: Oh I think you have to have-
Chris: Median. That's a median.
Amos: I think you have to have eight lines of Def delegate before you actually do anything.
Chris: Yeah. And you should use application config. What else do you want to throw in here to, to make me mad Amo? Go for it. I'll just, this is your floor- I'm gonna, I'm gonna, I'm gonna, I'm gonna, I'm gonna concede the floor to Mr., to Mr. King. Uh, you can now proceed. If you want to voir dire the witness here. I mean I know you don't voir dire a witness, but just whatever.
Amos: I'm just trying to get you worked up. And I, and when I get voices out of you I feel like I did that.
Chris: Is that how you pronounce that? Void air? Voy Dire. I've only, I've only been in jury duty once. And the guy was from Louisiana, the judge was from Louisiana. So you pronounced it vo-dere.
Steve: I don't even know what that, I don't even know what that was supposed to be.
Amos: I don't know what you're trying to say.
Chris: I think that's like where you question the jury, right?
Steve: Oh, ok. I've never actually been on jury duty.
Chris: So like during jury selection, the lawyers there are like, they go through this process called like, well voir dire. And then they, uh, that's how they, and they're like, they're allowed to question witnesses.
Amos: I think what I did they really, they didn't really, uh-
Chris: I think that's a part of it. Maybe I'm getting that wrong.
Amos: They didn't really call it anything. They just set us all at the juror box and started asking us questions.
Chris: In any case, this is your moment in time. You want to make me mad talking about- go ahead and mention anything you want to mention right now, get it all out of your system about things people shouldn't do in libraries.
Amos: Things that people shouldn't do ?
Chris: Yeah, go for it.
Amos: Well, I don't think you can build a good system without Def delegate for one.
Chris: That's true. Probably. Yeah.
Amos: I mean, if you don't have a couple layers of indirection, you don't have enough abstractions. Um, so, so that's, that's step one.
Chris: I think those are called ports and adapters these days. For sure. So, yeah.
Amos: Extra extractions.
Chris: Yeah.
Amos: So, and, and then, you know, on top of that, like if you're not putting specs on everything and the very first thing that your function should do on any data coming in is, is make sure that it is the right type. You gotta have like 15 lines. Once you get out of the Def delegates that are just checking all of the types of everything. And if you have a map, you have to check nested all of the types first, and then you don't really return anything except for an exception that just says bad Paramas.
Steve: (laughing) Bad params.
Amos: And it doesn't even tell you what they are.
Steve: No, you don't return the exception, Amos. You, you have to throw the exception, that's the other-
Amos: Raise, right?
Steve: Raise it, raise it, raise it.
Amos: Yeah. I forget what language I'm in.
Steve: It has both I think.
Amos: And see the great thing about that is we, you've, you've read Fred's post. It's let it crash, right. That's what you're supposed to do. So that's, that's what I did
Chris: See the, the real thing that makes me mad now, because here's the thing is that was, you, that's an old meme. The old meme is to raise exceptions, right? The new meme is to use the Elixir result type, which is to say, okay, error tuples. Everybody just uses okay error tuples these days.
Amos: Oh yeah. Or match error.
Chris: Literally for things that, for things that don't have a result, like they'll just put those, but they'll still use them. Like I've seen them used in like, people will build like little caches and stuff, and then they're like, they, they return like, okay, the thing or error missing or whatever. And it's like, that's not a result. That's not, that's not what that's for you. You did it wrong. And here's what I'm gonna say to that is, uh, raise exceptions you cowards. All of you need to start raising exceptions and crashing things more often
Amos: Raise all the exceptions.
Chris: Explicitly, explicitly match on things. Stop using width so much, explicitly match on things and raise exceptions. I dare you. I double dog dare you.
Amos: All match errors. Perfect.
Chris: At least 60% of the time. 60% of the time-
Amos: it works every time.
Chris: Yeah. But also 60% of the time you should, you should not be returning a result type.
Steve: I think 50%. Cause that way, if you have a coin, you can just flip it. If it's 60, you have to have a dice.
Chris: Oh, that's the right. See, there you go. Galaxy brain decision-making right there.
Amos: That was brilliant, Steve. This is why Steve's an author.
Chris: This is why he gets paid. This is why he lives that fast paced lifestyle that you and I will never understand Amos.
Amos: I know.
Chris: He's living the author life. He's going to do a book tour soon. That is actually a real thing, right? Like you go to conferences with your book ?
Amos: He had the sick brag earlier that he has lots of shoes.
Chris: Oh yeah, exactly. Right. Like I don't have a lot of shoes. Do you know how many pairs of shoes I have? One. I have one pair of shoes.
Steve: Just send me your credit card info. I'll buy you some shoes with your credit card. And you don't have to worry about it.
Chris: Oh, sure, that will be convenient.
Steve: I thought about that as a service-
Amos: It's like Uber, but for sneakers.
Steve: So I thought about this. People are a little afraid of spending money sometimes. So you just give me your info and I will spend the money for you and you'll get some cool stuff sent to you every month. And I get the joy of picking it out. So it's like, it just works out.
Chris: It's like making a mixed tape for someone. Except you get to spend their money.
Steve: Yup. I think, I think, I think there's some people that could benefit from that, but that's just my thoughts.
Amos: Don't they have that, that like with a monthly subscription where they just send you clothes every month, like here's a box of clothes.
Steve: I think they have that. I guess that's, that's sort of the same thing. I wonder if often they're sending you like cheap stuff that's like, so like, well, we couldn't sell this. So let's send it to these, to these people.
Amos: I don't know. But if they're sending expensive stuff, we need a service for cheap stuff. Thrift store in a box once a month. You get a thrift store in a box.
Steve: Yeah. There's people that love going, like thrift store shopping and stuff. I bet you, if they could then just they're doing it for other people, you know? And it's like, I mean, there's people that do that. Like a, you know, like, uh, some like consignment shop owners actually do stuff like that. You know, set, set, set aside inventory and whatnot, but that'd be a good service.
Amos: Should we cut this out of the show and go write that? Thrift shop plus-
Chris: Oh yeah. So we don't wanna let anybody know, right.
Amos: Cause Steve does have the best shoes.
Chris: I agree.
Amos: When you go to a conference, you're like, that's Steve, and you can't even see him, he's like standing behind people, you just see his shoes.
Steve: That's uh, that is, that is true. Yeah. That's a good way to identify me. Look for the crazy, the crazy shoes.
Amos: I'm like bright shoes, Steve.
Steve: So, so Chris said sometimes, I don't know that, I don't know the etiquette here. I don't want to look like a nerd. So you mentioned the, you mentioned the book tour. So I do have some conferences lined up. I think it'll be before the book's in print, but I'm sure though, I'm sure I'll apply to some after the books in print too. Like if you show up and like, let's say you had copies of your book, like, are you like, you're selling those? Like, how does that work? Is it like, all right, like there's some cash and you can get this book and then I'll sign it for you. I don't know how stuff like that works. Is that just like, ah, don't really do that. Just go and do your presentation and like pitch your book in it. Like, like what's the etiquette there?
Amos: I think you just stand up and be like-
Chris: I don't know why you're asking me.
Amos: We're not authors, but I think you can do this.
Chris: I don't work for Prag. How do I know?
Amos: You just go up on stage and you say, uh, I have, I have 10 copies of my book here. I'll sell them to you at $20 over the value on the book. And if you buy it, I'll sign your face. Seems legitimate.
Chris: You're going to have to get really good at, at face signing, you know.
Steve: I have a bad signature, so I will have to practice that.
Chris: Oh Yeah. 100%. I would, I would bring some glossies with me as well. You know, different headshots to pass out.
Amos: Use a different signature than what you use for checks. That's what I've been told.
Steve: Interesting, interesting advice, Amos. I wouldn't have considered that
Amos: Pro tip.
Steve: Just a big S. That's all it's going to be.
Chris: Oh yeah, exactly. With like a single line through it.
Steve: And a sneaker.
Amos: There you go. I'll never wash my forehead again.
Chris: I mean yeah you could draw some Jordans on there real quick.
Steve: I wish I was a good artist. Cause like, I think it's cool when people like do a little doodle with it too. I can't do that. I'm not, I wouldn't do that. I wouldn't even take that, that, that chance.
Amos: Yeah, you gotta risk that. Practice. Practice your doodle. Just a smiley face and a sneaker.
Steve: That's it. That's going to be the signature.
Chris: So yeah, but I mean, I, but yeah, it was, so if there's, I mean, I assume you'll get some discount codes. I seem to see those crop up every now and then, uh, and I'll say it's a book, it's a book worth picking up. It's good. I like it.
Steve: Thank you. And I know there's discount codes for the ebooks. I don't know what they're going to be doing for paper books because they are not selling, I think it's public now, but it's going to be all, all Amazon based. And if there's like bookstores that pick it up. So I doubt that there will be like discount codes for paper, which is, you know, that's just sort of the way the cookie crumbles, I guess. But, uh, you know, ebook, obviously the Elixir, like you can go to the Elixir forum and get the ebook discounts there, which I recommend to people, if they're listening here and they're, and they're looking for any Prag book and you know, they want a good discount.
Chris: Yeah. For sure.
Steve: It's basically the best discount you're going to get. It's like a really good discount.
Chris: And they come up a lot too.
Steve: Well there's just a persistent one.
Amos: There's one for the forum.
Chris: Oh. Huh. Rad.
Steve: I didn't know how big of a secret it is, but I'm like, it's a really good discount so I would use it every time.
Amos: Yeah. Just let the, let the secret out. So what conferences, you said you had some coming up, what are you going to?
Steve: I'll be doing the training at, uh, Lone Star Elixir. That'll be the real-time systems training, which will be very inspired from the book. And then I'll be doing, um, sort of like the anatomy of a real-time Elixir app at, uh, at Code BEAM SF, so that'll be in, so the Lone Stars, uh, what is it, uh, February 27th, the 29th. Um, and then the, uh, the Code BEAM's like the next week, I think it's like the 5th and 6th or something like that.
Amos: Well, I'm, I'm definitely going to be at Lone Star cause we're sponsoring and you can just sit at our table and sign books and I'll stand up and act like a bodyguard for ya.
Steve: Nothing will be in print by them though. It'll take a full, take like two to three months once it's-
Amos: Oh, I didn't realize that.
Steve: After the end of January. Cause there's um, like, so like the way it works is like, I'll finish, I have this beta finished up, right. It's actually, it should be going out like, um, like the 9th or 10th of January. So by the time this is live, the beta, the final beta will be out, but then there will be like additional edits that I'll make. And then there'll be like, after it goes into production, there's like rand, uh, like grammar, editors, spelling, editors, formatting, editors, indexing. There's like a lot of stuff that happens that I don't even have to worry about. I just sort of say hands up, I read it twice, I think, they'll give me like two chances to read it and then it just goes, it's done. Um, so it'll take, like, it takes like two to three months to do that whole process. Cause there's like people involved and they're, you know, doing, doing the reviews and stuff.
Amos: So I guess the only thing you'll have to sign is people's faces.
Steve: People's faces will be the only thing. I'll give, like an IOU. I'll sign it and then you can, you can cut it out and put it in the book.
Chris: Yeah, exactly.
Amos: Nice. Nice.
Chris: You just cut your face out and then put your face in the book.
Amos: Alright. I'm going to show up a bunch of. "Hello my name is "stickers that you can sign and give to people.
Steve: Perfect.
Chris: Oh, there you go. That's a real idea. That seems good. Oh, that'd be fun. I think. Yeah. We'll probably end up running into you at conferences and stuff like that at some point, I don't know what my schedule is for this year yet going to be, I don't know. I don't know. It's probably gonna be silly, but I'm sure-
Steve: Yeah I'm not even sure yet what I'll be doing.
Amos: I'll be at Lone Star, possibly Code BEAM. I have a ticket, but I don't know if I have the time, unfortunately.
Chris: Yeah.
Steve: Just do in and out same day. Just do a fly in, do, do, do a day of the conference and fly out.
Amos: Really, that's the only reason I go to California is for In and Out.
Chris: Yeah. I'm a, I'm a Five Guys, Five Guys, person myself. Yeah.
Amos: See, see, I went to junior high in LA, so.
Chris: You went to junior high?
Amos: Barely.
Chris: That's cool.
Amos: Um, I can't believe they let me, but I was in Los Angeles.
Chris: I didn't know that about you.
Amos: So I, ate a lot of In and Out.
Chris: I can't believe they let you out.
Amos: Me too.
Steve: I had a weird Five Guys experience. And I, uh,
Chris: You can't get past it.
Steve: They took the French fries and instead of putting salt on it, I'm like 99% sure they put sugar on it. It was just really weird. And I was like, these fries taste off. And ever since then, I haven't been able to get past that.
Chris: Oh man. Just broken. Broken forever. Uh, what a shame. What, uh, what a loss, what a loss, a loss to society and to you. Oh man. All right. Well, we should wrap this up.
Amos: Yeah. I got to get goin' today.
Chris: I got to do work at some point.
Amos: I think I'm going to try to, I said last time that I was going to try to do some open source stuff. So -
Chris: How'd that go for you ?
Amos: Today's the day. No, today's the day.
Chris: Today's the day.
Steve: Do you know what you're doing yet? Like what do you got going on?
Amos: Uh, I think, I think I'm going to hop in with, uh, Connor Rigby. If he'll allow me and, uh, we'll do some Nerves work together. Otherwise we also have an idea for, for, uh, a fun little project to set up at Lone Star, if we decide that we could get it done by then. Maybe I'll tell you guys offline.
Steve: I believe in you Amos. I believe in you.
Chris: You can do it. Don't let your dreams be dreams, Amos.
Amos: They're usually nightmares. All right. Have a great day.
Chris: All right. Later y'all
Amos: See you Steve. Thanks for coming.
Steve: See ya.