Episode 71: Double Speed

The Elixir Outlaws now have a Patreon. If you’re enjoying the show then please consider throwing a few bucks our way to help us pay for the costs for the show.

 

Episode Transcript

 

Amos: Welcome to Elixir Outlaws the hallway track of the Elixir community.

Chris: We can do a little bumper with the, with the, with the music.

Amos: Nice.

Chris: That'd be fun. We don't really do bumpers that often. Is that what those are called, bumpers? Like on a car?

Amos: Music at the beginning and end?

Chris: Just a little like a, the outtake that you play before the show starts.

Amos: Yeah. I don't know.

Chris: Teaser?

Anna: I have no idea.

Chris: It's an amuse-bouche for the ears.

Anna: There you go.

Chris: You show up to your favorite podcast listener and say (in comical French accent) Oh ho ho! Well, monsieur, we have a very special treat for you tonight. Yah, please listen to this joke that wasn't good enough to make it into the show. Enjoy the rest of your audio recording.

Anna: Keathley how much coffee have you had today?

Chris: Um, I don't know what you're talking about.

Amos: (also in an accent) He's on fire!

Chris: (in accent again) So if, uh, if you would like to adjust, your playback speed, that is it possible for you? We can, we do. We do recommend the listening at one X, but that we understand if you need to do the 1.5 X, if you listen to two X you out, I was throw you out of this establishment.

Amos: So when you listen to podcasts, do you speed them up Chris?

Chris: No. As I'm not a monster.

Amos: Uh, I do.

Chris: So you're a monster. That's what we've established on this, on the show.

Amos: And then I usually start out at, well, I start at one, so I get used to the voices and then I'll change it to one two five, so I'll get used to that. And then one five, and then just straight to two.

Chris: Um hmm. You're boiling the frog. But the frog is your ears. Make sense.

Amos: I don't do it with all podcasts.

Chris: I don't listen to- here's the thing about podcasts and me and podcasts. I don't listen to podcasts that I don't enjoy. Like I, you know, there's no like listening to, there's no information inside of a podcast that I'm so desperate to get to that I need to listen to it as fast as possible, because I'm just not that busy in my life. Like, there's nothing I need to rush off to. There's no, and there's no salient piece of information that any podcast has where I'm like, oh, I've got to get to the, to the nugget inside of this, inside of this, uh, this, this gooey caramel center inside of the chocolate bar.

Amos: I just know that I'm going to get interrupted 15 times while listening. So if I play it at two times the speed and it's only seven and a half times that I'm interrupted.

Chris: That's math, that's math. That's just science. Okay.

Anna: Definitely. Oh man. How's it going y'all?

Amos: I'm good.

Chris: I mean, you know, it's going, it's going as good as it can go. I feel like.

Amos: I'm enjoying life. A lot of home cooked meals.

Chris: What else can you do?

Amos: Right.

Chris: I feel very, uh, I feel thin. I feel, I feel like I'm spread thin in my mental, what I can keep in my head at the moment. I've got too many things. I failed to say no to too many things, especially because , here's the thing, here's how they get ya. It's easy to say no when other people ask you for things, it's much harder to say no, when your own brain is like, that sounds like fun. You should do this. And then I have to be like, brain, stop it. All right, listen, get behind me here. You can't, you can't do that right now.

Amos: I mean, we have a lot more, uh, Chris Keathley open source projects coming out.

Chris: Just so many, so many. I don't know why I feel so compelled to be like, oh, well I need a, I need yet another ETS cache for this project. So instead of building another ETS cache, I'm finally going to extract the one that I use at work. Just make that a library. And so then it's like, but it's like before I could work on this other thing, I needed that thing done first. And then at some point I was like, I should build a template for all the libraries that I'm working on right now. And of course that takes time and yeah, it's just, you know, continually writing more code to solve problems. At some point I'll be able to solve them all. I promise. It'll work that way.

Anna: Uh-huh. That's how it works.

Chris: That's how it's going to work. That's really, I think the takeaway you can really solve most engineering problems

Anna: By writing more code.

Chris: -by just writing more code . A hundred percent.

Amos: I've worked on those projects before.

Chris: Um-hmm. Um-hmm.

Anna: We all have. Just keep writing , right? And eventually it stops.

Chris: The galaxy brain is you, you look at a human problem and you need to solve a human political thing. And you're like, you know what I could do for this? More code.

Anna: More code. Yeah.

Chris: Okay. So I'll just write more code. That'll solve it. One of these days it'll be enough.

Amos: So you're working on open source.

Chris: Uh, yeah.

Anna: What are you working on?

Anna: What am I working on? I'm finishing up actually a project today.

Amos: Woo!

Anna: And then I roll onto something new on Monday.

Amos: Are you excited?

Anna: I don't know enough about the project yet. I don't know how much I'm allowed to say about this project.

Amos: Okay don't then.

Chris: That's okay. We don't have to talk about it.

Amos: Just be safe. always err on the side of caution.

Anna: I just don't know yet cause it's a new project.

Chris: Kickoff days on a new project are always, uh, they're always very interesting because they, they come with the hope of a brighter tomorrow and you're like, this is going to be the best project ever. This might be the one, this is the best one.

Anna: I'm actually rolling off of a pretty awesome project right now. Um, and this project has already been going, my new one. So I'm just rolling onto it.

Chris: I gotcha.

Anna: It will be interesting.

Chris: I could, when I could never guess which like the projects that were coming through the door, I could never guess the ones that were going to be interesting versus not interesting.

Anna: Right.

Chris: Because there is, you know, let's just, let's just be honest about it and say there are a subset of projects that aren't super interesting from a technical standpoint. Uh, some of them are interesting from a personality standpoint where it's like, oh, like I'm going to have to get really good at managing this person. Some of them are more interesting than others.

Anna: That is true too.

Chris: Uh, but I could never figure it out . In the same way that I don't think I could ever figure out, like I would see all these startups come in and literally all of them, I was like, that's never going to make money. But some of them did. Some of them do fine. And then the ones where I kind of thought like, this will be fine. There's a market for this. They were like, yeah, we're shutting down. We're shutting it down.

Anna: Right?!

Chris: So I just, I, at this point, I, I was talking to a friend about this. It's like, I've given up on trying to guess at what's going to be successful and what's not. Like I've just kind of given up on. I feel like there was a, there was a time in my life where I was, I had questioned a lot of the business decisions for the companies I was working at. Like, not like in an ethical way, but more just like, I'm like, why aren't we doing that when there's this a market of this stuff? And it doesn't make any sense to me. And at this point I've given up and I'm just like, you know what? I trust you. Like, you seem like you know what you're doing.

Amos: Is it because so much of that is more marketing than whether the product solves the best problem?

Chris: I just think, I think what it takes to run a successful business is fairly nebulous. And so much of it is kind of luck and-

Anna: Timing?

Chris: -timing, and just being in the right spot at the exact right time and seeing the opportunity when it presents itself to you, that it's just too hard to guess at it. And it's too hard to armchair it. You know what I mean? The people who built the people who successfully build this, okay. Now we're really getting into it. This is something I've been thinking about a lot. Um, it's very funny to me when people talk about like, I'm a, you know, I'm a serial entrepreneur and it's like, so you just like fail at stuff, like all the time? That's like, what I hear when people say that it was like, oh, so you just ,like, consistently fail to build businesses. Cool. Or like when they're like, oh, this, this entrepreneur like was really successful. So like, so, okay. So you got something right once. Okay. That's interesting. So you, so one time you were right, which I think is, is a fun way to phrase that because it puts it in, in real perspective, right? And I think so much of that is unknown and the people who can sort of successfully build consistently good businesses, the way they do it is, um, they're billionaires. And they just have so much money to play with that it doesn't matter.

Anna: That's true.

Chris: And they can just prop up things and like use their money to buy power and then change the laws. And I don't know, it's like, whatever, like the people who are like consistently good at it are already like, have such a leg up on the world, right.

Anna: That's true.

Amos: Right. And how many, you probably don't hear about every project that they do that fails.

Chris: Right. Yeah. Or, or the deck is so stacked towards them being able to complete the project. In any case, I just don't question business decisions that often anymore, because like, I mean, what do I know? I don't know. They probably know better than I do and it's, and I can never guess what stuff's going to work and what stuff's not. So I just kind of, don't push back on stuff that I think is silly anymore, where I used to as a, um, as a young engineer, I'd be like, that seems dumb. It doesn't make any sense why we would build the product that way. I think a lot of people go through that. Um, a lot of my friends have gone through it or they're like, I just don't understand what the business is doing this. It's like, yeah. But like, that's not your job to understand that.

Amos: I'm still a little bit that way.

Chris: Yeah. That's why you, that's why you run your own business.

Amos: I guess.

Chris: I've just given up kind of what, I don't know, it sounds defeatist, but at some point, like I just realized that the thing that I'm good at is not knowing how to make smart business decisions. And I trust that those people know what they're doing and if they don't, then the business will go under and I'll find a different job. But until then, I'm going to do not like, just like carte blanche, whatever they ask me to do. But like, I'm going to help them fulfill whatever it is they think they need to do to like make the business successful. Cause that's all, that's all, that's my job. I don't know. What were we talking about?

Anna: I dunno.

Amos: Oh! What we've been, what we've been working on. And then that led to business. New projects.

Anna: I don't know what I'm working on next, yet. But it might be blockchain related!

Chris: Exciting. I didn't realize it was still 2018. I mean, I knew that the economy had crashed back to 2010. Uh, but I didn't realize that, you know, we'd gone back in time in terms of technology too. That's awesome. (an alarm rings in background)

Anna: Hold on a sec.

Chris): Oh man.

Amos: Should we leave in her alarm going off?

Chris: Absolutely. This is the show. Everything that's in the show is in the show.

Anna: Turns out that JavaScript might be part of the project as well as blockchain.

Chris: JavaScript and blockchain.

Amos: Oh, uh, I'm sorry.

Chris: That is awesome. Fun things can be put on there. And it all runs in Internet Explorer 7.

Amos: Are you trying to build mining stuff that people can just plug into their websites, and

Anna: No. I actually don't know what the project is about really. I just know that there is a blockchain component and there's some JavaScript. I'm working with Weld.

Chris: That's exciting.

Anna: Yeah. One of my, one of our former, well Chris's former coworkers, that I really enjoy working with

Amos: You guys are doing all the fun stuff. I'm doing tool sharpening.

Anna: That sounds fun.

Chris: Literally?

Amos: Yeah. Like make my Emacs better.

Chris: Oh, I thought you meant you like had an ax that you were sharpening whetstone on.

Amos: Oh no. I'm not very good at that. I've tried that for many, many years. Like sharpening knives and stuff. I don't know. I don't know if I read stuff about it and I've asked people to show me and I just don't seem to have it unless I spend like hours and hours and hours to do it. And then I'm like um, I'll just buy a new knife. Or get an electric sharpener and move on.

Chris: You know there are people in the world who can sharpen that knife for you.

Amos: Yeah. Well right now I can't go to there.

Chris: Well, that's true. That's true. Uh, so you're making your Emacs better.

Amos: Those sharpener things to sharpen knives but it's more like keep them sharp its not to resharpen them. The honer, whenever it's called.

Chris: Yeah it hones, it hones the knife.

Amos: It hones the knife.

Chris: Hones.

Amos: Yeah, I don't know. I just, I've been wanting to play around with, uh, uh, Emacs for a while. I did Emacs 20 years ago.

Chris: Would you say that you're honing your Emacs or sharpening it?

Amos: I'm honing my Emacs skills, I think.

Chris: As opposed to sharpening?

Amos: I kind of threw away spacemacs.

Chris: Oh, tell me about that. Actually, no, don't, I'm bored.

Amos: Yeah, I don't blame you.

Anna: That's not very nice.

Amos: I saw someone managing their AWS clusters and everything inside of their Emacs.

Chris: In the spacemacs?

Amos: Well, not in spacemacs. They just did it, in regular Emacs, but.

Chris: That is pretty cool.

Amos: We do it in spacemacs. And actually the interface is a heck of a lot better than the AWS console I'm just saying.

Anna: Well cause that thing is garbage.

Chris: The thing is the worst thing anyone's ever seen. I mean universal, universal disdain for the AWS UI. All of them.

Anna: All of them.

Amos: The only thing worse than AWS is JIRA.

Chris: No. See, I actually think JIRA is a more intuitive API like user interface.

Anna: I agree, than AWS.

Chris: Like not comparable.

Amos: I tried to mass move a whole bunch of tickets from one sprint to another. When you go to do that and you bulk edit in JIRA, you have to check a box saying that you're changing the sprint. And then sprints all have names everywhere in JIRA its referred to as a name. There, it asks you for the sprint ID. Guess how you get the sprint ID? You go back to a sprint and you right click on it and you inspect the source. Or you dig through the URL and find it.

Anna: That's really good.

Amos: It is not listed anywhere on the site. And it has been that way for years. And it's in the JIRA, if you go to their, their like help site where people are asking questions, it's been a common question for years. And the common answer is look in the URL or look in the source to the webpage.

Chris: Classic.

Anna: I mean that sounds like JIRA.

Amos: I'm like "What?"!

Anna: And yet, people really love it.

Chris: Someone loves it. I don't know who it is that loves it. And I don't know what sort of love it is for them. Like, I don't know, like, you know, like it's gotta be a love that just has no sense of satisfaction involved. Like it's completely one-directional.

Anna: Oh yeah, that's true.

Amos: I think whenever your tool creates job titles, like JIRA admin is someone at your work, the tool is broken.

Anna: That's true.

Chris: Yeah. I mean, it's sort of, it's, it's JIRA is an interesting thing. And along with like Salesforce, right? Which is again a UI that's so sort of universally bad that you needed, that you created an entire cottage industry or of people who build things on top of it because they just don't want to use the normal thing. That's amazing.

Anna: Yeah, people specialize on working in Salesforce.

Chris: Talk about a business model. That's genius. Here's what we're going to do, gang. We're going to get so big that no one cannot use us. And, but we're also just not going to care. And then we'll just allow everybody else to build their own million dollar businesses around us. Like that's amazing. But so you've been doing, uh, so you've been, you've been back on the spacemacs. I know you threw away spacemacs. Now you're just going straight back to Emacs.

Amos: I have two computers right now. So the one that I'm still using, day-to-day at work has spacemacs, because I can't learn all those new configurations and everything that quickly. I'm working on that. That's been fun. There's a book called Mastering Emacs that I've been reading through as I'm doing it, just to try to learn it a little better. I used Emacs 20 years ago, and then switched VIM at some point, used VIM for like 15, 18 years. Something like that. Trying to think of when I switched and then, uh,

Chris: And then you're back.

Amos: Yeah. Now I'm back. Mainly because I don't like Vimmel. And I think Elisp's just a whole lot more fun to write and I get tired of configuring my VIM. And I'm not, I've not been happy with any, any, like, I'm going to call them commercial editors, but not all of them cost money, but just any of the other, like more-

Chris: The modern slew of editors.

Amos: Yeah. Like VS code and stuff like that. I'm just, I've tried them. And it just doesn't, it doesn't make me as happy. Mainly cause it's harder to customize, feel like, than a lot of the other stuff. And that's , so that's what I've been working on. That, and um, just other things like all the little things that we do in development, that there are solutions to automate it, that I'm trying to get rid of, like updating my encryption keys for SSL. I've been doing it by hand for a while. And I know that you don't have to. So that's what I'm working on this morning is getting SSL certificates to auto enroll. So I don't have to deal with it.

Anna: Nice.

Chris: That's exciting.

Amos: Cause who wants to do that?

Chris: I don't, I don't want to do anything like that.

Amos: No. I mean, I already scripted moving their certificates, but then I had to go restart engine X and all that. And it's like, you know what? This should just happen on the server. Done. That's what I'm doing is trying to eliminate the little things.

Chris: Toil.

Amos: Yeah.

Chris: Eliminate the toil.

Anna: Just write more code.

Chris: You can solve, you can solve code problems with more code. That's a, well, that's a well-known um, that's a well-known adage .

Amos: And you can solve people problems with permissions in your system.

Chris: Um-hmm.

Anna: Yeah, just write more code.

Chris: It's all, it's all about more code. See the secret is, it's just more code.

Anna: It’s all about more code. You just keep writing code and eventually

Chris: If nothing else, you're going to be so distracted with all the code that you're writing, that you don't have time so the people problems.

Anna: And you won't have any problems. You'll just be like, I'm too busy, ignoring all the problems.

Chris: Just write more code, make it someone else's problem.

Amos: Hire interns.

Anna: Yeah, just like pass the buck. This is a really exciting podcast.

Amos: I know, I know. What are the things day-to-day that, that you did that you try to automate or get out of the way ?

Chris: Mostly human interaction. So like, I've just got a bot set up that, uh, auto response for me, it's just a Markov chain of myself and it just responds on Slack for me. Instead of actually me talking .

Amos: It's got a lot of stuff, a of text to work with.

Chris: Yeah. Getting back to sort of our using code to solve all the problems,

Anna: How do you reflect all of the accents and stuff?

Chris: I'm not even here. I mean, right now. This is a bot talking to you currently. Like I'm really in secret lair still working on code.

Anna: Have you just reach singularity and it really doesn't matter whose responding?

Chris: Oh man! Would we even know? We wouldn't even be able to know.

Anna: How would we know?

Chris: It would be impossible to know.

Anna: Keathley's consciousness is sitting on a server somewhere.

Chris: If bots do become self-aware what I believe we should do is just give them lots and lots of money. Because if there's anything I know about people with lots and lots of money, they are not self-aware at all. Solved. Problem solved.

Anna: That was good. That was good Keathley.

Amos: How long you been workshopping that one?

Anna: Yeah, where'd you get that one?

Chris: Sunday morning breakfast, breakfast, cereal.

Amos: Woo!

Anna: Nice. How are your kids? How are your kids doin with all of this crazy indoor no school situation?

Amos: Have you ever seen kids really complain a whole lot about no school?

Anna: That's true. Are your kids like psyched, they're like "No school!"?

Chris: My daughter is surprisingly, like, she misses school, but I think that's a largely, she just like misses her friends and she misses like the stability or like the schedule of going to school.

Anna: Right, right.

Chris: That aspect of it, you know, I, that parts, that part, she misses. And, but actually right now we're on like quote unquote, spring break week. So like we're not even doing school at home. Um, and that's actually been kind of nice cause it's just like hanging out and reading books and they're outside. We got to slip and slide. It's the most Mondo like baller, slip and slide ever. It's got like a giant ramp on it. It's this big inflatable thing and you can take like little surf boards. You can ride down the ramp into the water. It's super dope. So they've been out there doing that.

Anna: That's awesome.

Amos: Is it big enough that you can get on it?

Chris: Oh yeah. I drove through it. It was awesome.

Amos: Yeah!

Chris: We, we used a little bit of our like very precious soap at the moment to like slick the thing up and it was, aww man, you can get some, you get some real speed going.

Amos: I'd like to get something out like that here, but we can't tell what the weather 's gonna be. It was 85 yesterday. We have a freeze warning tonight. It's supposed to be 29. We have two days of snow predicted this next week.

Chris: You're in that, um, you're in that prime like Missouri, Kansas, maybe like three weeks where you can wear shorts and a hoodie.

Amos: Perfect. It's true.

Chris: But all Midwesterners know that this is a, this is an honored time. This is a, this is a beautiful time of the year. That perfect time of the year where you get to wear both a shorts and a hoodie and probably like sandals, but with socks,.

Amos: Yes, this is the most important time of the year to wear layers because=.

Anna: I live in San Francisco. That's just my life.

Chris: Oh, shorts and a hoodie? Oh, layers.

Amos: Does the temperature swing heavily in San Francisco?

Anna: No, but like you're so used to a very finite temperature range, that like, even if it's nice outside, it's not going to be nice outside all day.

Chris: The needle only moves like 10 degrees.

Anna: Yeah, but you're like freezing by the end of the day.

Amos: We have 50 degree weather changes in one day.

Anna: When you live somewhere where the weather doesn't really change that much you just start to notice the, like when I was in San Diego, like ruined me. Cause like you're used to it being like 80 and then it's like 75 and then it's like 67 and you're freezing. I kid you not. I was like once there was like one summer where like dipped into the sixties and people started wearing like their winter jackets.

Chris: Started bringing out the Patagonia's

Anna: Got the puffy's on.

Chris: Yeah. The, the micro, the micro down.

Anna: Exactly. And like hats and stuff cause its cold.

Chris: Fingerless gloves.

Anna: I moved back here after living there for so long. And I was like, my first summer here, I was like, it's freezing. And my parents were like, "What is wrong?" Like at 68 degrees I'm like, yeah its cold. Like-

Amos: I got to get my leg warmers out.

Anna: I do miss being able to wear flip flops all the time though.

Chris: I mean you could still do that, I mean, well, listen, I'm just saying you can still do it.

Anna: But it’s cold! I mean seriously-

Amos: It'd be nice to live somewhere where I never have to own pants. I can just wear shorts all the time.

Anna: I think, the CEO of Carbon-5, it’s like his mission to just never wear real shoes.

Chris: Oh yeah. I, I, that was, that was my takeaway.

Anna: Um, he's always in flip flops, almost always.

Chris: Or barefoot.

Anna: Or barefoot. But he's in Santa Monica.

Chris: So I mean, that's just par for the course. Whadda ya gonna do?

Amos: So as everyone else's life gets stuck at home. This is what the Outlaws sound like when we're stuck at home.

Anna: I've kind been, not that everyone stays healthy, but I've kind of been like enjoying it.

Chris: Yeah.

Anna: I mean like life has gotten a lot simpler. Like all the things ,like I go to the beach, I run to the beach, I work out. I cook a lot more than I have before cause I'm like home and I was never home. I'm like reading books I want to read. I'm working on some side projects. You know, I'm just like, I dunno, life got a whole lot simpler really quickly.

Amos: Yeah, I'm not running kids from event to event to event.

Anna: Yeah, exactly. And I don't have kids at home, so my life got real simple.

Chris: It feels awkwardly, I mean, it, it feels awkwardly like guilty a little bit, right? Like where you're like, I know that this is not good, but my immediate day to day kinda, yeah, like.

Anna: I also feel super privileged about that.

Chris: Yeah, that's the guilt part of it, I think.

Anna: I'm getting groceries delivered so I don't have to go out anywhere. My house is in a pretty quiet neighborhood.

Chris: Yeah.

Anna: My living situation is pleasant.

Amos: Yeah.

Anna: I'm still working, but from home, right? There's a lot of privilege.

Chris: Yeah. It's weird to me how quickly this has become normalized in my head. Like, I wash my hands constantly. Like, we don't leave the house. Even Alice. It's like, it's, it's weird to see Alice normalize it where she's like, she comes in from outside and immediately washes her hands, which is like not a thing we ever enforced her. And she just did, but she, like, somebody had to come over. We had to sign a thing and some house stuff doesn't matter. But you know, like we, like basically he like put it on the porch and then stepped back. And then I went up and like sign the thing with a pen. And then I immediately like was cleaning my hands and stuff. And like even Alice was like, she came outside and then stayed away and then immediately came in and like washed her hands. It's like, it's weird to see her normalize it.

Amos: Yeah. We've had stuff delivered. And for the most part, that's like where people used to, you would open the door and they'd say hi and hand it to you. And now they like the doorbell, set it down. And then back six feet away or further. I don't know. It's a little strange. I wonder how much this is going to change society as we come out of. I don't know. It's going to be weird.

Amos: I bet I will wash my hands more.

Chris: I feel like, yeah, that's a thing that’s going to stick with me for the rest of my life.

Anna: Well I hope that sticks with people, cause like it'll make flu season a lot better.

Amos: And I'm going to spend a lot more money on hand lotion.

Anna: Yeah! Alright, y'all, I gotta go to a meeting in like two minutes just FYI.

Chris: No worries. I did start an interesting project. Like I'm working on, we've had some interesting movement on a thing. I have an actual Elixir thing to talk about.

Anna: Oh, ok. I'm glad we waited for the last two minutes that I have.

Chris: Yeah well, ya know.

Anna: I also didn't mention that I have to run.

Chris: We also got started late.

Anna: That's true.

Chris: No, like, so we've been working on this, um, on this new HTTP client, which I realized, yes, the world did not necessarily need another HTTP client. We have a couple of those already, but yeah. So I started it based on some stuff that Jose had done. And then, um, some other folks have gotten involved, uh, Nico has got involved and did done like a ton of work on it. And he's really done like the lion's share of the, of the real substantial work on it at this point. But yeah, it's really exciting. So the idea is, is really just it's, it's another HTTP client. Uh, it's another HTTP client, but it's really, really heavily focused on performance. That is like, the goal is just really high performance. And so far the benchmarks are looking really promising, like really, really promising.

Amos: Is this Finch, that you sent me?

Chris: Yeah. So the project is called Finch. Um, I'll post a link to it and as part of this, but um, so far, yeah, we're w we're getting really good results and it's pretty close. So it's been, it's built on top of Mint, um, and uses a pooling solution that Jose had been working on. Um, which is a pretty, it's a pretty high performance pool that plays really, really well with this type of thing. Like it plays really well for this use case. And, uh, yeah, it's still, like I said so far, we've been having really, really good results partially just because the, I mean, a big part of it's just because the pool is a higher performance than a lot of the other stuff that's already out there.

Amos: How does it, how does the pool work differently than like, Poolboy's pool?

Chris: Yeah. So there's a couple of things that we're doing in Finch, um, that allow us to sort of achieve better performance than a lot of stuff. Uh, the first is that we open up multiple pools per, per host. So we'll start several pools with many connections inside of them. Uh, and that tends to, that tends to spread out load because the, you know, the pool is actually really where like at high scales, the pool is where you're going to like see the biggest problem. Cause that's where all your queuing is coming from. I mean, that's the wrong, it's kind of the wrong way to say it. Obviously the queuing is coming from the fact that it's like, it's slow to make an HTTP to make an HTTP 1.1 requests, right? You can't multiplex them. Like it's just a bad protocol. If you want to actually do something high performance, right. Um, HTTP just sucks. HTTP 1.1 just sucks.

Amos: {inaudible}.

Chris: What'd you say ?

Amos: There's a lot of overhead.

Chris: Yeah. It's overhead. Well, and it's just like, you're not using any of the stuff efficiently. Like you're not using your single TCP connection efficiently, which is obviously why HTTP 2 is a thing and et cetera, et cetera. But for HTTP 1.1, uh, what we do is we start multiple pools for, for each host and then those each have their own connections. And so when you go to get a connection, you're really getting one of the, we can like round robin you across all the pools and then the pool can be faster because there's less contention on each connection. And then the sort of novel thing about the way the pool works, and this is where a lot of the benefit comes from, is instead of copying the request itself into a process and then waiting for the response in a different process, and then copying it back to the caller process and having all this like message passing and copying going on. Instead what we do is the pool hands the calling process, the Mint connection. So it only copies the connection in there. And so the entire request and response is happening in the caller process. So the pool is like a pool of connections. It's not a pool of workers, if that makes sense. And it just hands you the connection, you do stuff with it, and then you put it back into the pool, right? So it's a resource pool around connections-

Amos: It’s kind of like the check-in check-out part of what Poolboy does except Poolboy hands you a worker.

Chris: Right. And because we don't need, and because Mint is, is built to be processless, uh, we can do that. So we just hand you the like Ment struct, and we're like, I dunno, you deal with it. And then the caller can do it and then put it back in there. And so, because of that, like we've been benchmarking it on some C5, 4x larges on AWS. And right now, you know, we're getting we're as, out of the box immediately, we were, we were more, we were sort of, we sort of beat Mohito, which is the other kind of major Mint based, uh, HTP client. Um, so sort of immediately we were, we were higher we were getting higher throughput then they then Mohito does, and we beat Hackney by quite a bit just because we're, um, we're able to, like, you could probably achieve similar stuff with Hackney actually. Like I bet you could achieve similar performance to what we're seeing with Hackney, but you'd need to sort of build it yourself because it doesn't, like, you would need to construct all the pools and then sort of build, like tell Hackney which requests are using certain pools and stuff like that. Like you can do that now with Hackney. Like that's how we, that's how we use Hackney at work currently, but you have to kind of build all that from, from whole cloth. Um, and Finch is just doing it for you. So sort of, if you compare out of the box, then we're able to get higher throughput, but, but if you actually make it a fair comparison, because it's not a fair comparison to Hackney. It's a fair comparison to Mohito because they're actually doing the same thing we're doing. If you compare, like it's not a fair comparison to Hackney because Hackney uses a single pool per host by default. So if you actually do the comparison correctly with Hackney where you just like limit yourself to one where you limit to Finch to one pool per host, with the equal amount of connections, we're slightly more efficient than Hackney. But I think that that's probably, I bet it's not as statistically, um, significant, like we're basically right online with how with, with Hackney. It's just that you'd have to set up all the extra stuff with Hackney if you want to get really higher, higher throughput.

Amos: Yeah.

Chris: Uh, there we're, we're like the benchmarks were that we're slightly more efficient than Hackney, but I bet that's, I kind of bet that'll come out in the wash once we really, once we add in the overhead of, of some metrics that we're not, that we don't have right now, once we add in telemetry, which we don't have right now, like, you know, once it's producing that sort of stuff and it's needing to check the system time and all this kind of stuff, like I'm betting that you add a little bit more overhead and we get, and we're probably right in line about in line with where Hackney's at. That's my hunch. So, but yeah, but so far it's looking really good and, um, it's really exciting to see something that takes very little configuration that you can kind of just say, like, I don't know, default it to these numbers and, uh, and get pretty good performance out of it. So that's exciting so far, it's still early, but I'm really, really excited about that as a, uh, as a project. I think it's, it's going to be really useful. I know that we're like already looking to swap over to it.

Amos: At work?

Chris: Yeah. We're, we're ready to start testing it out just because I think it's, I think it could have some pretty substantial wins for us.

Amos: So is it, does it only do 1.1?

Chris: No. Uh, well, yes. Uh, the answer as of right now, today as we record this is, yeah, it supports 1.1, technically speaking, it'll open an HTTP 2 connection, if that HTTP 2 connections available. But the pooling strategy it's going to use is based around 1.1. So it's not gonna do multiplexing in that kind of stuff. I'm working on the branch that adds the HTTP 2 support. The trick is that the pooling and queuing and connection management is so different between 1.1 and two, like the strategies are so different because you can do multiplexing that we're going to have to require that you pick what kind of pool you want when you configure it, because they're just too different to pick like the, there there's two different to, for us to, to guess, and then try to like fall back between them or something like that. Like the pools are going to be completely different implementations. So we're going to have to ask you to like pick one, but if you pick the, if you, if you want to do HTTP 2 like we're going to support that out of the box and have a really high performance, uh, implementation of that.

Amos: Nice. I started to dig into it the other day and didn't make it very far. Maybe that's what I'll do as part of my afternoon, after I get this Let's Encrypt stuff fixed up.

Chris: It's, it's really exciting to see the benchmarks. Cause I just, it's something that, you know, obviously it's, it's one of the most key sort of critical pieces of our, of our system because it underlies like how we talk between services. Like we use HTTP to talk between these services and we'd like to be able to use HTTP 2, um, more than we're using it right now, but all that stuff right now in the current sort of, uh, ecosystem, it's just really hard to set up. And none of it gives you access to do it very well. A lot of it was built around to supporting 1.1 really efficiently. And so it turns out that like it's hard to do. Um, I'm hoping that with this new client, we'll be able to just support that and that'll just become like a solved problem.

Amos: So are a lot of your services like JSON type requests?

Chris: Yeah. It's a lot of it. A lot of the existing stuff is just, and this is something we're, we're really moving away from, but a lot of this stuff has just been really bare bones, like Phoenix controllers, REST-ish JSON and all that stuff sucks. Like not Phoenix, I like Phoenix, but I just mean that like, if what you want to build is a high performance set of services. You don't want to use REST and you don't want to use JSON and you don't want to use HTTP 1.1. Like, it's just, that is just the truth of it.

Amos: I was going to ask if you like, is that, do you think that like a web socket connecting type thing is even feasible for the level of stuff you're doing?

Chris: Yeah. I mean, you could do web sockets. I wouldn't. I would just do, well, okay, so for us specifically, like there's a bunch of options that you could go with here, right? Like the, honestly the easiest and the one that if you want to get actual high performance, like you do what everybody else does, which is you open up a TCP socket and you build a frame protocol on top of it. And then you just multiplex RPCs and you use something like flat buffers or prodo buff or something real small to do a binary, like do some silly binary encoding thing. And then-

Amos: That's like taking all the seats out of your car so that you can go faster.

Chris: Yeah, but sometimes you want to do that. I mean, in fact, a lot of times you want it, like if you actually want to really actually engineer a real solution, that's what you should do. You should just like open up a TCP socket and go for it.

Amos: That's the closest you can get to being on the same system.

Chris: You know, I mean, one point like, you know, using HTTP 1.1 is like, not, not it, that's not the, that is not the way you want to, you want to, you want to solve it like, and optimizing it in some ways is kind of like a fool's errand because there's just only so much you're going to get out of optimizing that stuff. I still believe in the project. Like I still believe in building Finch because I think it should exist. I think like it should be there and people should be able to use it. And it should be high-performance out of the box because I think we do want that to exist in the ecosystem, right. But if you're talking about building a real solution, that is, you know, gonna give you more scale, that's just, you know, you're going to work hard to get a 3X improvement, but you could probably get an order of magnitude improvement, if you just like, don't do that. If you actually like solve the problem holistically, right. What we're looking at doing is, so we're looking at changing up a lot of this. Like we're actually moving towards an RPC thing these days, uh, like, or, I get that HTTP is RPC, but we're, we're moving towards a structured efficient way to send messages across our services. Right now, the lead, the, the winner for us today is a technology called Twerp, which is, um, RPC it's kind of came out of Twitch. Um, it was one of their jams, but it's, it's RPC using protobuff as the, as the service definition language and as the encoder decoder for your messages. But it's built on top of HTTP, 1.1 semantics. So by default, you can just use an HTTP 1.1 request response model. Uh, you're just sending protobuff over the wire and that's, that's all it is. And there's a bunch of benefits to that, and the reasons why we're looking at it. The benefit, uh, the first benefit is that it's drop-in, you don't need any new load balancing. You don't need NLBs. You don't need like the new network load balancers that you can implement the full HTTP 2 spec for GRPC or anything, any of that stuff. Because the obvious thing that people talk about is like, well I just use GRPC. And I'm like, well, because like, I don't need 50 to 80% of what GRPC does. You don't either. And, and it's a huge, huge operational investment, massive operational investment. And it's hella complicated. Like you cannot look at that code and figure out what it's doing in an afternoon. I've tried. Like, like I promise you, it is way more effort than you want for very little gain over, like for what you're doing. I promise like, like I just, I guarantee it, like, you don't need half of what it's doing, but people who say that they do and like are willing to take on the massive amount of operations- Cause like nothing works with it, right? Like, so you got to support it holistically out of, for, in your entire organization. Like your organization needs to be ready to adopt it, right? Like operation down, like you can't just like put an HAProxy in front of your service anymore. Like you gotta, you gotta think this stuff through now, right? Cause it just has all these extra requirements. So, and it, and it like, it's so finicky, like it has, you know, all the requirements that it, that it needs. Like it needs all of them. If you don't have any of them, none of it works. Or none of it works or some of it might work, but it's falling back such that like, you're not getting any of the benefits out of it anymore.

Amos: I feel like it's a lot like pulling in Kafkas, that you now have another full-time job.

Chris: Yeah. Cause now you've got to run a zookeeper. You didn't know you needed a zookeeper just to run Kafka, but you do. And now you got two problems. Literally two problems. Actually you have six problems cause you needed to run three zookeeper instances. So, uh, those multiply out. No, the benefit of Twerp to me is that you don't need any of that. You can load balance with engine X or with HAproxy still, like you can, you know, you can drop it into Phoenix, right? So like I built the implementation for Twerp. You could read that code today, like easily because it's just a plug. It's just a plug that you add to your, you know, to your, uh, to your existing plug router or Phoenix app. Done. And like, and then it always works. So you don't need to run a new container. You don't need to run a sidecar thing. You don't need to run a separate service on top of the stuff. I mean, it's just like, it's, it's so easy to add in. The other benefit is that if you do want to use HTTP 2, like, you don't need the full spec. Like you can, you don't need all of the other stuff that like GRPC needs to maintain its long load HTTP 2 connections and do streaming and all this stuff. Like you're still just making requests and responses. So if you multiplex over the same TCP connection, like now you're just faster, but you didn't fundamentally change the ways in which your services have to work, but you're sending efficient payloads now, you're now multiplexing connections. You know, you can it's, it's just, it's, it's, it's a lot of benefit. And the other major things is like, because you have an SDL, now you can do things like auto-generate clients, auto-generate servers. You can verify the payloads to some level of verification, right? Like you can say like this, this is supposed to be an integer and this is supposed to be a string and all that kind of stuff, which is, you know, useful ish. But the payloads are much smaller. So there's smaller going over the wire and less network contention, you know, sending, turns out, sending less packets over the network means that they'll arrive there faster because it's just, there's less of them to send. So there's just a physics aspect of it. All the packets will arrive faster if you only need to send one. Like, so yeah. So for us, those are all the benefits and, and you know, the, to some level like the thing that we're really trying, we're trying to solve, we're trying to solve a scaling problem to some deg, well, it's not a scaling problem. We're trying to solve a performance problem because scaling is not about, uh, latency. Scaling's about how easy it is to run more of a thing, right. So we're trying to solve a performance problem, but we're also trying to solve like a human thing, which is like, how do you describe the data moving in our system right now? That's really hard right now. And we have enough services that it's become pretty untenable to just know anymore, uh, and to document it, uh, well, and to verify those documents and to, you know, every service now has to build its own clients, right. It has to, and it has to tune them. So like, you got to configure the Hackney pools, you got to do all this stuff, right? So that's the motivation behind doing all these things. That's the motivation behind Finch is like I don't want people to have to, you know, configure all this stuff. I want you to be able to like trivially configure a high performance thing or not configured at all and just it be high performance. So the client we generate can just have all the deadlines in, already embedded in it. It can have all like these, you know, it's not going to have a five-second request like timeout, like, cause that's not-

Amos: What is it? Just curious.

Chris: Oh, 250 milliseconds.

Amos: Oh, nice.

Chris: So that's the default. So that's my, that's my level of, that's my 99th-That's what I, that's like, you have to be under that 99th. Percentile latencies need to be under 250 for anything, um, critical. That's a very generous 99th, as far as I'm concerned.

Amos: Yeah, it's pretty high.

Chris: So, you know, we obviously shoot to be well under that, but that's, that's, that's where I start. That's where I'm basically like, nope, you just get dropped now. Cause it's just like, that's just taken too long. But you know, I mean, like I said, configuring all that, building clients using those clients is just, it's, it's really, it's really hard for people to do it all well. There's too many steps to go through and it's just not tenable. And we need to be able to make more substantive improvements to our RPCs like across the company at this point, doing things like building in load shedding, uh, building in, um, just all kinds of stuff, all kinds of, all kinds of interesting things, deadline, propagation, tracing all this kind of stuff. So if we just have all that available to people, the goal being that, uh, you get less stuff wrong if it's just all kind of built in that way. Which is again, another example of, uh, solving a human problem, aka communication, with just a little bit more code.

Amos: It's all you need. You just brought it back around.

Chris: So that's what we're looking at right now. I think I'm really excited about it. I think Twerp is actually a very, uh, pragmatic, good solution. I don't have any love really for protobuff, but it is an, it is a reasonable solution for this. And I think like choosing something pragmatic is important. And if you don't need to adopt a thing that has a massive amount of complexity, aka GRPC, when you don't need it, you shouldn't. And I, I contend that like, I, I like, I look at some people who do it and I'm like, why do you need this? Like, I promise you, we don't need this. And I can almost guarantee that like we have, like, our performance problems are like, it just, it kind of is interesting to me. I don't know. I don't, I'm not trying to gatekeep. I just, I just don't understand it. Like I get that, you know, it's fancy, but do you actually need, like any of the things it's doing?

Amos: If somebody needs all of GRPC, let us know. We would, I would love to hear about it.

Chris: Yeah. I'm going to go ahead and say, most people would be like, well, I really liked the streaming. And I'm like, yeah, but like, did you need that at all? Like, I don't know. Whatever. Like I'm sure people like it. And then they're also like, it's like three people at a startup who are also happened to run like a Kubernetes cluster, because that was also fun to do. And I get it. Like, dude, we'll fun stuff is fun. I just don't um, you know, I don't want to adopt and make my entire team adopt a bunch of complexity if they don't need it.

Amos: Right. It's more, more training, more overhead, more thought.

Chris: Yeah, it's just like a jerk move because you don't do that to your team if you're like, if it's just going to make your team's lives harder. So yeah. So anyway, that's what I've been working on.

Amos: I really like COBOL. So that's what we're using. Actually, I was just reading an article just as a side note-

Chris: Is it about New Jersey?

Amos: We're coming up to the end. Uh, well, I don't know, but there's a lot of states that are having problems processing people's unemployment because their COBOL system's written in the eighties and they can't find developers. So I imagine there's going to be a lot of, uh, a lot of COBOL money out there if anybody's really into that at the moment. Yeah. I don't know how you deploy remotely to most COBOL systems though. Very well. But I don't know.

Chris: You need to get your token ring network hooked up to the mainframe.

Amos: Right?

Chris: I don't know how you do that.

Amos: Star network.

Chris: Something. I don't know. Who thought the name token ring was good. Come on y'all. They knew what they were doing. They knew what they were doing. The second they called it a token ring.

Amos: One to rule them all. I'm terrible. See, it's funny because J. R.R. Tolkien in an author who wrote

Chris: I got it. I got the joke.

Amos: Just wanted to make sure. On that note, I better get out of here. Get some lunch. Get some stuff done. Got off to a little slow start today. I appreciated the run down on Finch and how the pooling is different.

Chris: It's fun. It's a fun project. And then also again, it goes, it's a, you could probably read that code. I don't know, in an afternoon, there's not much in there.

Amos: Sweet.

Chris: So yeah.

Amos: Maybe I'll go see if there's any way that I can help out.

Chris: It's exciting. Um, I am very excited to see what's going to come of it. It's a fun project. I think it has a lot of potential. There's a lot of potential there. So.

Amos: Cool. Thanks for hanging out today.

Chris: Yup. Talk to you later.