Chris Keathley
The Elixir Outlaws now have a Patreon. If you’re enjoying the show then please consider throwing a few bucks our way to help us pay for the costs for the show.
Episode Transcript
Amos: Welcome to Elixir Outlaws the hallway track of the Elixir community.
Chris: Born born of a bottle from heaven's hand. And now, you know, and here I am. I'm so glad I could introduce this song to you.
Amos: That was awesome. I still think that starting out the podcast with just us listening to a song silently where nobody else could hear it. Yeah. Because, because, you know, fair use policy probably doesn't include five minutes.
Chris: And, you know, five-minute anthem, really.
Amos: That's true. That's true.
Chris: And anthem for a generation. Yeah. I, I, uh, for, in case people are wondering, I, um, I recently, as in, in the last five minutes introduced Amos to Japandroids the house that had been built and, uh, it's a really good song and this, this band is so deep in my fricking wheelhouse in terms of my musical tastes. It's uh it's yeah. I love being able to introduce this band to people.
Amos: Yeah. It's like a nice, uh, I dunno, I said punk, but like the longer I listened to it, the more I it's just, it's just like a rock anthem
Chris: They're in that like sort of new wave punk scene, right? Like, like they're in the same scene that I w I mean, this is overly, uh, reductionist. Right. But they're in that same, like sort of new wave punk of that pup is in and it's yeah, there, there are a lot of fun. I mean, it's just like open throttle, sloppy punk music, which is like, I'm all about, I love it.
Amos: Love it. Who's your favorite old school punk band?
Chris: I don't know. I mean, when you say old school, do you mean like, you mean like the Ramones, like, or do you mean like, like what, what, like misfits, like what do you, what, what era are you? Are you going back to?
Amos: I don't know. So I guess for me, like all of them before 19, or before 99, are old school, whether they're Sex Pistols or Misfits, I really, I really liked the Misfits. Nirvana? Is punk?
Chris: Yeah. I thought it in there. Oh yeah. Nirvana has got like a real like that. There's a punk through line there, I guess, for sure. For sure. You know, there are a derivative of it, right? Like they're not, not in the sense that like they're derivative, but they're there in that vein of, of music.
Amos: Well, I just like to welcome everybody to Music Outlaws. Yeah.
Chris: Where are we talking about? Really? I don't know, fricking anything about music. Um, much like everyone my, my musical tastes stopped at, you know, 17 or whatever. It's all just, you know, uh, highly embarrassing skate park, punk music that like defines my musical, my musical tastes. So yeah, there's some bands in there that, uh, you know, we won't, we won't necessarily talk a whole lot about.
Amos: Semi-embarrassing.
Chris: Oh yeah. A hundred percent. A hundred percent. That it's either that, or like, you know, like new wave bands, you know, it's like, like new, new wave, poppy kind of stuff. It's good. It's, it's, it's good though. I mean, it gets me going. Oh, it gets me going.
Amos: The pop gets my head bobbing in the car. But, uh, after a little bit, I just am annoyed with it.
Chris: Really?
Amos: Yeah.
Chris: Oh, man. Yeah. There's a certain stuff, I guess, I suppose, but like, oh man. But that song is like so good. That's all it is super good.
Amos: That's good. Then I'm going to, I'm going to go on my way home and listen to like MX PX or something like that. That's that really bad punk.
Chris: Yeah, sure. Yeah. Yeah. Real Big Fish.
Amos: That's like a ska-punk, new, new-age ska.
Chris: Right? Yeah, yeah.
Amos: Yeah. I like Real Big Fish.
Chris: Okay. Yeah. A lot of Horns, really big into Horns.
Amos: I love some Horns. Yeah. And songs like, I want your girlfriend to be my girlfriend. Or I hate your guts.
Chris: Yeah, yeah. Yep, yep. Yep. How so? How listen. Okay. We have to change the topics. How's it going? Uh, I haven't seen you in a long time.
Amos: You know, like coming out of, uh, coming out of the election hole, um, from working on election software, it's, uh, it's been slow going, trying to catch up on everything that I missed and continue to give people the data that they're asking for. But that's good. That project is coming to an end. If anybody knows a new one, looking for some extra project help or greenfield help, I'm just going to go ahead and throw this out here. I know that we don't advertise, but uh, I'm advertising,
Chris: Because you need to advertise. Sometimes you just gotta do it. Yeah.
Amos: So, uh, so you can email me - amos@binarynoggin.com - and uh, I'd love, we'd love to hear from you. We've got Connor Rigby, Nerves guy, myself, and we have someone new. It's a surprise coming into January.
Chris: I didn't even know that.
Amos: Yeah.
Chris: Wow. Congrats.
Amos: Getting super excited.
Chris: That's awesome. Now you're like, now you're like a real company. Like you gotta, like, you got to, you have to get insurance and stuff.
Amos: We are getting insurance, too. Like I'm becoming a businessperson and still trying to be a developer. It's uh, yeah, it's fine. I don't, I'm never going to get away from development. It's too much. I can't let the company get big enough that I can't write software.
Chris: Yeah. Yeah. That's the, that's the trick, right? How do you continue to do what you want to do while also building and doing the things you have to do?
Amos: Right. Right. And I, I think what, what helps? One of the things that I've since day one, starting the company was, um, you know, our noggin days that we do on Thursday, even when it was just myself, I did Noggin Days on Thursday. So that gives me a chance that no matter what type of project I'm on, or, or if I'm knee deep in taxes, I still get a data to do what I want to do and to learn about what I want to do.
Chris: Right. And having that, that sort of time to really dedicate to it and where you're not just trying to grab an hour here or a lunch there, or a couple, you know, 15 minutes to read a blog post or whatever. Right. And like really sort of dig into it.
Amos: How do you, how do you do that?
Chris: How do I do that?
Amos: Yeah. Yeah. Cause you're, you, you pump stuff out like new projects all the time. It's got blog posts that you come up with, your recording podcasts, like where, where do you find the time?
Chris: The podcast one is easy because it's just stream of consciousness. You know, it's just showing, it's just showing up and saying words, um, pretty, I've got now 31 years of practice just saying words so...
Amos: Except for other podcasts. The other one that we do, the Elixir Wizards one, right?
Chris: Yeah. We're doing much more scripted. We're very scripted. We're very on time. We keep everybody moving. We have an agenda. I don't know. I was thinking of this thing the other day because it kind of came up about just in terms of projects I've worked on and stuff like that. And it goes in fits and spurts. Like it goes in waves of high productivity and high, not - the opposite of whatever the opposite of that is. It's it's. I just, I don't know. I don't know whether it is, I mean, a lot of it is really sort of, I don't know, it's the, the, the thing is no one is able to really quantify and it sounds sort of bro-ey when you talk about it, but like, there is sort of a getting into the zone kind of thing. Like if I'm really interested in a project or, or, or a problem to solve, I'll just go solve it. Like, as I can't put it down, I can't, I can't turn my brain off from thinking about it. And even if I'm sitting there, uh, not thinking about actively working on it, I'm still thinking about it and I'm still mulling it over in my head, like stuff like that takes hold in me and it won't let go.
Amos: Do you ever feel like, “Oh, my poor wife”?
Chris: Oh, yeah. Oh, constantly. I'm I'm guilt-ridden all the time that I worry and worry about not being present. This is actually a thing. This is like not to get too real, but that's a, that's a thing I struggle with constantly is feeling like a bad parent and a bad, um, partner, because I'm like, it's, it's really hard for me to put stuff away and like become present in the moment.
Amos: The moment. I mean, I'm, I'm right there with you full conversations with my wife. And then the next day she says, Hey, we've got, so-and-so coming over. Don't forget. And I'm like, wait, when did that happen? She was like, right. You picked the time yesterday. Right.
Chris: So I think about that a lot and not in the, not being able to turn it off, um, kind of stuff. And I've worked really hard to do that, uh, to, to get better at that. It's easier to do that with pure work-related things to just like, yeah. If it's a work thing, man, I I'm pretty good at being able to just walk away from actual work and just not do actual work. It's the, all the other adjacent things that are fun, you know, it's all the little problems that are just fun. And to some degree, I really like writing Elixir. I just like writing code. I enjoy that process. I enjoy creating a thing and yeah. And, and otherwise it's just in terms of - I don't know how to talk about this nicely. So without it coming out, sounding super hella arrogant, but I, well, no, I just, I think I'm pretty fast at stuff like I tend to, I'm not sure what it is. I I've worked really hard to try to cultivate a type of thinking that I, that lends itself to being able to visualize the problem. This is something Tesla talked about a lot.
Amos: You didn't sound cocky to you compared yourself to Tesla.
Chris: No, no, no. And it's that, that, that, that's a, that's a bad comparison. It's just my, my brain is kind of going off on some of this. Um, I think that's the originally where I got this sort of idea and this is, and then I've since read about it in other places, but, uh, Tesla, not the, the, the, the genius mad scientist, not the car company that only produces like eight out of 10 cars that work or something. Uh, so not the one, not the one owned by the guy who happen, to make it easy for people to pay money on the internet one time and was successful once at one at all of his ventures. Anyway, no Tesla would do this thing where he could just imagine a problem in his head. And so it led him to, and he, and he would be able to turn it over in his head and think about it and think about all the different pieces that we'd have to go into it. And then it would lead to this problem where he would then like, not write stuff down, cause he like he'd solve the problem already. And he'd solved all of them because he could just like visualize it. And I don't think I can do that. But I remember reading about that as a younger person and thinking like, that sounds like a superpower. I want to be able to do that.
And so I worked, I've worked like to try to cultivate the ability to like think through the problem. It also happens that I just do a lot of problems. Like, you know what I mean? Like I, I, I try to keep myself really sharp on algorithms, like kind of core algorithm and data structure stuff. Um, I try to read papers and then like the more you do that, the easier that all gets. And it also happens that I have pretty good understanding at this point of Elixir. And I can, I don't have to think about any of the mechanics of doing anything in Elixir, even the quote unquote like kind of complicated bits and the stuff that you don't use that often. And so, and then it's a rich get richer thing. Like I have a set of libraries that I've built for myself effectively, that other people happen to find useful, but I really just built them for me. And then they build on each other. Now it's easier to do the next thing. Cause I already had this thing solved.
Amos: Right. It's probably easier to build your knowledge too. Like the steps feel smaller. I bet. Yeah.
Chris: Yeah. And so then you're not adding a whole bunch of other stuff to your, you don't have to add 20 things to your, to do list, to do one thing. And that's typically how projects go it's I need to do 20 thing. I need to do this one thing, but that one thing relies on these other things. And those things rely on things. And in all that's knowledge and the more knowledge you just shove in your brain, the more base-level knowledge, the more you can just look at it and be like, that's a graph. I know how graphs work. I don't need to grab a library. I don't have to spend any time learning a library. I don't have to spend any time. I just know how graphs work and now I can just go and work on the graph problem.
Amos: So do you find it easy or difficult to communicate the ideas in your head to teammates that you're working with?
Chris: It's super difficult. I find it crazy difficult. This is actually a thing I've been struggling with a lot is that I am, I'm getting worse at my mentoring and it's something like I've been really challenged by personally as a, as something I've been thinking about as in the, well, all the ways that I've just failed as a mentor, as a person, as, as trying to build up the next generation of people and not to say that I'm like some old dude now, but you know, it's, It's more like there's juniors people like one year out of school and they're like, I don't know how to do this stuff. Like, what is, uh, how does these data, but how these database indexes work? How does it explain work? How does any of that stuff work? And I think I've done like a pretty poor job of being available as a mentor. And part of it is because I forget a lot that there are people who don't have a decade of programming experience where they've been bit by all this stuff. They need to learn these things. Right.
Amos: I'm going to project a little. Do you get frustrated with yourself when you're trying to communicate something?
Chris: I'm, I'm at baseline frustration with myself almost all the time. Um, in terms of my interpersonal abilities, that's just, that's, that's just where we're at. Like my baseline is like, I'm like, God, I've gotta be better at this because my communication skills are just, I think it's partially that it's partially, it's the remote thing. I'm actually gonna like, throw a remote out there as a, as a real deterrent for mentoring.
Amos: I definitely think it's a lot to train and mentor when you can't just grab a piece of paper and draw something on the desk, like right there, or turn around to the whiteboard. It's it's th there are tools out there to do these things remote, but they're not as readily available. I don't think to reach for some online whiteboard versus a piece of paper. Right. And it's also a lot of times harder to write and draw on those online things.
Chris: There's just enough barrier to entry to that. If you're at all nervous about asking a question, you won't right, and you won't just pick it up by osmosis either. Cause a lot of those things aren't, aren't being said. The other thing that I get really frustrated about with myself is just my own impatience. You know, when I'm going slower than I feel like I could go or really, that I know that I could go, I get frustrated by that. And that's not healthy either, really, especially. And you have to like, really push that aside because you have to let, if you're going to mentor people, you have to just be able to let them, you have to be able to turn them loose and trust them. Right. And just trust that, you know, you're not there to tell them what to do. You're there to answer questions they have and you're there to let them fail on their own.
Amos: And that is a big internal struggle that I have to like, whenever I just want to say, give me the keyboard, it's going to take me 30 seconds. Otherwise, I'm going to explain to you for an hour and a half, or when I feel like I I've explained it 10 times in different ways and you just don't get it. Like that is something that's really hard for me. I struggle with it. I think I'm pretty good about it. About not letting that show most of the time. I think the explaining it multiple times. Um, I do when I'm teaching get excited and talk really fast and I get louder and I've been told, yeah, you need to chill that out. Cause you sound like you're getting mad and I'm like, I'm not mad. I'm excited. Um,
Chris: I'm passionate.
Amos: Yeah. But not passionately angry. I'm passionately excited. Like I love teaching. And so I, I keep trying to remember that and trying to remember that just because I have some metaphor in my head that helps me someone who's brand new, that metaphor might not mean anything to them.
Chris: Right. Right. The metaphor part is, is hard. And that's just, it's a thing you have to, I don't think it's teachable to some degree in the sense that I don't think you can tell anyone, this is the metaphor you need to hold in your head. You can suggest that. But it's only really enforced when that person goes through that process and learns it. Experientially. That's been my that's, that's my take on it. Having watched people go through this, I think you just have to write a lot of code and, and you begin to understand the metaphors more as you get deeper into it.
Amos: Well, and I think figuring out your own metaphors too helps you get to that, that thought - the Tesla thought process, I'm going to call it that now. Cause I, I, I feel like I think that way too, or, or try to think that way most of the time and like you, you and I had different approaches to that. It sounds like, like you very intentionally tried to get there. I drove a hundred miles each way to work for seven years.
Chris: Right.
Amos: And, and I would program in my head or think through these problems in my head to and from work at well after the first year, the first year I listened to music. And then at the end of the first year, you're like, okay. I listened to everything in the music catalog from 1955 to 2006 and you're like, I'm, I'm done. Then it just became thinking, which was also scary when you show up at home. And you're like, I just left the office, like what happened to everything in between. But that, that forcing myself to sit kind of where I couldn't do anything else really, really put that, those thought processes into my head. Yeah.
Chris: I, the, the more you do it, the more you develop an intuition for it and you develop an and the more you use any given tool, the more you're going to develop an intuition for that tool and the things that are capable of being done in that tool. And that's going to naturally influence the way you think about the problem. And you're going to be able to draw from those experiences and begin to cultivate solutions from whole cloth a lot faster because you just understand the things that you did previously. Part of it's also, I think you have to push yourself and do really hard things, implementing stuff from papers. That was a huge leap for me when I first started attempting to do that, because it's hard to read a paper and then translate that into something that is meaningful, that you can turn into code that works and like achieves what you want it to do. Um, that was a skill that I think I had to develop. And it's much easier to have an intuition first that stuff.
Chris: No, that that's, that's my next step. So, I like reading white papers. I did it when I was in college. I got all excited about it when I got out, I was just working and driving. Right. So I didn't have that. And I've been back to reading papers the last few years, but I have a hard time necessarily reading a paper unless I get a suggested from somebody else reading a paper and realizing how it's applicable. So how, what is your, um, I guess paper process, paper learning process. Are there any things that you can say, “Hey, this, this way of reading, it really helps me.”
Chris: Uh, so, so step one is, um, you need friends who can point you at real things. You know, you need a why you need friends who have a wide who have a wide range of experience who have read papers and then can point you in the direction of papers that are meaningful to your, to the research that you're doing, you know, to the thing that you're trying to accomplish. Um, you know, if you're trying to, for instance, you're trying to, uh, drop traffic. So you don't overwhelm your servers and you have a bunch of friends who can say, “Oh, hey, there's all this network research from, you know, 30 years ago, that is totally relevant to the problem that you're trying to solve. And you should read these and learn about congestion control and learn about, you know, buffer, bloat and learn about all these, these ideas and the ways that you avoid it.” That is - having that as your first step. Cause there's a lot of stuff out there. There's a lot of research has been done out there in the world. And some of it's great actually, you know, I'll, I'll say that like a lot of almost all of it's great, right? But not all of it's relevant, even if it seems relevant and not all of it has aged as well. And some of it's, you know, some of it, you need to read the prior art and some of it, you can just jump to the new thing. So that's, but that's step one is find reasonable sources for stuff, read papers with friends, you know, have a, have a group that, where you do that kind of stuff. And then you go through like a greatest hits and just learn to read papers. That's the other thing. And the way I read papers, if I want to, I mean, whether or not I'm trying to get something out of it practical for my immediate purposes, or if I'm just reading it to read it, to read it, to learn a new thing. I always read the same way. And I do, I basically read it three times. The way I read it is I read it. I read it three times.
Amos: We should put that, that white paper about reading white papers in our notes.
Chris: Yeah. Yeah. It's, it's, I'll find it and link to it. I follow this thing that I read in a paper about reading white papers and it's it basically, it explains this notion of reading it three times. And when I say reading it three times, I don't mean you literally read it three times. You do three passes over it. You read so, and knowing that, and that helps you form at a more holistic, it gets your mind ready to actually deeply understand the thing. So the first pass, if you're going to read a paper, this is what I do. And this is what is explained in the paper. The first pass, you read the introduction, you read all of the titles in between the introduction and the conclusion and you read the conclusion and that's it. And all you're trying to get at that point is you're trying to figure out what it is they're even talking about, right? What is this even presenting to you? And also where are they going? What, what is, how are they, what is their methodology by which they're going to explore, like achieve the thing that they're talking about and how did they do? And so you get all that from the introduction and you get, and you get that from the conclusion and you see their path in the, in the titles, in between here and there.
Amos: You get to find out without reading a ton. Do I even want to continue with the next pass?
Chris: Yes. Right? Yeah. Maybe this isn't even relevant necessarily. Or maybe I need to go at the questions you asked. And then at that point, if you're trying to find information about something, are, is this actually relevant to what I need? Uh, secondarily, if it is, do I have enough information to understand this yet? So for instance, if you're reading about, uh, a CRDT, you, that paper that you're reading, if it's modern, let's say if it's recent, just came out this week, do you have enough knowledge in your head about the CRDTs and CRDTs generally, and all of the things that make up this world to understand that paper? Well, if it's a very new paper, the odds are probably that you don't, and you're going to have to go and find the references and go back and get some more prior art, because that paper is a conversation. The papers that you're reading are all conversations that researchers are having with each other, and you're necessarily coming into it very late. And they're going to assume that, you know, stuff and they're going to reference it. But as far as the reader goes, their intended audience, they're going to assume that you know things right. And they're going to cite those things and you can go find them. And then you can go find those papers and start there and figure out like what the level that you need to go to is to really understand what it is this thing's going to talk about. Um, you know, you, can't, I'm trying to think of a really have a really good example, but you know, if you want to read about raft, you kind of need to understand Paxos, at least at least tangentially. It's probably, if you're going to read raft, it's worth reading. You know, Paxos made live by Google first because they're going to talk about all this stuff and they're going to explain the problems. And then when you read the raft paper, you understand why they made choices in the RAFT paper that they did because you understand the problems with it. From the first thing, if you're going to read the dynamo paper, you first need to understand DHTs - distributed hash tables. So you need to go find the paper on DHTS before you can even begin to understand the dynamo paper. Cause they assume that, you know it. If I remember correctly.
Amos: I haven't read the dynamo paper. I can't help you.
Chris: It's another, it's another really good paper with reading, but in any case, all the, and obviously all of these papers I'm referencing are distribute systems papers, because that's like the stuff that I read a lot about. But, but anyway, so those are the questions you have. You ask yourself after the first pass and the more you do this, the more you build your head Canon, the more you do, you know, the more you build your, your mind palace that you can walk through, or you have your notes that you keep, you know, in your research journal, or you have a kettle stench in deeded, whatever that the, whatever the kettle kettles settle Castin is a thing I don't remember. Or you have a bullet journal or you have a whatever, right. You know, whatever it is that you're keeping your, your notes, then you have, uh, you have an even org mode, doc, you know, whatever, whatever it is, um, that you're doing all your research and you have all that stuff. And I do recommend, like you write stuff down as you go for me personally. I also I'll say this. I print papers out a lot because I'm a pretty tech, tactile person. I read them on paper with a pen. I typically make the font larger, but I like read them on paper with a pen and I underline stuff and I write notes in the margins. And I'm like, what does this mean? And it helps my mind stay engaged.
Amos: Maybe making the font larger is, is a life hack. I never thought of. But as soon as you said it, it's like, heck yeah. Cause that kills me. It makes me I start to get sleepy.
Chris: Yeah, yeah. Yeah. So, and also like, I like reading in bed and I like, you know, I like being on a lay on the couch and read and I just don't like sitting there with my laptop or something open to try to read stuff, you know? So anyway, so that's so, but that's just pass one. Right? Don't take any notes on past one. You're just looking at the introduction. You're looking at the conclusion. You're trying to ask yourself the question, like, is this gonna even get me close to what I need? If you decide that it is, and you need to read it further past two is now you actually read the paper and you start again at the introduction. Uh, and you read all the way through the paper this time, all the way through the conclusion. And yet don't, don't skip the testing section. Don't skip the metrics at the end. You know, don't skip all that stuff, all their testing methodology at the end, read all through it. And now you take notes. You know, you ask yourself some questions in the margins as you go, you say, uh, and you can criticize the paper. What, what the heck does this mean? Uh, this seems like a really bad idea. Be honest about it. You know? I mean like, like engage your brain in that. Even if it means being snarky as you're reading it, you know, I've been known to write just like whatever nonsense that comes to my head in there for no other reason that just keeps me engaged with the paper but skip all the math. If there is any, you can typically skip the majority of the code. Like don't work through all the code. If there's code, if there's like pseudo code in there and skip all the graphs, skip like skip all the pictures and stuff, just read the content of the, of the thing and get through it. And then if you need to go back again and then like, let it digest, right? Let it sit, take some notes on it. Write down what you think it said at the end of it, write down like what the, what you believe the outcomes of all this stuff, where, and then you go back and then you actually really deep dive into specific parts of it. Third pass is basically like, not a pass as much as it is like, okay, how does this algorithm work? And you actually sit down and you work out the algorithm and like, you actually sit down and map out if it's a data structure in memory, right. You start working out how, okay, how do these blocks of memory get allocated? How do they get managed? What does it look like? What does the shape of it look like? How does this tree get rebalanced? You all, whatever it is, right? You started looking through all of those things and then you start puzzling them out, right? You start working through the proofs. If you need to, if it's, if there's math in there and it's relevant, you're gonna have to work through the proofs and understand how it works at least cursorily, right? You need to at least start to understand, get an intuition for the proof and, and that's, and, and then, uh, you can start to look at the diagrams and understand like what it is that they're really saying and all that sort of stuff. So, but that's how I do it. I only do that though for, I only do that third pass part of it for stuff I really want to deeply understand stuff. I'm either going to implement or stuff that I just want to have in my brain. Like algorithms, I think ought to be in my brain. So otherwise I don't think you need to do all that, that level of depth on it.
Amos: What do you think that, that last little bit, like, I know every paper's going to be different, but on average, like what percent of time does that add?
Chris: 98 it's 98% of the time - Whatever that time is. Yeah. That's the hardest part. The third part is the hardest part, because now you actually have to like dig in and understand it. And in certain cases, like when I really started doing this, I had read papers. So the first thing they ever did, like this was raft was I built a, uh, a raft implementation. Uh, and it was really hard. It took me a long, long time to understand that paper at a fundamental level, at a level to which I could then build something.
Amos: What got you to read that paper?
Chris: I don't even remember at this point, I think I read it already. Right. I had already read it. It was almost like a dare. It was like, I mean really. Yeah, no, I mean, it really was like, I don't remember how I got off on that, but I just decided to build it. I don't know why I have no, I really don't understand why I did it. Um, and I just sort of did. And, um, it was it's again, it's like one of those, it's what we're talking about earlier. It's just a problem that took hold. And then when let go, and some of it was to prove to myself that I could, it was like, I think I can do this. I could enter this, this, this, this is a, this is a problem I could try to solve. Part of it was, yeah. So part of it was just like having was, I wanted to prove to myself that if I had enough gumption, I could just take a problem that I had no idea how to actually work on, read the paper and figure it out and figure out how to do it. Partially. It was to prove to my friends that I could do it. You know, I've, I've a bunch of really smart friends and I wanted to like for some degree, and not that they had any sort of competition. Right. But to some degree, it's like, I wanted to hang. Like I wanted to show that like, I could, I could hang if I had to, you know what I mean? It was like, it was like a little bit of a, that's probably unhealthy, probably. So that's probably, I want to be able to say I did it because I thought it would be impressive. I mean, there's no other nice way to say it. Like I thought it would be an impressive thing to be able to say, yeah, I just, I went through the pain and did it.
Amos: How did you feel after?
Chris: Oh, that I fell way short. Cause now I know too much. Like now I know, I know all the ways it doesn't work. Um, it felt way short and to some degree I wanted to do it because I thought it would be a cool project to have, I think it would be a cool project to have if it had worked. So, but in any case, I, yeah, I don't know. I don't, I don't know. There, there was, it was a certain amount of, um, yeah, I just wanted to do it to be able to show, to show off a little bit and be able to say like, guys, I did this and I was just really excited about it. I was like, I think if I can do this, you know, it'd be, it'd be, just be really cool. And yeah. And at some point it was, I think it went even beyond any of those things and it became much healthier and it was really just like, I'm going to do this because I refuse to let this problem beat me. I refuse to give up at this point. And there were times, cause it took a, it took a shocking amount of time to get it kind of working. And I, uh, yeah, it was just a lot of me like fighting and just being basically saying like, I refuse to let this beat me,
Amos: Those are wake up at two o'clock in the morning and start programming things for me, like wake up with sweats and you're like, I gotta get this out of my head right now.
Chris: Yeah. Yeah. Well, and, and, and I had to go back to the paper a lot and figure out like, how does this, when does this happen with this? When does this, you know, what States can these messages be received and what do I do with it? And then trying to, and it's easy to write that algorithm out in pseudo code. It's much harder to write it in a way that works as, you know, in a way that it is a real thing. And the paper's written like, okay, so you have a set of three nodes and then all know how to talk to each other already.
Amos: So you got to even get there.
Chris: Yeah. And so, yeah, of course. Right. So the first step that you have to actually solve is how do I even get these nodes to talk to each other? And how do I get them to join the cluster? And of course, cluster management is this thing that gets hand waved over in the paper. So you gotta, so that's the first thing you got to figure out how to solve, right? Which means you got to go read the really, it was now you got to go read these other papers. And so I had to read a thesis by somebody and I had to read another paper that people had written and, you know, it was difficult and took a lot of research and a lot of Googling. And I went and read through a ton of the raft mailing lists threads. And, uh, you know, I looked at, I read through HashiCorp's raft algorithm, which is wrong or was wrong time and like pointed it out. And I had a friend. It was really funny, serendipitously a good friend of mine, went and worked at HashiCorp provincially and had to work on RAFT. And I was like, okay, so here's the deal. I know where your problem is. Now we actually talked about it. And he, and he was like, you're totally right. And then it ended up somehow also getting linked back. Like there was like an issue somebody had opened that then got linked back to my raft algorithm. And he, and like that was a whole moment in time or whatever. It was very funny. But yeah, no, at some point it became like, I refused to let the problem be better than me because I was like, I will, my brain can do this. I refuse to believe that I can't do this.
Amos: And that's your first paper?
Chris: It wasn't first paper I'd ever read, but it was the first time I'd ever taken something from a paper and tried to implement it. Yes.
Amos: That's a big one to start with.
Chris: It's just a hard problem. Right. But it's also, I think you need that a little bit of like naivete. Like you need to, you need a little bit of gumption. You kinda need to go into anything like that. You kinda need the attitude of like, I refuse to let this beat me. I just refuse to give up. I don't know. That's my, that's my feel on it.
Amos: That's uh, yeah. I still think that's a big, big, first paper to try to implement.
Chris: Yeah. It's not small.
Amos: I don't remember what it was, uh, what the paper was, but I was in college, so it would have been, it's been a while. Uh, and so it had been like 99 to 2006, somewhere in there. And I had, uh, upperclassmen who just suggested it to me and it was all, it was just like an out, like a sorting algorithm. And he's like, you should go, you should go read this and write it. And so that's, that's what I did. And I had the same thing as you, except for, it was, it was doable in like a week to read the paper and figure it out and write it. And then he, he fed me a few things. Um, before he graduated, he was a graduate student actually. And, uh, I tried to read one. I can't man. I wish I could even remember the guy's name. I tried to read one of his things that he was working on when were in college and he was trying to generate music with artificial intelligence. And that was a rabbit hole that I never got to the bottom of it. One day I was finally like, you know, I don't really even care about this.
Chris: Yeah. Right. So I'm done. But I think that, I think that that really is the most important skill is you have to get uncomfortable with stuff and then you can't let it. And then you, and you have to pick a problem that's way outside of your comfort zone. You just have to pick, I mean, to this day, raft - at the - it's all contextual. Right. It's all based on who you are at the moment. Raft might have been the hardest problem I ever attempted to solve out of whole cloth on my own time and all that sort of stuff. Right. It's just technologically. It was out. It was way outside of anything that I was comfortable doing.
Amos: Do you think, do you think doing that has guided your career since then?
Chris: I don't know about all that. Like, I don't know if it's that big. Right. But I think that mentality is, has guided my career. I think having that, uh, that the, the sort of attitude of I'm going, like, it's, it's just code at the end of the day. I can, I can solve it at the end of the day. Like I will be able to figure this out. It's gonna take a long time and it might be a lot of hard work. I mean, it took weeks and weeks of work to figure it all out and get it working or kind of working on it. I mean, it doesn't even work half the time. Right. You know what I mean? It's got all these edge cases in that that just don't work, but - I mean it's, well, I mean, it works if everything goes well, which is of course not the point, um, and any, and there's problems with it needs to be fixed and tuned and all of these kinds of things. Right. But the, but more importantly, it's having the attitude of, I have no idea how to do that, but that's fine. We'll figure it out is that's the important thing. That's the thing worth cultivating and being uncomfortable is a thing worth getting used to, because it just necessarily means that you're not going to, you know, when the really hard thing comes along, you're not going to have to shy away from it. You're going to be able to just go for it. Uh, and that's, I think just generally it's much more fulfilling, it's fun, but it also leads to much more meaningful solutions because if it was easy, everyone would do it. It's the thing I always say about businesses. It's if your business was solving a trivial problem, will there be no point for you to have a business because someone would just replace you with an Excel spreadsheet. So, so you're the problem domain necessarily has to be normal. Like it has to be hard. Right? And he's like, you need to get used to the, the normal being difficult. Right. Because otherwise, why does your business, what problem could you possibly be trying to solve? If it's trivial? Why would anybody pay for that?
Amos: Yeah. You got to have a, you want to have a, if your target market is willing to pay for the trivial stuff, it won't be around that long.
Chris: Yeah. Right. It's just, it's the spreadsheet's role. If you can be replaced by a spreadsheet, you will be. And, and yeah. So it necessarily means that your core, the thing that's at the heart of your business has to be something that it's really hard to do. It's really hard for everyone else to do. And maybe that's just a level of effort thing. So a lot of times it's just level of effort. We could replace Slack with our own web socket-based Phoenix app, but there's just so much other stuff around Slack that it's hard to justify that. So, you know, it doesn't happen. That's just like a level of effort thing. And then there's problems that are just really, really hard to solve. And somebody figured out how to solve them and that, and that's what sets you apart. So my, my opinion on it is just generally, I just always encourage people to just get uncomfortable. Don't use libraries for stuff, learn how stuff works. Like, you know, don't install library if for a data structure, when you can just learn how the data structure works. Right. You know, don't install library to do fuzzy searching, just learn how fuzzy searching works, build your, build your own fuzzy searching algorithm. You're going to benefit more from that. You're going to learn more from that. And you're going to get farther in this career doing that. And then you put it in your, in your, in your bag of tricks, then you'll be able to pull it out whenever you need it and just build stuff. Right.
Amos: You can't, you can't always do that, but whenever you have the capacity to do so, you should be doing it and make, yeah. Don't, don't build the same thing over and over. Like you said, you gotta be comfortable with being wrong. I think that's where we learn the most. Yeah.
Chris: Yeah. So, yeah. So that's, that's, those are my tricks. Um, read papers, you know, and read, read books on computer science stuff. Do problems. Do advent of code having them codes out right now. Advent of code has great problems. They're super fun. They typically get variable brutal. I never can't get through it. Not necessarily. I've never gotten through it unnecessarily. It's like, it's the time-consuming part.
Amos: Well, that's what I mean. They get very - pretty. If you don't have like a very wide range of, of like a wide bag of tricks, uh, they get very time-intensive because you're still going to spend most of your time just searching for the like, all right. What, what about, what's the solution that lets this complete in under 10 minutes.
Chris: Right. I feel like, you know, at some point the brute force just stops working,
Amos: Feels like project Euler. Yeah.
Chris: Right. It's, it's very much that. So, you know, and, and sometimes you need to know the trick to be able to solve it. And then you can spend all day looking for the trick. And so it can be pretty draining on your time. They're on day three as we record this, and they're not that bad.
Amos: You just got to do what I do. I find other people that I like reading their code and I let them do it. And then I just read their solutions.
Chris: So far. I've been doing it an Elixir and in rust. Cause they've been pretty easy.
Amos: Or say, do you write it? And Elixir, and then you rewrite it and rust?
Chris: Yeah, exactly. I literally rewrite it rust. I don't tell people that I write. I've actually written a non-trivial number of thousands of lines of code of rust and my career so far. I just keep that to myself.
Amos: Well, now everybody knows
Chris: They're gonna they'll keep it to them. They'll keep it to themselves too. Don't worry about it. Yeah. Uh, no. See I'm doing it in rust and I'm also current. I'm trying to write it in Keathlang.
Amos: Okay. What's Keathlang?
Chris: Well, it's my own language for me.
Amos: Yeah, but tell me about it.
Chris: It's not done yet. I gotta, I gotta finish writing the interpreter.
Amos: Oh, okay. So that you can't quite use Keathlang. You write it on paper could do. Yeah.
Chris: You could do parts of it at this point, but not all of it for sure. One day?
Amos: Are you writing a book on Kaitlin?
Chris: No, no, no, no, no, no, No, no, no. Just go read crafting interpreters or go read. Yeah, go recopying interpreters. That's a better book than I'll ever write about languages.
Amos: I have not read that one.
Chris: It's really good. It's really, really real.
Amos: I have to go pick it up. I am, uh, I'm supposed to be reading. I'm reading two books right now that are programming books. One is a genetic algorithms and Elixir, the prep product book. Um, I am not a reviewer for it. I just picked that one up. But then I'm working on reviewing, uh, another book for prag product. That's going to be coming out.
Chris: I don't get asked to review things anymore because of my scathing review. That's not really true.
Amos: These are why I get asked to do it.
Chris: Yeah. It is a concurrency models in Elixir and how to choose the right ones. Yeah. Yeah. It's supposed to be one of their short books. Who knows though, you know, when you start writing, the author may decide to write a long book, but yeah, I'm looking forward to, to reviewing that and I may have started on a project. That's been consuming a lot of my time.
Amos: Oh yeah? RAFT 2.0?
Chris: No, no, no, no. But in the same vein I wrote that. I think I sent you that the screen shot. The spoiler.
Amos: What? I don't remember.
Chris: It was a while ago. It was like, probably like before the election.
Amos: Oh, geez. I don't remember anything from before the election. Uh, I had to write down your name just so that I would remember who you are.
Chris: Yeah. That's yeah, obviously, no, I, uh, I I'm considering, I'm considering starting a, in the same vein as, uh, learn new Erlang publishing, uh, uh, resource, nice on building and building and deploying, which is the key thing. Like essentially stateful systems in Elixir. Nice. I look forward to it. I wrote the intro. I wrote the intro and I've started the first chapter. And so now I'm telling people because then it will motivate me to keep working on it.
Amos: I'm just going to ask you every time we record. How, how much have you done since last week?
Chris: Yes, sir. Well, hopefully once I'm going to, if I can get, here's the thing, here's what I'll say. If I can get the intro and chapter one done, I'm going to go ahead and publish those on the website that I'm building for it. And then I will probably start like a mailing list or something like that as for chap, for new chapters as they come out and just release it like piecemeal. That's probably my goal.
Amos: I can't wait to, to get my hands on it.
Chris: It should be fun. It should be fun. We'll see.
Amos: Well, we started late in the day compared to our normal time and we're running up on an hour. We've got some other things going on.
Chris: Now. Have we given the people what they want?
Amos: Don't know that we ever give them what they want, but we give them what they need.
Chris: That's true. You just might find you get what you need. Alright. Born of a bottle. Thanks from heaven's hand. Thanks.
Amos: Coming in Mick Jagger. It's nice to work with you today.
Chris: Uh, obviously. Listen Amos.
Amos: Yes, sir.
Chris: When they love you, and they will, you tell them all the love and your shadow.
Amos: Perfect.
Chris: And listen. And when they, and when they, when they try to slow you down, you tell them all to go to hell.
Amos: And, uh, I hope everybody, uh, figures out where that comes from because I was totally enlightened today.
Chris: Oh no, they will.
Amos: All right, sir. Thank you. Alright, later. Have a good one.
Chris: You too. Bye.