Chris Keathley
The Elixir Outlaws now have a Patreon. If you’re enjoying the show then please consider throwing a few bucks our way to help us pay for the costs for the show.
Episode Transcript
Amos: Welcome to Elixir Outlaws, the hallway track of the Elixir community. So it's been a good morning.
Chris: It has been a good morning, so, okay. So let's get this out of the way. And then we don't. So we're not gonna talk about this anymore.
Amos: Let's not talk about it at all.
Chris: As I think we should, we should at least acknowledge it. So as of, as we record this, it is Thursday, the, uh, January 7th, we're still all a little bit recovering from the fact that, uh, yesterday armed riders broke into the Capitol building. Uh, and we're sort of like dealing with that. If we come out sounding, we're not gonna talk about it anymore after this, because smarter people than us have talked about that. And it would just devolve into like depression and whatever. Uh, so we're not going to bring it up anymore, but if we sound a little off that's, that's, what's going on in the back of our minds right now. And, uh, we hope you're all staying safe and healthy and, and happy out there in the world.
Amos: You said it all. It's perfect. All right, let's move on.
Chris: All right. Um, let's, let's talk about let's, let's give the people what they're here for, which is an escape from this, um, blasted hellscape and let's talk about Elixir.
Amos: Yeah. 2021 is going to be a good year.
Chris: Yeah, it's already as bad as before.
Amos: Alright. Uh, there was recently a new version of Elixir, really just a few bug fixes, pretty nice.
Chris: Nice patch patch, release,
Amos: A patch release. Nice. I was just trying, I'm trying to keep up with the good things that have happened so far this year. So we've got that.
Chris: I got a 3D printer.
Amos: Holy crap. Uh, we should talk about that cause I've never had one. And I, I look at them like every, probably two to three times a year, I look at them and then I get so overwhelmed by which one of these dang things should I buy.
Chris: Yep, absolutely. Absolutely. And there's a huge amount of price variance as well. Uh, which you're probably not, well, I mean, you could certainly do the research on into it, right. But as an, as, as a neophyte, it's hard to, uh, determine where you want to spend your spend your coin and you can go, you can get really, I mean, okay. Let's caveat again with, you know, you're talking in, in, you can get a 3D printer in the hundreds of dollars. That's kind of the, the, you know, that's the target amount of money that you're talking about. Right. And you can get that in the low hundreds of dollars or the medium hundreds of dollars or up to the thousands of dollars. And it's really hard to know, like what it is you even care about, uh, as a, as a novice, I feel like, So yeah. So I've asked related about it for a long time and it was always sort of, I mean, I've done, I've played around with it with 3D printing before we had a 3D printer back in college, but it was one it's like before, you know, this is in the era where the Arduino was still like a new fancy, like toy project for people, you know, it's like, that's kind of the era you're talking about. And so I think this 3D printer was like, I mean, I'm not, I think it might've been like a hundred thousand dollars. It was the university's 3D printer and it was the, you know, it does it all out of the, um, I'm forgetting what the technique is. It's the one where you put like the effectively, you know, printer, toner into it and it fires a laser into it and hardens it. I don't remember that. What that's called. It's not resin as it's not resin, like liquid resin. It's like a powder. It's basically toner. It's basically printer toner that you then like, you know, harden. So in any case, yeah. No, but it's been super fun. I got a, any Any Cubic I-3 Mega and I've been, you know, toying with just getting it dialed in and leveled and playing with.
Amos: Have you printed anything?
Chris: Oh yeah. Printing all kinds of stuff.
Amos: Do you have anything on your desk? You can just lift up.
Chris: I don't, I literally don't have anything. No, no. I have my dice jail. I made a jail. I made a jail to put dice in that are misbehaving.
Amos: All right. I'm going to describe it to the people. If you can hold it up. Can you?
Chris: It's not, it's not in here currently. It's it's out. Um, um, I sprayed filler primer on it so I can paint it.
Amos: Oh, cool. Cool. Sounds fun. I should just go use the one at the libraries. What I really should do. Yeah. Well, as soon as the library is open again, I can't go inside. Bring me books to my car, but I can't go inside.
Chris: Yeah. That's basically our situation as well. I think our library has two, um, printers, like two 3D printers are used to, I don't know what they have anymore, but I would say like, it is simultaneously more difficult than you would. If what you want to do is sit down and print stuff. It is more involved than that. You're going to need to learn more stuff than that simultaneously. It is not nearly as complicated as it seems to be. When you first look at it, you can pretty much sit down, fiddle with the thing, get it leveled and start printing stuff. And yeah, like some of your prints fail and you need to like tune it and get it dialed in. But it is kind of like, it is a little, for sure. Certain operations. It is a little bit like clicks of buttons in it in a way it goes.
Amos: Sounds like being a barista. You can spend your time dialing in the machine or you can just click a button and you can get something that's okay.
Chris: Yeah. Basically, I'm like, whatever the Tim Horton's of 3D printing is, you know what I mean? Like, um, I'm the Dunkin Donuts of 3D printing right now. Like, um, I like I'm uh, I'm it's serviceable at best.
Amos: They have good coffee.
Chris: Tim Horton's or Dunkin Donuts, or both?
Amos: I don't know.
Chris: Do they, though? Do they though, Amos?
Amos: Well, it's, it's, it's decent. They had some of the best coffee before I was really the coffee snob I am today.
Chris: But I mean, yeah, I get it. I grew up on Dunkin as they say.
Amos: You are in the south.
Chris: Yeah, Florida specifically Circle K. Actually Circle K was our, we did a lot of Circle K.
Amos: Okay. And Waffle House. I remember standing on the street in Augusta, Georgia. And you could look down the road and you could count eight Waffle Houses. And they would be full. They would all be full there. It was like two o'clock.
Chris: Right. Well, yeah. It's a queuing theory problem.
Amos: That's true. That's true. Are we going to turn this into Little's Law now?
Chris: I mean, that's, it's queues all the way down. Let's be clear. It's always cues all the way down, but no, 3D printing is super rad. I've made a dice jail. I made some toys for the kids. I made one of those. I mean like a flexible dinosaur for the kids. I started printing, um, mask straps for all of us. So there's basically like you, if you have a mask, when you're going out, you hook it onto these loops. It sits on the back of your head and you hook the mask straps into the, into this plastic thing that you print sort of con you know, bins it's thin enough that it will sort of bend and conform to your head. And then it's a, it takes all the strain off of your ears and hold your mass tighter and such. It's been actually super convenient.
Amos: That's always what bothers me is after I, if like, if I have to go someplace and I have my mask on for like an hour, my ears start to hurt.
Chris: Yeah. Yeah, exactly. So this is, this takes all the strain off your ears and it holds your mass tighter to your face. It's good stuff. It is good. It's good stuff. Good stuff. So anyway, 3D printing is super rad. Do highly recommend it. I had to buy a pair of calipers, which I did not own. And I really had to talk myself down off the ledge of buying the really nice calipers. They were in, they were in the basket.
Amos: Uh, honey, this one measures in Pico meters.
Chris: No, I mean, really like really well. And it was like, I was, I had justified it to myself about five different ways of like, well, these are the, these are exactly like what I used in college, you know? And like, I know how to use these already. And I was like, looking at the price tag, look it at them. And I'm like, this is the only calipers I'll ever need to buy it. They, you know, yeah. They have the thumb wheel and the whole thing. Yeah. They were really nice. But I talked myself out of it. I talked myself myself, I bought the $5 pair instead.
Amos: And, and that whole, like, I already know how to use them thing. They're calipers. They're really not that difficult.
Chris: They're just digital caplipers, man. They're not that hard to figure out. And also like, I don't need to measure anything, but within any amount of precision really.
Amos: Right. Unless you're going to 3D print a car.
Chris: Yeah. Or, or, I don't know. I don't even know. Like, I guess for electronics, maybe I would have done it, but like yeah. Nothing I do needs to be within that amount of tolerance. So I bought the cheap one and stuff and it'll be fine. But in any case, yeah, I'd highly suggest the 3D printer is super fun. That's been a really fun way to spend some time.
Amos: I've always wanted a 3D printer and a laser cutter.
Chris: Yeah. Well, there are there, you know, that there's now like dual morals. Like you can get one. That'll do both. Yeah, man. I mean, I don't know, like, I'm not going to tell you to spend a bunch of money, right? It is, it is money for, for reals money and like, who even knows if you know, who even knows what the world's going to bring tomorrow.
Amos: Right. I mean, our money might not be worth anything tomorrow anyway.
Chris: Well, you actually literally do not know that at this point and it's on the table is, is the scary, the really scary thing that Vanguard account you've been working on may or may not be there tomorrow. So in any case, uh, but I do recommend it. It's super fun. Um, it's a great little new hobby. I'm planning on printing out some stuff that I can like do some art projects and go, they break the airbrush out and start, you know that along with that.
Amos: Nice. That sounds like a lot of fun. I need to...
Chris: It's been super fun. It's been super fun. Nice to have a distraction,
Amos: You need to put your printer online so I can just send files to it. And it'll print in the middle of the night. Yeah.
Chris: Are you going to pay for shipping to get me to, to for, for the prints?
Amos: Yeah, sure. It's what you need to do.
Chris: I'll just print you stuff if you want stuff. But you need to, what do you want to print?
Amos: I don't know yet. There's all the times that I'm, I'm at home, like doing something -
Chris: You just want to print.
Amos: I think so, but I it's like at home I'll be doing something and I think, you know, this, they should have a tool that does this and I can like picture a design in my head. That's super simple. Um, for a long time, I thought of this thing. Uh, so we have an SUV and like cleaning the top of like an SUV. And we used to have like a 12-passenger van. Cause we're a giant family - cleaning the top of it. It's nearly impossible without like a ladder. And I was like, man,
Chris: No one has invented technology, right? To fix cleaning off the top of a large car.
Amos: It would be cool if I could like step up there, if there was a step on the car. And then I thought about, uh, the door, the little, the U-shaped thing that the door actually latches into is that if you could have some kind of step thing that hooks on that, and then you could step up and then the other day I saw one advertised and it seriously looks like it's just been 3D printed.
Chris: See, that could have been you.
Amos: I would be living on the, as sold on TV storefront right there.
Chris: You'd be on the as sold on TV isle in Bed, Bath and Beyond.
Amos: Which is the whole store. Isn't it? Uh, man, that would be super awesome.
Chris: I'm sending you pictures of things that I have 3D printed.
Amos: I'll have to add, I'm going to have to I'll describe them to people.
Chris: I mean, it's a dinosaur, it's a thing that hooks masks.
Amos: Yeah. He's got the, uh, it's orange. The thing that wraps around the back of the head, um, and holds the mask and it's got different, um, points to put the mask around for the different sizes of your head. So you don't have to print individual ones for each person that are custom.
Chris: Well, each mask. Right. It's a little bit different. Has different loops.
Amos: They kind of look like, um, it, it looks like it's made out of a pillar, uh, in ancient Greece. It's got a little scroll across the top. That's what they hook across. Oh. And then the diced jail. It's got a little lock on it. A little round bar.
Chris: It doesn't actually not, not working.
Amos: It's not a working lock. It's just a pretty luck.
Chris: I'll make it the show art. How about that? So you don't have to do, you don't have to do this.
Amos: Well, this is fun. This wait, this dinosaur looks like one of those snakes that I had when I was a kid. Can you hold on that tail? And it wiggles wiggles, but it's-
Chris: It's a dinosaur that flexes. It's very popular. It's a very common thing that people have 3D printed. I made one for my middle child.
Amos: Everything is in orange. So now I know that if I'm going to buy Chris something though that it should be an orange because that's the color that he approves of.
Chris: That was the color that was available.
Amos: Not your favorite.
Chris: No, I mean, I like orange. I got other - orange is just, you know, I didn't pick it. I didn't look at that. And it was like, Ooh, it's orange.
Amos: I would have orange, orange and purple are my favorite colors,
Chris: Orange and purple orange. I would not have pegged you for a purple man.
Amos: I like a dark purple.
Chris: Orange is a good - man. You want to, you want to make a design pop. As they say, the client comes back to you and is like 'it needs to pop'. You just gotta toss a little orange in there
Amos: I bought an orange shirt. One time. I have the wrong skin tone to put that thing on.
Chris: I bet you basically just become one orange blob.
Amos: I bought it. I got home. I put it on. And I was like, uh, I can't see myself. We need to get rid of this shirt. And so I, I took it back. Oh man. All right. So we we've got 3D printers.
Chris: Yeah. 3Dprinters are super fun, you know? I mean, I'm not telling anybody anything that they probably don't already either suspect or know, but 3D printers are very fun.
Amos: Yeah. I can't imagine it being not fun unless you got like a bad 3D printer. It's like buying, buying a bad guitar is your first guitar will make you not want to play guitar. Right. If you can't keep in tune.
Chris: Yeah. The guitar is hard to play. Yeah. That's a, that's a surefire way to not play any more guitar. Right.
Amos: I could see the same thing with the 3D printer. So finding one that actually is useful. You have to have a certain tolerance level. Right. So what, what do you have going on? Uh, like now maybe not work, but programming world wise.
Chris: I don't have anything.
Amos: Cool. Sometimes that's really nice.
Chris: No, I, um, no, I got, I got some stuff. I don't know. I I'm really the energy levels I have for programming stuff. No, actually, no, I have something we can, I have an interesting thing we can talk about. So sorry. I started working on this book. I bought the domain. The intro should be published soon. Now a lot of people basically come back and just be like, please publish this now.
Amos: What's the domain link?
Chris: It's uh, well it's not, uh, it's not live yet,
Amos: But will it be by the time this is out?
Chris: Yeah, yeah, yeah. It's just gonna, it's just stateful, Elixir.com is the domain. That's where it'll all be. So in any case, that's where stuff will be eventually, maybe by the time this comes out. Probably not.
Amos: Just so people know, I don't have any inside knowledge on stuff like this when it comes to Keathley. Sometimes I'm just as surprised as everybody else when it gets heard. I just get to hear it a week ahead of time.
Chris: Yeah. Yeah, exactly. And it's just going to be the intro up there and uh, and none, you know, I'm just trying to see if it's interesting to people or not, and probably start moving forward with that, uh, slowly but surely I got to figure out how to do some sort of newsletter I think is probably what I'm going to do. Um, yeah, I'll probably link to my GitHub sponsor's page just in case anybody does want to throw some money at the problem, but, uh, but not explicitly expecting any. Um, and otherwise it'll just be sort of a free resource for people.
Amos: Uh, can you, can you, do you have like your first topic in plan?
Chris: So my, okay. But yeah, so my main idea, this is, here's the thing I reserve, uh, absolute authority to change anything I'm about to say. Perfect. So don't get your hopes up. Uh, I think there's two super big parts of this. Um, one is a lot of the knowledge that's out there about the distributed parts of Erlang. Um, don't they give you like all the tooling, but then they're sort of like, “Eh, have fun,” and that's not like you need something that's more opinionated. I think. So, um, this is going to be a surprise to everybody, but, um, I'm super opinionated about that. And so it's going to be filled with my opinions on like, here's how you build. So I literally broke Amos. Wow. Uh, so, um, it's going to be filled with like sort of my opinions about it and, and, and very real world stuff. Like, I want you to come away from reading this with like, okay, I know how to build into play this stuff. And then the deploy part is also important because deployments with stateful systems are totally different. You have to think about them totally different than you think about deployments with a non-state with a stateless. So a so-called stateless,
Amos: Is this the, uh, there be dragons book?
Chris: Yeah, I hope so. Um, so the goal I believe is going to be, to demonstrate how to build a few real-world things. So we'll actually sort of go through a couple of different projects. I'm still planning out the roadmap as it is, as it were a couple of different projects. So you build basically from scratch that do distributed real world-ish distributed stuff. And at some point it has to be slightly contrived because like, you gotta, you know, you, you can't write the universal, you just have to write a book that people actually read. So, um, but hopefully it'll demonstrate. And the, the goal is to demonstrate a bunch of really important real world distribute systems properties. So like, how do you look up processes in the cluster? Like, how do you, uh, how do you deal with data? You know, how do you, what does item potency look like? Uh, what is, how do you deal with ordering, you know, all this sort of stuff, how do you do a failures? So that's really the big goals.
Amos: So if anybody took your distributed systems, uh, trainings in the past, should do they, is that kind of like a preview for this? You think?
Chris: Yeah, I would say that that would be like a, a lot of that stuff will carry over partially just because I've already written all that code. Some of it we'll get a lot of it, we'll get more detailed and we'll do the thing that we it's just really too hard to do in the training, which is actually talking about deployment and sort of say like, okay, here's how you actually deploy the, deploy the stuff. And here's the things you have to think about. And then another part of it's testing, like how do you spin up a cluster and then break it so that you can induce failures.
Amos: Cool. Uh, that sounds really awesome. Um, so deployment wise are like, when you're talking about deployment with it, um, what are, what are the biggest issues that you, you think you need to address that you see a lot of people not doing currently?
Chris: Yeah. So the biggest issues with are one part of its discovery. Like, you need a way for the nodes to discover each other just with today's setups, right? It's also, it's possible obviously to have like five dedicated boxes with IP addresses that you just slap in like a file somewhere or putting Etsy, you know, house or whatever you want to do. And that's also reasonable and I'll probably like try to address some of that as well. But I think for realistically, a lot of people who are deploying existing systems to their cloud infrastructure stuff, which is probably running in somebody's Kubernetes somewhere, um, you know, you need a way to discover this stuff. So you can either use service discovery. That's built in like Kubernetes has a mechanism for finding other nodes, or you need to like construct something yourself. And, um, and the downside to using something like Kubernetes service discovery is it doesn't help you at all with leasing or locking that it, it doesn't give you enough primitives that you need to be able to do other things as well, to like manage operationally, like manage certain deployments. And then all that goes back to, to like how much you care about your data. And so that's the other side of it. I think that's, that's the other really big part is once you start putting data in your cluster, you necessarily have to figure out how much you care about that data. You know, how much do you users care if you see the wrong stuff, do they care a lot or a little, and if you put important stuff in there, then now your deployments get really hard because it, well, I say that your deployments get really hard if you also, you know, want to achieve certain uptime guarantees, if you can afford just to like take the whole cluster down and like spin it back up again, then it's not that bad.
Amos: So I guess you'll have some like hot code deployments.
Chris: Yeah. Um, um, so, um, so that's part of it is I want to talk through hot, hot code deployments, uh, in the real-world stuff. But I also realize that most people don't want to do that. And, um, really just want to see, like how do I get my Docker containers to talk to each other? And so that's going to be interesting too. So I want to talk through a lot of patterns on like, here's how to maintain state in your application and also sync it to a database, like just make that the driving force behind the book, um, and also to play it. So those are the, those are the main things. Um, and because of that, like the book's going to assume, you know Elixir it really well. And to some degree that you're comfortable jumping back and forth between the Elixir and Erlang, it's not going to be an easy beginners’ book. Um, but there's plenty of resources out there that do that, that address those things. This is not going to be one of them. Cool. Like you need to read all that other stuff. What do you use? And that's probably good anyway, like you don't want to do this first anyway. That's true, honestly.
Amos: Figuring out how to deploy a single instance of an app is the first step to even being able to deploy a clustered. Right. And dealing with the cap theorem and, and all that, that stuff is, I assume you're going to be into that in this book-to-book newsletter, whatever you want to call it.
Chris: Yeah. So, yeah, it'll be really, I think it will be fun. Um, at least for the first few chapters, and then it'll become a slog, but, uh, I'm having fun so far. And I it's made me do some other research into sort of different parts of Erlang and the beam and that sort of stuff. And that's been also pretty fun. I've been breaking amnesia or Nisia as they say.
Amos: Is that really that difficult?
Chris: No, it's no, actually here's the thing it's super not, it was more difficult to get it to even kind of work,
Amos: Right. Like that's, that's what I've seen with local development. Even in local,
Chris: It was harder, dude, just getting three nodes running locally on my machine. I kept screwing it up. I found, you know, uh, a veritable cornucopia of ways to break in Amnesia locally before I ever even got to like the distributed part. I mean, and it's all like, quote, unquote, working as intended, which is to say, you know, there's no bug report to file. It's like, you did it wrong. You just issued the wrong operational commands to it. But man, like a little big oomph.
Amos: Yeah. I've, I've run into a lot of issues with being able to get data consistency with amnesia and a cluster and being able to like the whole, I just blanked out. Never mind. Forget it.
Chris: Okay. All right. I'm going to take the wheel.
Amos: I had data consistency issues in, in amnesia whenever I had, um, not even bad flaky networks, like with one or just the network in general. Uh, I couldn't imagine trying to run it on any network that had, that was very flaky.
Chris: Yeah. I mean, so the tests were basically like, okay, so we have a distributed Nisia, let's see what happens. If you have three nodes and you start to partition them just to say, you let them live. You don't take the boxes down, but you cut off their ability to talk to each other. And so I wrote a, I built a repo that does this and, uh, and all this started because I was like, people have been talking for whatever reason, like people aren't talking about amnesia more and more recently. And, um, I was sort of fascinated. I was like, I should go in and play with it. And like, I've, it's been a long time since I looked at this, I looked at that amnesia. And so I should try to figure out like, what's going on with it. So I was digging into it and like just trying to get networking stuff, working and whatever. And yeah, no, it's like, if you partition some nodes and then you just issue rights to either of the nodes and you don't do any and you just leave the default settings the way they are ill is very, I mean, it obviously, right, this is well known. It just very happily overrides data. And then they refuse to start talking to each other again, or, well, rather like there, they just alert you and say that they're partitioned. And then, you know, they're like, wait, we have inconsistent state fix it. And then, and then you're, it's up to you to fix it. And, and so, yeah, there's no healing, right? Like you have to determine the healing part of it. And there are modes where you can tell Nisia to do things like, uh, use a majority. So when you build the tables, you say here's all the notes, here's all the disc copies of this. Oh, here's all the nodes that have discoveries of this. And that that's part that's essentially what becomes the cluster at that point for that table. And so you can put it in majority mode, which is like, if I can write this to three other nodes and I have five nodes in my cluster and three of them, get it, then, then we'll consider that good enough. And you're on the wrong side of that partition. Then you aren't allowed to write stuff, but like that comes with its own caveats, which is that, you know, it's still Nisia reads to you. And at which point you just, your reads are wrong, which is like, that's not that surprising. It's, it's like you expect, it needs just sort of like a cash, right. More than a database. Like it's, you know, it's mostly all just in memory and then occasionally backs it up to disk, do some journaling and stuff like that. Uh, well, uh, depending on the mode that you're using to write to desk, if you're using desk copies, it, it writes it as like a journal file and all this sort of stuff.
Amos: So do you think, think there's, uh, depending on what the date is you're putting in the, do you think that there is a, a good way to tell it how to heal that would work for many circumstances?
Chris: There's two ways to do it. Oh. And by the way, healing, it is not is non-trivial. We also like healing. It's super non-trivial because if you don't have ordering setup, right. Go, this goes back to what we talked about last week. If you don't have some sort of clock give up on ordering, right. And now you're just picking you just pick one at random. There's no such thing as last write wins. If you don't have ordering, when you, when you feel right. You know what I mean? Like, it, it there's no time. If there's no timestamp, then who's the last, like you don't even know. So just pick one at random and you hope it's the right one, or you go, do you try to do something fully? You know, you, you try to build some sort of CRDT thing, but even that's like, you're going to have to have some sort of clock for ordering prefer most of CRD Ts that don't need to be garbage collected. Well, like, like brand-new, cutting-edge stuff, you know, like CRD already work well, but old, like CRDTs as they exist today, like you have to garbage collect them. And it takes a ton of time. And like, they're, non-trivial. So that's one side of it. Like you can go full AP and attempt to rebuild your Nisia tables by just overlaying and picking the one that you believe, you know, is the most recent, or you store a list of like operations that you're then going to need to like issue to each other and like somehow reconcile that state that way. Yeah. It's like, that's, that's one way to do it. Or I think the, the flip of it, and I think this is what the rabbit people are working on. As far as I understand is you don't allow yourself to get into a bad state. Right. Which is that you actually build real consensus over the top of it. So like, once you, if you're, you know, you only allow that to write to tables based on it consistently.
Amos: So when you're, when you're, I'm going to go back a little bit and what you were saying, uh, sure. We're talking about ordering last week in ordering in amnesia. So I know that they have an ordered set in amnesia, but do you know how they order that across the cluster?
Chris: Well, I mean, the ordering is just done in an order. It just in the ordered set table, it's just done by a key, like, and you pick the key.
Amos: Okay. So you have to tell it what to order by. So you could put a, uh, HLC or vector clock or something in there?
Chris: If you want to use it to the store operations. Yeah, yeah, yeah. That would be, that would be a thing you could do and then to rebuild, but to rebuild the world, it gets tricky because you need to ensure that everyone has seen all the same stuff you need some way to ensure that here's the scenario, here's the scenario. You have two notes, they should be talking to each other, but for whatever reason, they're not, maybe you're doing a deployment or you're just haven't connected yet or whatever. There's a problem. Change the cookies, you know, who knows what? Right. Most of the time I find in production, like real-world net splits are really just synonymous with deployment. So, uh, cause you're like, I think the thing's alive, but it hasn't like connected yet or it's, you know, rolling or whatever. So in any case, so it's, so they're not talking to each other just yet, but you issue two different rights and you store those as a list of operations. This is the whole idea. This is like, you know, uh, event sourcing without, without the baggage is, you know, you start issuing these, these operations. And then as long as they're all, you know, competitive, you'll arrive at the same stuff. Right. The implicit statement in there is that if they eventually see all the operations-
Amos: So when they heal, you have to get them to share events. Yeah. All of it.
Chris: So how do you know, how do you know if you've seen all the operations, Amos?
Amos: If you've gotten all the operations?
Chris: Oh, I know. How do you know? But how do you know.
Amos: Neither side knows, right?
Chris: Right. So you need a - now you need a new thing. Now you need a new thing on top of that. And it, and it, and it rhymes with trees typically. But really though, that's you use a data structure like that, right. That can start to tell you, like, assuming that there is a place to have that thing. And you also send that thing all the way around that you start to say like, that is, you know, we're missing bits, we're missing chunks. We're missing parts of it because the tree says we should have these other things.
Amos: So it feels like you could, you could get, if you design your data with a lot of these things in mind, then amnesia could work for you.
Chris: I mean, yeah, yeah. It still has that whole, like, it's not dirt, it's durable-ish. You know, problem, by my opinion is like, you should view - You should view Nisha the same way you view a single node, redness. Like it's going to give you basically nothing in terms of clustering, nothing real that you want to, you actually use right direction. And you're going to have to figure out how to make that work. But, and it has basically the same kind of like journaling semantics, right. Of like, it's going to flush this stuff to disk periodically and probably it's going to save it all. And if not, like it needs to not wreck your day
Amos: Except for when it doesn't.
Chris: Right. Yeah, exactly. And if it doesn't, it doesn't and that, and we just deal with that, but ideally it saves it for the most part. It's going to save it. And then when it's not there, you just fall back to the real data store. Like if you treat it like redness, you know, you're in pretty good shape.
Amos: I want to caveat that with, if you treat it like you should treat Reddis because I've also seen Reddis treated like -
Chris: Primary database number.
Amos: Our, we know it's never going down or it's never going to lose data. And then when it does people crap their pants, um, not my redness, my redness never goes down.
Chris: My redness is here to last.
Amos: Uh, I, that doesn't that just go against what reddis says it is anyway. Um…
Chris: I mean, you know, it's programmers know the benefits of everything and the trade-offs of nothing. I keep feeling it over and over. So yeah. I, I think, I don't know, Nisha could be really interesting in terms of like a cap solution. It's, it's neither it's neither C or A, like, it doesn't pick either of them really. I would, in my estimation,
Amos: Does it try to leave it up to you more? Or is it like,
Chris: Everything up to you, but it's also just a product of its time. You know what I mean? It's like invented when you were literally running a machine that had a, where the network was a physical back plane. No. Yeah. You know, I mean, it's like, it's just the, the, the, the, the constraints were different. And so if the network was down the back plane didn't work and like, you had a bigger, a much bigger problem at that point, which basically meant nothing. Right? Like the whole system was down. If the network was down, right. That's the, because the physical back plane would've broken at that point. So I know it's, it's a product of its time. And I think people are in, I think people come into Erlang and it is unfortunate because it's one of those things that I think is a huge differentiator. Like how many languages, how many runtimes, right. Have they built in high-performance database? Not many, and can support that. You know what I mean? Like how many, how many, like, and it, it does what it, it, sorry, let me say this. Nisha does do what you mostly expect it to do in certain circumstances. And so it's really cool to see that, to have that capability. The problem is, is I it's, I don't want this to come out sounding like Nisha is broken. I don't think that that's accurate. I think people's expectations for what you, for what, for what the words database means have changed a lot, you know, database, like people hear the word database and it is synonymous with Postgres or my sequel.
Amos: Where everything is in one place. And if it's down, it's down, if it's up, it's up like-
Chris: And it's, and like, for the most part it's consistent. Right. And it has, it's, you know, slow for certain operations, but, but you know, it's fast for others. And it's pretty much like going to reliably store your data, barring some sort of just gross oversight, right. Or just some sort of absolutely catastrophic problem. Those databases are just going to store your data. And you are not going to have to think about it. Like you never questioned whether or not MySQL or Postgres is actually stored a thing when you issue a right?
Amos: If you get a success back, you know-
Chris: -it's there. And if you get a read back, you know, it was the right value at the time that you issued the read. Like there are, there are caveats to that. You don't know things that are going to happen in the future and transactions are trying to run concurrently. And it depends on your transaction level and all that sort of junk. But for the most part, you rely on your database. And again, databases are very much people. People have this association with the word database, mean my SQL Postgres SQL server, whatever it is and that's, and you just, you come to rely on a certain amount of comfort and knowing that barring something absolutely catastrophic happening, this thing is storing my data when it says
Amos: Even, even those databases or any relational databases that provide a distributed implementation, have all these caveats. When you, when you read in there about what happens when you distributed. And even if you just turn on readers, your readers, when you read from a reader, it might not be the most up-to-date. You gotta be thinking about that.
Chris: And you can have weird issues in file systems. Like there's, there's still that thing where Postgres can attempt to write to the file system and it can fail. And then it will crash the Postgres because Postgres, at that point goes, “Yo, something bad has happened.” And we, the safest thing for us to do is to terminate at this point. Yeah. Because like file systems are all fundamentally broken and SSDs are all fundamentally broken, you know, all these problems or whatever, right? So like, yeah, I get it. Like, I'm not here to say that post-stress will never lose your data. It's more just that, you know, the, the likelihood of Postgres losing your data is, is markedly less than redness losing your data for instance. And that's markedly less than Nisha losing your data for instance.
Amos: Well, any, any time that you have to have cooperation between more than one system, the more, the more of those links that you have, the, I guess, less likelihood you have of getting what you want out of it without extra. There's a lot of extra overhead and work that needs to go in to make it sustainable. And then your, your trade-offs are bigger.
Chris: And you're going to have to learn a lot more. I mean, you're, you're managing you're, you're now literally managing a database inside of your application code, which is super interesting, right? Like that's fascinating. And again, I think it's a huge differentiator as part of the runtime. I just think people's expectations for what the word database means are so different than what is giving you out of the box, especially with its default choices that it's going to be really surprising to people. And that's where people get the feeling that Nisha is broken or wrong. I don't know that amnesia is wrong to be the way that it is. I do think that people need to be aware of what it is and to be responsible about that choice and accept it for what it is of like, yeah, man, if you want a really fast in memory, cache is backed more or less by a database and you just know that you can be wrong and your recovery strategy is like, I don't know, I've got an HLC or some other logical clock. I's pick the one that's like most recent. And we'll just deal with that. Yeah. I think he's just like,
Amos: As a blockchain and then we own them. We throw away the, the newest 10 until we have a consistency. Okay. My, my, my internet voice is not as good as yours. No, that was-
Chris: -that was strong. That was, that was, uh, that was strong. I mean, I hated you the moment you started talking. And so it's like, I feel like your mission accomplished might.
Amos: Uh, I got to tell this story now because voices. So my son yesterday, we were out and he started like, I don't know. I guess the best way I can describe it as is talking hick. And, uh, so then I just started talking like Mickey Mouse, every time he would talk like that, we're out in public and I'm like, so Josh, how's your day today? And he's like, dad, how about I stop? And you stop. I was like, “Oh, you're growing up. You've decided to compromise. Growing up so fast.” It was pretty funny. The rest of the kids found it quite entertaining.
Chris: You think your mom wants to have smokes? Oh boy, there's a deal on eggs. Okay. That's the, that was pretty much our dinner yesterday. So honey house today, now I'm going to go home and do that today. Like some smoked salmon dish. Did you see that? Six people still just contested?
Amos: Okay. I think that's a good place to end it. Uh, seriously though. I do. I do have, uh, another meeting coming up and we have a new person that started today. Chad, welcome Chad. Um, so I have got to go get some stuff done.
Chris: This is so bad. We can't release this.
Amos: Because I guess, I mean, we can't, we can't go any lower or any higher than where we just were. So I guess we can just take it on out. Yeah.
Chris: They look at, everybody looks at us and they're like, you know, now we know about those guys, but guys are high.
Amos: High on life, baby. Chris, thanks for coming today. It was nice seeing Anna today. Uh, even though I don't think she made it into the recording, she had, she had a meeting come up. She stopped by, she has meetings. She's been super busy. She misses everybody, not Chris and I. All right. Well take it easy, Chris.
Chris: Have a great day.
Amos: Later. Bye.