Episode 97: Successfully Vamped

 

The Elixir Outlaws now have a Patreon. If you’re enjoying the show then please consider throwing a few bucks our way to help us pay for the costs for the show.

Episode Transcript

Amos: Welcome to Elixir Outlaws, the hallway track of the Elixir community.

Chris: (Laughing)

Amos: This is the best beginning of any episode.

Chris: Oh, that's great! I've got to pour my coffee. Oh, coming in hot. All right, hang on. Vamp. Vamp for me while I pour my coffee.

Amos: Do what for you?

Chris: Vamp.

Amos: Vamp?

Chris: Yeah, hang on.

Amos: I don't have any blood to suck over here, so I don't know how to do that. What am I doin’?

Chris: Well, that's well, I'm back, coffee poured.

Amos: Okay. Whew.

Chris: Successfully vamped. Good job.

Amos: Yeah. Good. I didn't know what I was going to do without you.

Chris: Whew! How have you lived the past three weeks then?

Amos: I know we've been gone and things have been crazy. Um.

Chris: Here's some inside baseball. Summertime with recording in the summer-

Amos: (Whispers) Terrible.

Chris: -is always tough. This happened last summer, too.

Amos: Yeah.

Chris: We've been doing this little song and dance for a while now. Enough to have had a previous summer upon which to reflect.

Amos: Why would we reflect on things? Are we trying to change and improve?

Chris: No, absolutely not.

Amos: Okay, phew! Phew. I wasn't sure.

Chris: I believe in plowing ahead headlong into danger.

Amos: I mean, we're not getting paid for it, so just, just go forward and enjoy.

Chris: Yeah. You come over here and mow my lawn, and then I'll reflect. Okay. I'll spend a little of a- I'll do some introspection.

Amos: Uh, yeah, I've had, I've had crazy stuff going on, a broken window in the backyard, like, like ceiling to floor eight foot.

Chris: Oh no!

Amos: Yeah. What happened, it's all spider web right now where you've got to get somebody to come over and take a look at it. Went on vacation, uh, fly fishing in Montana, it was fun. What else have I done in the last few weeks of work, work, work in a happy way. I'm having a lot of fun, um, digging into Ecto more and more, which is something that I didn't do too much of before. Like most of my Ecto stuff was fairly basic. Like writing data from sensors out. So, yeah. We should talk about that. I mean, really, we have your blog post and then we also have your comment that nobody should use. Well, you said you wish more people would just use, uh,

Chris: Yeah. Don't- don't listen,

Amos: Don't use belongs to and has many and stuff like that, right?

Chris: Don't make, don't make bold statements for me.

Amos: No, I'm not making them for you. You made them. I'm just repeating.

Chris: I didn't say that though. I didn't say you should never blah, blah, blah.

Amos: What did you say? You should go find what you did and-

Chris: I didn't say you should never blah, blah, blah.

Amos: Okay. Go find what you said.

Chris: What I said was I think it would be, I kind of wish more people had, would use Ecto with just basic schemas and queries. Just try that. Try it out.

Amos: Explain that to me.

Chris: So when we were at, uh, Le Tote before the mass exodus-

Amos: LT, as the cool kids call it.

Chris: The, the, the, the Elixir diaspora of Le Tote, of which Lance was the only remaining, uh, individual. Um, he, you know, when we started there, he had put into place this pattern of- we had schemas and those schemas- and we had an existing data model that we were interacting with. And so that was part of the motivation behind us and an existing data model in an old database that we were interacting with is my SQL. I say old; it- it just meant the database that had been propped up when the company was started and the Elixir stuff needed to interact with that data model. And one of the things Lance had done is that, uh, we, we, we didn't really write to that database all that much when we did, we used chain sets, but for the most part, we were querying from it. And instead of bringing over all of the associations and whatnot, Lance just set up a bunch of schemas that had the foreign keys written out as foreign keys, didn't use any associations or anything like that. And then whenever you needed to do things, you just wrote queries. You just wrote explicit queries to go get and interact with the stuff that you actually needed.

Amos: So if you need to insert something, you, you did like a, like if you needed to insert multiple things, you would, they're dependent on each other. You do like a multi. Y.

Chris: Yeah. Yeah, yeah, yeah. And also the writes were a little bit interesting, writes were more complicated for a whole host of technical reasons, um, that were largely had to do with like, you know, the fact that lots of people were interacting with this singular set of data- data. So they went through a kind of a different pattern, but for the querying logic specifically, yeah. We just wrote queries. So what does that mean? Well, what's interesting about that is you would end up writing a lot of very specific queries to do the thing that you actually needed to do. So you would like go get the piece of data that you actually needed from a select statement and bring it back in and, uh, in the most convenient form.

Amos: So it'd be a little more and perfect for what the, or your exact use case at that moment.

Chris: Yeah. And you did the joins explicitly cause like you, you didn't preload because there's, there's no, there's no association to preload or cast into.

Amos: Okay. So one of the things that I deal with is let's, let's say that you have, um, you're doing an invoice, right? And it has a lot of line items and you require at least one line item. That's one thing that's really nice with cast_assoc is being able to say that you require at least one in your change set. Um, and then some of the other things are like the unique constraint versus unsafeunique, give you some nice things there. So I've found that when, whenever I've created like a form change sets, right, or form schemas that are, they're not back directly. ‘Cause it's like crazy multiple tables. And whenever I need to do those kinds of validations, like the unique constraints, it's like a really, a lot more going on now than if, if I use the like has many and, and cast_assoc with require one. So I'm, I'm curious at how that changes and what that looks like.

Chris: Yeah. So, I mean, on the writing side of it, uh, you, you just get more explicit. So you either like have, you know, just use repo dot transaction and just write the various inserts that you need, uh, for your various casting and all that sort of stuff.

Amos: The validations is what I'm talking about. Like the, the cast_assoc like, cast_assoc, um, line items for an invoice required true. Makes you have to have at least one.

Chris: Yeah. So I mean, yeah, yeah. I get what you're saying. I think those things can be useful. It is my feeling though, and this is something that we also kind of figured out at LT and stopped doing to some degree is, changesets aren't all that reusable. Like, chainsets aren't a reusable idea in the sense that like you really can't use, it's why people end up with a create thing, changeset, and an update thing changeset and a blah, blah, blah thing changeset. It's like, you have a chainset per action, but you're going to deal with it because you kind of care about different stuff depending on the use case. And so like we ended up hoisting all of that logic out of our schemas into sort of this higher level intermediary. I mean, we didn't have context, we didn't call them contexts. Like, I don't think contexts were a thing when we were working on this, we had modules and we had functions that did stuff and provided a layered API. But yeah, a lot of the changeset stuff ended up on like in the layer above the actual schema definition. And so you would end up building the changes that you needed for that action. And then, and so like, yeah, you may not have been able to use cast_assoc or whatever, but like you kind of don't care because cast_assoc is predicated on the idea that there's a specific reusable changeset that you want to use on that other schema. You know what I mean? Like, it's like, it's built on the idea that like you have some other reusable thing and it's like, well, you actually don't like, that's none of that logic's reasonable. And so just write it all in the layer that you're actually using it where you're actually doing the inserts and such.

Amos: Okay. So how I, I, I did similar, I guess, I guess it's still in that I put another module out there where I combine, uh, different changesets from different things like the invoice changeset and a line item, chainset.

Chris: Yeah. Your context.

Amos: Yeah, I guess. If that's what you want to call it. So I guess what always gets me when, when going with this approach, like it's powerful, it's super flexible, which in a lot of ways makes it simpler to maintain. Uh, it doesn't mean it's like quick or easy, but it's, it's simple to follow and see what's going on. But when it comes to creating a changeset so that your front end form shows validations and stuff without a lot of extra work on your part, that's a pain too. Like rebuilding a form changeset with all those errors. So what do you do any, do you like not use those forms? Do you do all of it by hand on the form? Like how do you deal with that?

Chris: I don't know because I, to be honest, don't have a lot of experience dealing with that kind of stuff in my Elixir work. I mean, truthfully, like that's just not, that's not how I use Elixir. It's not, I don't build a lot of like static web apps and stuff like that. That's just not the work that I do. So I kind of don't even know how to answer. I don't know if I would be able to answer your question very well because.

Amos: You're building a lot of API stuff.

Chris: It's just, yeah. It's not the wheelhouse I live in and it's not the world that I inhabit at all. And so I, my inclination is that- I don't know what my inclination is. I think, I think something about, I'll say this, something about those layers, the layers as encouraged for like static web apps, feels very off to me in a way that I cannot describe more than like a gut feeling. And it's just a gut feeling based on like 10 years of doing this. And to be honest, I don't deal with that problem enough. So I haven't sat and thought about it in the hammock long enough to really figure out what it is about that that feels off to me. But something about it just feels a little off. I don't know how I would handle that because yeah, you kind of have to have a changeset for the form to work the way that you want it to work and you have to because it, because of that, that's how you get like your errors back into the form and blah blah and like do the foreign validations and all that sort of stuff. Like, yeah. It's, I don't know. I don't know if I have a good answer for that, to be honest.

Amos: How in, in the non-web app, the services version, how are you returning errors?

Chris: It depends on the contracts of the API, but what I like to do is, uh, have a generic error struct. Have a generic error struct that can be used for any errors that occur in the system. And that thing is actually an exception. You know, you do def exception, but an exception is a struct that also happens to be raisable and happens to have like a message. Uh, you know, it happens to conform to a specific contract and stuff like that. So I generate that and then have a bunch of functions on it that are like not found and internal and, uh, you know, not authorized or whatever. And, um, all of my various operations return those, return, return an error struct. And then when you get out to the edge of the system, you can either embed-I mean, those are, those are exceptions that you can either raise. If you want to raise and just blow up the request or blow up the background job or whatever it is you're going to do with it, or because of the structs, you can implement all these protocols with them and either convert them into text or convert them into a response, right, cause you can say, “Okay, well it's a not found error.” So that's a 404. So send back a 404 with this HTML in the body or whatever. So like, then you can have a generic thing that allows you to support all those operations. So that's how I like to handle it, but forms get real messy because forms, you actually want all of the individual fields returned, right. And really what you want to do is you want to render HTML that happens to like have certain new HTML tags in it and maybe new CSS selectors that mark a field as an error. I mean, that's what Phoenix is doing for you, um, with like some of their, you know, generated HTML and stuff, it's, you have an error page, an error version of that, which is that if there's error messages for this field, you render it with like a red border around it because it has a special CSS selector and you render additional HTML elements and that sort of stuff. So what's cool about the error struct thing is that you can return that and it's arbitrary what you stash inside of it. And so you could theoretically generate that HTML from that error struct as well. I mean, that's the, at the end of the day, that's what chainsets are doing for you. That's the same, it's a similar, very similar pattern. And because then you get the changeset back and it has all the pieces that you need to, to render that stuff appropriately. And you can do the same thing with changesets, if you'd want to speak, changesets throughout your entire application, you could do that with changesets as well and convert those into JSON errors or whatever.

Amos: That's what I'm doing now. And I, whenever I have to split things out and do like a, a special, form changeset versus what I'm using to talk to the database, I just have to rebuild that changeset to send out the errors. So I have to build one for the front end. Um, and I just didn't know, like sometimes it felt like, am I messing something up? Because like, this is way more complicated, but it-

Chris: Yeah, I don't know.

Amos: And so I just wasn't sure if there was another way that you were doing it.

Chris: No, I don't. Yeah. I don't, I don't really have a good, good patterns for that sort of stuff. Mostly just cause it's like, yeah, it's not the world I live in. I don't need patterns for a lot of that stuff. My inclination would be to like, not use changesets, to be honest, like my own, my own proclivities are to like handle it myself instead of, because that's, and that speaks more to my ignorance of like, how does Ecto work with how do changesets even work? You know? Like it's built more on that ignorance where I'm like, well, I think I can just write this myself. And it'll be really simple if I just write it myself and then I'll be able to understand it. Cause it's going to be like one big function that I can just read and understand. So that would be my inclination, but I'm sure there's probably a good pattern for this stuff with Ecto proper. I don't really know what it is.

Amos: I haven't figured it out. I've asked around and like, it feels like not many people have figured it out if they have, or I'm just asking the wrong people. I don't know.

Chris: Yeah. I don't know. I think you could probably, there's definitely people out there to ask. I just, you know, I'm not one of them, I'm also like a less is more person. Like I want to re- I want to rely less on a lot of these like niche features of, of, of a library.

Amos: That's fair.

Chris: That's, you know, that's my own sensibilities.

Amos: When you get into those like really niche features, they- they seem to change a lot too. So if you're kind of implementing that yourself and you have less changes that you have to deal with whenever there's a library upgrade or anything. I mean, I guess at the same time, you might lose any optimizations or anything that they put in. How do you, how do you find it is to bring somebody new onto a project whenever you're not doing it the, the prescribed way?

Chris: I think bringing new people onto a project is hard no matter what.

Amos: Fair.

Chris: And all of the arguments about like, “Well, it's, it's conventional, so everybody should be able to understand how all of this application works," is total bogus. Like none of that's, that's all BS. There is no substitution for like experience and being in an application and there's nuances that you're going to need to learn on every application. And I'm sitting here in, uh, in my current work, looking at a application that is 350,000 lines of Elixir and it's all conventional, but it's really, really, really freaking hard to understand how anything works because it's a giant application. It's like using context. It's used an umbrella app. It's like, did all the things, all the quote unquote correct ways. And it's impossible to understand, you know, if you're brand new, I've been here three months and I'm just now beginning to kind of get a sense of like where stuff is in the application, how it's all put together.

Amos: That's kind of my thing about being useful at any new project. If the project already exists is, is about three months. And actually, if you're making a brand new project from scratch, I feel like you, unless you've been studying the field, that that project is in, it’s probably about three months before you're useful there too. Like really useful.

Chris: Yeah. The people who pitch you on the whole idea of like, “Well, every Rails application, it just looks like every other Rails application.” Those people are consultants and they work on tiny applications that barely do anything. Get, get over yourself. Like that is, that's not a real thing.

Amos: I'm a consultant, but I don't feel that way.

Chris: Especially if you want to- I know. But the people who say that you'll notice that they're basically all consultants. Uh, and it's like, yeah, because it makes your ability to get into an application and start being a tactical tornado and making things worse for everyone else that's there so that you can make your billable hours. You know, get those billable hours. Listen, I was a consultant. I used to live that life and felt like I was doing a really good job. I, when I say tactical or tornado, I'm talking about me, I'm talking about young Chris, like who didn't know any better, who didn't realize that like what they were probably doing was making a lot of other people's lives really hard by being more productive than everybody quote, unquote, more productive than everybody else. Because you're not improving the system. You're getting, you know, your work done for today, which is not the same thing. I would argue those two things are at odds with each other: the faster you want to go, the more mess you're probably making for someone else to clean up, especially if, as a consultant where you don't have to live with it six months from now.

Amos: No matter, no matter whether you're- you're a consultant or not, anytime that you're trying to move fast, you are going to make a mess.

Chris: Right.

Amos: If you're not taking the time to step back, make sure that the changes you're making are actually improving the system.

Chris: Right. Right, right. Yeah, absolutely.

Amos: So I think we have, uh, we have another thing to talk about.

Chris: Okay.

Amos: Your blog post. Good and bad Elixir.

Chris: Sure. Hit me. We want to talk about?

Amos: Well, you had a lot of ideas in there. I wish I had just read this right beforehand. Cause now you said -

Chris: You didn't even read the post?! And you want to talk about it?!

Amos: No, I read the blog post, but like you said, it's summertime. So it's been three weeks since I read it.

Chris: I got it. I got it.

Amos: I read it the day that you came out with it and I posted it to everybody.

Chris: Are you're the one who shared it to Hacker News?

Amos: I didn't, no. I don't get on Hacker News. I don't get on Discord.

Chris: I don't, either. I had to find out. So, so this is going to sound like a weird flex. And I guess it kind of is, but like I didn't, I didn't know it had even been submitted to Hacker News until someone at Frame like posted in a public engineering channel that was like, uh, "Keathley's post is now number one on Hacker News. Congrats." And I was like, “I didn't even submit that crap. Who did that?”

Amos: I don't go there, and I don't go to Reddit. They're just time sinks to me. I can't.

Chris: Yeah. Yeah, for sure. For sure.

Amos: Well, we could just start at the beginning and go through, and then anybody who hasn't read it doesn't need to.

Chris: Yeah, for sure. So, okay. So I do need to explain a little bit about this post because I think it it-

Amos: I got it pulled up.

Chris: I think it is, it's important to explain the context, right? So this post started as me writing up an internal memo for all, essentially for the, for the team that I'm now working with of like patterns that I believe would lead to a more maintainable application. And simultaneously it's stuff that I've picked up- it's stuff I've picked up over the years, uh, of doing Elixir, but it's a lot of, it was formalized at Bleacher Report because Bleacher Report was unique for me in that we had a ton of services. They were all, we had a ton of services for the size of team that we were dealing with. And yet it just wasn't that much code. They were all pretty self-contained. And while some of them, you know, were in very, all of them to some degree were in various stages of neglect because we just had way more services than we had people to maintain them. And our maintainership was kind of weird, they were all pretty reasonable. Like, you could get in there and figure it out, for sure. All the flows were pretty reasonable and it was harder to understand the overall, you needed to understand the overall architecture, but overall, like once you did that, each individual service was pretty okay. Some of them were outliers to that, but overall it was pretty okay. Um, so a lot of these patterns started to be formalized for me at BR where it was like, “Oh, we can just build apps with like, way less code.”

Like, we don't need most of the ceremony that people think that you need. Um, we can get away with a lot, a lot less. And actually it's way simpler when we get away with a lot, when we do less here. When we do less design, things kind of were better. When we askewed all of the various rules that people state about how you ought to build Elixir systems, especially Phoenix apps and stuff, it was like, yeah, well, all that stuff's wrong. Like, like it was way easier for us to maintain things when we didn't, when we ignored a lot of those ideas and it was a thing that we kind of fell into backwards. So that's where a lot of these ideas in this post are coming from, right. And, and, and then the other part of it is tons of libraries. Like, I've maintained a bunch of libraries over the- over the years. These are all things that I now endeavor to do in my libraries because it makes maintenance on them easier. It makes understanding them easier and it makes the APIs for using them better. So, yeah. So that's all, that's all the context of this. Anyway. So where do you want to jump in? What do you want to talk about? This is your show.

Amos: We know we all know it's the Keathley show.

Chris: This is your show. You're driving the bus.

Amos: I'm more like the guy playing the piano on, on The Tonight Show. You just talk to me once in a while.

Chris: You just, you just point and look over there, "Hey, hey, ha-ha!"

Amos: Hey, there you go.

Chris: Very good, sir.

Amos: The first one is map get keyword get I'm going to throw in the second one and fetch functions versus access.

Chris: Yeah.

Amos: So you talk about using access over those because it allows you to change the underlying structure and not have to maintain the code. Like to me, that's like use protocols over directly calling things, if you can, because it allows it to be more expandable. My question here is, do you end up using or implementing Access for a lot of things internally for yourself?

Chris: Something that I had forgotten that we did at one point is we had, for a lot of our Ecto stuff, I had forgotten that we always used like a, a specific, our own, like, um, what is it? Oh, what's the term? Uh, no like we had our own module that we would like use Ecto schema in. And so we never used Ecto schema directly. We always use like BR dot schema and that had, and because it was like pre set up to do primary, like primary keys that were like either UIDs or HLC or whatever. And like, it was set up to do like custom stuff. I had forgotten that we added for every single struct, a default access. So all of our Ecto stuff implemented access already.

Amos: Nice.

Chris: So I had forgotten that part, which is also important to like know because then you can use Access everywhere and you don't run into that situation of like, “This struct doesn't implement Access yet,” and that kind of thing. Like, you don't hit those bugs.

Amos: What always kills me with Access is when you have like, you know, params have string keys and other things have symbol keys and Ecto works, like change setswork with both for attributes and sometimes they don't work for both. And it's really frustrating. But I guess you could, if you implement your own Access, you could have your Access to work with either type of key.

Chris: Right.

Amos: Which is probably what they're doing on underneath. I haven't gone to look, but.

Chris: I haven't looked, I forgot the other part of it, which is like when we were doing just JSON APIs. So for, I mean, the majority of, of Bleacher Report APIs are JSON APIs still. We started very slowly started moving towards, uh, using protobuf. And we've talked about this before, but using protobuf and RPC, um, with protobuf, if you get a struct, you know the struct, because it's a defined contract, it's not arbitrary J keys going over the wire. It's like, this is what I'm sending you. Um, and so with those, you get a struct and you can actually just like use the dots and tax on all of it. ‘Cause you know, what's in there. Like, you can go look at the code and say like, “Well, this is the, literally the exact fields on this, on this struct.”

Amos: And that's a lot different than if you're working with web forms.

Chris: Yes. Yeah, absolutely. But the other side of that is when we were using JSON, we never use atom keys. We never converted to atom keys. We just used string keys everywhere. So all of our maps were string keyed.

Amos: But your internal structs are not.

Chris: What internal structs? Like, that's what I'm saying. Like, no, like a lot of our services were calling other things and dealing with Kafka and dealing with JSON and then calling other stuff and blah, blah, blah. And it's like, why, why are you going to convert that to a struct? Just like maps with string keys, whatever, not a big deal. So like, well, you got type two more characters, one more character, actually like, then using the using atoms.

Amos: Yeah.

Chris: It's like, who cares? Um, so yeah, we did, we just used maps with- with string keys everywhere. Cause it was simpler and it was not a big deal. Uh, so,

Amos: So do you always use the arrow syntax on all of your maps?

Chris: No. No, no. I mean, if they're, if they're atoms, I don't, if they're string keys, yeah.

Amos: Okay. I'm just curious.

Chris: Yeah. No, I mean, it just depends, but yeah, but a lot of our stuff was literally just string keyed pass between services and that sort of stuff. So, you know, you just didn't care. So that was the, that was, that's also part of it. The bigger thing here though, is just like, and it's funny ‘cause this, this, this very first point is like one of the most contentious, amongst people, like this is the one people dis-

Amos: I thought it was the easiest one.

Chris: This and width were the two that people disagreed with the most. My big thing is like, people were like, “Well, I've never been bitten by like, you know, whether or not it's a keyword or a map.” And it's like, “Okay, you haven't run as much Elixir as I have.” So like I don't know what to tell you, like I have.

Amos: I get bit by it all the time.

Chris: I've totally been bitten by that. And I've been bitten by that libraries that don't- that like arbitrarily support one or the other. And it's like, yeah, but I have a map right here. Like don't make me convert it into a keyword list. They behave identically for the purpose. I mean the semantics of them do, right? And also here's the other thing. These aren't giant, you know, when you're, when you're accepting, here's the thing, when you're using map dot get and keyword dot get, it's not like you're dealing with maps that have hundreds of thousands of keys in them. You're not dealing with keyword lists that have hundreds of thousands of keys in them and so the performance difference is literally noise. It does not matter. There's no performance degradation between using a map with 10 keys and using a keyword list with 10 keys. Come on. Like, there, any performance issues in that is literal noise. So it doesn't matter from a performance standpoint. And, semantically, they give you the exact same result. Like they behave identically. It's so funny. ‘Cause someone on Hacker News was like, "Well maybe they do. Maybe they don't." And I'm like, “No, they literally do. Like, you can go look at the code.” And he's like "Well, I don't know. You can't know that." And I'm like, “No, I can literally know that I looked at the code. They behave the same.” I was like, they behave. If you're using dot get specifically, it's not like dot get on a keyword list supports, I mean, the main difference between keyword lists and maps are that you can have duplicate keys in a keyword list. Um, you can't have duplicate keys in a map, but if you're using dot get, duplicate keys don't work anyway. It just gives you the first one.

Amos: Okay. Right the whatever it runs into first.

Chris: Right. And so it's like the performance difference is totally negligible. And there's really no reason to, to lock yourself into one thing. Because like, what if I have a map? What if I already have a map right here? And I have a map because you know, I'm trying to configure a GenServer and I decoded a JSON uh, file or like call it an S uh, a rest API to, and got back. JSON and now I have a map. I don't have a keyword list. I don't want to convert this crap into a keyword list just to pass arguments into a GenServer, come on. Like, that's dumb. So work with both. Like it doesn't- it's not that big a deal to work with both and it, and forget operations. Done. Easy.

Amos: Well, and then, and then you also, by being able to use other things other than keywords and maps, like you can, you can add access to a service if you want. So you can ask a service for something by, by just doing the keyword Access. Watch it though. I think that this turns into people trying to implement objects by using Access,.

Chris: Well, you're just using protocols, right? Like you're basically at that point, just using protocols and like dynamic dispatch. And that's kind of where you want to live. Like, you want things to be more generic because it means that your stuff's more reusable. And especially for people who are writing libraries, right. This is especially, especially useful for people who are writing libraries. You've got a library, be more permissive about what I can give you. Don't arbitrarily break because I happen to have a keyword list or a map here. And you, you just demand that you have that you want one so that you can use dot get. Like, that makes no sense. Or convert it for the user. Like, if you really want keyword lists, do it on their behalf. It's a safe operation to call keyword dot new on a keyword list.

Amos: Right. Or on a map. You're gonna get the same thing back.

Chris: Basically, yeah.

Amos: So you said the, the, the other thing that people hit up the most was, uh, your, your discussion on width and probably specifically using an else block with the width.

Chris: Here's the thing. I think people read that, I think people didn't read what I wrote. I think people read the headline and the H two block there and didn't actually read what I wrote underneath it, because I got a ton of comments about like, “I use else all the time and it seems fine.” It's like, I didn't say don't use else. That's not what I said. I said, “Don't, if you're using else to handle every possible error, you don't want width.”

Amos: What do you want instead? Case?

Chris: Cases, nested case statements. Because if you're handling every else case, it necessarily means that the errors matter. And I actually, I think it's more obvious when you start talking, start talking about the whole tagging operations with tuples thing. Have you seen this in the wild before?

Amos: Tagging with tuples?

Chris: Okay. So the idea is you have a width, a set of width, uh, the quote unquote happy path, which we're going to come back to as an idea. Because I've done a lot of thinking about this Amos and I'm pretty sure the happy path is not a real thing.

Amos: I have too, and I use width differently than almost everybody else I see use it, so.

Chris: Side note, one of the things people got real tilted about was that I used width to match on an error in one of my examples. They got real tilted about that because it because "width is only for happy path stuff."

Amos: No, I use it for errors.

Chris: And it’s like, actually, that's not how width- that's not what with is intended for. That's what you use width for. That's not what width was built for.

Amos: I use width for like pulling something out of a cache. And then when I get an error, the due tells me how to get it-

Chris: Yeah. I mean,

Amos: -for real, yeah, whatever. Okay.

Chris: Uh, this is just pattern matching. It's just de-structuring and pattern matching. It's that complicated, but whatever. Uh, like you're, you're ascribing more to it than–all the people who are upset about that are doing, are ascribing more, uh, convention to, width–than it actually holds. But that's neither here nor there for the moment. So the notion is with this tagging idea is that you have a width, set of width statements and you tag each of them with like the operation name and you do it by using a tuple. And so the first part of the tuple is like the operation name. And then you do a second part, the actual call. And so then if you have an error down below, you can figure out, “Well, what operation failed and let me see the error and I can do stuff with it?” If you're doing that, it necessarily means that the errors matter. It necessarily means that the error condition is a core part of the control flow of that function, like tacitly, right? Like we can, we have to, I think we'd have to agree that the control flow, that the control flow has to include errors, if you're doing this tagging thing at all.

Amos: Right.

Chris: Otherwise, there's no reason to do it.

Amos: Yeah. What I always see it used for is like, where did this error come from? Because I could have the same error tupal from two different things. Uh, but, but then, like you said, now I have, to me, I've separated where I know things about that error from where the error happens. And there's a lot more cognitive overhead in matching these.

Chris: Yeah. Now, I have to read the thing and then jump to the else block and figure out like, okay, but like, where did this come from? And why does it matter? Or I could just use nested case statements.

Amos: This is the same problem with piping to a function like handle result.

Chris: Yes. Right. Exactly.

Amos: Because you now hide that piece of information. And it's, it's really valuable information and handle result the name doesn't tell you anything.

Chris: Yeah. I mean, that was the other thing is a handful of people were actually like, no, you should totally pipe your results from previous functions into the next function. And I was like, I, we, we don't even have common ground to stand on. Like, we'd be like, like you want different things from your APIs than I do. But yeah. So the tagging operation thing necessarily means that you care about errors. And if you care about errors, then you care about errors and you should talk about them as a first-class thing. Something that I think is an important context here is that in my work, specifically at BR, there was no such thing as a happy path. I don't even know what that phrase means. Like, I get what people think it means, but it's not real. The errors matter in your application and how you handle them. So you either need to make them generic, AKA you have a big exception struct that you can then use for all your operations, so you can compose your operations arbitrarily, or you admit that it matters that they're- they're a core part of your control flow. And so you want to do things like have a giant function, say giant and be like 20 lines long. It's not that big a deal. And it would say like, check the fuse here. If the fuse is good, then do this, handle those error, you know, call the downstream service. If the downstream service comes back with an error, go look in the cache for it and just do all that in one function. Like, don't make me go read 30 things to figure out how all that works. Like, I just want to read the one function and I want to read the whole flow and see it because this is the only function that actually knows how to do all that stuff. That actually knows the composition of all those operations. And so, yeah, like if you're in that situation, then you should be more inclined to just use nested case statements because it's way easier to read. And it's much, much easier to change in the future. Much easier to change. And in fact, case statements generally are much easier to change than width statements. In my experience, in my experience, it is way easier to swap out various bits of a case statement than it is to go into a width, figure out where you're calling certain operations, figure out what they're returning, understand all that stuff. It is much easier to maintain a nested case statements long-term. Much, much easier. And on the flip side of that, if you don't try to maintain case statements, where you will naturally end up is having to figure out how to unify all your various errors. And in which case you will end up back at re-inventing def exception, having a, you know, an exception that you use for everything. Like that's the end result is that you're going to end up reinventing that. Width is really, really, really good, when you can have a unified set of errors that pop out of it and you just fail out wherever. Like, if you're going to use it for this so-called happy path thing, you want them all to return the same type of, of, of structure so that you don't care where you fall out. That's the benefit of the width. You don't care where you fall out. If you care where you fall out, you don't want width, and that's the thing about else. Like, it's not that you shouldn't use else. I use else all the time and I use it for like logging or updating a counter or updating a metric or, or, you know, updating tracing, like what- else is useful. But if you're handling every type of error, you really it's way less maintainable and it's way harder to understand and change in the future.

Amos: The, the times that I've seen and heard a lot about complaining about using case instead of width, this is when you're matching on all of those cases when there's a lot of code in there. But to me, that's where I, where I pull a method out, like I match on the case. And then I have a- I have a function that has a name that means something to me because I probably need to do that in multiple places, too. And then I pass to that function.

The thing about the pipelining, too, outside of the width, cause that's related, I think is in that handling, is the now if I have four functions that I'm piping into and there's an aircraft for the first one, now each one of them has to have two function heads, one to pass that error on, which I guess is what width is supposed to be for. But I find it's not often used that way very well, because then you have the, the else and all the matches inside the else.

Chris: Yeah, no, I agree with that. Um, along the, along this happy path thing, I just think it's important to re-emphasize this, like, I don't think it's kind of, I kind of don't think it's a real thing for most of my work. Most of the work that I personally do, just doesn't amount to like, like the, the, like when everything goes well logic is uninteresting. And it's also, if all, if you can get away with caring about just what happens when things go, well, I really envy you because that means your application is, is like, it doesn't have to deal with error conditions that much, that's what it implies to me. And it so happens to be that a lot of the work that I do is about resiliency and about dealing with errors directly. And the work that we did at BR is about dealing with errors directly because you want to build a system that gracefully degrades. And so we ended up acknowledging that there's really no way that you can just say, “Well, I call this service and this service and I compose the results and I move on.” It's like, yeah, that's not, that's actually not useful because what you care about is I call this service, but it fail, so I'm going to try to get it from cache. But if I can't get it from cache, I'm going to have to go over here and try to do this thing and degrade this part of the result, or I need to fail this part of the result but that part of the result is critical to this request. So I'm going to just have to raise in this case. It's like, those are choices you have to make. And there's not really, I mean, there are generic ways to talk about that. And you could do, you know, functional compositions in certain ways with certain data structures that rhyme with, you know, that, that, that all rhyme with monad, uh, to, to talk about that stuff. But whenever you lean on those things, you're necessarily assuming that you can treat different fallback scenarios generically, and that's really not often the case in my experience. It's good to get to that position, but when you're getting there, you typically can't. So you kind of have to, you kind of have to figure out what's permissible, what's not. And- and, and you have to deal with that stuff directly. And it just leads me to a point where I'm like, “I can use width for certain things.” And I do, and I enjoy it, but there's just a lot of stuff where I'm like, I would, you know, there's, there's an example in the blog post, it's like a 15-line function that- that shows calling fuse and calling a service and handling those errors and calling a cache. I've written that function, people were like, “Yeah, but like, you'd break this up.” And I'm like, “No, I actually would, this is literally what I would push to production.” This, this block of code right here, I have written before. It's basically verbatim what I would write it. And I would leave it in this one function call so I could see it all because anybody can read this in artc and understand what's happening here. And if I force them to go read five other functions, it's going to be less maintainable and less useful.

Amos: I don't know where to go from here.

Chris: Well, what's- what's the- what's next on your list?

Amos: Well, I'm running out of time, but there's the, like, I call it the bonus of your blog post, it’s the very last thing. About using for when checking collections and tests. It's perfect. Like, I don't, I don't know that I'd say more. That is.

Chris: And yet some people still disagreed with that one. It’s hilarious, really, actually.

Amos: When I write tests, I'm trying to make the errors tell me what's going on so I don't have to actually dig through and figure out what's going on. And when you use things like enum all in a test, then you just get, well, it should have been true, but it was false. And now I've got to dig in to figure out what's going on.

Chris: Exactly.

Amos: So tell me where the error is. That's what I want a test to do.

Chris: Yeah, exactly. Yeah. And that's, that is easily the best way to do that. That's a trick, I don't, it's so funny. Like I was bouncing these ideas off of, uh, my friend, Matt, who I work with now.

Amos: Rock climber, Matt?

Chris: Yeah.

Amos: Hey, tell him, I said “Hi!”

Chris: I will.

Amos: Hi, rock climber Matt.

Chris: And it was funny cause we were talking about this and we both were talking about how we both use for. And it's like a thing that we arrived at individually, but it's like, yeah, you don't get good error messages is you use, you know, enum dot all. Cause it just says, “Nope, it's not true,” or whatever. And you actually want to know that stuff. And so slipping, like, uh, swapping it around like that, uh, and using for if you're checking collections of things is very useful.

Amos: The funny thing is, is I know this and I use it but I haven't internalized it yet. So a lot of times when I'm first writing the test, I just do enum all and then move on. And then as soon as it fails and I'm like, ugh, it didn't tell me why. Then I go change it to a four. I got to get that internalized.

Chris: Yeah. Yeah. For sure. So yeah, it's a good post there's lots of stuff in there.

Amos: I really enjoyed reading it.

Chris: Hacker News disagrees with basically of it. Um, which I assume means that most of my ideas are good. So my favorites too, my favorites too, where the people who are like, “Oh, well this person clearly doesn't understand Elixir conventions or idioms. It's very clear that they have not spent time with Elixir.” I'm like, “Okay, dude. Sounds good.”

Amos: I mean, I agree with them, but-

Chris: Anyway, but it's- it's yeah. I hope it helps people. It's definitely like, I don't know, at the end of the day, it's like, it's stuff that I- these are all ideas that I utilize, um, and encourage others to utilize because I do believe they lead to more maintainable systems long-term and if that's useful for people I hope, I mean, I hope it's useful for people. It seems to have struck some kind of chord with people.

Amos: I just say before you write it off, try it. Your mileage may vary.

Chris: Yeah, yeah. Yeah. I mean that like, you know, yeah. You might have totally different opinions than me on this stuff. That's fine too. I- it's just been my experience that this, this stuff tends to be easy. It leads to code that is easier to maintain a year from now and to understand what it's doing, so. And, and building systems out of less code, like, you know, code is lines of code or are a, lines of code are a liability. So if you can have less of them, but then some of them, some, you know, within some reasonable amount, you're kind of better off. That's my take.

Amos: Good job.

Chris: Thanks. Anyway, we're going late. We had to start late.

Amos: I feel like we're ending this like kind of mellow compared to how we normally do, but-

Chris: No, that's good. Yeah. We got to keep this tight. You gotta get out of here.

Amos: Yep. I got to go. Thanks for hanging out today. I missed it. I'm ready to get back into it.

Chris: Okay. Yeah, we'll be back next, next time. More next discussion.

Amos: Yeah. Hope maybe next week, but it's summertime. So who the heck knows?

Chris: Who knows? Who knows?

Amos: All right. See you later, Keathley.

Chris: Later.

Amos: Have a good day.

Episode Transcript

 

Amos: Welcome to Elixir Outlaws, the hallway track of the Elixir community.

 So it's been a good morning.

 Chris: It has been a good morning, so, okay. So let's get this out of the way. And then we don't. So we're not gonna talk about this anymore.

 Amos: Let's not talk about it at all.

 Chris: As I think we should, we should at least acknowledge it. So as of, as we record this, it is Thursday, the, uh, January 7th, we're still all a little bit recovering from the fact that, uh, yesterday armed riders broke into the Capitol building. Uh, and we're sort of like dealing with that. If we come out sounding, we're not gonna talk about it anymore after this, because smarter people than us have talked about that. And it would just devolve into like depression and whatever. Uh, so we're not going to bring it up anymore, but if we sound a little off that's, that's, what's going on in the back of our minds right now. And, uh, we hope you're all staying safe and healthy and, and happy out there in the world.

 Amos: You set it all. It's perfect. All right, let's move on.

 Chris: All right. Um, let's, let's talk about let's, let's give the people what they're here for, which is an escape from this, um, blasted hellscape and let's talk about Elixir.

Amos: Yeah. 2021 is going to be a good year.

 Chris: Yeah, it's already as bad as before.

 Amos: Alright. Uh, there was recently a new version of Elixir, really just a few bug fixes, pretty nice.

 Chris: Nice patch patch, release.

Amos: A patch release. Nice. I was just trying, I'm trying to keep up with the good things that have happened so far this year. So we've got that.

 Chris: I got a 3D printer.

 Amos: Holy crap. Uh, we should talk about that cause I've never had one. And I, I look at them like every, probably two to three times a year, I look at them and then I get so overwhelmed by which one of these dang things should I buy.

 Chris: Yep, absolutely. Absolutely. And there's a huge amount of price variance as well. Uh, which you're probably not, well, I mean, you could certainly do the research on into it, right. But as an, as, as a neophyte, it's hard to, uh, determine where you want to spend your spend your coin and you can go, you can get really, I mean, okay. Let's caveat again with, you know, you're talking in, in, you can get a 3D printer in the hundreds of dollars. That's kind of the, the, you know, that's the target amount of money that you're talking about. Right. And you can get that in the low hundreds of dollars or the medium hundreds of dollars or up to the thousands of dollars. And it's really hard to know, like what it is you even care about, uh, as a, as a novice, I feel like, So yeah. So I've asked related about it for a long time and it was always sort of, I mean, I've done, I've played around with it with 3D printing before we had a 3D printer back in college, but it was one it's like before, you know, this is in the era where the Arduino was still like a new fancy, like toy project for people, you know, it's like, that's kind of the era you're talking about. And so I think this 3D printer was like, I mean, I'm not, I think it might've been like a hundred thousand dollars. It was the university's 3D printer and it was the, you know, it does it all out of the, um, I'm forgetting what the technique is. It's the one where you put like the effectively, you know, printer, toner into it and it fires a laser into it and hardens it. I don't remember that. What that's called. It's not resin as it's not resin, like liquid resin. It's like a powder. It's basically toner. It's basically printer toner that you then like, you know, harden. So in any case, yeah. No, but it's been super fun. I got a, any Any Cubic I-3 Mega and I've been, you know, toying with just getting it dialed in and leveled and playing with.

 Amos: Have you printed anything?

 Chris: Oh, yeah. Printing all kinds of stuff.

 Amos: Do you have anything on your desk? You can just lift up.

 Chris: I don't, I literally don't have anything. No, no. I have my dice jail. I made a jail. I made a jail to put dice in that are misbehaving.

 Amos: All right. I'm going to describe it to the people. If you can hold it up. Can you?

 Chris: It's not, it's not in here currently. It's it's out. Um, um, I sprayed filler primer on it so I can paint it.

 Amos: Oh, cool. Cool. Sounds fun. I should just go use the one at the libraries. What I really should do. Yeah. Well, as soon as the library is open again, I can't go inside. Bring me books to my car, but I can't go inside.

 Chris: Yeah. That's basically our situation as well. I think our library has two, um, printers, like two 3D printers are used to, I don't know what they have anymore, but I would say like, it is simultaneously more difficult than you would. If what you want to do is sit down and print stuff. It is more involved than that. You're going to need to learn more stuff than that simultaneously. It is not nearly as complicated as it seems to be. When you first look at it, you can pretty much sit down, fiddle with the thing, get it leveled and start printing stuff. And yeah, like some of your prints fail and you need to like tune it and get it dialed in. But it is kind of like, it is a little, for sure. Certain operations. It is a little bit like clicks of buttons in it in a way it goes.

 Amos: Sounds like being a barista. You can spend your time dialing in the machine or you can just click a button and you can get something that's okay.

 Chris: Yeah. Basically, I'm like, whatever the Tim Horton's of 3D printing is, you know what I mean? Like, um, I'm the Dunkin Donuts of 3D printing right now. Like, um, I like I'm uh, I'm it's serviceable at best.

 Amos: They have good coffee.

 Chris: Tim Horton's or Dunkin Donuts, or both?

 Amos: I don't know.

 Chris: Do they, though? Do they though, Amos?

 Amos: Well, it's, it's, it's decent. They had some of the best coffee before I was really the coffee snob I am today.

 Chris: But I mean, yeah, I get it. I grew up on Dunkin as they say.

 Amos: You are in the south.

 Chris: Yeah, Florida specifically Circle K. Actually Circle K was our, we did a lot of Circle K.

 Amos: Okay. And Waffle House. I remember standing on the street in Augusta, Georgia. And you could look down the road and you could count eight Waffle Houses. And they would be full. They would all be full there. It was like two o'clock.

 Chris: Right. Well, yeah. It's a queuing theory problem.

 Amos: That's true. That's true. Are we going to turn this into Little's Law now?

 Chris: I mean, that's, it's queues all the way down. Let's be clear. It's always queues all the way down, but no, 3D printing is super rad. I've made a dice jail. I made some toys for the kids. I made one of those. I mean like a flexible dinosaur for the kids. I started printing, um, mask straps for all of us. So there's basically like you, if you have a mask, when you're going out, you hook it onto these loops. It sits on the back of your head and you hook the mask straps into the, into this plastic thing that you print sort of con you know, bins it's thin enough that it will sort of bend and conform to your head. And then it's a, it takes all the strain off of your ears and hold your mass tighter and such. It's been actually super convenient.

 Amos: That's always what bothers me is after I, if like, if I have to go someplace and I have my mask on for like an hour, my ears start to hurt.

 Chris: Yeah. Yeah, exactly. So this is, this takes all the strain off your ears and it holds your mass tighter to your face. It's good stuff. It is good. It's good stuff. Good stuff. So anyway, 3D printing is super rad. Do highly recommend it. I had to buy a pair of calipers, which I did not own. And I really had to talk myself down off the ledge of buying the really nice calipers. They were in, they were in the basket.

 Amos: Uh, honey, this one measures in Pico meters.

 Chris: No, I mean, really like really well. And it was like, I was, I had justified it to myself about five different ways of like, well, these are the, these are exactly like what I used in college, you know? And like, I know how to use these already. And I was like, looking at the price tag, look it at them. And I'm like, this is the only calipers I'll ever need to buy it. They, you know, yeah. They have the thumb wheel and the whole thing. Yeah. They were really nice. But I talked myself out of it. I talked myself myself, I bought the $5 pair instead.

 Amos: And, and that whole, like, I already know how to use them thing. They're calipers. They're really not that difficult.

 Chris: They're just digital caplipers, man. They're not that hard to figure out. And also like, I don't need to measure anything, but within any amount of precision really.

 Amos: Right. Unless you're going to 3D print a car.

 Chris: Yeah. Or, or, I don't know. I don't even know. Like, I guess for electronics, maybe I would have done it, but like yeah. Nothing I do needs to be within that amount of tolerance. So I bought the cheap one and stuff and it'll be fine. But in any case, yeah, I'd highly suggest the 3D printer is super fun. That's been a really fun way to spend some time.

 Amos: I've always wanted a 3D printer and a laser cutter.

 Chris: Yeah. Well, there are there, you know, that there's now like dual morals. Like you can get one. That'll do both. Yeah, man. I mean, I don't know, like, I'm not going to tell you to spend a bunch of money, right? It is, it is money for, for reals money and like, who even knows if you know, who even knows what the world's going to bring tomorrow.

 Amos: Right. I mean, our money might not be worth anything tomorrow anyway.

 Chris: Well, you actually literally do not know that at this point and it's on the table is, is the scary, the really scary thing that Vanguard account you've been working on may or may not be there tomorrow. So in any case, uh, but I do recommend it. It's super fun. Um, it's a great little new hobby. I'm planning on printing out some stuff that I can like do some art projects and go, they break the airbrush out and start, you know that along with that.

 Amos: Nice. That sounds like a lot of fun. I need to...

 Chris: It's been super fun. It's been super fun. Nice to have a distraction,

 Amos: You need to put your printer online so I can just send files to it. And it'll print in the middle of the night. Yeah.

 Chris: Are you going to pay for shipping to get me to, to for, for the prints?

 Amos: Yeah, sure. It's what you need to do.

 Chris: I'll just print you stuff if you want stuff. But you need to, what do you want to print?

 Amos: I don't know yet. There's all the times that I'm, I'm at home, like doing something -

 Chris: You just want to print.

 Amos: I think so, but I it's like at home I'll be doing something and I think, you know, this, they should have a tool that does this and I can like picture a design in my head. That's super simple. Um, for a long time, I thought of this thing. Uh, so we have an SUV and like cleaning the top of like an SUV. And we used to have like a 12-passenger van. Cause we're a giant family - cleaning the top of it. It's nearly impossible without like a ladder. And I was like, man,

 Chris: No one has invented technology, right? To fix cleaning off the top of a large car.

 Amos: It would be cool if I could like step up there, if there was a step on the car. And then I thought about, uh, the door, the little, the U-shaped thing that the door actually latches into is that if you could have some kind of step thing that hooks on that, and then you could step up and then the other day I saw one advertised and it seriously looks like it's just been 3D printed.

 Chris: See, that could have been you.

 Amos: I would be living on the, as sold on TV store front right there.

 Chris: You'd be on the as sold on TV isle in Bed, Bath and Beyond.

 Amos: Which is the whole store. Isn't it? Uh, man, that would be super awesome.

 Chris: I'm sending you pictures of things that I have 3D printed.

 Amos: I'll have to add, I'm going to have to I'll describe them to people.

 Chris: I mean, it's a dinosaur, it's a thing that hooks masks.

 Amos: Yeah. He's got the, uh, it's orange. The thing that wraps around the back of the head, um, and holds the mask and it's got different, um, points to put the mask around for the different sizes of your head. So you don't have to print individual ones for each person that are custom

 Chris: Well, each mask. Right. It's a little bit different. Has different loops.

 Amos: They kind of look like, um, it, it looks like it's made out of a pillar, uh, in ancient Greece. It's got a little scroll across the top. That's what they hook across. Oh. And then the diced jail. It's got a little lock on it. A little round bar.

 Chris: It doesn't actually not, not working.

 Amos: It's not a working lock. It's just a pretty luck.

 Chris: I'll make it the show art. How about that? So you don't have to do, you don't have to do this.

 Amos: Well, this is fun. This wait, this dinosaur looks like one of those snakes that I had when I was a kid. Can you hold on that tail? And it wiggles wiggles, but it's.

 Chris: It's a dinosaur that flexes. It's very popular. It's a very common thing that people have 3D printed. I made one for my middle child.

 Amos: Everything is in orange. So now I know that if I'm going to buy Chris something though that it should be an orange because that's the color that he approves of.

 Chris: That was the color that was available.

 Amos: Not your favorite.

 Chris: No, I mean, I like orange. I got other - orange is just, you know, I didn't pick it. I didn't look at that. And it was like, Ooh, it's orange.

 Amos: I would have orange, orange and purple are my favorite colors,

 Chris: Orange and purple orange. I would not have pegged you for a purple man.

 Amos: I like a dark purple.

 Chris: Orange is a good - man. You want to, you want to make a design pop. As they say, the client comes back to you and is like 'it needs to pop'. You just gotta toss a little orange in there.

 Amos: I bought an orange shirt. One time. I have the wrong skin tone to put that thing on.

 Chris: I bet you basically just become one orange blob.

 Amos: I bought it. I got home. I put it on. And I was like, uh, I can't see myself. We need to get rid of this shirt. And so I, I took it back. Oh man. All right. So we we've got 3D printers.

 Chris: Yeah. 3Dprinters are super fun, you know? I mean, I'm not telling anybody anything that they probably don't already either suspect or know, but 3D printers are very fun.

 Amos: Yeah. I can't imagine it being not fun unless you got like a bad 3D printer. It's like buying, buying a bad guitar is your first guitar will make you not want to play guitar. Right. If you can't keep in tune.

 Chris: Yeah. The guitar is hard to play. Yeah. That's a, that's a surefire way to not play any more guitar. Right.

 Amos: I could see the same thing with the 3D printer. So finding one that actually is useful. You have to have a certain tolerance level. Right. So what, what do you have going on? Uh, like now maybe not work, but programming world wise.

 Chris: I don't have anything.

 Amos: Cool. Sometimes that's really nice.

 Chris: No, I, um, no, I got, I got some stuff. I don't know. I I'm really the energy levels I have for programming stuff. No, actually, no, I have something we can, I have an interesting thing we can talk about. So sorry. I started working on this book. I bought the domain. The intro should be published soon. Now a lot of people basically come back and just be like, please publish this now.

 Amos: What's the domain link?

 Chris: It's uh, well it's not, uh, it's not live yet,

 Amos: But will it be by the time this is out?

 Chris: Yeah, yeah, yeah. It's just gonna, it's just stateful, Elixir.com is the domain. That's where it'll all be. So in any case, that's where stuff will be eventually, maybe by the time this comes out. Probably not.

 Amos: Just so people know, I don't have any inside knowledge on stuff like this when it comes to Keithley. Sometimes I'm just as surprised as everybody else when it gets heard. I just get to hear it a week ahead of time.

 Chris: Yeah. Yeah, exactly. And it's just going to be the intro up there and uh, and none, you know, I'm just trying to see if it's interesting to people or not, and probably start moving forward with that, uh, slowly but surely I got to figure out how to do some sort of newsletter I think is probably what I'm going to do. Um, yeah, I'll probably link to my GitHub sponsor's page just in case anybody does want to throw some money at the problem, but, uh, but not explicitly expecting any. Um, and otherwise it'll just be sort of a free resource for people.

 Amos: Uh, can you, can you, do you have like your first topic in plan?

 Chris: So my, okay. But yeah, so my main idea, this is, here's the thing I reserve, uh, absolute authority to change anything I'm about to say. Perfect. So don't get your hopes up. Uh, I think there's two super big parts of this. Um, one is a lot of the knowledge that's out there about the distributed parts of Erlang. Um, don't they give you like all the tooling, but then they're sort of like, “Eh, have fun,” and that's not like you need something that's more opinionated. I think. So, um, this is going to be a surprise to everybody, but, um, I'm super opinionated about that. And so it's going to be filled with my opinions on like, here's how you build. So I literally broke Amos. Wow. Uh, so, um, it's going to be filled with like sort of my opinions about it and, and, and very real world stuff. Like, I want you to come away from reading this with like, okay, I know how to build into play this stuff. And then the deploy part is also important because deployments with stateful systems are totally different. You have to think about them totally different than you think about deployments with a non-state with a stateless. So a so-called stateless,

 Amos: Is this the, uh, there be dragons book?

 Chris: Yeah, I hope so. Um, so the goal I believe is going to be, to demonstrate how to build a few real-world things. So we'll actually sort of go through a couple of different projects. I'm still planning out the roadmap as it is, as it were a couple of different projects. So you build basically from scratch that do distributed real world-ish distributed stuff. And at some point it has to be slightly contrived because like, you gotta, you know, you, you can't write the universal, you just have to write a book that people actually read. So, um, but hopefully it'll demonstrate. And the, the goal is to demonstrate a bunch of really important real world distribute systems properties. So like, how do you look up processes in the cluster? Like, how do you, uh, how do you deal with data? You know, how do you, what does item potency look like? Uh, what is, how do you deal with ordering, you know, all this sort of stuff, how do you do a failures? So that's really the big goals.

 Amos: So if anybody took your distributed systems, uh, trainings in the past, should do they, is that kind of like a preview for this? You think?

 Chris: Yeah, I would say that that would be like a, a lot of that stuff will carry over partially just because I've already written all that code. Some of it we'll get a lot of it, we'll get more detailed and we'll do the thing that we it's just really too hard to do in the training, which is actually talking about deployment and sort of say like, okay, here's how you actually deploy the, deploy the stuff. And here's the things you have to think about. And then another part of it's testing, like how do you spin up a cluster and then break it so that you can induce failures.

 Amos: Cool. Uh, that sounds really awesome. Um, so deployment wise are like, when you're talking about deployment with it, um, what are, what are the biggest issues that you, you think you need to address that you see a lot of people not doing currently?

 Chris: Yeah. So the biggest issues with are one part of its discovery. Like, you need a way for the nodes to discover each other just with today's setups, right? It's also, it's possible obviously to have like five dedicated boxes with IP addresses that you just slap in like a file somewhere or putting Etsy, you know, house or whatever you want to do. And that's also reasonable and I'll probably like try to address some of that as well. But I think for realistically, a lot of people who are deploying existing systems to their cloud infrastructure stuff, which is probably running in somebody's Kubernetes somewhere, um, you know, you need a way to discover this stuff. So you can either use service discovery. That's built in like Kubernetes has a mechanism for finding other nodes, or you need to like construct something yourself. And, um, and the downside to using something like Kubernetes service discovery is it doesn't help you at all with leasing or locking that it, it doesn't give you enough primitives that you need to be able to do other things as well, to like manage operationally, like manage certain deployments. And then all that goes back to, to like how much you care about your data. And so that's the other side of it. I think that's, that's the other really big part is once you start putting data in your cluster, you necessarily have to figure out how much you care about that data. You know, how much do you users care if you see the wrong stuff, do they care a lot or a little, and if you put important stuff in there, then now your deployments get really hard because it, well, I say that your deployments get really hard if you also, you know, want to achieve certain uptime guarantees, if you can afford just to like take the whole cluster down and like spin it back up again, then it's not that bad.

 Amos: So I guess you'll have some like hot code deployments.

 Chris: Yeah. Um, um, so, um, so that's part of it is I want to talk through hot, hot code deployments, uh, in the real-world stuff. But I also realize that most people don't want to do that. And, um, really just want to see, like how do I get my Docker containers to talk to each other? And so that's going to be interesting too. So I want to talk through a lot of patterns on like, here's how to maintain state in your application and also sync it to a database, like just make that the driving force behind the book, um, and also to play it. So those are the, those are the main things. Um, and because of that, like the book's going to assume, you know Elixir it really well. And to some degree that you're comfortable jumping back and forth between the Elixir and Erlang, it's not going to be an easy beginners’ book. Um, but there's plenty of resources out there that do that, that address those things. This is not going to be one of them. Cool. Like you need to read all that other stuff. What do you use? And that's probably good anyway, like you don't want to do this first anyway. That's true, honestly.

 Amos: Figuring out how to deploy a single instance of an app is the first step to even being able to deploy a clustered. Right. And dealing with the cap theorem and, and all that, that stuff is, I assume you're going to be into that in this book-to-book newsletter, whatever you want to call it.

 Chris: Yeah. So, yeah, it'll be really, I think it will be fun. Um, at least for the first few chapters, and then it'll become a slog, but, uh, I'm having fun so far. And I it's made me do some other research into sort of different parts of Erlang and the beam and that sort of stuff. And that's been also pretty fun. I've been breaking amnesia or Nisia as they say,

 Amos: Is that really that difficult?

 Chris: No, it's no, actually here's the thing it's super not, it was more difficult to get it to even kind of work,

 Amos: Right. Like that's, that's what I've seen with local development. Even in local,

 Chris: It was harder, dude, just getting three nodes running locally on my machine. I kept screwing it up. I found, you know, uh, a veritable cornucopia of ways to break in Amnesia locally before I ever even got to like the distributed part. I mean, and it's all like, quote, unquote, working as intended, which is to say, you know, there's no bug report to file. It's like, you did it wrong. You just issued the wrong operational commands to it. But man, like a little big oomph.

 Amos: Yeah. I've, I've run into a lot of issues with being able to get data consistency with amnesia and a cluster and being able to like the whole, I just blanked out. Never mind. Forget it.

 Chris: Okay. All right. I'm going to take the wheel.

 Amos: I had data consistency issues in, in amnesia whenever I had, um, not even bad flaky networks, like with one or just the network in general. Uh, I couldn't imagine trying to run it on any network that had, that was very flaky.

 Chris: Yeah. I mean, so the tests were basically like, okay, so we have a distributed Nisia, let's see what happens. If you have three nodes and you start to partition them just to say, you let them live. You don't take the boxes down, but you cut off their ability to talk to each other. And so I wrote a, I built a repo that does this and, uh, and all this started because I was like, people have been talking for whatever reason, like people aren't talking about amnesia more and more recently. And, um, I was sort of fascinated. I was like, I should go in and play with it. And like, I've, it's been a long time since I looked at this, I looked at that amnesia. And so I should try to figure out like, what's going on with it. So I was digging into it and like just trying to get networking stuff, working and whatever. And yeah, no, it's like, if you partition some nodes and then you just issue rights to either of the nodes and you don't do any and you just leave the default settings the way they are ill is very, I mean, it obviously, right, this is well known. It just very happily overrides data. And then they refuse to start talking to each other again, or, well, rather like there, they just alert you and say that they're partitioned. And then, you know, they're like, wait, we have inconsistent state fix it. And then, and then you're, it's up to you to fix it. And, and so, yeah, there's no healing, right? Like you have to determine the healing part of it. And there are modes where you can tell Nisia to do things like, uh, use a majority. So when you build the tables, you say here's all the notes, here's all the disc copies of this. Oh, here's all the nodes that have discoveries of this. And that that's part that's essentially what becomes the cluster at that point for that table. And so you can put it in majority mode, which is like, if I can write this to three other nodes and I have five nodes in my cluster and three of them, get it, then, then we'll consider that good enough. And you're on the wrong side of that partition. Then you aren't allowed to write stuff, but like that comes with its own caveats, which is that, you know, it's still Nisia reads to you. And at which point you just, your reads are wrong, which is like, that's not that surprising. It's, it's like you expect, it needs just sort of like a cash, right. More than a database. Like it's, you know, it's mostly all just in memory and then occasionally backs it up to disk, do some journaling and stuff like that. Uh, well, uh, depending on the mode that you're using to write to desk, if you're using desk copies, it, it writes it as like a journal file and all this sort of stuff.

 Amos: So do you think, think there's, uh, depending on what the date is you're putting in the, do you think that there is a, a good way to tell it how to heal that would work for many circumstances?

 Chris: There's two ways to do it. Oh. And by the way, healing, it is not is non-trivial. We also like healing. It's super non-trivial because if you don't have ordering setup, right. Go, this goes back to what we talked about last week. If you don't have some sort of clock give up on ordering, right. And now you're just picking you just pick one at random. There's no such thing as last write wins. If you don't have ordering, when you, when you feel right. You know what I mean? Like, it, it there's no time. If there's no timestamp, then who's the last, like you don't even know. So just pick one at random and you hope it's the right one, or you go, do you try to do something fully? You know, you, you try to build some sort of CRDT thing, but even that's like, you're going to have to have some sort of clock for ordering prefer most of CRD Ts that don't need to be garbage collected. Well, like, like brand-new, cutting-edge stuff, you know, like CRD already work well, but old, like CRDTs as they exist today, like you have to garbage collect them. And it takes a ton of time. And like, they're, non-trivial. So that's one side of it. Like you can go full AP and attempt to rebuild your Nisia tables by just overlaying and picking the one that you believe, you know, is the most recent, or you store a list of like operations that you're then going to need to like issue to each other and like somehow reconcile that state that way. Yeah. It's like, that's, that's one way to do it. Or I think the, the flip of it, and I think this is what the rabbit people are working on. As far as I understand is you don't allow yourself to get into a bad state. Right. Which is that you actually build real consensus over the top of it. So like, once you, if you're, you know, you only allow that to write to tables based on it consistently.

 Amos: So when you're, when you're, I'm going to go back a little bit and what you were saying, uh, sure. We're talking about ordering last week in ordering in amnesia. So I know that they have an ordered set in amnesia, but do you know how they order that across the cluster?

 Chris: Well, I mean, the ordering is just done in an order. It just in the ordered set table, it's just done by a key, like, and you pick the key.

 Amos: Okay. So you have to tell it what to order by. So you could put a, uh, HLC or vector clock or something in there?

 Chris: If you want to use it to the store operations. Yeah, yeah, yeah. That would be, that would be a thing you could do and then to rebuild, but to rebuild the world, it gets tricky because you need to ensure that everyone has seen all the same stuff you need some way to ensure that here's the scenario, here's the scenario. You have two notes, they should be talking to each other, but for whatever reason, they're not, maybe you're doing a deployment or you're just haven't connected yet or whatever. There's a problem. Change the cookies, you know, who knows what? Right. Most of the time I find in production, like real-world net splits are really just synonymous with deployment. So, uh, cause you're like, I think the thing's alive, but it hasn't like connected yet or it's, you know, rolling or whatever. So in any case, so it's, so they're not talking to each other just yet, but you issue two different rights and you store those as a list of operations. This is the whole idea. This is like, you know, uh, event sourcing without, without the baggage is, you know, you start issuing these, these operations. And then as long as they're all, you know, competitive, you'll arrive at the same stuff. Right. The implicit statement in there is that if they eventually see all the operations,

 Amos: So when they heal, you have to get them to share events. Yeah. All of it.

 Chris: So how do you know, how do you know if you've seen all the operations, Amos?

 Amos: If you've gotten all the operations?

 Chris: Oh, I know. How do you know? But how do you know.

 Amos: Neither side knows, right?

 Chris: Right. So you need a - now you need a new thing. Now you need a new thing on top of that. And it, and it, and it rhymes with trees typically. But really though, that's you use a data structure like that, right. That can start to tell you, like, assuming that there is a place to have that thing. And you also send that thing all the way around that you start to say like, that is, you know, we're missing bits, we're missing chunks. We're missing parts of it because the tree says we should have these other things.

 Amos: So it feels like you could, you could get, if you design your data with a lot of these things in mind, then amnesia could work for you.

 Chris: I mean, yeah, yeah. It still has that whole, like, it's not dirt, it's durable-ish. You know, problem, by my opinion is like, you should view - You should view Nisha the same way you view a single node, redness. Like it's going to give you basically nothing in terms of clustering, nothing real that you want to, you actually use right direction. And you're going to have to figure out how to make that work. But, and it has basically the same kind of like journaling semantics, right. Of like, it's going to flush this stuff to disk periodically and probably it's going to save it all. And if not, like it needs to not wreck your day

 Amos: Except for when it doesn't.

 Chris: Right. Yeah, exactly. And if it doesn't, it doesn't and that, and we just deal with that, but ideally it saves it for the most part. It's going to save it. And then when it's not there, you just fall back to the real data store. Like if you treat it like redness, you know, you're in pretty good shape.

 Amos: I want to caveat that with, if you treat it like you should treat Reddis because I've also seen Reddis treated like -

 Chris: Primary database number,

 Amos: Our, we know it's never going down or it's never going to lose data. And then when it does people crap their pants, um, not my redness, my redness never goes down.

 Chris: My redness is here to last.

 Amos: Uh, I, that doesn't that just go against what reddis says it is anyway. Um…

 Chris: I mean, you know, it's programmers know the benefits of everything and the trade-offs of nothing. I keep feeling it over and over. So yeah. I, I think, I don't know, Nisha could be really interesting in terms of like a cap solution. It's, it's neither it's neither C or A, like, it doesn't pick either of them really. I would, in my estimation,

 Amos: Does it try to leave it up to you more? Or is it like,

 Chris: Everything up to you, but it's also just a product of its time. You know what I mean? It's like invented when you were literally running a machine that had a, where the network was a physical back plane. No. Yeah. You know, I mean, it's like, it's just the, the, the, the, the constraints were different. And so if the network was down the back plane didn't work and like, you had a bigger, a much bigger problem at that point, which basically meant nothing. Right? Like the whole system was down. If the network was down, right. That's the, because the physical back plane would've broken at that point. So I know it's, it's a product of its time. And I think people are in, I think people come into Erlang and it is unfortunate because it's one of those things that I think is a huge differentiator. Like how many languages, how many runtimes, right. Have they built in high-performance database? Not many, and can support that. You know what I mean? Like how many, how many, like, and it, it does what it, it, sorry, let me say this. Nisha does do what you mostly expect it to do in certain circumstances. And so it's really cool to see that, to have that capability. The problem is, is I it's, I don't want this to come out sounding like Nisha is broken. I don't think that that's accurate. I think people's expectations for what you, for what, for what the words database means have changed a lot, you know, database, like people hear the word database and it is synonymous with Postgres or my sequel

 Amos: Where everything is in one place. And if it's down, it's down, if it's up, it's up like,

 Chris: And it's, and like, for the most part it's consistent. Right. And it has, it's, you know, slow for certain operations, but, but you know, it's fast for others. And it's pretty much like going to reliably store your data, barring some sort of just gross oversight, right. Or just some sort of absolutely catastrophic problem. Those databases are just going to store your data. And you are not going to have to think about it. Like you never questioned whether or not MySQL or Postgres is actually stored a thing when you issue a right-

 Amos:- If you get a success back, you know-

 Chris: -It's there. And if you get a read back, you know, it was the right value at the time that you issued the read. Like there are, there are caveats to that. You don't know things that are going to happen in the future and transactions are trying to run concurrently. And it depends on your transaction level and all that sort of junk. But for the most part, you rely on your database. And again, databases are very much people. People have this association with the word database, mean my SQL Postgres SQL server, whatever it is and that's, and you just, you come to rely on a certain amount of comfort and knowing that barring something absolutely catastrophic happening, this thing is storing my data when it says

 Amos: Even, even those databases or any relational databases that provide a distributed implementation, have all these caveats. When you, when you read in there about what happens when you distributed. And even if you just turn on readers, your readers, when you read from a reader, it might not be the most up-to-date. You gotta be thinking about that.

 Chris: And you can have weird issues in file systems. Like there's, there's still that thing where Postgres can attempt to write to the file system and it can fail. And then it will crash the Postgres because Postgres, at that point goes, “Yo, something bad has happened.” And we, the safest thing for us to do is to terminate at this point. Yeah. Because like file systems are all fundamentally broken and SSDs are all fundamentally broken, you know, all these problems or whatever, right? So like, yeah, I get it. Like, I'm not here to say that post-stress will never lose your data. It's more just that, you know, the, the likelihood of Postgres losing your data is, is markedly less than redness losing your data for instance. And that's markedly less than Nisha losing your data for instance.

 Amos: Well, any, any time that you have to have cooperation between more than one system, the more, the more of those links that you have, the, I guess, less likelihood you have of getting what you want out of it without extra. There's a lot of extra overhead and work that needs to go in to make it sustainable. And then your, your trade-offs are bigger.

 Chris: And you're going to have to learn a lot more. I mean, you're, you're managing you're, you're now literally managing a database inside of your application code, which is super interesting, right? Like that's fascinating. And again, I think it's a huge differentiator as part of the runtime. I just think people's expectations for what the word database means are so different than what is giving you out of the box, especially with its default choices that it's going to be really surprising to people. And that's where people get the feeling that Nisha is broken or wrong. I don't know that amnesia is wrong to be the way that it is. I do think that people need to be aware of what it is and to be responsible about that choice and accept it for what it is of like, yeah, man, if you want a really fast in memory, cache is backed more or less by a database and you just know that you can be wrong and your recovery strategy is like, I don't know, I've got an HLC or some other logical clock. I's pick the one that's like most recent. And we'll just deal with that. Yeah. I think he's just like-

 Amos: -As a blockchain and then we own them. We throw away the, the newest 10 until we have a consistency. Okay. My, my, my internet voice is not as good as yours. No, that was-

 Chris: -That was strong. That was, that was, uh, that was strong. I mean, I hated you the moment you started talking. And so it's like, I feel like your mission accomplished might.

 Amos: Uh, I got to tell this story now because voices. So my son yesterday, we were out and he started like, I don't know. I guess the best way I can describe it as is talking hick. And, uh, so then I just started talking like Mickey Mouse, every time he would talk like that, we're out in public and I'm like, so Josh, how's your day today? And he's like, dad, how about I stop? And you stop. I was like, “Oh, you're growing up. You've decided to compromise. Growing up so fast.” It was pretty funny. The rest of the kids found it quite entertaining.

 Chris: You think your mom wants to have smokes? Oh boy, there's a deal on eggs. Okay. That's the, that was pretty much our dinner yesterday. So honey house today, now I'm going to go home and do that today. Like some smoked salmon dish. Did you see that? Six people still just contested?

 Amos: Okay. I think that's a good place to end it. Uh, seriously though. I do. I do have, uh, another meeting coming up and we have a new person that started today. Chad, welcome Chad. Um, so I have got to go get some stuff done.

 Chris: This is so bad. We can't release this.

 Amos: Because I guess, I mean, we can't, we can't go any lower or any higher than where we just were. So I guess we can just take it on out. Yeah.

 Chris: They look at, everybody looks at us and they're like, you know, now we know about those guys, but guys are high.

 Amos: High on life, baby. Chris, thanks for coming today. It was nice seeing Anna today. Uh, even though I don't think she made it into the recording, she had, she had a meeting come up. She stopped by, she has meetings. She's been super busy. She misses everybody, not Chris and I. All right. Well take it easy, Chris.

 Chris: Have a great day.

 Amos: Later. Bye.

 

Episode Transcript

 

Amos: Welcome to Elixir Outlaws, the hallway track of the Elixir community.

 So it's been a good morning.

 Chris: It has been a good morning, so, okay. So let's get this out of the way. And then we don't. So we're not gonna talk about this anymore.

 Amos: Let's not talk about it at all.

 Chris: As I think we should, we should at least acknowledge it. So as of, as we record this, it is Thursday, the, uh, January 7th, we're still all a little bit recovering from the fact that, uh, yesterday armed riders broke into the Capitol building. Uh, and we're sort of like dealing with that. If we come out sounding, we're not gonna talk about it anymore after this, because smarter people than us have talked about that. And it would just devolve into like depression and whatever. Uh, so we're not going to bring it up anymore, but if we sound a little off that's, that's, what's going on in the back of our minds right now. And, uh, we hope you're all staying safe and healthy and, and happy out there in the world.

 Amos: You set it all. It's perfect. All right, let's move on.

 Chris: All right. Um, let's, let's talk about let's, let's give the people what they're here for, which is an escape from this, um, blasted hellscape and let's talk about Elixir.

Amos: Yeah. 2021 is going to be a good year.

 Chris: Yeah, it's already as bad as before.

 Amos: Alright. Uh, there was recently a new version of Elixir, really just a few bug fixes, pretty nice.

 Chris: Nice patch patch, release.

Amos: A patch release. Nice. I was just trying, I'm trying to keep up with the good things that have happened so far this year. So we've got that.

 Chris: I got a 3D printer.

 Amos: Holy crap. Uh, we should talk about that cause I've never had one. And I, I look at them like every, probably two to three times a year, I look at them and then I get so overwhelmed by which one of these dang things should I buy.

 Chris: Yep, absolutely. Absolutely. And there's a huge amount of price variance as well. Uh, which you're probably not, well, I mean, you could certainly do the research on into it, right. But as an, as, as a neophyte, it's hard to, uh, determine where you want to spend your spend your coin and you can go, you can get really, I mean, okay. Let's caveat again with, you know, you're talking in, in, you can get a 3D printer in the hundreds of dollars. That's kind of the, the, you know, that's the target amount of money that you're talking about. Right. And you can get that in the low hundreds of dollars or the medium hundreds of dollars or up to the thousands of dollars. And it's really hard to know, like what it is you even care about, uh, as a, as a novice, I feel like, So yeah. So I've asked related about it for a long time and it was always sort of, I mean, I've done, I've played around with it with 3D printing before we had a 3D printer back in college, but it was one it's like before, you know, this is in the era where the Arduino was still like a new fancy, like toy project for people, you know, it's like, that's kind of the era you're talking about. And so I think this 3D printer was like, I mean, I'm not, I think it might've been like a hundred thousand dollars. It was the university's 3D printer and it was the, you know, it does it all out of the, um, I'm forgetting what the technique is. It's the one where you put like the effectively, you know, printer, toner into it and it fires a laser into it and hardens it. I don't remember that. What that's called. It's not resin as it's not resin, like liquid resin. It's like a powder. It's basically toner. It's basically printer toner that you then like, you know, harden. So in any case, yeah. No, but it's been super fun. I got a, any Any Cubic I-3 Mega and I've been, you know, toying with just getting it dialed in and leveled and playing with.

 Amos: Have you printed anything?

 Chris: Oh, yeah. Printing all kinds of stuff.

 Amos: Do you have anything on your desk? You can just lift up.

 Chris: I don't, I literally don't have anything. No, no. I have my dice jail. I made a jail. I made a jail to put dice in that are misbehaving.

 Amos: All right. I'm going to describe it to the people. If you can hold it up. Can you?

 Chris: It's not, it's not in here currently. It's it's out. Um, um, I sprayed filler primer on it so I can paint it.

 Amos: Oh, cool. Cool. Sounds fun. I should just go use the one at the libraries. What I really should do. Yeah. Well, as soon as the library is open again, I can't go inside. Bring me books to my car, but I can't go inside.

 Chris: Yeah. That's basically our situation as well. I think our library has two, um, printers, like two 3D printers are used to, I don't know what they have anymore, but I would say like, it is simultaneously more difficult than you would. If what you want to do is sit down and print stuff. It is more involved than that. You're going to need to learn more stuff than that simultaneously. It is not nearly as complicated as it seems to be. When you first look at it, you can pretty much sit down, fiddle with the thing, get it leveled and start printing stuff. And yeah, like some of your prints fail and you need to like tune it and get it dialed in. But it is kind of like, it is a little, for sure. Certain operations. It is a little bit like clicks of buttons in it in a way it goes.

 Amos: Sounds like being a barista. You can spend your time dialing in the machine or you can just click a button and you can get something that's okay.

 Chris: Yeah. Basically, I'm like, whatever the Tim Horton's of 3D printing is, you know what I mean? Like, um, I'm the Dunkin Donuts of 3D printing right now. Like, um, I like I'm uh, I'm it's serviceable at best.

 Amos: They have good coffee.

 Chris: Tim Horton's or Dunkin Donuts, or both?

 Amos: I don't know.

 Chris: Do they, though? Do they though, Amos?

 Amos: Well, it's, it's, it's decent. They had some of the best coffee before I was really the coffee snob I am today.

 Chris: But I mean, yeah, I get it. I grew up on Dunkin as they say.

 Amos: You are in the south.

 Chris: Yeah, Florida specifically Circle K. Actually Circle K was our, we did a lot of Circle K.

 Amos: Okay. And Waffle House. I remember standing on the street in Augusta, Georgia. And you could look down the road and you could count eight Waffle Houses. And they would be full. They would all be full there. It was like two o'clock.

 Chris: Right. Well, yeah. It's a queuing theory problem.

 Amos: That's true. That's true. Are we going to turn this into Little's Law now?

 Chris: I mean, that's, it's queues all the way down. Let's be clear. It's always queues all the way down, but no, 3D printing is super rad. I've made a dice jail. I made some toys for the kids. I made one of those. I mean like a flexible dinosaur for the kids. I started printing, um, mask straps for all of us. So there's basically like you, if you have a mask, when you're going out, you hook it onto these loops. It sits on the back of your head and you hook the mask straps into the, into this plastic thing that you print sort of con you know, bins it's thin enough that it will sort of bend and conform to your head. And then it's a, it takes all the strain off of your ears and hold your mass tighter and such. It's been actually super convenient.

 Amos: That's always what bothers me is after I, if like, if I have to go someplace and I have my mask on for like an hour, my ears start to hurt.

 Chris: Yeah. Yeah, exactly. So this is, this takes all the strain off your ears and it holds your mass tighter to your face. It's good stuff. It is good. It's good stuff. Good stuff. So anyway, 3D printing is super rad. Do highly recommend it. I had to buy a pair of calipers, which I did not own. And I really had to talk myself down off the ledge of buying the really nice calipers. They were in, they were in the basket.

 Amos: Uh, honey, this one measures in Pico meters.

 Chris: No, I mean, really like really well. And it was like, I was, I had justified it to myself about five different ways of like, well, these are the, these are exactly like what I used in college, you know? And like, I know how to use these already. And I was like, looking at the price tag, look it at them. And I'm like, this is the only calipers I'll ever need to buy it. They, you know, yeah. They have the thumb wheel and the whole thing. Yeah. They were really nice. But I talked myself out of it. I talked myself myself, I bought the $5 pair instead.

 Amos: And, and that whole, like, I already know how to use them thing. They're calipers. They're really not that difficult.

 Chris: They're just digital caplipers, man. They're not that hard to figure out. And also like, I don't need to measure anything, but within any amount of precision really.

 Amos: Right. Unless you're going to 3D print a car.

 Chris: Yeah. Or, or, I don't know. I don't even know. Like, I guess for electronics, maybe I would have done it, but like yeah. Nothing I do needs to be within that amount of tolerance. So I bought the cheap one and stuff and it'll be fine. But in any case, yeah, I'd highly suggest the 3D printer is super fun. That's been a really fun way to spend some time.

 Amos: I've always wanted a 3D printer and a laser cutter.

 Chris: Yeah. Well, there are there, you know, that there's now like dual morals. Like you can get one. That'll do both. Yeah, man. I mean, I don't know, like, I'm not going to tell you to spend a bunch of money, right? It is, it is money for, for reals money and like, who even knows if you know, who even knows what the world's going to bring tomorrow.

 Amos: Right. I mean, our money might not be worth anything tomorrow anyway.

 Chris: Well, you actually literally do not know that at this point and it's on the table is, is the scary, the really scary thing that Vanguard account you've been working on may or may not be there tomorrow. So in any case, uh, but I do recommend it. It's super fun. Um, it's a great little new hobby. I'm planning on printing out some stuff that I can like do some art projects and go, they break the airbrush out and start, you know that along with that.

 Amos: Nice. That sounds like a lot of fun. I need to...

 Chris: It's been super fun. It's been super fun. Nice to have a distraction,

 Amos: You need to put your printer online so I can just send files to it. And it'll print in the middle of the night. Yeah.

 Chris: Are you going to pay for shipping to get me to, to for, for the prints?

 Amos: Yeah, sure. It's what you need to do.

 Chris: I'll just print you stuff if you want stuff. But you need to, what do you want to print?

 Amos: I don't know yet. There's all the times that I'm, I'm at home, like doing something -

 Chris: You just want to print.

 Amos: I think so, but I it's like at home I'll be doing something and I think, you know, this, they should have a tool that does this and I can like picture a design in my head. That's super simple. Um, for a long time, I thought of this thing. Uh, so we have an SUV and like cleaning the top of like an SUV. And we used to have like a 12-passenger van. Cause we're a giant family - cleaning the top of it. It's nearly impossible without like a ladder. And I was like, man,

 Chris: No one has invented technology, right? To fix cleaning off the top of a large car.

 Amos: It would be cool if I could like step up there, if there was a step on the car. And then I thought about, uh, the door, the little, the U-shaped thing that the door actually latches into is that if you could have some kind of step thing that hooks on that, and then you could step up and then the other day I saw one advertised and it seriously looks like it's just been 3D printed.

 Chris: See, that could have been you.

 Amos: I would be living on the, as sold on TV store front right there.

 Chris: You'd be on the as sold on TV isle in Bed, Bath and Beyond.

 Amos: Which is the whole store. Isn't it? Uh, man, that would be super awesome.

 Chris: I'm sending you pictures of things that I have 3D printed.

 Amos: I'll have to add, I'm going to have to I'll describe them to people.

 Chris: I mean, it's a dinosaur, it's a thing that hooks masks.

 Amos: Yeah. He's got the, uh, it's orange. The thing that wraps around the back of the head, um, and holds the mask and it's got different, um, points to put the mask around for the different sizes of your head. So you don't have to print individual ones for each person that are custom

 Chris: Well, each mask. Right. It's a little bit different. Has different loops.

 Amos: They kind of look like, um, it, it looks like it's made out of a pillar, uh, in ancient Greece. It's got a little scroll across the top. That's what they hook across. Oh. And then the diced jail. It's got a little lock on it. A little round bar.

 Chris: It doesn't actually not, not working.

 Amos: It's not a working lock. It's just a pretty luck.

 Chris: I'll make it the show art. How about that? So you don't have to do, you don't have to do this.

 Amos: Well, this is fun. This wait, this dinosaur looks like one of those snakes that I had when I was a kid. Can you hold on that tail? And it wiggles wiggles, but it's.

 Chris: It's a dinosaur that flexes. It's very popular. It's a very common thing that people have 3D printed. I made one for my middle child.

 Amos: Everything is in orange. So now I know that if I'm going to buy Chris something though that it should be an orange because that's the color that he approves of.

 Chris: That was the color that was available.

 Amos: Not your favorite.

 Chris: No, I mean, I like orange. I got other - orange is just, you know, I didn't pick it. I didn't look at that. And it was like, Ooh, it's orange.

 Amos: I would have orange, orange and purple are my favorite colors,

 Chris: Orange and purple orange. I would not have pegged you for a purple man.

 Amos: I like a dark purple.

 Chris: Orange is a good - man. You want to, you want to make a design pop. As they say, the client comes back to you and is like 'it needs to pop'. You just gotta toss a little orange in there.

 Amos: I bought an orange shirt. One time. I have the wrong skin tone to put that thing on.

 Chris: I bet you basically just become one orange blob.

 Amos: I bought it. I got home. I put it on. And I was like, uh, I can't see myself. We need to get rid of this shirt. And so I, I took it back. Oh man. All right. So we we've got 3D printers.

 Chris: Yeah. 3Dprinters are super fun, you know? I mean, I'm not telling anybody anything that they probably don't already either suspect or know, but 3D printers are very fun.

 Amos: Yeah. I can't imagine it being not fun unless you got like a bad 3D printer. It's like buying, buying a bad guitar is your first guitar will make you not want to play guitar. Right. If you can't keep in tune.

 Chris: Yeah. The guitar is hard to play. Yeah. That's a, that's a surefire way to not play any more guitar. Right.

 Amos: I could see the same thing with the 3D printer. So finding one that actually is useful. You have to have a certain tolerance level. Right. So what, what do you have going on? Uh, like now maybe not work, but programming world wise.

 Chris: I don't have anything.

 Amos: Cool. Sometimes that's really nice.

 Chris: No, I, um, no, I got, I got some stuff. I don't know. I I'm really the energy levels I have for programming stuff. No, actually, no, I have something we can, I have an interesting thing we can talk about. So sorry. I started working on this book. I bought the domain. The intro should be published soon. Now a lot of people basically come back and just be like, please publish this now.

 Amos: What's the domain link?

 Chris: It's uh, well it's not, uh, it's not live yet,

 Amos: But will it be by the time this is out?

 Chris: Yeah, yeah, yeah. It's just gonna, it's just stateful, Elixir.com is the domain. That's where it'll all be. So in any case, that's where stuff will be eventually, maybe by the time this comes out. Probably not.

 Amos: Just so people know, I don't have any inside knowledge on stuff like this when it comes to Keithley. Sometimes I'm just as surprised as everybody else when it gets heard. I just get to hear it a week ahead of time.

 Chris: Yeah. Yeah, exactly. And it's just going to be the intro up there and uh, and none, you know, I'm just trying to see if it's interesting to people or not, and probably start moving forward with that, uh, slowly but surely I got to figure out how to do some sort of newsletter I think is probably what I'm going to do. Um, yeah, I'll probably link to my GitHub sponsor's page just in case anybody does want to throw some money at the problem, but, uh, but not explicitly expecting any. Um, and otherwise it'll just be sort of a free resource for people.

 Amos: Uh, can you, can you, do you have like your first topic in plan?

 Chris: So my, okay. But yeah, so my main idea, this is, here's the thing I reserve, uh, absolute authority to change anything I'm about to say. Perfect. So don't get your hopes up. Uh, I think there's two super big parts of this. Um, one is a lot of the knowledge that's out there about the distributed parts of Erlang. Um, don't they give you like all the tooling, but then they're sort of like, “Eh, have fun,” and that's not like you need something that's more opinionated. I think. So, um, this is going to be a surprise to everybody, but, um, I'm super opinionated about that. And so it's going to be filled with my opinions on like, here's how you build. So I literally broke Amos. Wow. Uh, so, um, it's going to be filled with like sort of my opinions about it and, and, and very real world stuff. Like, I want you to come away from reading this with like, okay, I know how to build into play this stuff. And then the deploy part is also important because deployments with stateful systems are totally different. You have to think about them totally different than you think about deployments with a non-state with a stateless. So a so-called stateless,

 Amos: Is this the, uh, there be dragons book?

 Chris: Yeah, I hope so. Um, so the goal I believe is going to be, to demonstrate how to build a few real-world things. So we'll actually sort of go through a couple of different projects. I'm still planning out the roadmap as it is, as it were a couple of different projects. So you build basically from scratch that do distributed real world-ish distributed stuff. And at some point it has to be slightly contrived because like, you gotta, you know, you, you can't write the universal, you just have to write a book that people actually read. So, um, but hopefully it'll demonstrate. And the, the goal is to demonstrate a bunch of really important real world distribute systems properties. So like, how do you look up processes in the cluster? Like, how do you, uh, how do you deal with data? You know, how do you, what does item potency look like? Uh, what is, how do you deal with ordering, you know, all this sort of stuff, how do you do a failures? So that's really the big goals.

 Amos: So if anybody took your distributed systems, uh, trainings in the past, should do they, is that kind of like a preview for this? You think?

 Chris: Yeah, I would say that that would be like a, a lot of that stuff will carry over partially just because I've already written all that code. Some of it we'll get a lot of it, we'll get more detailed and we'll do the thing that we it's just really too hard to do in the training, which is actually talking about deployment and sort of say like, okay, here's how you actually deploy the, deploy the stuff. And here's the things you have to think about. And then another part of it's testing, like how do you spin up a cluster and then break it so that you can induce failures.

 Amos: Cool. Uh, that sounds really awesome. Um, so deployment wise are like, when you're talking about deployment with it, um, what are, what are the biggest issues that you, you think you need to address that you see a lot of people not doing currently?

 Chris: Yeah. So the biggest issues with are one part of its discovery. Like, you need a way for the nodes to discover each other just with today's setups, right? It's also, it's possible obviously to have like five dedicated boxes with IP addresses that you just slap in like a file somewhere or putting Etsy, you know, house or whatever you want to do. And that's also reasonable and I'll probably like try to address some of that as well. But I think for realistically, a lot of people who are deploying existing systems to their cloud infrastructure stuff, which is probably running in somebody's Kubernetes somewhere, um, you know, you need a way to discover this stuff. So you can either use service discovery. That's built in like Kubernetes has a mechanism for finding other nodes, or you need to like construct something yourself. And, um, and the downside to using something like Kubernetes service discovery is it doesn't help you at all with leasing or locking that it, it doesn't give you enough primitives that you need to be able to do other things as well, to like manage operationally, like manage certain deployments. And then all that goes back to, to like how much you care about your data. And so that's the other side of it. I think that's, that's the other really big part is once you start putting data in your cluster, you necessarily have to figure out how much you care about that data. You know, how much do you users care if you see the wrong stuff, do they care a lot or a little, and if you put important stuff in there, then now your deployments get really hard because it, well, I say that your deployments get really hard if you also, you know, want to achieve certain uptime guarantees, if you can afford just to like take the whole cluster down and like spin it back up again, then it's not that bad.

 Amos: So I guess you'll have some like hot code deployments.

 Chris: Yeah. Um, um, so, um, so that's part of it is I want to talk through hot, hot code deployments, uh, in the real-world stuff. But I also realize that most people don't want to do that. And, um, really just want to see, like how do I get my Docker containers to talk to each other? And so that's going to be interesting too. So I want to talk through a lot of patterns on like, here's how to maintain state in your application and also sync it to a database, like just make that the driving force behind the book, um, and also to play it. So those are the, those are the main things. Um, and because of that, like the book's going to assume, you know Elixir it really well. And to some degree that you're comfortable jumping back and forth between the Elixir and Erlang, it's not going to be an easy beginners’ book. Um, but there's plenty of resources out there that do that, that address those things. This is not going to be one of them. Cool. Like you need to read all that other stuff. What do you use? And that's probably good anyway, like you don't want to do this first anyway. That's true, honestly.

 Amos: Figuring out how to deploy a single instance of an app is the first step to even being able to deploy a clustered. Right. And dealing with the cap theorem and, and all that, that stuff is, I assume you're going to be into that in this book-to-book newsletter, whatever you want to call it.

 Chris: Yeah. So, yeah, it'll be really, I think it will be fun. Um, at least for the first few chapters, and then it'll become a slog, but, uh, I'm having fun so far. And I it's made me do some other research into sort of different parts of Erlang and the beam and that sort of stuff. And that's been also pretty fun. I've been breaking amnesia or Nisia as they say,

 Amos: Is that really that difficult?

 Chris: No, it's no, actually here's the thing it's super not, it was more difficult to get it to even kind of work,

 Amos: Right. Like that's, that's what I've seen with local development. Even in local,

 Chris: It was harder, dude, just getting three nodes running locally on my machine. I kept screwing it up. I found, you know, uh, a veritable cornucopia of ways to break in Amnesia locally before I ever even got to like the distributed part. I mean, and it's all like, quote, unquote, working as intended, which is to say, you know, there's no bug report to file. It's like, you did it wrong. You just issued the wrong operational commands to it. But man, like a little big oomph.

 Amos: Yeah. I've, I've run into a lot of issues with being able to get data consistency with amnesia and a cluster and being able to like the whole, I just blanked out. Never mind. Forget it.

 Chris: Okay. All right. I'm going to take the wheel.

 Amos: I had data consistency issues in, in amnesia whenever I had, um, not even bad flaky networks, like with one or just the network in general. Uh, I couldn't imagine trying to run it on any network that had, that was very flaky.

 Chris: Yeah. I mean, so the tests were basically like, okay, so we have a distributed Nisia, let's see what happens. If you have three nodes and you start to partition them just to say, you let them live. You don't take the boxes down, but you cut off their ability to talk to each other. And so I wrote a, I built a repo that does this and, uh, and all this started because I was like, people have been talking for whatever reason, like people aren't talking about amnesia more and more recently. And, um, I was sort of fascinated. I was like, I should go in and play with it. And like, I've, it's been a long time since I looked at this, I looked at that amnesia. And so I should try to figure out like, what's going on with it. So I was digging into it and like just trying to get networking stuff, working and whatever. And yeah, no, it's like, if you partition some nodes and then you just issue rights to either of the nodes and you don't do any and you just leave the default settings the way they are ill is very, I mean, it obviously, right, this is well known. It just very happily overrides data. And then they refuse to start talking to each other again, or, well, rather like there, they just alert you and say that they're partitioned. And then, you know, they're like, wait, we have inconsistent state fix it. And then, and then you're, it's up to you to fix it. And, and so, yeah, there's no healing, right? Like you have to determine the healing part of it. And there are modes where you can tell Nisia to do things like, uh, use a majority. So when you build the tables, you say here's all the notes, here's all the disc copies of this. Oh, here's all the nodes that have discoveries of this. And that that's part that's essentially what becomes the cluster at that point for that table. And so you can put it in majority mode, which is like, if I can write this to three other nodes and I have five nodes in my cluster and three of them, get it, then, then we'll consider that good enough. And you're on the wrong side of that partition. Then you aren't allowed to write stuff, but like that comes with its own caveats, which is that, you know, it's still Nisia reads to you. And at which point you just, your reads are wrong, which is like, that's not that surprising. It's, it's like you expect, it needs just sort of like a cash, right. More than a database. Like it's, you know, it's mostly all just in memory and then occasionally backs it up to disk, do some journaling and stuff like that. Uh, well, uh, depending on the mode that you're using to write to desk, if you're using desk copies, it, it writes it as like a journal file and all this sort of stuff.

 Amos: So do you think, think there's, uh, depending on what the date is you're putting in the, do you think that there is a, a good way to tell it how to heal that would work for many circumstances?

 Chris: There's two ways to do it. Oh. And by the way, healing, it is not is non-trivial. We also like healing. It's super non-trivial because if you don't have ordering setup, right. Go, this goes back to what we talked about last week. If you don't have some sort of clock give up on ordering, right. And now you're just picking you just pick one at random. There's no such thing as last write wins. If you don't have ordering, when you, when you feel right. You know what I mean? Like, it, it there's no time. If there's no timestamp, then who's the last, like you don't even know. So just pick one at random and you hope it's the right one, or you go, do you try to do something fully? You know, you, you try to build some sort of CRDT thing, but even that's like, you're going to have to have some sort of clock for ordering prefer most of CRD Ts that don't need to be garbage collected. Well, like, like brand-new, cutting-edge stuff, you know, like CRD already work well, but old, like CRDTs as they exist today, like you have to garbage collect them. And it takes a ton of time. And like, they're, non-trivial. So that's one side of it. Like you can go full AP and attempt to rebuild your Nisia tables by just overlaying and picking the one that you believe, you know, is the most recent, or you store a list of like operations that you're then going to need to like issue to each other and like somehow reconcile that state that way. Yeah. It's like, that's, that's one way to do it. Or I think the, the flip of it, and I think this is what the rabbit people are working on. As far as I understand is you don't allow yourself to get into a bad state. Right. Which is that you actually build real consensus over the top of it. So like, once you, if you're, you know, you only allow that to write to tables based on it consistently.

 Amos: So when you're, when you're, I'm going to go back a little bit and what you were saying, uh, sure. We're talking about ordering last week in ordering in amnesia. So I know that they have an ordered set in amnesia, but do you know how they order that across the cluster?

 Chris: Well, I mean, the ordering is just done in an order. It just in the ordered set table, it's just done by a key, like, and you pick the key.

 Amos: Okay. So you have to tell it what to order by. So you could put a, uh, HLC or vector clock or something in there?

 Chris: If you want to use it to the store operations. Yeah, yeah, yeah. That would be, that would be a thing you could do and then to rebuild, but to rebuild the world, it gets tricky because you need to ensure that everyone has seen all the same stuff you need some way to ensure that here's the scenario, here's the scenario. You have two notes, they should be talking to each other, but for whatever reason, they're not, maybe you're doing a deployment or you're just haven't connected yet or whatever. There's a problem. Change the cookies, you know, who knows what? Right. Most of the time I find in production, like real-world net splits are really just synonymous with deployment. So, uh, cause you're like, I think the thing's alive, but it hasn't like connected yet or it's, you know, rolling or whatever. So in any case, so it's, so they're not talking to each other just yet, but you issue two different rights and you store those as a list of operations. This is the whole idea. This is like, you know, uh, event sourcing without, without the baggage is, you know, you start issuing these, these operations. And then as long as they're all, you know, competitive, you'll arrive at the same stuff. Right. The implicit statement in there is that if they eventually see all the operations,

 Amos: So when they heal, you have to get them to share events. Yeah. All of it.

 Chris: So how do you know, how do you know if you've seen all the operations, Amos?

 Amos: If you've gotten all the operations?

 Chris: Oh, I know. How do you know? But how do you know.

 Amos: Neither side knows, right?

 Chris: Right. So you need a - now you need a new thing. Now you need a new thing on top of that. And it, and it, and it rhymes with trees typically. But really though, that's you use a data structure like that, right. That can start to tell you, like, assuming that there is a place to have that thing. And you also send that thing all the way around that you start to say like, that is, you know, we're missing bits, we're missing chunks. We're missing parts of it because the tree says we should have these other things.

 Amos: So it feels like you could, you could get, if you design your data with a lot of these things in mind, then amnesia could work for you.

 Chris: I mean, yeah, yeah. It still has that whole, like, it's not dirt, it's durable-ish. You know, problem, by my opinion is like, you should view - You should view Nisha the same way you view a single node, redness. Like it's going to give you basically nothing in terms of clustering, nothing real that you want to, you actually use right direction. And you're going to have to figure out how to make that work. But, and it has basically the same kind of like journaling semantics, right. Of like, it's going to flush this stuff to disk periodically and probably it's going to save it all. And if not, like it needs to not wreck your day

 Amos: Except for when it doesn't.

 Chris: Right. Yeah, exactly. And if it doesn't, it doesn't and that, and we just deal with that, but ideally it saves it for the most part. It's going to save it. And then when it's not there, you just fall back to the real data store. Like if you treat it like redness, you know, you're in pretty good shape.

 Amos: I want to caveat that with, if you treat it like you should treat Reddis because I've also seen Reddis treated like -

 Chris: Primary database number,

 Amos: Our, we know it's never going down or it's never going to lose data. And then when it does people crap their pants, um, not my redness, my redness never goes down.

 Chris: My redness is here to last.

 Amos: Uh, I, that doesn't that just go against what reddis says it is anyway. Um…

 Chris: I mean, you know, it's programmers know the benefits of everything and the trade-offs of nothing. I keep feeling it over and over. So yeah. I, I think, I don't know, Nisha could be really interesting in terms of like a cap solution. It's, it's neither it's neither C or A, like, it doesn't pick either of them really. I would, in my estimation,

 Amos: Does it try to leave it up to you more? Or is it like,

 Chris: Everything up to you, but it's also just a product of its time. You know what I mean? It's like invented when you were literally running a machine that had a, where the network was a physical back plane. No. Yeah. You know, I mean, it's like, it's just the, the, the, the, the constraints were different. And so if the network was down the back plane didn't work and like, you had a bigger, a much bigger problem at that point, which basically meant nothing. Right? Like the whole system was down. If the network was down, right. That's the, because the physical back plane would've broken at that point. So I know it's, it's a product of its time. And I think people are in, I think people come into Erlang and it is unfortunate because it's one of those things that I think is a huge differentiator. Like how many languages, how many runtimes, right. Have they built in high-performance database? Not many, and can support that. You know what I mean? Like how many, how many, like, and it, it does what it, it, sorry, let me say this. Nisha does do what you mostly expect it to do in certain circumstances. And so it's really cool to see that, to have that capability. The problem is, is I it's, I don't want this to come out sounding like Nisha is broken. I don't think that that's accurate. I think people's expectations for what you, for what, for what the words database means have changed a lot, you know, database, like people hear the word database and it is synonymous with Postgres or my sequel

 Amos: Where everything is in one place. And if it's down, it's down, if it's up, it's up like,

 Chris: And it's, and like, for the most part it's consistent. Right. And it has, it's, you know, slow for certain operations, but, but you know, it's fast for others. And it's pretty much like going to reliably store your data, barring some sort of just gross oversight, right. Or just some sort of absolutely catastrophic problem. Those databases are just going to store your data. And you are not going to have to think about it. Like you never questioned whether or not MySQL or Postgres is actually stored a thing when you issue a right-

 Amos:- If you get a success back, you know-

 Chris: -It's there. And if you get a read back, you know, it was the right value at the time that you issued the read. Like there are, there are caveats to that. You don't know things that are going to happen in the future and transactions are trying to run concurrently. And it depends on your transaction level and all that sort of junk. But for the most part, you rely on your database. And again, databases are very much people. People have this association with the word database, mean my SQL Postgres SQL server, whatever it is and that's, and you just, you come to rely on a certain amount of comfort and knowing that barring something absolutely catastrophic happening, this thing is storing my data when it says

 Amos: Even, even those databases or any relational databases that provide a distributed implementation, have all these caveats. When you, when you read in there about what happens when you distributed. And even if you just turn on readers, your readers, when you read from a reader, it might not be the most up-to-date. You gotta be thinking about that.

 Chris: And you can have weird issues in file systems. Like there's, there's still that thing where Postgres can attempt to write to the file system and it can fail. And then it will crash the Postgres because Postgres, at that point goes, “Yo, something bad has happened.” And we, the safest thing for us to do is to terminate at this point. Yeah. Because like file systems are all fundamentally broken and SSDs are all fundamentally broken, you know, all these problems or whatever, right? So like, yeah, I get it. Like, I'm not here to say that post-stress will never lose your data. It's more just that, you know, the, the likelihood of Postgres losing your data is, is markedly less than redness losing your data for instance. And that's markedly less than Nisha losing your data for instance.

 Amos: Well, any, any time that you have to have cooperation between more than one system, the more, the more of those links that you have, the, I guess, less likelihood you have of getting what you want out of it without extra. There's a lot of extra overhead and work that needs to go in to make it sustainable. And then your, your trade-offs are bigger.

 Chris: And you're going to have to learn a lot more. I mean, you're, you're managing you're, you're now literally managing a database inside of your application code, which is super interesting, right? Like that's fascinating. And again, I think it's a huge differentiator as part of the runtime. I just think people's expectations for what the word database means are so different than what is giving you out of the box, especially with its default choices that it's going to be really surprising to people. And that's where people get the feeling that Nisha is broken or wrong. I don't know that amnesia is wrong to be the way that it is. I do think that people need to be aware of what it is and to be responsible about that choice and accept it for what it is of like, yeah, man, if you want a really fast in memory, cache is backed more or less by a database and you just know that you can be wrong and your recovery strategy is like, I don't know, I've got an HLC or some other logical clock. I's pick the one that's like most recent. And we'll just deal with that. Yeah. I think he's just like-

 Amos: -As a blockchain and then we own them. We throw away the, the newest 10 until we have a consistency. Okay. My, my, my internet voice is not as good as yours. No, that was-

 Chris: -That was strong. That was, that was, uh, that was strong. I mean, I hated you the moment you started talking. And so it's like, I feel like your mission accomplished might.

 Amos: Uh, I got to tell this story now because voices. So my son yesterday, we were out and he started like, I don't know. I guess the best way I can describe it as is talking hick. And, uh, so then I just started talking like Mickey Mouse, every time he would talk like that, we're out in public and I'm like, so Josh, how's your day today? And he's like, dad, how about I stop? And you stop. I was like, “Oh, you're growing up. You've decided to compromise. Growing up so fast.” It was pretty funny. The rest of the kids found it quite entertaining.

 Chris: You think your mom wants to have smokes? Oh boy, there's a deal on eggs. Okay. That's the, that was pretty much our dinner yesterday. So honey house today, now I'm going to go home and do that today. Like some smoked salmon dish. Did you see that? Six people still just contested?

 Amos: Okay. I think that's a good place to end it. Uh, seriously though. I do. I do have, uh, another meeting coming up and we have a new person that started today. Chad, welcome Chad. Um, so I have got to go get some stuff done.

 Chris: This is so bad. We can't release this.

 Amos: Because I guess, I mean, we can't, we can't go any lower or any higher than where we just were. So I guess we can just take it on out. Yeah.

 Chris: They look at, everybody looks at us and they're like, you know, now we know about those guys, but guys are high.

 Amos: High on life, baby. Chris, thanks for coming today. It was nice seeing Anna today. Uh, even though I don't think she made it into the recording, she had, she had a meeting come up. She stopped by, she has meetings. She's been super busy. She misses everybody, not Chris and I. All right. Well take it easy, Chris.

 Chris: Have a great day.

 Amos: Later. Bye.