Chris Keathley
The Elixir Outlaws now have a Patreon. If you’re enjoying the show then please consider throwing a few bucks our way to help us pay for the costs for the show.
Episode Transcript
Amos: Welcome to Elixir Outlaws, the hallway track of the Elixir community.
Chris: Here we go. You're not going to see this, but I'll send you a photo of what we have. I'll send you. I'll send you this of what, uh, of what I'm working on over here. What I got going on.
Amos: I'm gonna send you a picture of what I would buy you for Christmas, if I was going to buy you anything,
Chris: What does that mean? I don't even know what that, what that's about. That was, that was so spiteful in a way that I don't quite understand. Hang on. I'm almost, I'm gonna send you the photo. Hang on. Here we go.
Amos: I'm not feeling spiteful other than the fact that I'm running low on coffee.
Chris: Amos, I'm on my third cup. It's a bad day.
Amos: Alright. Maybe we should start out the day where I just stand here and make a pour over near the microphone.
Chris: This is what I'm drawing currently, on my, on my iPad.
Amos: All right. And I'm going to describe it for everybody.
Chris: It's it's just, it's it's coming in. It's very rough, very early, very early.
Amos: So we, uh, we have a pretty nice map goin' with a one, two, three, four, five-ish, mountain ranges
Chris: I'm just bringing in the mountain ranges first, see? You gotta set the, you gotta have other stuff. Yeah. Yeah. This is the mountain. This is the world map for my homebrew D&D campaign setting.
Amos: I, I'm enjoying it. There's a few cities in there. An Island off to the side.
Chris: Yeah, it’s got, it's got, it's got a lot going on. It's got a lot going on.
Amos: One of my favorite things about maps for role-playing games in, is that when you draw the entire world, almost all of them are way closer to Pangea than we are here. Like everybody draws like one, maybe two big continents and a couple of islands
Chris: Yes. Yes. Exactly. As far as I'm concerned. You know, yeah. Yeah. Well, listen, there's only so much time in the day and I've been taking a long time to draw what I have. Anyway. We shouldn't talk about this. No, one's going to see it.
Amos: Well, that's why I tried to describe it. It was beautiful. It was a well done map. It looked like a cartographer. I mean, not quite, but
Chris: I mean, it's- Listen, everyone needs an escape from this, from the fresh hell. That is every single day.
Amos: So how deep do you get into this? Do you start like worrying about weather and like how the mountains affect weather.
Chris: Yeah, sure, if its thematically appropriate, if it's thematically interesting. It's all about the narrative, Amos.
Amos: That's true. That's true. And but, but you gotta make it feel real too, right?
Chris: Yeah. You gotta make it feel lived in. Any who, so that's been what I've been doing, um, because.
Amos: I'm still waiting for you invite me to play.
Chris: I did invite you to play, but to do anything else would be to invite madness.
Amos: You told me you would, you would talk to the team, the group and see if I could.
Chris: I did.
Amos: But then you never invited me.
Chris: Well, I was just hoping you'd read between the lines.
Amos: I assume they said, "No," and I moved on.
Chris: I didn't want to say no.
Amos: I understand. I understand.
Chris: No, I mean, listen. It's gonna happen at some point. We'll make it happen at some point.
Amos: So I, I was talking to you the other day.
Chris: I'm closing, closing all the tabs. Yes. Talk to me about Elixir-related subject.
Amos: Okay. Um, tangentially Elixir, sort of like most things, um.
Chris: Amos, take the wheel.
Amos: So you're writing this D&D world. Um, I assume it has time in it.
Chris: Yes.
Amos: I'm pretty sure you know where I'm going. So I was rereading the hybrid logical clock page the other day.
Chris: Yes!
Amos: Just like I'm reading it. It sounds too good to be true. Right.
Chris: It, ‘cause it is.
Amos: Spelling out how many awesome things. And then you said, if it sounds too good to be true, it probably is. And, and yes, I agree with that. But you alluded to some issues with a hybrid logical clock, and I was just curious what, what those issues were that you, you had in your head?
Chris: Sure. Yeah. We can talk about this. Do you want to like elucidate? What are, what a hybrid logical clock is to our audience?
Amos: Uh, I, I just have to say elucidate cracks me up because I'm taking a great courses plus thing on expanding your vocabulary. So,
Chris: Aah!
Amos: You always keep me on my toes.
Chris: Is the great courses plus course doing this podcast with me?
Amos: Um, no?
Chris: Okay. Well.
Amos: Maybe we should, you're, you're supplemental to the course. I'm pretty sure. I'm pretty sure he mentioned it. Elixir Outlaws, but, but don't listen to that Amos guy. Cause he never has any good vocabulary.
Chris: And so it's like, it's like, you're studying a foreign language and then you're doing an immersion course.
Amos: That's right.
Chris: I get it. Yeah, for sure. You gotta, you want to, you, you gotta get the native dialect. You can, you know how to conjugate, but you don't know all the slang. All the colloquialisms.
Amos: Right. Right. Especially when you start, when you start dropping the slang, then I'm really lost. Um, I just, I just.
Chris: Same girl, same.
Amos: I just asked my daughter I'm like, “What does Keithley mean?” She fills me in. Alright. So hybrid logical clock. Okay. So I think that we've talked enough on here about how time is problematic in a distributed system in ordering things. Um, I I'll start out as recently, I was trying to analyze some logs and get an order of events and really never trust time is, well, what, what the theme of this is, but we had events that look and at first glance I even was like, Whoa, wait a second. And then I re you know, remembered, never trust time. But we had events that we had an end event before we had the start event. Cause there was no, there were supposed to be events in between, and itt's like, Hey, I need to sync. Okay. Sync is over. But because they hit two different servers, those events, we had a synch over before the synch. So for a moment, you know, you have this flash of, Oh my God, what did we break? How did this happen? And then when you look at the timestamps and they're like a hundred milliseconds apart, that's not insignificant in computer time, but it's pretty normal for NTP to drift. Especially if you have like different data centers going on. So their hybrid logical clock is one solution to that. I mean, you could have message counters and just count things. That's still called a clock, but a hybrid logical clock combines the ideas of, uh, in a, in a way, the ideas of a counter and the ideas of a clock. And then also uses the law or the idea of clock physical time, like real world time. Like when you look at your watch, you don't look at, oh, it's third event of the day. You look at time, that kind of time.
Chris: Wall clock time.
Amos: So it combines the two of those. Wall clock. There you go. Thank you. So it combines the two of those, but also has a mechanism for dealing with drift to try to kind of come up with like a, a sync between disparate systems, um, so that you can still order events by this time clock, wall clock. And it be in order.
Chris: Right.
Amos: Real, real time.
Chris: Yeah, we needed to. So we needed to define some terms.
Amos: Alright.
Chris: Because when you say in order, boy, is that loaded, right. That brings to mind all sorts of connotations.
Amos: Can we skip the physics part of it though?
Chris: Yeah, yeah sure. But I want,
Amos: Okay, good, that gets crazy.
Chris: Let's get, let's get, listen, Amos, let's get down to business. So distributed systems are fundamentally they're concurrency problems. It's a problem of concurrency and it's made worse by the fact that it's much harder to distribute systems as we kind of know them today. Right. It's made worse by the fact that we're going over networks and we're doing all this, but it's fundamentally a concurrency problem. What is the fundamental I'm going to ask you, then we're going to be, we're going to be Socratic. What's the fundamental problem with concurrency?
Amos: Really knowing when things are concurrent.
Chris: Right. And if you're competing, if you have multiple actors attempting to compete over a resource and change that thing in some way, what's the problem, like what's a race condition at its at its base at its sort of fundamental, What's the fundamental problem of a race condition. Uh,
Amos: Really who got, who, who acted first?
Chris: Yes, exactly. A hundred percent. Yes. It's an ordering problem. Concurrency problems outside of, uh, pervasive issues such as deadlocks, right? There's, there's an, there's a whole class of issues with, with concurrency, but the most nefarious are ordering problems and they're nefarious because they don't deadlock. They're nefarious because it's something that is non-obvious. So if you've got a database and you're storing data in it and you've got multiple different writers competing on the same bit of resource, they can do operations out of order. And if you don't notice it, if you're not analyzing your data, you can be really, really screwed because you don't know what's happened. It's hard to figure out the lineage of, of operations that have that have, that have occurred on that piece of state.
Amos: So let's, let's put it in like a really basic contrived real world example. But uh, your bank account, right? This is not how they work typically. But uh,
Chris: Bank accounts are always the example and it's always wrong.
Amos: Well, because it's not how they work.
Chris: Because the metaphors are always bad.
Amos: Right?
Chris: But it turns out banks actually solve this problem. And they don't, and they're not acid compliant.
Amos: Well in a lot of them wait until the end of the day. And then they do all deposits and all, all withdrawals. But.
Chris: Right.
Amos: If, if they were trying to be real time, um, then you have, if you don't know what order, the withdrawal and the deposit happened, you could have deposited and then withdrawn. But when the system receives those messages, they might get the withdrawal first and now you're over your limit, right? You can't withdraw something like that.
Chris: Another example that I think is slightly more accurate is not accurate, but it's a little more telling is let's say you unfriend someone on a social site, you comment about them. So they shouldn't be able to see it anymore. And then you friend them again, if you get the order of those things wrong, that has repercussions on your users. Data problems are a hundred percent, the most damaging thing. People, people don't, I don't think, think about this enough, but in terms of like your business data problems are, are easily the most potentially damaging operations to get wrong to your business, to the fundamentals of your business. But all that to say ordering is really hard. And when you talk about ordering, there's actually well-defined well, I say well-defined, there's actually defined. Not always well understood, well understood by by practitioners, but there are well-defined strictness levels. Let's say when it comes to ordering operations. So if every operation that ever occurs in your system, if you can guarantee that it all happens in the order that it happens in and you always write things correctly into the, in the order that happens in that is way stricter than saying something a little more hand wavy, like these two operations that are related, this one caused this one, right? Those are different strictness levels. Because one of those things, in one case, you're talking about two independent operations, right? You're talking about two, two isolated events in a, in a sea of other events. And those two things are causal, but you don't know anything else. You know, that those two things are causal. You know, this happened before HB happened before this other thing, but you don't know anything else about it. You don't know where, where in the world, those two things occurred, versus I know where every single event occurred and I can prove it. Those are different things. And they have conseq- depending on what you choose, they have consequences.
Amos: Right.
Chris: So all of this to say, there's this, in order to combat the concurrency problem of ordering there's a family of algorithms and data structures, because it's always algorithms and data structures, that provide a way to deal with ordering. And databases use these sorts of things internally as well, especially, you know, databases that have that distribute load across a cluster and that sort of thing. Um, but you know, just your, your Postgres, your Postgres has the, has notions of ordering inside of it. That's what all those transaction levels are about. Know that's what all those isolation levels are about. It's about ensuring ordering of events takes place in the right order. And so when you pick an isolation level, right, or you pick the level that you're going to set your transaction to , your that's, why it's called strict, strict serializability, You know what I mean? It's like, because that's the, that's the term for it, right. And that's, that's what you're getting at the heart of, right. You're getting at this notion of these events are ordered. I should not be able to see things that happened simultaneous to me or before me or after me or whatever, right? And you're dealing with all that sort of stuff. So there's a whole family of algorithms around this and in distribute systems contexts, uh, this goes back to Lamport, uh, as, as most things do.
Amos: Good old Lamport.
Chris: Yeah. And, and, and the note and the, his notion of what is now termed Lamport clocks. And a Lamport clock is,
Amos: It's just a counter, right?
Chris: It's a vector, right? It's a, it's a, it's a, yeah, it's a counter that, Oh, and I always get confused. I always have to go back and reread this. What the, the yeah. Lamport clocks are just a counter. And then you've got vector clocks, which are a vector of counters. Um, but like
Amos: One for each system that you have.
Chris: Yeah. And the notion is every node, uh, the notion of a vector clock is easy to explain the very highly naive algorithm. And they're hard to implement in practice, um, cause garbage collections a thing. But the notion is that you have every node gets its own counter, which is its representation of the state at that point in time. And when they share them around, they, you have a way to reconcile that. So you can say, well, this person's a, has these counters. This is, and you have to reconcile, like when we were at this counter, we agreed on this version of the state and, you know, et cetera, et cetera. And like, and that's how you maintain the clock.
Amos: So using vector clocks, um, the issue that pops into my head immediately is like, so if I have "A" set of events, you have A1, right, And maybe B and C both risk do something with that. So you get A1 B1 and A1 C1. And now you don't know between those two, you don't know ordering necessarily. So you have to pick. And then also if you get large systems, you have so many things in the vector that your storage for passing a vector clock is huge.
Chris: Yes. Right? And if you have, if you have a hundred nodes, you need a hundred things. And if those nodes come and go, you need a way to identify which nodes came and went and you need to be able to distribute that around the cluster. And everybody knows they can garbage collect old nodes out of the vector, et cetera, et cetera. That's all really hard.
Amos: Right. Too many problems.
Chris: Um, but those, and it can be done. There are algorithms to do it. I think so. I mean, this is out of my area of, um, it's, it's far enough out of my area of expertise that I I've failed to, I struggle to explain it in ways that probably make sense auditorily. Like I could draw you pictures of it on a, on a whiteboard. And it might make more sense.
Amos:
We, we should that. A Mimecast sometime where we just don't talk for an hour.
Chris: But honestly, this is outside of my wheelhouse just enough. Um, it's out of my brain just enough that I don't know that I can speak totally intelligently about these algorithms anymore, but all that suffice to say, I think dotted version vectors are still the go-to for vector clocks at the moment. And I think that's what, uh, react used to achieve causal delivery of because they needed causl- I'm totally forgetting this. Sean will be able to tell me all the things I'm getting wrong about this. But if, um, but if I remember correctly, uh,
Amos: Sean had a show.
Chris: Friend of the show, Sean!
Amos: Sean Cribbs?
Chris: Uh, is yeah,. Oh, okay. Would have to, he, all those folks would be like, explain it to me better. But like, uh, if I remember correctly, because Erlang does not have, does not support out of the box causal delivery, meaning I'm failing to explain this correctly, but it's, it's, uh, essentially you need a, you need a way to, to fake causal delivery and for the er SWAT CRDT thing to work. And so they had to use, I think they use dot version vectors for that. I I'm forgetting some of this stuff, but in any case, so those are a very, those have a higher level of ordering guarantee, right? There are stricter ordering guarantee, then the paper you're looking at, which is a hybrid logical clocks. Hybrid logical clocks, they take a super position on NTP clock timestamps. And they basically chop like the, the last bits off of the, of the clock of the physical wall clock, uh, known as physical time in the, in the litter literature. And instead place upon it, a logical timestamp. So you have, you know, it starts at zero zero. And then as you broadcast these clocks around the cluster, they have a way to merge on top of themselves so that you're always incrementing the clock itself. And once you roll over the physical time, is there any events that occur at the same physical time, according to the clock, the wall clock, those are all treated as like this happened at the same moment in time. And then you have logical time that attempts to give you a reasonable ordering of events. It's worth noting though, that that ordering events is not causal. It's only not across the cluster. It's causal per box. It's causal per node, but they're wide scale.
Amos: It, it's still kind of a best guess on-
Chris: -Yes.
Amos: When things happen in what order. So you could have two of them that have the same stamp and you're just going to be like -(muttering)
Chris: You just pick, and then you picked it and then you also don't know what happened, happened before just means happened before doesn't mean happened before. So the takeaway from all this is that if you, you know what, you know, if you have two clocks that, that, you know, if you have an ordering, what you know of, of events, what you know, is those events either happened before, or they happened at precisely the same time.
Amos: Right.
Chris: And we don't know, and we just, don’t.
Amos: But you know, they didn't happen after,
Chris: Uh, yeah.
Amos: From what, from what, I’m trying to remember, there's three, three.
Chris: Yes. It most likely came before, or it happened at the exact same time. Right?
Amos: Three properties. Yeah.
Chris: Yeah. So that's what you get with a hybrid logical clock. It also has limitations in that the hybrid logical clock, if, if NTP does drift outside of a maximum window, you stop making logical, you stop making progress, you refuse to create a new timestamp, a new clock.
Amos: Oh, I didn't realize it refused.
Chris: Well, I mean, that's the, that's the, that's the trade-off right. Is like, it requires you to be within at least some reasonable bounds.
Amos: Right?
Chris: Yeah. And it's, and it's, and that's like, I'm making that more dramatic than it sounds because it's only if it reverses, like it can only ever make forward progress.
Amos: Okay.
Chris: So if it skips ahead, that's fine.
Amos: But it can't go backwards.
Chris: Yeah. And you, and that works and that works because of their merging algorithm. So the idea is that,
Amos: That you get to pick your tolerance.
Chris: Yeah, and you send this to us and you send this stuff around and then the, all the nodes eventually get all of these new clocks, like disseminate it to them periodically, um, by each other, they all send them around to the other, the other nodes. And then you do this merging operation where you split apart the bytes and you take physical time, and you take logical time. And then there's emerging operation and algorithms actually like really trivial, like the pseudo code for its two lines of code.
Amos: Oh, yeah. It was, it was like, you pick the max of the physical time.
Chris: Yeah.
Amos: And then the max plus one of the logical time and put them together.
Chris: Yeah.
Amos: And then I see this is the same number of bits as NTP time. So then you can, you can do that. What I, I found useful is that I can have, I can progress toward it by having, uh, some of my systems are still straight in TP and I don't have like with a vector clock, I got to get everybody on it before it's that useful where this I can, I can slowly evolve my system is why I went toward the hybrid logical clock. But then you said there were problems. And I was like, uh-oh,
Chris: Well, there are, I mean, it's it's, but there's, this is the nature of distributed systems, right? Is that there are always trade offs. There's always constraints around this stuff. And that's partially because like the only way to solve these problems is to bound them. Like, that's the takeaway of harvest and yield. Like, that's the takeaway of cap. Is that these problems are bounded necessarily. Because if you don't bound the problem, you literally can't solve all of it.
Amos: Right.
Chris: It is not physically possible to solve all of it. And people get hung up on cap and like, people get hung up on the idea that like, Oh, you just can't have, you know, the blah, blah, blah. Like then people get, people get hung up on all this sort of stuff. But like, if you don't do anything to bound the problem you do end up in this situation where it's like, well, you can't solve it. You can't solve, you can't make a CAP system. So like, you do run up, but not, not with one primitive.
Amos: Right.
Chris: You can make something that, that pretends to be a CAP system.
Amos: Right.
Chris: Like you can make something that, that lives on that spectrum. The harvest and yield spectrum, but you can't have it all. And so yeah, the problems are necessarily bounded. So like dotted version vector is way more complicated and they're, and, and like, they have to, it's an algorithm that has to exist because you have to solve all the problems with vector clocks in the naive implementation of it. Right. You have to garbage collect it eventually. Hybrid logical clocks you don't have to garbage collect. There's no bounding, like, er, rather, they're already bounded. So you don't have to do anything about that. The bounding also presents a problem.
Amos: They're 64 bits.
Chris:
Yeah. The bounding also presents a problem because eventually you run out of logical time.
Amos: Right. Well, yeah. What if in the same second you have 2000 events while with adding the logical time on your you're like, well, I can't, I can't handle this.
Chris: It's actually much more than that, but yeah. It's like many, many, many more events that you'd have to, you'd have to create, but like, but yeah. But your point stands, which is like, yeah, if you have more than a hundred thousand events, you know, you're going to run out of logical time.
Amos: So how do you deal with this in a system where maybe you have an endpoint that runs in some sort of offline mode because it is there and then syncs back up later and it's collecting events, even when it's offline. Is there, is there some kind of clock that you think works well there? Cause a hybrid logical clock, it's not receiving events very often.
Chris: Right. I think HLCs actually do work with that still. Okay. The HLCs are actually very good at being able to handle like bursty sort of like synching operations like that.
Amos: Okay.
Chris: The other thing that we do, so, okay. And also, I should make it clear, like these trade offs are known things and you have to just pick, you have to make those sorts of choices. Right. That doesn't mean that they're the wrong option. We use these at Bleacher Report. We use, we tag every event, every thing with an HLC and they're working, it's working pretty well for us. Like, you know, there are problems, there are things to be aware of, but it's more like there's things to be aware of with all this stuff. So you need to just like pick the thing that makes the most sense. And for our operations, like that made the most sense.
Amos: Are you still using the, are you using the tonic systems HLC?
Chris: Yup. Yup. Yup. Yup. That's right.
Amos: That's that library I think like, if, if you don't know what an HLC is and you don't want to read a white paper, the white paper is going to give you a lot of information, but if you go look at that library, it's super straightforward.
Chris: Yeah, yeah. Yeah. Its super straight forward. Ours is slightly modified. That HLC library is also slightly modified in that it also adds a unique, uh, essentially a node name, but it's not the node name. It's like, it's a unique crypto secure set of bits. It basically adds more entropy to the name and that makes them unique. And because they're unique or yeah. It just, it, as long as you never have a collision in those crypto secure bits across your cluster, then it's unique. So that is again, you know, a thing to be aware of.
Amos: So do you use that for ordering?
Chris: Yeah. Final tiebreaker for ordering is the known name, but more importantly, it provides a unique ID and they're still electrically ordered.
Amos: Do you jam that into the 64 bits?
Chris: No, we extend it. We just add, we just add extra bits to the end of it.
Amos: Okay. Okay. So it's no longer straight NTP compatible at that point. Yeah, yeah, yeah.
Chris: We give up that, but we now we have, we have an, uh, hybrid, unique logical clock
Amos: But if everybody's using that, then you don't, it doesn't matter. You don't need the NTP compatibility if you'd have everybody using it.
Chris: Yeah. And they still fit in a binary column and Postgres.
Amos: Yep.
Chris: And you, and they're orderable. So you can say order by this and it works. Cause it's like they're lexically ordered. Cause they're, they're stored like in NDN or wherever. And so they're, they're still lexicaly ordered. You can sort them directly in the database. We have an ecto adapter. So when it pulls it out of the database, it rips it back into like a, a struct version of the thing and et cetera,
Amos: That was what I really liked about the logical clock is that I don't have to write anything special to sort it with a vector clock. I mean, there's a lot of libraries that'll do it for you, but there's like special sorting to sort factor clocks. And with the hybrid logical clock, it's like just sort this, like you would any timestamp, go.
Chris: Yup. Yup. Yeah, absolutely. And that, so all those things. And so at that point, they're really closer to like flake IDs. If you're familiar with that.
Amos: I do not know what a flake ID is.
Chris: A flake ID is a case sortable ID that can be created, uh, what's the term I'm looking for without, um, coordination. So each node can, can build their own. They're only, they're only physical time though.
Amos: Right.
Chris: And so it's, uh, it's very close to like a flake ID, but we get causal-ish, right, is, is like the, not the actual term, but it's like kind of-ish causal-ish. Yeah.
Amos: Okay, I've done something similar where I did a timestamp plus a machine ID plus a process ID and then a count order.
Chris: Yeah.
Amos: And that worked really well. It didn't always get everything in order. Cause we weren't trying to merge the clock. Like the hyper logical clock. We were just using whatever the machine time says plus this and this and this.
Chris: Right, right, right., right. And that's the thing is when you're doing all this merging, part of what you're merging is the physical time. And so that's the drift window thing. Like if all of a sudden you, you know, you drift outside of that, you you're, you're all of a sudden in an error state. So you have to find ways to reconcile that. But in any case, like I said, all this stuff lives on, uh, all of these things live on like a spectrum of quote unquote correctness, right.Of, of like of ordering guarantees. And we picked one that gets us close enough and is now unique. And we can use it as like an ID for all the things. So we just, we tag it all. We refer to them as content IDs. So every single thing gets a content ID. They're all HLCs. And then we're able to use those as a generic way to talk about any entity in the system.
Amos: And they're unique because of your cryptographic part. Because if you just do a regular HLC, sometimes you do have collisions where they're exactly the same.
Chris: Right. Yeah. So we use the, and we, like I said, we use that as the final tiebreaker, but you could totally, if you had, and like the, my friends who introduced me to all these things, this is what they did at their, at a thing that they're working on. They had, you know, iPads and we're using them to tag stuff on an iPad and then sync it across the wire occasionally. And they use the Mac address of the iPad.
Amos: Interesting. This is why I'm doing this,
Chris: As the node name. And so then you all, yeah. Right, exactly. So like that, and that works for them. As long as they're like talking and kind of gossiping the clocks around often enough, you get, you stay within the snapshot window. Which is like described in the paper. But in any case, that's the, the, the bigger thing to take away from this, right is like, yeah, there's, there's lots of interesting things about HLCs generally. And I think they're really cool. They're a very useful tool, uh, in my experience. But, and the real takeaway though, is that like, you need something, you need something like this at some level though, that's why everybody has like a case sortable ID library. You know what I mean? That's like why flake IDs exists, It's why people use flake IDs. It's why people will use, you know, the whatever segments case K squid or case sued or whatever, like their case sortable, unique IDs or whatever.
Amos: Ulids.
Chris: Um, yeah, you need stuff like that because ordering starts to matter. And, you know, getting close is useful. Just being able to, sometimes you don't need like a total order. Like you don't need to know the entire spectrum of all events that have ever taken place, that's just dumb. Um, sometimes you just get close and that's good enough. And as long as you can get, like in the ballpark, you're like, sweet. That's that solves my, my business problem. I can like solve the rest of it with emails.
Amos: Yeah. I think if you know what events can cause others on top of an HLC, it feels like a pretty good alternative because I've run systems where like, I'm, I'm basically caching and memory up until I get an event that looks like it was slotted at the right time. And then I get rid of that cache where with the HLC plus that it shortens the number of events that I actually need to pull to my cache because I can say, Hey, these are in order this event, I know has a causer that I haven't seen yet, so I can hold onto it for just a really short period of time and should be good to go. And if I don't that I know I need to maybe ask for like, Hey, I think I missed an event.
Chris: And let's say that, let's say the cool thing is you can push that all the way to the edge. Like you can push that, the clock stuff all the way out to edge of your system and then have it like bring it back in. And that makes sense because like the clients that you're dealing with, I mean, they're part of your distributed system,.
Amos: Right.
Chris: They're not dumb, right? They're, they're, they're loosely connected a lot of times and users just know that if they refresh it, we'll probably fix it. But, but they're connected. They're our clients, they're clients of your district system. Thus, they are, I mean, do people realize how smart, like a lot of like the Postgres clients have to be to work? Like, you know what I mean? Like do people like the clients, part of the, the thing
Amos: Are you even many web front ends are starting to be really smart and do a lot of, a lot of things. And I mean, that's good because you can take some power off your, off your server and let the other end maybe do some filtering or things that would be expensive, which, you know, actually you talked about the other day on intercept on a Phoenix channel, is that for every every message that goes up for every client connected intercept has to run where if you can do that on the front end and let the front end filter those out, that's great. As long as it's not a security issue, don't.
Chris: Right. Right. Right.
Amos: Because that's the thing about sending to the front end. If they're passing their own clocks back, you either have to trust their clocks or, or ignore them basically, Uh, and then you, then you worry about message receive order. You can add your own clock, but
Chris: Right. Yeah. And then it's whoever gets to me first, basically.
Amos: Yeah.
Chris: Right. That's as much as we know at that point, whoever gets to me first,
Amos: Which I mean, depending on your tolerances and your system, that's all things that you have to figure out yourself.
Chris: And that's, that's why you need to know the trade-offs. Right. It's important to be clear eyed about the trade-offs. And I think people are too squeamish about calling stuff garbage. Like when it, when it is, and like, people don't acknowledge the trade-offs that exist, but also don't like live within them, right. Like, yeah. I know that HLCs have these problems, quote, unquote problems. It's more like they have these limitations for what they're trying to do, but it's like, that's fine. You can totally live within that, that choice, but you can't make a choice if you don't actually have data. Right.
Amos: Right.
Chris: You can't make a choice if you just refuse to acknowledge that there are problems. So I don't know. I think people don't are not, uh, yeah. I think people aren't are like too squeamish about that generally. Or they like, or are they spend all the time pontificating about all the various options that are out there as though that's helpful. It's like, no, like have an opinion. I like,
Amos: So, you know, so where nobody can agree, which trade off to actually take. And so you spend lots of time trying to figure out and complain about the corner cases of each individual thing, which I think is valuable to a point. But at some point you might just need to make a decision.
Chris: Yeah. And, and like, our trade-offs are totally unique to Bleacher Report, right. We're shooting sports news on the internet and we're letting people comment on it, like, come on, like, you know, you can, you can, you don't have to be perfect for that.
Amos: If you're trading on the stock market, you might get a little more complaints about ordering
Chris: Yeah. I mean, if, if you're doing all kinds of stuff right. Who knows, like you just change your business domain a little bit and then yeah, your trade-offs to might need to change too. That's fine. That's that's like, that's why it's called software engineering. Supposably. If you want to call yourself an engineer, friggin act like it. Like, like, you know, do some math occasionally.
Amos: Right. You just type and don't make any real decisions.
Chris: Yeah. If you don't look, I mean, yeah, yeah. If you're not making conscious choices about that stuff, you're not mentally making a choice at all. You're just like plowing forward, moving fast and breaking things. So I don't know. I, um, yeah, that's, that's the trade-offs. Those are the things that you should be aware of. Otherwise, you know, I think so far, they're, they're very useful tools, definitely drawbacks, er like, constraints, but otherwise super useful.
Amos: And go read the paper for hybrid biological clocks and implement your own just so you can, it's really easy.
Chris: It;s a good paper. Yeah. It's a, it's a it's. Yeah. They're not too bad. You need to, that's one of those that we talked about this last time, but it's one of those that kind of a little bit expects you to know some stuff already.
Amos: They try to give you a little bit though, too, when they, it starts out with a brief history of time and they give you a little bit, they don't tell you all of the trade-offs, but they're like, here's kind of a high level. What a Lamport clock is. Here's a high level of what a vector clock is.
Chris: Here's how they define HB. That's important.
Amos: Yep. Yep. Happened before.
Chris: Yeah. So, you know, they define some terms and stuff like that, but it is one that I think is speaking to an audience who they know already knows a lot of this stuff. Yeah. So,
Amos: And there's not so much math there that you're like, I need to go retake linear algebra to figure it out.
Chris: Yeah. For sure. And you know, you know, read it three times. That's right. And, and you'll be, you know, it'll be okay.
Amos: What if I'm on like time six of reading it. I read it, I read it like a year ago and now I'm reading it too.
Chris: Yeah, that's fine too.
Amos: It's funny when I reread, I go back to the three again, because I often am like, I don't remember if this is what I wanted or not.
Chris: Right. (laughing)
Amos: And I'm pretty sure it is.
Chris: Yeah.
Amos: Like, it sounds like the HLC is exactly what I'm after. Uh, I like the idea of adding on some kind of like cryptographic signature or something, just to, just to give things another order. I'd like to figure out how to maybe use that as like a priority, like this system it's events get priority over anybody else's so adding that to it would be interesting too.
Chris: Yeah. You could probably, if you have a way to link that back to something, yeah. You could do that essentially. I don't know. I might, my feeling on it is like, keep it, keep that stuff like simple.
Amos: Yep.
Chris: Keep the clocks as simple as possible. Cause they're clocks, they're just like IDs and then layer that other like that sort of stuff on via other methodology.
Amos: Yeah. Yeah.
Chris: Because you don't want to attach that to like a thing that a scheme that you, I don't know, what am I trying to say? That it seems like a conflating too many things I guess is maybe what I'm trying to say. I.
Amos: I can see that. Also, losing that backwards compatibility with NTP can be, can be a major drawback to piling stuff into it.
Chris: Yeah, for sure. For sure. Cool. But I don't know. I think that that's reasonable. It's not that many more bytes. You know what I mean? So anyway, so that's what I would, uh, yeah, I think they're, they're useful.
Amos: Thanks.
Chris: What else is going on in your world?
Amos: Uh, not a lot, you know, Christmas is coming up. Um, have like a small Christmas party for people here that, that aren't already on vacation. We've got some people out. Binary Noggin's growing, so that's cool
Chris: Yeah. I'm happy you hired somebody else.
Amos: Yeah, I hired somebody else. They start in January.
Chris: That's awesome.
Amos: And talking to some other people, so
Chris: That's awesome.
Amos: Try, trying to build a place where, our, our conversations at work are a lot, like the ones that you and I have.
Chris: Yeah. I've I feel that, yeah.
Amos: I like those. And, and uh, generally just like, that's what excites me about my work is learning and growing. That's why I picked tech. Cause I knew it probably wasn't going to slow down.
Chris: Yeah. It hasn't hasn't so far.
Amos: Yep. And other than that, I don't have much. How about you? Anything exciting? Any good news?
Chris: No, no quarantining. Quarantining. Yep. Quarantining. Quarantining.
Amos: That's great. Yeah. I've been thinking about going fishing. It's a good quarantine activity by yourself.
Chris: Yep. Quarantining.
Amos: Hope the kids are good. Wife's awesome.
Chris: Kids are, yeah. Kids are. Kids are okay. Everybody's we're just, you know, we're hanging in there. Just keeping on.
Amos: Sounds good.
Chris: Keeping on keeping on. Nothing much to report.
Amos: Well I better-
Amos: Huh?
Chris: Nothing to report nothing to report. Nothing, nothing new.
Amos: That's, that's sometimes a good thing.
Chris: In this, in these troubled times, no news is good.
Amos: Truth. Truth. I can't wait to put 2020 into bed and hopefully 2021 will be better.
Chris: Oh my gosh. Gotta be better.
Amos: Put me to bed or not another year of this.
Chris: Yeah. I'm done. Yeah. No, I'm I'm cooked. I'm cooked through. Yeah. I'm crispy. Yeah.
Amos: I need an in-person conference. That's all I'm saying. Like I.
Chris: I'm dude, even I want to go to a conference at this point.
Amos: I'm going to hug people that I don't know. Just so you know, next time that there is any sort of Elixir conference that's in person, everybody out there, if you don't want to get hugged by random person, don't come because I'm hugging everybody. Whether I know you or not,
Chris: I am, I am very ready for an in-person conference, which is not a thing I normally really say. Yeah, I am normally like, eh, I could just not go to this. I'm like ready to see, I was talking to Voytech about this the other day. It's like, I am ready to see people. I'm sure I'm going to get there, and then after day one, I'm like, okay, I had enough.
Amos: I'm good. I'm just going to stay in my room.
Chris: I'm topped up. Now I'm topped up to the tank. The tank is full.
Amos: I I'll tell you what, uh, it doesn't have to be at a lazy river, but I'm ready for another lazy river conf too where we just sit around and talk.
Chris:
I'll tell you what. Yeah. That's the big thing. I just wanted to be able to sit around and chat with people again.
Amos: Um-hmm. Yep. Cool. Well, I guess I'll see you in person when we can, but for now I better get back to it.
Chris: Yeah, same.
Amos: Thanks, sir.
Chris: Alright. Have a good one, man.
Amos: Bye.