Thanks, Ian. 🙏 Plus Kevin Hoffman talks about empathy and the joy of logging ⚡
This show is supported by you, our listener. Stick around to live for the news to hear more about that. This is Cup of Go from 05/16/2025. Keep up to date with the important happenings in the Go community in about fifteen minutes per week. I'm Jonathan Hall.
Shay Nehmad:And I'm Shay Nehmad. Hi, Shay. Hello. You with the fancy equipment and the new camera.
Jonathan Hall:I've got
Shay Nehmad:my Congratulations on finishing your home setup.
Jonathan Hall:I got my old microphone set up again, my old webcam, my old big screen. You can't see it, but I'm enjoying it right now. Feels good to have a desk again after nine months.
Shay Nehmad:What sort of screen you were working with?
Jonathan Hall:What kind of screen?
Shay Nehmad:Yeah. How how many inches? A curved? It's a double wide what's the the Samsung I
Jonathan Hall:don't remember their their model number. They're famous.
Shay Nehmad:Wait. They don't sponsor us, so they can go No. F off. It's it's some brands.
Jonathan Hall:It's it's a it's a generic brand of generic monitor that's really big and cool.
Shay Nehmad:I miss my my old monitor at home, man.
Jonathan Hall:So we have a great show for you today, I think. We have a great interview for you at least. So if the rest of the show is terrible, at least the interview is cool. Just finished recording it. Gonna be gonna
Shay Nehmad:go Skip ahead. We're gonna be interviewing Kevin. We're actually recording this episode in such a weird way. We record the ad break first. So we have to talk about some things.
Shay Nehmad:I one of them is gonna be my meetup that's coming up. Yeah. And then we did this is gonna be the ad break we already recorded, and then there's gonna be the interview, which we already recorded as well. And then we're Forgive us if this show is kind of fuzzy. Let's just dive into it, start with meetup corner, right?
Shay Nehmad:Anyway, I was saying in
Jonathan Hall:the interview, I wanna finish that teaser. We're interviewing Kevin Hoffman, founder of Spark Logs. I mentioned it in the lightning round a couple of weeks ago. You want to hear both about why you should use Spark Logs and more important or more interesting, maybe why they chose Go and the benefits of Go in a serverless environment on Google Cloud and how it compares to Rust and all that sort of fun stuff. Stick around for that interview right after the ad break, right after the news.
Shay Nehmad:Which already happened.
Jonathan Hall:Which already happened.
Shay Nehmad:I need the
Jonathan Hall:the time to Okay.
Shay Nehmad:So meetup corner real quick. Do it. I am arranging a meetup, and I've talked about this in previous episodes, but now I feel more responsibility to do it because I've been upgraded to the role of organizer, co organizer of the Go San Francisco meetup group, which is huge, has like 5,000 members. So for me, it's like pretty pretty crazy. And many people signed up for the event already, like 17 people, which is a lot more than what, we originally planned when I think Andy originally was like, hey, I'm gonna be in San Francisco.
Shay Nehmad:Do you wanna sit for a beer or something? And then, Josh was like, yeah, sure. I'll drive over from Marin. Yeah. Whatever.
Shay Nehmad:Let's do a meetup. Now it's a big sponsored thing with the group and everything at the Elasticsearch's offices. So Tuesday, 05/27/2025, '5 PM at Elastic's offices in San Francisco. Let's meet there, meet cool gophers and have a good time. It's sponsored by Elastic.
Shay Nehmad:Josh Bleecker Snyder is gonna talk. Ishai Shore is gonna talk. And I'm gonna do a couple of go episode live together with Jonathan on stage. And there's no way that that's gonna go wrong in any way. Never.
Shay Nehmad:Technology never fails us. Never. So, yeah, it's gonna be cool. That's it. Awesome.
Shay Nehmad:Any other meetups coming up by the way that are or conferences that we haven't talked about yet?
Jonathan Hall:I don't think so. I'm probably gonna speak at Go West in in October, but I'll talk about more of that later when it's when it's official. Cool. Cool. Cool.
Shay Nehmad:Oh, now we're in the same country. Maybe I'll actually fly out and see you talk.
Jonathan Hall:Oh, that would be fun.
Shay Nehmad:How do you deal with hecklers?
Jonathan Hall:I wait until it's dark. All right. All right, so the big news this week, everybody's been talking about it. If you haven't been, what rock are you under? Ian Lance Taylor is leaving or has left the Go team.
Jonathan Hall:Which is a big He's one of the core contributors, of the like, I don't know, core figures behind it. He responds to many of the issues. In fact, the first thing that went through my mind when he retired is, crap, that message I just sent him on GitHub probably was after he was gone. Issue is probably never going to get fixed now. I don't know, I'll try to figure out who else to ping to get my issue fixed.
Shay Nehmad:Retired is an interesting word to use there because we don't know exactly what's going on there, right?
Jonathan Hall:Yeah. He doesn't say in his announcement exactly what he's doing. I hope it's good. We wish him the best. I certainly know that my experience of Go is improved, thanks to Ian's contributions, So Yeah.
Jonathan Hall:The the top thing on Reddit this week
Shay Nehmad:By the way, usually I open Reddit to see like if there's anything interesting going on and to bring stuff to the show. And the top thing would have like a hundred votes or, you know, a 50, Or if it's a really click click baity social media thing, has like 250 votes. This one's like 600. So Wow. Obviously, people care about that a lot.
Shay Nehmad:And almost all of the comments have been like end of an era, his calm and clear way to dealing with feedback was incredible, grateful for all the contributions over over the years. And also there's, you know, some people speculating about what this happens. There there are both Microsoft and Google are having big internal cultural shifts at the moment. Mhmm. Some people say it's due to AI, like LLM technologies taking over.
Shay Nehmad:I know from a friend at Microsoft that all senior people are expected to like, it's part of their performance review, like, how do you use LMs and things like that. And, you know, some cofounder, from Cygnus and Red Hat. So like, oh my god, he was the epitome of an ethical hacker. Humble, generous, wicked smart. I mean, people really, really seem to appreciate Ian.
Jonathan Hall:Absolutely. I mean, I don't know. He mentions this in his post, but one of his first contributions to the Go project was building the Go frontend for the GCC compiler, which I don't know how many people use that. It's not nearly as popular as the official Go compiler, but it's actually really, really important because was for a long time the second implementation of Go, which is really important for a spec driven language like Go, right? It is a little bit sad that that hasn't gotten the attention it needs really since Go 1.18, the generics, came out.
Jonathan Hall:So if you're using GCC Go, you're still stuck on 1.17. But it's still a really, really impressive thing that he really led that project of adding Go to GCC. And I hope that one day it gets the update it needs to run modern Go versions in GCC as well, because that opens it up to all sorts of architectures and platforms that the official Go team isn't able to support.
Shay Nehmad:Well, only thing we can say here is thanks a lot for all your contributions and for people who are interested more in details, it seems like he put a public announcement on his blog, which, you know, summarized is I was, in Go for a very long time. I left Google after nineteen years. Some self reflection about his approach, which is really cool. And then sort of an ominous summary of Google has changed, Go has changed, computer programming environment has changed. I am no longer a good fit for the Go project at Google and I have to move on.
Shay Nehmad:I don't know. These sorts of people who are so productive, I'm gonna have a hard time believing it's not gonna help other programmers someplace else, but he's gonna take a break for a while, which makes sense.
Jonathan Hall:Well, and and he ends the the post by saying, hope to be able to contribute to Go again in the future. So don't think he's leaving. He's not leaving the community. He's just
Shay Nehmad:changing his role within it. Maybe. We'll see. I don't know what, like, my opinion of Google is Google is a very big, enterprise. So, you know, I would protect them on the Chrome thing, you know, that the courts are trying to get them to sell off Chrome now, but I really hate how they manage their cloud stuff.
Jonathan Hall:Didn't they just announce that they're requiring all remote employees to move close to an office?
Shay Nehmad:I don't have opinions about that one way or another.
Jonathan Hall:I think Okay. But still, mean, it's disruptive and
Shay Nehmad:it's weird.
Jonathan Hall:If they have a policy of like, one of my best friends works remotely for Google. The nearest office is probably a three hour flight away. So it's not like it's reasonable for him to just pick up his family and move. So it's certainly disruptive and it's a change. I could certainly see somebody saying, I'm not a good fit anymore for a change similar to that, right?
Shay Nehmad:Yeah. All I'll say is these companies are so big that it's hard to have any sort of feeling towards them other than, oh, you're a huge machine that just, eats people and churns out profits for shareholders or whatever. Yeah. But when it hits, like personally with someone who would answer your, you know, GitHub issues threads and so thoughtfully, Yeah. I think that's why it, hits a lot of people.
Jonathan Hall:Exactly.
Shay Nehmad:Other than Reddit also, you know, they're they're obviously, the Go channel, the Go Slack has been some channels talking about it, including our own. But, you know, I think all the sentiments are are sort of similar, where most people are just like, end of an era, I hope everything went well, and he was really cool. He was really great. So it seems like most people super appreciate his work, which is cool, you know, having this sort of legacy over a project. Other than the big, that big announcement, which has more to do with the Go community than the Go language, we have, there isn't a lot of proposal work going on, maybe related, you know, I don't know.
Shay Nehmad:We don't know. But this week, the proposal has been kinda on a on a wind down, so we just have a couple of cool links
Jonathan Hall:to to To be fair, there were a bunch of proposals that they were all crypto things and adding things to the AST, which are important, but not really interesting to talk about in this format.
Shay Nehmad:And I'll say that too much about them.
Jonathan Hall:Yeah. But I don't wanna make it sound like the Go team is is not doing anything.
Shay Nehmad:No. Of course not. Of course not. It's not it's not what I meant to insinuate. I do wanna talk about a couple of blog posts, about security, and a couple of, links about AI that we found, specifically about Go.
Shay Nehmad:And I have my own, which is socket it's actually been languishing in our backlog for a while, but socket.dev's blog post about w gets to wipe out malicious Go models fetch destructive payload.
Jonathan Hall:Woah. And
Shay Nehmad:his name is John C. Yeah. It's a super I I have to say the the flowery language is like, come on. It's it's a bit too much. Like, introduction, the silent threat, Go's open ecosystem, a double edged sword,
Jonathan Hall:a breeding ground
Shay Nehmad:for malicious I mean, come on guys, it's code. We love we
Jonathan Hall:A single line of obfuscated Go code could wipe out entire disks.
Shay Nehmad:This is a bit too much. The actual content, if you look past the Flyeria language, is interesting. And I think it's worth knowing, typo squatting. We mentioned this a few times in the past. Sounds familiar.
Shay Nehmad:Because there's no not a lot of namespace validation in, the Go ecosystem, but at least built in, you know, it's it's possible to do supply chain attacks. Although the moment you're in a big enough company, you'll have something like, you know, Snyk or GitHub Dependabot or Jamie's dependency management data thing. Like, you'll have, things that look at your dependencies, right, at some point.
Jonathan Hall:Mhmm.
Shay Nehmad:But it is possible to to attack, you know, workloads by trying to deploy malicious Go models, having them have a nice looking Go name that's similar to other projects, and then having, developers accidentally install them. Has that ever happened to you in reality, by the way?
Jonathan Hall:So my first thought kind of answers that, isn't this kind of similar to the situation over the last three fifty years with NPM, where there's 6,000 different versions of the same thing? Sometimes it's even repackaged. The same thing has been repackaged by different people. You kind of have to really be careful. Like, is this the version of XYZ module I really want?
Jonathan Hall:Or is it some knockoff or is it some hacked version by some guy and it's a weird place? I feel like NPM has been like that forever. Are we just getting big enough, popular enough that we have to deal with the same problems?
Shay Nehmad:It's basically the same. The reason Socket is pointing those out is because they detect it by scanning packages, including NPM packages, by the way, for a code that looks malicious e and then trying to figure out why it's there and and how because it's not enough to publish a a malicious library. You people have to, like, install it for some reason.
Jonathan Hall:Right?
Shay Nehmad:Right. So they found the malicious library and this blog post about, well, how do people go and, install these? And the reason is like namespace confusion. You just import the module from GitHub, right, theoretically. If you don't use a specific like company managed proxy for your Go packages, right?
Shay Nehmad:It's really easy to to mess this up because you need to look at the whole the the package name is the same for for in this in this case, it's TLS proxy, right? Sure. Sounds pretty
Jonathan Hall:pretty useful. It
Shay Nehmad:sounds like super generic and whatever. And then you sort of look at the path and you try to figure out if it's the right one or not and it's really hard. The only the only way I know to do it that doesn't involve actually opening the code and looking at it, just looking at the number of imports and seeing like, all right, a ton of people import this. This is probably the one I want. Mhmm.
Shay Nehmad:Which is not not great.
Jonathan Hall:It's not the best, but it's better than nothing. Right?
Shay Nehmad:I we are super tight. Where I work right now, we're super tight about introducing new dependencies. I used to be very laissez faire about it, but now I'm like, let's just copy or, like, generate the the tiny piece of code we need instead of introducing new dependencies just because it's so much headache to deal with dependabot if you actually have to. Anyway, this blog post, just shows the the malicious module itself. The code in the library is trying to be like fancy about it, which is cool.
Shay Nehmad:It has like an array that has a string that's obfuscated and then a super weird way to execute it with super weird variable names, but it ends up just doing, you know, running a shell command. And when you decode it, it just fetches a super destructive shell script. One thing I've learned this week is if you look at this blog post, you can't click on the link, right, if you go to Decoded Nintendo or whatever, because it has been, defanged. You heard about this No. I don't know
Jonathan Hall:that term.
Shay Nehmad:So let's say this is really a malicious URL that really includes a malicious pass script. Right? Uh-huh. And you have a thing that's like crawling the site and trying to like go through all these links. Then it downloads it and it kills its machine.
Shay Nehmad:Right? Yeah. So for all these security blog posts or and and specifically when you do security disclosures as well, which is where I learned from it because my wife is a is a security researcher and she taught me, you have to, it's like common best practice to defang the URLs, like make them non clickable by replacing dots with, you know, brackets, dot brackets or something like that. Just so people or machines reading the report don't accidentally go to the actual IOCs, indicator of compromise and get a compromise themselves.
Jonathan Hall:So is that like when somebody says contact me at, in my case, Jonathan, and then space, the word x?
Shay Nehmad:AT, space.
Jonathan Hall:Is that called defanging as well?
Shay Nehmad:I don't know. It depends on why you do it. I think defanging is specifically for malicious URLs because otherwise then it probably
Jonathan Hall:is because I'm sure emailing me is a bad idea. For
Shay Nehmad:sure. And unfortunately, the the destructive payloads in this specific case
Jonathan Hall:Mhmm.
Shay Nehmad:Just copy zeros all over your shell, which is really an asshole thing to do. Don't do that. I don't know what's the benefit, like, what's the upside of the people developing that as well? Like, it's not ransomware. If you wrote zeros all over the disc, you you can't get any money out of it.
Shay Nehmad:Like, what's the point? Just being But, yeah, cool blog post from Socket.dev, a bit flowery, but two new terms that you might have not known. Three, obfuscation, typo squatting, and defanging. For me, one of these was new this week.
Jonathan Hall:Mark your bingo cards.
Shay Nehmad:For sure. And we haven't said AI yet, so your middle square should be still Reset
Jonathan Hall:the timer. Let's do one more. This is a blog post that comes from our friend, Christoph Burger.
Shay Nehmad:Christoph.
Jonathan Hall:Yay. The blog post is security, the habits that matter most. We talk a fair amount. It comes up pretty frequently as you would expect in a show about a programming language, security. But it's such a big topic.
Jonathan Hall:We've even had experts on the show whose whole job is security research and stuff like that. Most of us don't spend that level of effort on security, but this blog post is here to sort of give you the, I guess, how to tackle the lowest hanging fruit, would you say?
Shay Nehmad:Mhmm. I I wouldn't say the lowest hanging fruit because that implies it's the easiest things to do. I would Mhmm. Say the the most impactful, reasonable things to do. Okay.
Shay Nehmad:Sure. And I really like the the intro, which is like when I'm writing code, I don't worry about security because I wanna make it work and I wanna make it nice and I wanna make it maintainable. For me, that rings so true. Right? Like, you have to think about security because it's like an adversarial mindset.
Shay Nehmad:And when you program, it's more like a creative mindset. Right? You wanna make the software work. Mhmm. So I really like this, list.
Shay Nehmad:There are two lists here. There are three lists here actually. First one is about, just like code security. Second one is about development security. And the third one is about supply chain security.
Shay Nehmad:We could dive deep into each one of these topics, but let's just to keep the people coming to the actual blog post and reading it, you know, the top one thing in security measures to add to code according to Christophe is input validation. That makes sense. Right?
Jonathan Hall:Yeah. And and I I mean, think closer to standardization, which is, of course, on the same line there. You can't always validate something, but you can sanitize it. Yeah. Like query parameters in database string, for example, right?
Jonathan Hall:Maybe you validate that it's a valid email address, you at least use a query parameter so that it gets quoted properly when it's inserted in the database. I a real actual bug like this for a client just a couple weeks ago. It wasn't a database.
Shay Nehmad:Was A real open one staring at me right now, and because I don't use Go, but I use TypeScript, I'm suffering my way through trying to solve it properly. What was the bug for your client?
Jonathan Hall:Yeah. So it was, passing unsanitized, unvalidated query parameters from one from the end user to a back end API used for payments.
Shay Nehmad:Damn. It doesn't do that anymore. And, you know, for for Patreon members, we can tell you what the payment subscription to remove no. I'm just kidding. No special things for Patreon other than internal recognition.
Shay Nehmad:This is a really good blog post. I wish we had time to dive into every single one, but it just has really good tips. If you're, you know, some sort of the person who's writing the coding guidelines for your organization, underneath security, I would just put this link. Like, this is the 95% of what you need and then the extra 5% that's special to your case. So The it's really good.
Jonathan Hall:I said low hanging fruit, and act there is some low hanging fruit here, and I wanna call that out. When I when I join a new project, almost always the very first thing I do is I add static analysis and linters, which are two of the things on this list. It's such an easy, easy thing to do. It's often just adding one line to your CI script, assuming you have a CI script, and it it pays back so quickly. So that's that's an example of low hanging fruit.
Shay Nehmad:Yeah. And Christophe lists some specific examples here that are worth adding. Go vuln check, go sec, static check. That's pretty easy to get started with, I think. One last thing that's important specifically to what we just discussed, right, supply chain security is minimalism.
Shay Nehmad:Just like, hey, don't do too many, you know, don't bring too many things and you use like containers or Unikernels or whatever, which is cool. I don't I have never used UniKernels before, so I'm interested to see that as a recommendation.
Jonathan Hall:Well, once the proposal we talked about last week about, compiling to Go OS equals none is done, you could even deploy your Go application without an operating system.
Shay Nehmad:Goddamn. That's the And, you know, this is list of lists, and, there are a lot more options, which and and resources about it, which Microsoft, very generously listed at the end of the blog post with the sign off, stay secure, which is funny because usually the code rewrite doesn't keep us secure. It keeps us our users secure. But it just had the image in my brain of, like, angry users running after you, like, with pitchforks and torches. Stay secure from those angry villagers.
Shay Nehmad:Alright. We usually have a lightning round here, but we have an interview and this show has been going on for pretty long already. So we'll skip that and maybe we'll have an extra long lightning round next week. Let's do it. Onto the ad break and stick around for the interview.
Jonathan Hall:Do it. Don't leave. Don't hang up. Don't hang up. Wow.
Jonathan Hall:What interesting news that was. I can't wait to know what it is when we record it in the future.
Shay Nehmad:Yeah. For sure. That was very insightful and or, clickbaity and or interesting and or boring. Welcome to our heartbreak. We recorded before the show this time because reasons.
Shay Nehmad:As mentioned at the top of the show, this show is supported by you. And last week, because of a request from one of y'all, we introduced a new Patreon level, is slightly more affordable. And we have four new Patreons, which is I think the most we've ever onboarded in a week. And we really, really appreciate you joining to Cup of Gopher mini level. Remi Zendvichig, Peter Skezezi, Filip Vranzcevic, and Steve Niklin.
Shay Nehmad:We really appreciate it y'all. It's not about the, like, how much you give, it's just the fact that we have support. I think it's way more meaningful. So if you consider joining as a Patreon in the past and you were like, $8, that's a bit too much, we now have a cup of gopher mini level, which is $3 a month.
Jonathan Hall:Yeah. And big shout out to Steve, who you just mentioned. He's the one who emailed us and requested this level. So he's like already a four x Patreon member since his suggestion.
Shay Nehmad:A %. Hundred people on board. And therefore 50% of the pro what? No? Okay.
Shay Nehmad:In other ways to talk to us and support us and join the community, etcetera etcetera, you can reach us at kapago dot dev, that is kapago dot dev. At the Gopher Slack, we have a channel, kapago, kebab case with hyphens, or you can email us news@cupofgo.dev. That is news@cupofgo.dev. Other than supporting the show directly financially, which helps because this is a pretty expensive hobby, you can help in other ways by spreading the word, leaving a review on Spotify, Apple Podcast, Overcast, like all these ratings and whatever. They actually do help, put the show in new people's faces more algorithmically.
Shay Nehmad:But remember, Jonathan, the last like great recommendation you heard about a book or a movie or a podcast, was that from the algorithm TM or was that from, like, a person?
Jonathan Hall:It was from a person, for sure.
Shay Nehmad:Yeah. So, you know, go recommend this show to someone. We all love the algorithm TM and specifically for listeners of the show, the algorithm TM probably, you know, pays for a large chunk of your salary. But, recommend the show to a friend or a colleague, or a co student or just anybody around your Go community. Like I mentioned, during the news segment, now we have to remember to do that, I'm now a co organizer of Go San Francisco.
Jonathan Hall:Woo hoo.
Shay Nehmad:If you wanna catch up on the next meetup, we at Go SF, have, which is meetup.com/golangsf, have an upcoming event which I'm like doing and we're gonna record a live episode of Cup of Go for. So basically, if you're listening to this and you're in the area, I think you're the perfect audience. But even if never listened to Cup of Go, it's just gonna be a cool meetup with like that we mentioned, you know, in our meetup portion of the news. I just wanted to mention that because I'm arranging it and I really really want a ton of people to come because I don't wanna be there alone. That's gonna be a bit embarrassing.
Shay Nehmad:One last thing to mention is our swag store, which you can also find on, cupogo.dev. That it's store.cupogo.dev, where you can find some cool swag, cups, t shirts, sweatshirts, stickers, and a single, very sad, you know, wireless charger that someone bought once, like, two years ago, I think, at this point. Maybe it can be the second one. We've looked at other swag options, by the way. I've looked for YubiKey covers, but our printing, company doesn't provide those unfortunately.
Shay Nehmad:It's a bit too much of a niche item.
Jonathan Hall:I want refrigerator magnets.
Shay Nehmad:That's a cool idea. We we will look into refrigerator magnets and mostly from people who would actually buy it and it's not our money. We would love to hear your opinion about, what's the sort of swag you would like. We're really not into like I don't I don't wanna make swag that's just useless stuff.
Jonathan Hall:Yeah. Because of these print on demand things, we could go crazy and have thousands of things there, but they would just clutter
Shay Nehmad:the Yeah. It doesn't cost us anything. But we want stuff that you actually want to like our like our new Patreon label. So that's it for our ad break. Stick around for a super interesting interview coming up, perhaps a bit flaming, perhaps Rust versus Go.
Shay Nehmad:Spicy takes. See you there. Hey, Jonathan.
Jonathan Hall:Hey, Shay.
Shay Nehmad:How's my ACDC singer impression?
Jonathan Hall:Let let's let's compare it now to the real one and see how it how it compares.
Shay Nehmad:I'm actually I actually did a live cover of ACDC, but I don't know. I guess I don't know about I don't have that spark. I wish someone was here who could help us understand.
Kevin Hoffman:I could jump in on the piano.
Jonathan Hall:Oh, that would be sweet. I've never heard thunder start with a piano. Oh, wow. Hi, Kevin.
Shay Nehmad:Yeah. Hey, Jordan. Thank
Kevin Hoffman:you. Thanks for the invite.
Jonathan Hall:So I didn't know Kevin for many years. We worked together. He was my boss for a while, I guess, or skip level maybe. I don't know exactly how that was. It's been so long.
Kevin Hoffman:Good colleagues.
Jonathan Hall:Yeah. Yeah. But you're on a new startup now and that's what we're going to talk about. Before we talk about that though, tell us a little bit about you personally, who you are, what you do, maybe how that led to this interview.
Kevin Hoffman:Yeah, fair enough. So I'm a techie at heart, long time doing software engineering, started out in game development in my teens, like every teenager does almost, you know, and then jumped over to business software and computer telephony, voice recognition. Eventually, that led to my first startup, which took twenty years before it was a success. That was in data backup and disaster recovery, and had lots of adventures there eventually had hundreds of petabytes of data, lots of systems, to manage. And so recently, that company was acquired and I had this idea for a second company and being a techie at heart, you know, just scratching my own itch with the problems I faced some of the observability problems I faced in my first company.
Kevin Hoffman:And so now I'm trying to go and build the tool that I wish that I had ten years ago, sort of a thing.
Jonathan Hall:Nice, nice. And that's spark logs, right?
Kevin Hoffman:That's the Yeah, exactly. Spark logs, observability, logs, traces, metrics, etcetera.
Jonathan Hall:So I mentioned spark logs in our lightning round two or three weeks ago just because I've been using it as a client using their free tier and enjoying it. But we're going to dive into more. I don't know. I'm curious to learn about your vision for the product and what that itch is that you're trying to scratch. So yeah, is that a good place to start?
Jonathan Hall:Suppose what's the problem you're trying to solve here?
Kevin Hoffman:Yeah, and that'll lead right into, okay, why did I select Golang to write back end So at Acxiant, we had lots and lots of data and systems. So hundreds of petabytes of data, thousands and thousands of systems of various kinds running on prem in three different public clouds. And it was an absolute challenge to manage observability for those systems. So we're using a lot of open source tooling of various kinds, The Elastic ELK stack, you know, everyone starts there and then you branch out from there. But, it was really challenging because those tools, you have to have a lot of infrastructure to run.
Kevin Hoffman:So we'd have NVMe drives in our data center across various racks and literally a hundred plus VMs running this elastic cluster ingesting, you know, tens of terabytes of logs every month from the thousands of systems. And inevitably, the SaaS application might be having an issue. And during an issue, the log volume might increase by a factor of five because it's all kinds of bad things are happening. And then your logging tooling starts to break because you're running steady state and it's not, you know, it's not in the cloud. And so it's starting to break and then you need to do queries to figure out what's wrong and it takes two minutes and it's like, it's really frustrating.
Kevin Hoffman:Plus, like, the SRE team spent a lot of time managing those open source clusters at scale. And and so, like, you know, it's really good tooling, and you can do really well with those tools, but it requires a lot of infrastructure and expertise. On the flip side, like there's a bunch of really good commercial tools for solving those problems, but they're just super expensive. Like it would have been for us over a million dollars a year or more for commercial observability tool. And that was never in the cards.
Kevin Hoffman:And it's a lot of software engineers you can hire for that much money. And so we we never could. But, know, it just I got to thinking, like, there's got to be a better way. Can there be something that's serverless and can scale from zero to petabytes? Can there be something that has mostly automation and convention over configuration?
Kevin Hoffman:And can it be super cost effective, at least by a factor of 10 or 20, so that anybody could run this at scale and be happy? Finally have a good UX. So it's like, make it fun, make it scalable, make it affordable, you know, go for the trifecta, no compromises as possible. And so I started thinking through, yeah, what what could be done to solve that problem? And eventually, you know, Sparklogs was born as a company as we figured out how to do that.
Shay Nehmad:My first thought when, you know, I've I've I've done some, observability things in the past, like not as a just as a consumer. Right? So I'm always on the side that's like trying to write queries, trying to write dashboards, and I've done like the rounds. Right now I'm using a solution I'm super unhappy with, which is Azure like native KQL dashboards, which are just like so you mentioned the word joy, it could be this it's not exactly despair, but it's almost the other way around. But my common wisdom is, oh, if you're willing to pay infinite dollars for good observability, you get Datadog.
Shay Nehmad:And the rest of them are just the same. But there are like differences there.
Kevin Hoffman:Yeah. And you usually have to have like compromises and trade offs like cardinal cardinality limits. Like, don't, you know, control the cardinality in your metrics or, you know, worry about having a limit on the number of fields. Like, don't have more than a thousand custom fields with your logs or, you know, you have to manually set up schema mappings. And it's just all these really annoying things.
Kevin Hoffman:It's like, thought an AI was supposed to be doing most of this grunt work for me these days, you know, or it's not an AI couldn't couldn't it just, you know, do a lot of that work for me and make it more fun? Rather than wrestling with the tool, I'm I'm actually trying to solve problems or, you know, get better information. So
Jonathan Hall:So the this
Shay Nehmad:pricing comparison that's like it's not exactly top and center, but it does exist on your on your homepage of like monthly total cost, comparison, is mostly about how much, size you ingest in a month. Right? How how does that like end up? Because I not not to question the validity of the data or anything like that. Just remember that Datadog is pricing is so vague.
Shay Nehmad:Never could, like, even guess how much I'm gonna pay.
Kevin Hoffman:So Right. Well, it changes a lot as well, but typically the big commercial tools, they'll charge by how much you ingest and then how long you retain the data. And then sometimes they'll add how much are you querying the data. Some of the some of the new pricing models that I've seen is ingestion is free, but then we'll charge you a lot for querying, which I think is even worse because then it's like, well, you know, the worse things get, the more I'm gonna pay. And it's really hard to predict your costs.
Kevin Hoffman:Like, one of our customers was actually paying five times more for AWS CloudWatch querying than they were for ingestion.
Jonathan Hall:Oh, goodness.
Kevin Hoffman:You know? And so, like, thousands of dollars a month in query costs. And so like, ingestion cheap, cheap ingestion is actually pretty easy, you know, but because you can just stash it in s3 and or to a column oriented database, but how do you actually query petabytes at scale, and have it be cost effective is where we, you know, we introduce some some technology that allows us to have a fixed cost per query no matter the scale, and let people dive in hierarchically and explore their data. And that that lets the cost be so low even though, yeah, ingestion is 15 times lower than the ingestion price for Datadog, but then the query cost can be bundled in still and and and you have a flat cost still. The other interesting angle there in pricing is just disconnecting retention period with the cost of ingestion.
Kevin Hoffman:So because the, the back end is built around BigQuery and storage there is relatively inexpensive, it's like, hey, we can let you retain up to a year because it's highly compressed, and we have this fixed query cost. And so we're fine with you querying older data or large datasets because it doesn't actually cost us more than if you were querying only three days of data or fourteen days of data, the more traditional retention period for a for an observability vendor.
Shay Nehmad:That's interesting because you don't know how long you need to, it's always a bet, right, how long you need to retain. Like, maybe the bug was thirty one days ago, and I just now realized it.
Kevin Hoffman:Yeah. It's like, you know, I need to prove to my cloud vendor that the issue is theirs, and please give me this credit, or there's this compliance issue, or there's this security issue. So normally, what you do is you archive your old observability data, then you have to rehydrate it when you wanna query it. And it's kind of a big pain. And so it was, you know, another key thing was how do we unify and only have something that can be queried live all the time, even if you wanted to retain twenty years of observability data, etc.
Kevin Hoffman:So, yeah, so that's another unique requirement that we thought through when we were putting the foundations of spark logs in place.
Jonathan Hall:So you've talked about the sort of the high level goals, pricing. Let's talk about technology a little bit. So what languages did you consider? Obviously you settled on Go, which is why I invited you on the show, but I'm sure you consider other options. How did you come to that decision?
Kevin Hoffman:Yeah, so, you know, there's a lot of popularity in having a unified front end and back ends, you know, Node. Js or whatever. But I knew that this was going to be an at scale system where performance was really important because cost was kind of a primary factor. And so it's like, yeah, we should probably use a language that's known for being very efficient and productive at the same time. And so, at Acxiant, we'd been using both Rust and Go fairly extensively.
Kevin Hoffman:We did a lot of low level systems engineering, even some kernel development, but both on the on the client side as well as the server side. And we had at scale systems with Golang on the server side, and then kind of at scale deployments of low level Rust code. And I've, I've written a bunch of code on both sides, and we had excellent teams who had written even better code. And so it came down to, you know, in this case, like, okay, we knew we wanted to build on the public cloud. And so what was going to help us be very productive, and in this case, it's around the Google Cloud.
Kevin Hoffman:And Google has excellent tooling, of course, for Golang and their cloud SDKs and APIs. And so that was a big win. I'd most recently been using Rust and really love, like, memory safety and the the raw performance and the level of control that you have. It was also quite challenging, though, with, the development cycles and the compile times. And, I found that the the Go Cloud tooling for Rust was pretty immature a few years ago when we first started this.
Kevin Hoffman:And so, it just became clear, like, if we wanted to be really productive, sure, we might have 10% less performance than if we use Rust, perhaps. I mean, that's debatable of course, but Rust would allow, you know, Go would allow us to move probably 5x faster in terms of development speed and productivity in this case than if we had used Rust on the back end.
Jonathan Hall:That seems to be sort of the common theme that I keep hearing is that if you really want to get the best performance, Rust is going to generally be the winner over Go. But if you're looking for the best like velocity of developing new features, Go is going to be the winner. And I haven't used Rust, so I can't really comment on that, but that seems to be what you're saying as well as what I'm hearing elsewhere.
Kevin Hoffman:Yeah, and we can get into it if you want, but like performance, there were some big surprises, like because Go is garbage collected, if you don't know how to tune it out of the box, it can perform really poorly. And so it does need some tuning. And you can't assume like you have to profile. And that's why probably Go makes it really easy to do profiling and getting flame graphs for both CPU and allocations. And so we found we were doing that fairly early.
Kevin Hoffman:And we can talk more about test driven development, which was core to how we did things. But, Go's philosophy seems to be, you know, to iterate fast and to have make it easy to prove things about your software, including performance. And so that that became essential as things got to scale, you know, having those assurances built into the development process.
Jonathan Hall:Do you wanna say something, Shai?
Shay Nehmad:A million things. The the main thing I I keep hearing in this is, oh, I wish I had time to learn Rust or I wish I had time to program everything in Rust so everything will be more productive. If your main, or not main, but one of the tenants of the thing is it needs to be fast and needs to be cheap, seems like no brainer to pick like the best program the like the the cheapest programming language possible. But you also want to develop these features. Now, if this was a, you know, completely new idea, completely new market sort of thing, then iterating super quickly and figure out what people want is a super high priority.
Shay Nehmad:Don't you think that for something like Sparklogs, you already know what people want because you develop it for yourself. You know what you want. You've used all these competitors in the past. Maybe there are some new ones out there, you know. I I for one really like ground cover.
Shay Nehmad:Just they're also friends just to to say, but, you know, some people do try to innovate on these observability products. Most of them are like, can you show me the most recent errors? Can you show me spikes? Can you show me anomalies? Can you do it fast?
Shay Nehmad:Do I need to learn a new query language to figure all this shit out? Like most like most of the features the feature set yeah. Yeah. Exactly. So why is it important for you to move fast if you already know what you're gonna build?
Kevin Hoffman:Yeah. I mean, as a startup, you can either raise a bunch of money and get lots of resources and and go chase something really big as fast as possible. In this case, the philosophy is, yeah, we you know, I really deeply understand this use case and, we wanted to bootstrap. And so it was like, okay, limited resources, we know what we need to do, but it's like the faster we get to value and deliver value to customers and get more customers in the door, then the faster we can turn those resources again, to more innovation and value. And so the need to get to deliver value more quickly is essential to the business in a bootstrapped environment where every minute counts of how you're spending your time essentially, especially in the earliest days.
Kevin Hoffman:And so that productivity with the language is essential. Like, know, one of my favorite quotes is that software, you know, good software engineering is an act of empathy. Ultimately, what we're trying to accomplish is to understand the needs of customers and other actors in the system and deliver something that solves the problems and makes them feel some emotion. You know, in our case, we selected joy, for the experience, but it's just how do we work together? And that's often much easier in environment when a language and framework that's by convention focused on this iterative approach, this this very interactive, you know, simple convention based approach and has a big community and support.
Kevin Hoffman:You know, in our case, it was very aligned with the Google Cloud. And so we had a lot of things to build off of gRPC and all of the cloud APIs and and all of the things that we needed for highly concurrent, asynchronous, long running server side, processes, that was all really easy in Go, and Go is really well suited for that. So, you know, your use case could be totally different, and then you should select the tooling that will give you the biggest head start, for your particular use case.
Jonathan Hall:I'm curious if you found areas, you may in the future, maybe not yet, but if you have found areas where Go's working okay, but you just need a little bit more performance out of that, then you might consider rewriting something in Rust.
Kevin Hoffman:Yeah, I mean, the most cost sensitive portion of the whole product is in the ingestion layer. You know, that's where it's like all the data is flowing through, you know, terabytes to petabytes of data. And so, you know, even a 10% speed up at scale could mean a difference, a big difference in the cloud costs. We just there's so many other things to do before optimizing cost that that may not come for years, if ever. And so over time, hey, you can use managed Kubernetes clusters to run Cloud Run-in Google rather than the managed Cloud Run.
Kevin Hoffman:And you can you know, then that gets you to spot instances. And, like, there's other things you can do besides tweak your language that give you an order of magnitude better improvements well before you say, okay, I'm just gonna ditch my language of choice. And so, often yeah. There's often much, you know, many other ways to solve that same cost problem that will require less time or less distraction for your business.
Jonathan Hall:Right. Right.
Shay Nehmad:In terms of, I really liked what you said about, software development as a is an act of empathy. I'll I'll I'll definitely take that into my next design review and try to apply that.
Kevin Hoffman:And it's not my quote. It's I think I got it from like a first round
Shay Nehmad:of our We heard it here. You made it up. Folks, you heard it here first.
Kevin Hoffman:There you go. Yep.
Shay Nehmad:I'm just kidding. But I am worry I I am interested. What is, like, cool and joyful about Parksel? Let's assume I set it up and let's assume it's cheap. Let's assume it's fast.
Shay Nehmad:What's cool about it?
Kevin Hoffman:Yeah. I mean, right now, it's really good at logs and it doesn't do metrics, you know, so just that's the caveat. But it's really, really good at like, making it easy to query large data sets for logs, you know, we're think hundreds of millions or billions, you know, of results. And I can zoom in interactively with the histogram, and it was written in Flutter on the front end. And so it's like it's a native mobile app experience that's super smooth.
Kevin Hoffman:And yeah, it's running on the web right now, but it's, you know, it selected a framework that was designed to feel more like a mobile app that was highly responsive and interactive and more real time rather than, I've got this big clunky web app where most of the work's happening on the back end. There's actually a lot of work happening on the front end to make it feel interactive and snappy and, you know, just easy to zip around and explore your data. Cool.
Shay Nehmad:Have you do you like dog food, Sparklogs into Sparklogs? Have you had a moment where you used it and you're like, oh, cool.
Kevin Hoffman:Yeah. I mean, we don't use it to dog food yet because we don't actually produce that many logs. You know, like, the Google Cloud tooling is fine.
Shay Nehmad:You you just tail a text file at this point. Yeah.
Kevin Hoffman:Yeah. It's like, you know, Google Cloud has a logging thing. Right? It's like all the public clouds do and
Shay Nehmad:Is it good?
Kevin Hoffman:Open the cloud run. So that's fine. But, like Is
Shay Nehmad:it good?
Kevin Hoffman:I mean, it doesn't bring me joy.
Shay Nehmad:Azure one is just horrible. It's fine. Don't wanna I again, they I I don't wanna bad mouth it too much. I I didn't super go deep into it and try to become a super user, but it it's not like one of these product y products where you, like, ground cover or Datadog that are, top of the line you go in. It's like, my god.
Shay Nehmad:Everything works. Everything looks cool.
Kevin Hoffman:Yeah. It's really good for logs. Like, if you've struggled with terabyte scale logs before, you know, our first customers are those who had 10 plus terabytes of logs and, you know, really having a bad time with fragmented logs or other things in other places, and they've been really happy so far.
Jonathan Hall:So I'm the complete opposite end of that spectrum. I'm using Spark logs because I had an attractive free tier. Actually that's not even the main thing. The main thing is the UI wasn't as terrible as the competitor I was using before that had an attractive free tier. I'm a very early stage startup.
Jonathan Hall:Let me just pull up the dashboard here and I'll tell you how many logs we've ingested. Give you an idea of the tiny scale here.
Shay Nehmad:Just so you know exactly what number minus one to lower the free tier two. There you are.
Kevin Hoffman:The special Jonathan plan. Yes.
Jonathan Hall:In the last thirty days, I've ingested 67 megabytes of logs.
Kevin Hoffman:There you go.
Jonathan Hall:Oh my god. It was about was Just a trickle. Right?
Shay Nehmad:I thought you were going to say 67 logs.
Jonathan Hall:126,000 events for whatever that's worth. But I guess I was thinking about making this point earlier when we're talking about pricing. When you build a product that's designed to scale to petabytes, it also kind of works the other way kind of magically, right? That it also makes it really easy to offer a free tier that it doesn't cost you anything effectively to offer this to a small startup like me. And then when my presumably if my startup or some startup succeeds and we do have gigabytes or terabytes or petabytes of logs, we'll stick with you.
Jonathan Hall:So
Kevin Hoffman:Well, that's one interesting thing. Sometimes startups I see will have an enterprise plan and it's like, oh, run it in your own cloud environment. And they run the whole darn application in somebody else's environment. And, like, that's a real pain. Like, is this startup okay?
Kevin Hoffman:Like, now I've got potentially hundreds to thousands of instances of my software in other places. And so, like, in engineering this, it was like, how could we let people have a private cloud environment? Bring your own Google projects, but we'll only use BigQuery. There's nothing to manage. And so, yeah, now you have all the data sovereignty and other concerns of an enterprise, but all the compute logic still lives in the same place.
Kevin Hoffman:And so, you know, when designing for petabyte scale, some startups will say, okay, that just means we're gonna have separate, infrastructure for these large customers, and they scale it manually behind the scenes. So, like, to be serverless, you know, then you can get the advantage. Like, serverless, can I scale two petabytes on the one hand and down to zero on the other hand? And can I apply that? And so that's why it was easy to offer the free tier.
Kevin Hoffman:It's like, yeah, you're not actually costing anything because it's so cost efficient, but there's nothing special extra running. And so just if you're designing your own SaaS app, just think through who am I targeting and why am I ever gonna go down market or not, you know? And if so, can I do this in a serverless way? And do I really have to offer full private cloud? Or could I just put the data there?
Kevin Hoffman:You know, there's new ways, especially in the Google Cloud, but other clouds too, where you can link and authorize certain resources across accounts, and you don't have to do the traditional thing there.
Jonathan Hall:I would like to loop back to performance. We touched on that a little bit. I think that's an interesting topic we don't talk about very often on this show. So before we recorded by email, we talked about tuning the garbage collector and profiling and some of this. Why don't you just talk us through?
Jonathan Hall:What are some of the problems you had and how did you track them down and what was the solution?
Kevin Hoffman:Yeah. So one of the core features was this thing called auto extract, where it takes your raw text logs and it says, oh, yeah, I recognize all these variables that humans have scattered amongst this text in here. And I'm gonna automatically extract them out and put them into structured data in addition to your text log fields that you can search. And then I'm gonna recognize all the types. And so naturally, that's a lot more compute intensive than just ingesting plain text and shoving it into another system.
Kevin Hoffman:So there's a lot of compute and processing that happens over this textual data. And so step one was, okay, coming up with all the algorithms, improving it's correct, and all the unit test coverage. And then we did some basic profiling. It was like, wow, this is only doing like one megabyte a second per CPU core. I thought Go was supposed to be a fast language.
Kevin Hoffman:Like, what the heck is going on here? And so, you know, you start doing some profiling, and then we got into, you know, like, allocation is actually a really big deal if you're lazy about how you allocate data and go along the hot path. Like, you shouldn't do it everywhere. But with profiling, we found, like, 80% of the CPU time was spent just in the GC, you know? And we wouldn't have known that without the really good profiling tools.
Kevin Hoffman:So like, you know, the flame graph tooling that's built in and makes it easy for Versus code, like 100 recommended. And so anyways, you know, then it wasn't too difficult to see like, okay, we need to use a JSON library that manages its own internal pool of of, string data and other data and data structures and reuse them. And then we have to get into GoGC equals 200, you know, changing the ratio of how often the GC is triggered and then telling it with GoMemlimit, you should use the full two gigabytes of RAM that I've allocated this container so that it's only doing a garbage collection run occasionally. And then we're doing one run as far more efficient. And so like out of the box, Go tries to use as little RAM as possible.
Kevin Hoffman:And and so like, if you're running these workloads that are memory intensive and throughput driven on a on a controlled environment like a back end, then you should definitely tune the garbage collector that way. We also ran into trouble, and this is where, you know, it's like the Go versus Rust thing. The fast JSON library that we're using uses a lot of unsafe tactics. And so, like, if you, use the data that that library returns beyond the lifetime of when, you return the end of that, thing back to the pool, then you'll actually crash the program or corrupt the data or worse, things will start changing. And in a highly concurrent environment, like, lots of bad stuff could happen.
Kevin Hoffman:And so, you know, that was experienced early on in testing. And so we had to write a bunch of tests that, like, execute 10,000 concurrent Go routines at the same time and stress stress this to make sure that there are no uncaught, unsafe memory bugs, which Rust enforces at compile time and Go happily lets you do a bunch of unsafe things that, you know, may not be caught until runtime. So but that will only we only need to do that in the one area. That was the JSON the library that was crafting all of this JSON behind the scenes. That was the profiled sensitive throughput, you know, throughput sensitive area.
Kevin Hoffman:And it made the code a lot, you know, a lot more intricate there, but everything else was really clean and more traditional Go.
Shay Nehmad:The flame graphs you've, you know, you're collecting and using, is that something that you do regularly or is that something that you do when you see a performance thing go down? Because stuff like a compiler guided optimization and things like that, you might be tempted to collect profile data all the time, as well from all customers. But, you know, some customers but collecting profile data has its own overhead even though it has gone lower in recent years. Like, with the performance being such a high focus, I'm wondering like, what what do you do? Not what do you think it should be done, but prag like pragmatically what happens in real, you know, companies like yours?
Kevin Hoffman:That's right. Yeah. I mean, we haven't actually deployed any, profile guided optimization yet, even though some of the newer versions of Guling make that even easier to use. And the best practices, yeah, collect it on some portion of your production workloads and feed that data back in. I'm sure we could get an extra 10% of, you know, performance out of that probably if we we did that.
Kevin Hoffman:It's just we're not focused on costs anymore. It's like we got good enough to where cost isn't a factor anymore. So we have benchmarks that we run, you know, against every pull request. And if there's a performance regression, then we look into it. But as long as it's good enough now against this baseline, then we haven't gone further into some of those more advanced techniques.
Kevin Hoffman:But more often than you might think, as we enhance the product or make changes, you know, there are performance regressions, it's fairly easy to do in the hot paths. And so, you know, like, it's really important to have those basic benchmarks that you run with every code change. So you're aware, like, you know, this this could be really different.
Shay Nehmad:So this happens like on CI because you don't want, degradation. So in every code change?
Kevin Hoffman:Yeah. You know, we're using GitHub and pull request actions and those sorts of things. So that's really easy to do these days. And then that feeds into how we deploy with Cloud Run and you can deploy. It makes it easy to do canary deployments or percentage deployments and kind of watch how things as you roll it out more gradually, potentially that sort of thing.
Shay Nehmad:I wish more software companies did performance checks on every release. I'm looking at you, the software that, Jonathan and I use to manage our backlog and is getting slower every week.
Kevin Hoffman:Yeah.
Shay Nehmad:Which will not be named But
Jonathan Hall:you know what it is,
Shay Nehmad:I mean. Ends with hello. And oh my god, I can't stand it anymore.
Kevin Hoffman:And maybe some customers like infrastructure software, that's where I came from, you know, data back up and recovery, the non functional requirements is 90% of the work. And so, you know, it's used to having this rigor. Other companies, it may be, you know, the last thing on their mind and that's fine with their users. Right? So it just depends who you're targeting and why.
Kevin Hoffman:But I definitely agree with you. It's good rigor.
Jonathan Hall:I'm curious to hear what's coming next. So you you said that you're doing great with logs. You've also talked about doing doing some other things. What's what's next on the roadmap for Spark logs?
Kevin Hoffman:I mean, metrics is naturally the next big thing. I really want to solve the cardinality problem with metrics where you could just have, I could have 5,000,000 labels or a billion labels and it doesn't really matter in terms of my costs. And so we've been exploring various ways. How can we write the similar thing where, you know, logs can sail down to zero and scale up and very cost effective and metrics you need to have aggregations. Do we become a little more opinionated and have preformed aggregations, you know, I'm gonna automatically calculate fiftieth percentile on the ninety and ninety fifth and ninety ninth.
Kevin Hoffman:Or should it be powerful enough to let people do custom aggregations, but then still solve the cardinality problem? So there's trade offs that we're learning from users and the market at large, like what's the most painful and what do they really want, and how could we deliver a metrics engine? And ideally, it would be a Prometheus compatible, you know, drop in so people could bring their own Grafana in the beginning. You know, this would be the back end for that data. And then over time, sure, make it easy and pretty to do all the dashboards and other things.
Kevin Hoffman:So that's, you know, that's naturally a next step. Some of our customers are game studios, and so we've written plugins for the Unreal Engine. There's been some requests there to pull game metrics in and make that data so they can do machine learning on the back end. So for midsize studios, that's a pretty common use case. And then over time, it'd be really interesting to have, we have this automatic pattern analysis that takes the shape of your logs.
Kevin Hoffman:And so based on that pattern data, it would be really interesting to feed that into an LLM periodically and say, you know, you as the user could say, this is what I'm really interested in or concerned about or the patterns I want you to watch for me, and then have the LLM actually look at the change in those patterns over time and give you a change log of your, you know, of your application running based on the shapes of your logs or the anomalies or various things. But to have it be kind of, you know, chat style observability as it were, you know, it's it's trendy, but I think it would be very helpful. Like, for example, at Acxiant, four twenty nine response for a SaaS backup application from Microsoft was a real concern. Like, we wanna know, is Microsoft starting to throttle our our backup application through their APIs? And what are the trends?
Kevin Hoffman:And so being able to tell LLM, hey, I want you to watch for this shape of log patterns in my API and have a record of that over time, like a change log of those patterns, summarized for me as a human, based on what I'm interested in could be really interesting. And, you know, a few folks have started to do things like that. But I think that's a really interesting angle where you can make the tool work for you based on this auto extraction of data plus classification of patterns is a really good foundation to do some of those, you know, more more, AI type features on top.
Jonathan Hall:Mhmm.
Shay Nehmad:Other than the, you know, obvious, like, business choices, we have to move fast, we're Boots rep versus VC, all these like rational choices. I know that for me a lot of time, picking a tech stack or language is somewhat vibe based. Like, do I have the mental capacity to deal with Python right now? Like, am I smart enough right now to be able to write this Python script without typing that won't bite me on the ass? How many people am I gonna work on this project with and have to, you know, move the models library around in this types of project so they can all use this package or make sure that they all use the same node environment installed in their machine, etcetera, etcetera.
Shay Nehmad:So at least for me, I know it's somewhat vibes based. And a lot of the time Rust is like for for me, it feels like it requires a lot of upfront cost and a lot of investment, like a specific mental space. You mentioned you have some like pet peeves and like Rust versus Go. I was wondering if you could share them so I could help, inform my vibes based decision on when to use Rust and when to use Go. Well, I
Kevin Hoffman:can yeah.
Jonathan Hall:I can
Kevin Hoffman:give you some on both sides, you know. So when we first introduced Rust and Acxiant, not many knew the language, of course. And so, you know, there's a pretty steep learning curve just because of the concepts of ownership and borrowing and lifetimes and how you do a safe and synchronous and so forth. And so it took
Shay Nehmad:How you do a synchronous at all? That's a question for me. Yeah. So hard.
Kevin Hoffman:You know, it's like, yeah, because you can control the runtime for how asynchronous stuff works in Go, it's like or sorry, in Rust, then there's a lot of things you need to know just to get the basics going. And so it took about a year before people really got into it. You know, the the c plus plus team really started to like writing things in Rust. And probably two years before it was really like, hey, we want to write this new thing, and we're going to do it in Rust rather than c plus plus sort of a thing. It took a long time.
Kevin Hoffman:When the company adopted Go, it was adopted much faster, probably within a quarter. There was several people starting to write things and wanted to write more and more things. They were moving from Python and Python microservices to Go, as we were kind of doing higher scale services. And so Go, you know, Go is just a fun language, you know, put it that way. People felt productive.
Kevin Hoffman:But yeah, having written in Rust, like, their Go feels a little bit less feature rich. So like, I really like how Rust does error handling, and it's there's this native result type with the question mark operator, and it's really concise to do the right things for handling every error, and there's safety in error handling. In Go, it's like, how many million times do I have to, you know, type if error does not equal nil, etcetera, etcetera? You know, no ternary operator and other kind of powerful pattern matching and Rust. Everything's most every expression is a value, so you can be much more functional in your style.
Kevin Hoffman:And then in Go, you can shadow variable lifetimes with the colon equals operator. And I don't know how many bugs, you know, I've crafted personally just from that one aspect of Go. And it tries to tell you you didn't use that value, but if you have, like, for example, named return values, you can still shoot yourself in the foot. And so there's there's some gotchas there, you know, that just, give you some surprises, which is, of course, like any language, you just need to unit test or, have integration tests. REST, on the other hand, like, I remember we were experimenting with REST and, modifying vector, which is, you know, an open source agent for ingesting logs, and doing some basic extensions of that.
Kevin Hoffman:And it took literally, like, four minutes to compile. And I would modify my test and wait two minutes for the test to run every time. And so it's just like, wow, you know, it's really safe code. And sure, Rust encourages to think. And if it compiles, it'll likely be correct, which is true.
Kevin Hoffman:But, you know, I'd rather just like write the tests, iterate, you know, type out a little more code, run it again. And so it's a very satisfying feeling when you're developing like that, and you're running your tests every few seconds or every minute or two rather than, you know, I'm going to go get take a break every time I need to compile and run my test suite sort of thing. And so that's, you know, that's one of the things that really pushed me over the app vibe wise is, like, I just don't wanna I I feel a lot more productive iterating in Go than I do in Rust because of the compile times on these large projects.
Shay Nehmad:I super share that, feeling. Like, whenever I have to wait for my tools to give me feedback, I I I don't know. Maybe it's just my, my attention span is getting shorter and shorter or maybe my, expectations are getting higher and higher. I really want to just save my file and have all the linting done, all the typing done, all the tests run, all the just let me know if it works. And then having to wait, my current test suite needs to set up, like, end to end test suite on my back end project, needs to wait like six seconds to spin up a Docker composer because it uses test containers.
Shay Nehmad:And I'm like, these six seconds trying to remain like looking at the loading thing instead of going over to Slack real quick and suddenly losing 20 in a discussion. Right? Or or let's go trim the cup of go backlog or oh my, I have a email waiting for me or let's do the open LinkedIn. Like I just there's so many things that are are vying for your attention that I feel like this this feedback from the tools is super important. The fact that the Go compiles so fast, it's just like, it's been a godsend.
Kevin Hoffman:Yeah. Mean, that's one aspect of Flutter I've really enjoyed is like a live UX updates as you type sort of a thing. So it's very fast feedback cycle on the front end. And yeah, Go is not quite to that same level, but it is still very productive compared to compared to rest. And fundamentally, it's like, Oh, you can't modify a concrete struct type and extend it in other modules.
Kevin Hoffman:But that's why I go can compile so fast. So it's like there's some language irritations that I'm still annoyed with, but then at the end of the day, like I'm fine with that trade off, you know, because compile times are super fast.
Jonathan Hall:So if if our listeners are annoyed by how long it takes to load their logs and they're getting a cup of coffee or getting distracted on Slack while they're waiting for their log searches to happen, tell them what they should be doing instead.
Kevin Hoffman:Yeah, just head over to sparklogs.com and get a free account and just some logs, see if you get more joy in your life.
Shay Nehmad:Pretty straightforward. Yeah.
Jonathan Hall:Great. Well, yeah, that's one of the quickest plugs I think we've had on the show. Sparkloggers.com. Sign up. Boom.
Kevin Hoffman:You know, you're you're either gonna find that it works for you if you're tired of fighting with Elastic Stack or Loki and Cardinality. You know, if you have SQL logging pains, it could be for you. If that's not a problem for you, then don't worry about it.
Jonathan Hall:Well, we do like to end our interviews with a stumper question. Stumper isn't really the right word. It's hopefully. I mean, I guess it could be. But the first year it was more of a stumper question.
Jonathan Hall:The question is, who has maybe influenced you or maybe encouraged you the most as a Go developer? Obviously, you're clearly multilingual developer. Go isn't the only language you do, but if you narrow your focus to your time with Go, who's maybe influenced you the most?
Kevin Hoffman:Yeah, I'd say it's all the colleagues at Acxiant. You know, they pioneered Go at Acxiant and they wrote these incredible microservices that ran at petabyte scale. And, you know, I learned a lot, you know, I was not in the code every day there, but I saw a lot of the techniques and tactics they used and really appreciated what they did, you know. There's a lot of really cool open source projects written in Go, and so there's a lot to learn there as well. But yeah, my colleagues at Acxiom, you know, were the ones that taught me the beauty of joy of Go, I think.
Shay Nehmad:Nice. Can I ask if you were in office at the time, or was that a remote role?
Kevin Hoffman:We're almost a % remote engineering team.
Shay Nehmad:No way. And you managed to to get inspired by other people programming even remotely?
Kevin Hoffman:It's highly collaborative environment, you know, like even though we're remote, just people are working together all the time. And so there was an intensity there. You know, we went through some hard times and that brings people together and we had to solve some hard problems fairly quickly. And so there was a lot of intense collaboration going on at the time.
Jonathan Hall:I had the same experience at eFolder before it was acquired by Acxiant. I was of course living in Mexico for a year during that time while the rest of the team was And even though we were remote, I felt that we were It was probably one of the most highly collaborative teams I've ever worked on. So I can identify with that.
Kevin Hoffman:It was really good. Yeah, was fun working together with Jonathan and his passion. And so it's fun to be invited back and chat about technical stuff for a bit.
Jonathan Hall:Yeah, thanks for taking the time and best of luck with Spark logs and all the monitoring and metrics and everything else you'll be adding over time.
Kevin Hoffman:Very good. Yeah. Thanks. Enjoyed it.
Shay Nehmad:All right. That's it for our interview. Program exit out. Goodbye.
Creators and Guests
