Quick GopherCon recap, and interview with Kemal Akkoyun of Datadog
Most people are building things that users see. You're you're not even building the stuff that the engineers see. You're you're building the stuff that feeds the stuff that engineers see. So what excites you about this?
Kemal Akkoyun:We should have definitely talked about this, like, in earlier.
Shay Nehmad:We might put this as the hook. Right? Some people think it's boring. That would be a good hook.
Jonathan Hall:This is cup of go for some random day in July.
Kemal Akkoyun:Don't know.
Shay Nehmad:It's August 10.
Jonathan Hall:Oh my gosh. Let me try that again. This is Cup of Go for some random day in August. Keep up to date with the important happenings in the Go community in about fifteen minutes per week. I'm Jonathan Hall.
Shay Nehmad:I'm Shay Nehmad, and today, we're not gonna do either.
Jonathan Hall:Yeah. We're not doing this is a weird episode.
Shay Nehmad:Everything's wacky. We're recording on a Monday instead of a Friday. And our editor, Filippo, is on vacation. Hello, Filippo. I hope you're enjoying your vacation.
Shay Nehmad:So the edit is gonna take longer, and we don't have a lot of time. So we're not gonna do a normal episode. Things that we will talk about in the future include the new prerelease that's coming up, one twenty six point six was announced. We're gonna talk about CRAP scores. We had an interview about it and actually built it's, like, evolving in our Slack channel, which is really cool.
Shay Nehmad:All those meetups or whatever, but we're only going to talk about two things this episode.
Jonathan Hall:I think it needs to be three things. Yeah. We need to talk about Go For Con Israel CFP also because it's closing soon. Oh, right.
Shay Nehmad:Right. Right. Okay. So three things. Yeah.
Shay Nehmad:Two short ones, and then I'll ask Jonathan how Go For Con US was because he was cool and went, and I was at home and cried. So two short notices. Go SF. Go San Francisco. I'm arranging a meetup.
Shay Nehmad:It's coming up real soon, and we would love to have y'all. It's hosted by c one dot a I, Conductor One, and it's in San Francisco, and it's gonna be August 26 coming up real soon. Sign up. And the second thing about Go for Con Israel, the call for papers has been extended. So if you have if you wanted to put an idea and you haven't had time to do it yet, you have time until very soon.
Shay Nehmad:Right?
Jonathan Hall:Yeah. I don't
Shay Nehmad:see Five days from now. Yeah. I don't know when this episode will come out. It might be too late even.
Jonathan Hall:August 15. Hopefully, this comes out before then.
Shay Nehmad:Just two short notices.
Kemal Akkoyun:How
Shay Nehmad:was Go For Con US?
Jonathan Hall:I had a blast. It was fun meeting people, a bunch of people I'd never met at all who knew me through the show. Shout out to everybody who listens. Yeah. Cool.
Jonathan Hall:Met a bunch of people who've been on the show, but I have not met in person or otherwise have interacted with but not met in person. That was fun. I made some tentative arrangements for a number of interviews that will be coming on the show, details to be announced as they solidify. But we'll probably be talking You went recruiting. Oh, yeah.
Jonathan Hall:Oh, yeah. Just a little sneak peek. We'll probably be talking about some bees. We'll probably be talking about TinyGo and AI, and Oh. Yeah.
Jonathan Hall:A number of other things. Some interesting companies using Go that we haven't talked about before. So there's a few things that are in the pipeline. Hopefully, bunch of those will pan out to be good interviews in the future.
Shay Nehmad:The weather up in Seattle? It was in Seattle. Right?
Jonathan Hall:Yeah. It was in Seattle. It was beautiful weather, comfortable for walking around Downtown Seattle without a jacket. Shorts and T shirt was perfect. Had some great food, some great drinks.
Shay Nehmad:Did you have your GO tie on?
Jonathan Hall:I did not take that, actually. I did wear my Cup of GO shirt, but I didn't
Shay Nehmad:Did you wear your new shirt?
Jonathan Hall:I did. I did. Yeah. And Awesome.
Shay Nehmad:How is it? I
Jonathan Hall:haven't worn
Shay Nehmad:it yet.
Jonathan Hall:It was comfortable. Yeah. And a number of people are like, oh, are you related to Cup and Go? I'm like, yeah. What do you mean?
Jonathan Hall:So, like, people who knew this show but didn't know my voice, were like, how are you related to Cup of Go? Like, well, I'm one of the co hosts. Oh, I listen to that show all the time.
Shay Nehmad:For real? They didn't know your face when
Jonathan Hall:you knew voice? People. A few people did that. Most of the people would recognize my voice. I'd sit down and start chatting, and they go, oh, I recognize your voice from Cup of Go.
Jonathan Hall:Well, great. But, yeah, people will do the show, but not me. So Did you go
Shay Nehmad:to any of the talks
Jonathan Hall:or workshops? I did go to some talks. I didn't go to any workshops. I mostly attended the hallway track, but there were some good talks, and some of those talks led to me inviting people to to hopefully come on the show. One of the talks in particular, I think deserves a fair amount of attention on this show, and that is the one from from Austin who talked
Shay Nehmad:I about was worried it's going to be like a crime true crime podcast about our podcasts.
Jonathan Hall:No. No. So Austin had a proposal that we haven't talked about yet since we haven't been the show for a while. Or it's actually a discussion. It's not a proposal, but it's a discussion about a more robust proposal process.
Shay Nehmad:Oh, I've seen that. I've seen that, and I wanted to talk about it.
Jonathan Hall:And that was the one of the opening talks at the conference was Austin talking about this proposal they've created or this discussion they've created and sort of going through it. And so I'm I'm hopeful that we can get Austin on the show to talk about it too. I haven't reached out to them yet, but I intend to. But we can certainly, when we have time well, I don't know if it's today or another time, but we can go through some of this and talk about it on the show.
Shay Nehmad:In high level, his proposal is the backlog is growing too fast, and the priority isn't very clear. So maybe let's create, like, a better process for managing the backlog.
Jonathan Hall:Right. So it it's not like an overhaul. We've we've talked about the proposal process both directly and indirectly on the show many times. Right? It's not really an overhaul of that as so much as, like, a streamlining and adding abilities to make things get lost less often and some more community feedback.
Shay Nehmad:I'm I'm down. I I don't know. I feel like my very small contributions to the proposal process have not been very thoughtful on my end, usually. Usually, I I'm I'm more of a watcher. Like, look at these things.
Shay Nehmad:You're an actual contributor. You've you've fixed the line of documentation or something.
Jonathan Hall:Yeah. Right. Exactly.
Shay Nehmad:But it always feels like, you know, the issue is, like, at some level of quality, if it gets, like, you know, maybe four or five interested people, it goes very well. If it gets more than 20 interested people, just no way it's ever gonna get close until one of the maintainers is like, okay. This is what we're doing. Stop bike shedding about everything. I assume a lot of them, especially ones that we don't see, are just like a proposed, and then like, okay.
Shay Nehmad:It's moved to the backlog column and just dies there. I mean, it is an incredible amount of issues. Right? How many issues does the Go project, like, have on GitHub?
Jonathan Hall:So issues, I mean, I I don't know how many open ones there are. There are over 80,000.
Shay Nehmad:There are more than 5,000.
Jonathan Hall:Yeah. I mean, like
Shay Nehmad:Oh, almost 10,000. That's, like, rough.
Jonathan Hall:But I don't know what percentage of those are proposals. I'm sure it's a small percentage, but even a small percentage of 10,000 is a large number. So
Shay Nehmad:Yeah. Anyway, interesting proposal. We may dig into it deeper with Austin or without him, I guess.
Jonathan Hall:Mhmm.
Shay Nehmad:Yep. Other than what is the hallway track? Like, is it
Jonathan Hall:the That's
Shay Nehmad:just talks or whatever?
Jonathan Hall:No. That's just the hallway. It's euphemism for hanging out with people. Alright.
Shay Nehmad:Yeah. That's the best place in a conference, not actually the the talks.
Kemal Akkoyun:Because the talks,
Shay Nehmad:you can watch on YouTube later.
Jonathan Hall:You can, generally. Much later sometimes.
Shay Nehmad:Who's who's who are the big sponsors? Like, the big sponsor's name. Just shout shout them out if you had such a good time.
Jonathan Hall:They had a bunch of sponsors this year. The big ones were, of course, Google, Microsoft, Capital One, Bayer.
Shay Nehmad:What's Bayer? Capital One's a credit
Jonathan Hall:card. Aspirin company.
Shay Nehmad:Oh, for real?
Jonathan Hall:Yeah. Yeah. They do biotech stuff, biotechs and pharmaceuticals and whatnot.
Shay Nehmad:Awesome. Maybe they use the
Jonathan Hall:JetBrains, DNA which is Filippo Valvassori's company, was one of the sponsors. Stable Kernel, it's a company I recently interviewed with. By the way, I'm looking for work, so if you're if you have an opening on a on your team for a Go developer, reach out. Yeah. There were a bunch.
Jonathan Hall:I'm not gonna remember them all.
Shay Nehmad:Well, good on the sponsors for making such a nice event happen. I'm really sad I missed it. Hopefully, next year I'll be able to justify going.
Jonathan Hall:Word on the street is that next year will almost almost certainly, very likely be in Toronto. Toronto?
Shay Nehmad:Yeah. Interesting. I hope, like, East Coast, because I didn't get a chance to be in the East Coast yet. Maybe New York. Let's see.
Shay Nehmad:I don't know if New York is a big, like, go city. Think it's
Jonathan Hall:a Montreal. That's East Coast ish.
Shay Nehmad:It's East Coast ish. I haven't, like, been to The Atlantic.
Jonathan Hall:I mean, it's, like, almost to New York, but not quite.
Shay Nehmad:Yes. Cool. So I'm glad you had a good time. It, of course, messed with our programming schedule, but that's worth it for you to collect all these interviews and get all the star power and whatever. That's awesome.
Shay Nehmad:Yeah. No ad break, you know, as usual. We you can find everything in cupofgo.dev, including the link to our Patreon, which is the best way to support us. Thanks a lot to all our Patreon supporters for helping us maintain this rather expensive hobby. Mhmm.
Shay Nehmad:It's getting more expensive now that we're doing video as well. Yeah. Hey, if you're watching on YouTube, let us know if we're doing a good job with the shorts or not. I'm still I'm still working out how to do it. Considering how much time I spent on, like, trying to kill my brain with short form videos on social media, I'm not very good at creating them, I guess.
Shay Nehmad:And stick around because we have a very interesting interview. What's coming up, Jonathan? Let
Jonathan Hall:him know. Yeah. We're gonna be talking with Kamal from Datadog about observability.
Shay Nehmad:Yes. Trace tracing my stolen bike. Let's see how that goes.
Jonathan Hall:With with the go run time.
Shay Nehmad:Yeah.
Jonathan Hall:Alright. Stick around for that. And we'll hopefully be on a regular schedule with regular programming again in about a week.
Shay Nehmad:It's the summer, so we'll
Jonathan Hall:see. Yeah.
Shay Nehmad:We'll see. Everything's kinda haywire. We will get back to normal programming pretty soon.
Jonathan Hall:Alright. See you then.
Shay Nehmad:Jonathan.
Jonathan Hall:Hi, Shay.
Shay Nehmad:What's up? I'm I'm into a really serious detective case right now.
Jonathan Hall:Really? What are you selling?
Shay Nehmad:Someone stole my bike in San Francisco. It's really bad. Yeah. If only I could, like, trace it. If only could someone help me trace things.
Shay Nehmad:Oh, hi, Kamal. Hello. Maybe the worst maybe the worst the worst jokes we've had. I'm sorry.
Jonathan Hall:Where's all this line? We're just trying to see how low we can go.
Kemal Akkoyun:Now I've seen how it happens. It's it's fun.
Shay Nehmad:Well, the joke is, you know, the best jokes are explained, that tracing is something you do with eBPF, and Kamal is here to talk, about that. How about you introduce yourself to our beautiful listeners?
Kemal Akkoyun:Yep. Yeah. My name is Kemal. Kemal Akkoyun. I'm originally from Turkey, but I'm living in Germany, Berlin now.
Kemal Akkoyun:I work for Datadog. I've been working on observability tooling and goals since, I guess, at this point, 2020 like, 2017, 2018. Right now, I'm mostly focused on instrumentation side of things that includes eBPF, compile time instrumentation, which I'm mostly here because of that. We're gonna I'm I wanna talk about a lot of the cool things that we've been building as part of OpenTelemetry and Go. Basically, yeah, That's what I've been doing.
Kemal Akkoyun:I yeah. Besides that, yeah. Observability, I am main maintainer of a lot of c c n c f projects. I'm helping to develop Prometheus. I used to work on TANOS, a lot of metrics.
Kemal Akkoyun:Yeah.
Shay Nehmad:That's Well, I I use all these things all the time.
Kemal Akkoyun:Awesome. Do they do you
Shay Nehmad:work on this stuff as part of your, like, job in Datadog? Are Datadog, like, sponsor sponsors of this? Or was it the other way around? You worked on these things and then joined Datadog?
Kemal Akkoyun:The other way around. I used to work for Red Hat, and I used to work in this team where we collect observability data for all these OpenShift clusters. So Kubernetes, Prometheus, Tanna, Sjager, all those sort of open source projects that I worked for, then I eventually moved to more of a data collection side of things where it's like I worked on profiling, I worked on eBPF, and then went into go internals. Because of that work, I ended up working for Datadog. I'm not doing metrics anymore as you joked.
Kemal Akkoyun:I am doing a lot of tracing, distributed tracing nowadays. And I that's what I do for Datadog. APM, Application Performance Monitoring, specifically focused on Go. I work for a team called language platform. We build and maintain SDKs on the collection side, whatnot.
Kemal Akkoyun:Actually, I think John Bodner was a guest. He used to work for this team. So I'm trying to fill in his shoes. And yeah. And a lot of things that we talk about, he actually started.
Shay Nehmad:Yeah. And you know our guests. I wouldn't even remember. I think That's amazing. John might have been an interview to Jonathan.
Shay Nehmad:Did you did it on your own. Right? I was like sick
Jonathan Hall:or something. You weren't able to make a decision for some reason.
Shay Nehmad:Somehow all the good interviews happen when I'm not here. I don't know.
Kemal Akkoyun:Oh, is this a bad interview then?
Jonathan Hall:Oh, no.
Shay Nehmad:Let's see. Let's see. Let's see how it goes. So not sponsored yet. I don't know if Datadog will sponsor us.
Shay Nehmad:That'll be freaking awesome. But APM is a really good product. Really, really good product. Like, I don't know how many of our listeners, like, know of these things. I sort of assume it is basic knowledge just because I've been working in distributed systems for so many years.
Shay Nehmad:But assuming our listeners don't really know what this stuff even means, just to, you know, get things started, what is, like, this APM distributed tracing product, and what are the, like, challenging parts of it you're solving with Go?
Kemal Akkoyun:APM stands for application performance monitoring. We basically try to collect performance statistics from the application layer. That also that means like in the simple terms, we are collecting a lot of tracing data, distributed tracing data. When we said tracing, tracing is basically you have an application, there's a request, it started here, it ended here, you spend that much time on this function or in this service, then we call the distributed system, and then you can just drill down and see okay, like I'm actually spending a lot of time, let's say 30% of this request in this microservice, and I need to maybe dive deeper, this shouldn't be supposed to be like that. And then you can also associate this data with a lot of other signals, with your logs, with your other metrics, your infrastructure metrics, whatnot.
Kemal Akkoyun:You can also attach profiling data to this and get the nitty gritty details of your process and check where you can optimize whatnot. So that's the gist of it. And yeah, we also provide basic health metrics, We call them red red metrics for online services, which is like request rate, error rate, and the duration.
Jonathan Hall:How does how does what you do differ from or expand on what things like PPROF do sort of out of the box?
Kemal Akkoyun:Yeah. Pprof is just it is for profiling. And profiling signals is about actually collecting the resource usages of a process. Right? It's not request bound.
Kemal Akkoyun:You are basically how pprof works, it gives signals to go runtime and each let's say that every second, like, collect 10 times of, like, snapshot of what CPU does. Right? It's just the stack traces associated to the CPU usage or you you can associate that to a memory usage allocations or like the lack contention or whatnot. That is profiling, but the tracing is mostly request based. Right?
Kemal Akkoyun:You got an HTP request, you got a gRPC request, and now we call that this is like transactional, so you have a transaction and within that transaction you are trying to gather like what are you doing. You are trying to get the timings of that data. And the nice part of this, you can also, like, attach a metric profile to this. So while you are handling this request, this was what's going on in your CPU or in memory. And then you can, using that data, find out the bottlenecks of your service, like, detect where to optimize, where not to optimize.
Jonathan Hall:Alright. So let's trace back to the beginning of this conversation.
Shay Nehmad:Yes. After all the technical difficulties. Yes. We compared we compared p professor to this, like, distributed tracing thing. I'm also interested you mentioned, you know, you had to go into, like, the internals of Go.
Shay Nehmad:I mean, Datadog, for listeners who don't know, is, like, a huge company. Right? You have the biggest customers and, like, whatever. Maybe you're the leader in this industry. I don't know.
Shay Nehmad:I assume you are. So what sort of, like, Go internal challenges did you have to go into to solve all this stuff?
Kemal Akkoyun:Yeah. So we are one of the biggest observability companies. Yes. And we use Go a lot. Like, pun intended, and we dog food the software a lot internally.
Kemal Akkoyun:So everything we build, build with Go, and we use, like, distributed tracing profiling a lot. And we run this at scale and we have like large customers and they run this software at scale and then we discover a lot of like problems with the runtime whenever some upgrade happens, whatnot. This is not the team that I'm working on, but we have a dedicated profiling and runtime team. And they're actually contributing back to Go runtime, like crazy. I think we are one of the biggest outside contributors, like, aside Google.
Kemal Akkoyun:And we are mostly focused on those runtime. So we have, for example, a component for, like, Go runtime metrics. Go runtime itself already, like, exposing these metrics. But sometimes these measurements are not correct, and we find these and report these back and sometimes go and fix them. Or like, we come across this interesting platform combined with, I don't know, some weird Linux version and some hardware, and then that was not tested for Go, but it's technically, you can compile that against Go, and you discover these different things.
Kemal Akkoyun:And then we reported a lot of things like that, for example, and I think we fixed that as well. But like what we do, in my team, we are focused on the compiled time instrumentation. This is what we try to provide for like an auto instrumentation zero touch observability for Go applications. This is easy for Java, Python, and Ruby because they have Ruby like, run times, and they allow a lot of dark magic. You have a load time.
Kemal Akkoyun:You can rewrite the binary code the byte code whatnot, so a lot of things that you can pull. But Go is simple, and it's really hard to it's really hard. It's not possible to do those things. Right? So our solution was Which
Shay Nehmad:which, just to clarify, is a good thing, right, in your opinion? Yes. Because on one side you said like, oh, it's easy in Java. On the other side, you said, it's like dark magic. So wait, is it good or not?
Kemal Akkoyun:It is it is bad. It's dark magic. Right? Also, what we are doing is also dark magic. Right?
Kemal Akkoyun:It's not sanctioned at
Jonathan Hall:all.
Kemal Akkoyun:So it's when I first, like, encountered this, okay, coming from all these, like, infra work go after years and like, who who whoever actually uses this in production, this is crazy. And then turned out, like, people love this. What we do with compile time, there is actually this, like, flag called tool and flag, which you can, like, pass a script, any binary, and then go tool chain itself would report any build tool chain step to that, like compile and link, whatnot with all the arguments that it's passed to. So the original idea behind this flag was to maybe measure the latency of these steps and like improve basically for good for Go team itself. Right?
Kemal Akkoyun:We basically piggyback onto this flag and started to rewrite the abstract syntax tree AST during these compilations, and we started to pull any dependencies that we want and like basically introduce the concepts for from like Java, like aspect oriented programming, right? You want to have some security, some observability, but this is independent from your business logic, you need to add those and we just manipulate the AST, add those code in place, and feed that back to the compiler, and feed that to the linker, and everything is instrumented automatically. So that's the dark magic part. And this is the tool that we build and we are injecting our own SDK for that. This tool is also open source, it's called Orchestrion, but then we wanted to like move this to the more neutral governance model.
Kemal Akkoyun:There were similar tools as well, so we donated this tool to the OpenTelemetry. And there were, as I told you, there was a competing tool from Alibaba, which is a Chinese company and they were doing things a little bit differently, but the same concept, same APIs, they were like changing. And then OpenTelemetry governance said that okay, let's form a special interest group and like you work on these tools. And this is what we've been doing for the past two years. And we built yet another tool.
Kemal Akkoyun:So this could be like second or third iteration of the same idea for Go. But now it's in a neutral governance model, and like you can just collect these things for OpenTelemetry, and you can send your data directly to an OpenTelemetry collector.
Shay Nehmad:So So basically, the options I have, if I understand correctly, is one, maybe the simplest one, but requires code changes, is to go into my code and be like, at these points and these points, send the data to OpenTelemetry. Right? Like register some metrics, register what can I send to OpenTelemetry other than metrics? Traces? Logs?
Shay Nehmad:Logs
Kemal Akkoyun:as Logs, metrics, and profiles. It's an Yeah. Alpha or better. Yeah. It's the fourth signal that you can send now.
Shay Nehmad:So I so I can send all these things just like manually. Right? Write some code in my app that loads an OpenTelemetry client and sends it to whatever configured OpenTelemetry server or whatever. Yes. My second option is to do it with eBPF where I, like, I don't want to change my code.
Shay Nehmad:And now what you're saying is you can do it in compile time as well when I don't wanna change my code. Right? Yes. So there's clear there are clear, like, two options. One that requires work for me, which is the manual instrumentation, and then eBPF and compile time.
Shay Nehmad:But they both don't require me to change the code, so why would I pick one over the other? Seems like they're both, like, sorta the same. They require me to change some variables or some things in my build process or whatever, but not change the code. So what's the difference between them?
Kemal Akkoyun:Yeah. That's a great question. So the the simplest answer is, one is for runtime, the other one is compile time, right? When you think about those advantages, when we say eBPF, it's a subsystem for Linux. So it only works for Linux, right?
Kemal Akkoyun:And you need to have certain events, event hooks, and you need to have an eBPF program that you attach to your kernel, and you run that against that. So the facilities that OpenTelemetry eBPF instrumentation, which is OBI use is Uprobes. Uprobes is the facilities where you can attach any piece of code to function entries or if you're using UreturnPro function exits. And using these things, you can collect, build your traces, build your spans, agreed metrics, and you collect this data. But these are like depend on the application binary interfaces.
Kemal Akkoyun:Right? Things can change when you change your compile time, when you add another dependency whatnot. So you need to do a lot of like either preprocessing in compile time or runtime preprocessing. Right? So one thing to just like summarize, it's for Linux only.
Kemal Akkoyun:It is, let's say, like, little more brittle than the compile time approach because you need to you make a lot of assumptions on the runtime itself. Right? On the other end, compile time is basically this is just go code. Right? This is abstract syntax stream manipulation.
Kemal Akkoyun:So this is nearly like source code level. And as you described, like the the manual code that you would have written, we are just we grab that. We just inject that for you. We just don't see that. It ended up in here, so the performance characteristics of that is the same, and it works everywhere that Go works.
Kemal Akkoyun:Either Mac OS, Windows, any architecture that you wanna run this on. So it is more like platform agnostic, let's say. But in the end, these are the same. Like these solve the same problems, right? They have similar characteristics of like you we call them instrumentations, like you let's say that you are using Redis, you are using Postgres, and you are using this third party libraries, and you want to have special labels for these and you need or like these have different, let's say middle layers or like functions that when they call to interact with these external systems and you would like to trace those, right?
Kemal Akkoyun:You need to add support for these. You need to add specific code for to handle all these third party integrations, either in eBPF or in the AST for Go compile time. So similar things, similar projects, just the biggest difference is runtime versus compile time. If you want to get into nitty gritty details, someone like the eBPF programs needs to work in the kernel space, so if you're running this in a Kubernetes cluster, let's say that you need a daemon side and you need to assign certain capabilities to that daemon side, or if it's running on a host you need to give some privileged access, whatnot. Maybe you don't like that.
Kemal Akkoyun:Maybe you don't want to give you have a different security posture and you don't want to give those capabilities to this third party agent, then this is a different story. Also, what the compile time instrumentation does, it's like the blast radius is just a single service, like the service that you instrument for Go. So if something goes wrong for whatever reason, because this is the code that you are running, and if you introduce, let's say, some performance overhead, it is just within that service. But if something goes wrong for those eBPF programs, then it you could take down to a whole host. Right?
Kemal Akkoyun:But these are just the edge cases. Like this is not something you would face your day to day. Yeah. It's really hard to saturate. Yeah.
Kemal Akkoyun:Just to give some examples. Different trade offs in the end. Pros and cons, you should decide what to which one to use.
Shay Nehmad:I think I don't know. You tell me what you think, Jonathan. But always when I look at these things and it's just like one magic solution at like compile time or eBPF or whatever, and one where I can just add a few lines of code and I sorta know what's happening, I tend to, like, prefer manual instrumentation. Even though it's, like, more work and ostensibly from what you're explaining, not actually better. You might miss some cases or whatever.
Shay Nehmad:What do you think?
Kemal Akkoyun:One thing I can add, I think, sort interject, like, I forget to mention that e b p f one works for multiple languages. Right? It's that if you have something more than Go in your, like, mix, if you have heterogeneous services
Shay Nehmad:Why would you have something more than Go? This is the company of a podcast. What are you talking about?
Kemal Akkoyun:Some people some crazy people, they prefer to use multiple languages in their system. So then
Jonathan Hall:You know, that's fine.
Shay Nehmad:There. I don't have any problem with it. Some of my best friends
Kemal Akkoyun:here. So
Jonathan Hall:I I I have mixed feelings on that, honestly. Like, on the one hand, I like I like avoiding the magic, so I like doing it manually. On the other hand, I know that it's really a chore to manually add the telemetry you care about, so I don't know. Maybe the answer is just don't do do observability. Just just skip it all.
Shay Nehmad:Just don't write bugs. Just write perfect software.
Kemal Akkoyun:That's why we we are trying to make it this, like, happy for the customers. Right? Like, users for auto instrumentation, either eBPF or compile time instrumentation. Instrumentation. For example, like, you you might upgrade your third party Redis library and maybe something has changed.
Kemal Akkoyun:Now you need to change that code and it's breaking whatnot. Like, basically, we are providing all these integrations and you don't need to think about. Like, we apply all the best practices. We make sure that you are propagating your context correctly between your service boundaries, which is, again, if you are, like, talking to multiple services in a microservice environment. We basically also, like, packing with all these integrations in this auto instrumentation scenarios, like, provide the expertise for you.
Shay Nehmad:Mhmm. And in general, you mentioned, you know, all this stuff. You do it as part of Datadog, but you did mention a lot about governance and stuff like that. It's happening as part of OpenTelemetry. So this is some open source, like, a governance thing that I don't fully understand just because the open source projects I've worked on are either very small scale, you know, just like a library I whipped up and put out, or something that's backed by a company.
Shay Nehmad:Like, you I worked at a company and they had that open source project, and it was theirs. OpenTelemetry, CNCF, these are, like, not really companies, but they are they do have money. They do have people working there. And you work at Datadog, which is not that company, but your project is part of that CNCF thing or OpenTelemetry thing. Do they give you money back?
Shay Nehmad:Do you sponsor them? Like, how does it work? Do you are you part of that relationship, or do you just, like, write the code and let the lawyers deal with the other stuff?
Kemal Akkoyun:I wish they have given me money. I've been doing this for years and nobody paid me anything. So I think this is best of the both worlds. Right? You have certain vendors for CNCF, say that for Kubernetes.
Kemal Akkoyun:Like, you have Google, Reddit, Microsoft, and Kubernetes is, like, a huge beast, and a lot of people needs to maintain that. And these vendor companies basically giving these developers, letting them work on these projects, or they sponsor the events, they directly contribute financial contributions to the CNCF and they make sure these projects are healthy, right? But in these equations, they don't pay any of the contributors whatnot. Usually companies let their engineers to work on these projects, right? Same for the OpenTelemetry.
Kemal Akkoyun:The nice part of this, now you are like vendor neutral, right? You don't need to think that, okay, like, I put the Datadog SDK in my process and now I wanna use some other vendor. Like, what I'm going to do about this? Like, you don't need to think about that. OpenTelemetry is already neutral.
Kemal Akkoyun:You are putting the OpenTelemetry SDK in there and then you can easily switch to whatever vendor you would like to go. So this is more democratizing for the users and it's the benefit of the users are what isn't it for the vendors? It's like it's again, that flexibility sometimes help to that vendor, sometimes to other vendor, and they we we also would like to this community to thrive and these tools to be basically grow. Because nobody makes money out of instrumentation. Let's be, like, clear about it.
Kemal Akkoyun:So services and people pay for the services. Nobody pays for the instrumentation or the data collection, but it's the hard engineering part and someone needs to do that. So so I'm, like, super happy that, like, OpenTelemetry and CNCF exists, and my company allow me to work on these open source projects.
Shay Nehmad:And I think there's a question that always comes up with open source projects. Right, Jonathan? Whenever we mention one, people are like, how do we get how do we join? How do we get started? Or whatever.
Shay Nehmad:At at least if there are more junior engineers listening to the show right now. So with other open source projects we present, you know, I think last week we had Peter Downs, right, with his PG test thing. So it's just like a guy, then like, oh, just talk to the the dude and get, like, get started. With CNCF Project, it's a it's a bit more, I think, of a process or a more official thing, but it actually should be easier for juniors to to get in on that. Right?
Kemal Akkoyun:Yes. It is easier. It could be intimidating, but it's easier. Everything is documented. For OpenTelemetry, there is an OpenTelemetry community report, for example.
Kemal Akkoyun:In that, you can find all the SIG instrumentation links, the calendars, like when the meetings happen, where are the Slack channels, what are the project that depends on that special interest groups. Same goes for Kubernetes and all other CNCF projects, right? These are all public knowledge. You can just go into the meeting, say, hi, I'm a new contributor. I'm looking for new issues, like, I would like to contribute back.
Kemal Akkoyun:What is the best way to do that? For the easy stuff, usually there is GitHub issues tagged with, like, good first issue, whatnot. But for more complicated issues, maybe you should have a conversation. But in any case, like, being in the room, like, listening to the meetings, you learn a lot. There are a lot of, like, open source veterans out there, like, contributing to special interest groups.
Kemal Akkoyun:Basically, this is free education. As an extension to that, LFX Linux Software Foundation, which is the parent foundation of, like, a CNCF Cloud Native Foundation, there are mentorship programs. And Google Summer of Code, for example, is a joint effort with CNCF and Google, I guess. There's LFX mentorship programs. And if you are, like, fresh out of college or or, like, if you are already in college, you can apply for these programs.
Kemal Akkoyun:And for three months, like, there are mentors or depends on the SIGs and the projects, they help you out to cut your teeth in the open source and all these projects. My In SIG context
Shay Nehmad:in this context, you would be like a Google open source, summer of code, whatever, mentor. Right?
Kemal Akkoyun:Yes. Exactly. I did it several times. I did this for LFX several times. I'm already doing it, currently I'm doing it for this Go Compile Time instrumentation, we have an LFX mentee, and like for three months over the summer we are working on a big project together, like they learn everything to writing, RFCs request for, like, comments, project proposals, like how to engage with the community, how to contribute back, everything.
Kemal Akkoyun:That's also another way to contribute back. Maybe you are not contributing already a SIG, maybe you are already, like, experienced, but you know how to do mentorship, for example, you can join these SIGs and, like like, basically donate your time to help others.
Shay Nehmad:Did you ever get a chance to mess around with these open source programs, Jonathan?
Jonathan Hall:Not not like that. No. No, I haven't.
Shay Nehmad:I had a Google Summer of Code, like, mentorship position when I was working on this thing called Infection Monkey. This was, like, 2019 or something like that. It was very interesting because, like, they pay the same for all the interns, so it sort of biases itself because it's not a a lot of money, it biases itself towards countries where the same amount is a lot of money. Mhmm. And people, like, would try really hard, and you would get, like, incredibly bright engineers.
Shay Nehmad:So it's it's a pretty funny program in that regard. I don't know if they changed it because this was, like, seven years ago, but you get this, like, the best engineers from, like, rural India and then, like, very mediocre engineers from The United States. You know? Like, of course, I'd rather and they're all of them are, like, juniors just starting fresh out of college. Like, well, of I know who I'm gonna pick.
Shay Nehmad:I'm gonna pick the really bright, like, hardworking ones. And for them, it's like, you know, like a full salary. But, yeah, I used to do that. It was it was really it was a really good program. I didn't know that CNCF, like, these sorts of projects work for it as well.
Kemal Akkoyun:I don't know about the recent changes of, like Google Summer of Code, but for LFX, for example, the stippens amount changed recently and it's like adjusted to the inflation of the country whatnot. So it's not like a flat amount anymore. Oh, So I guess it's more like democratic right now. But then we have a lot of we get a lot of applications from certain countries compared to others, I guess. It's more competitive and people see this as an opportunity to, like, be better in the job market, whatnot.
Shay Nehmad:Yeah. No. It's a great it's a great opportunity. But even, I think, if you're not part of these programs, that, Jonathan, you can for sure say because you've done a ton of open source. Starting to hack on these projects, like, all on your own can be intimidating, but now if you have, like, a little bit of a community, like a Discord or a Slack or a whatever, it can make it a lot easier.
Shay Nehmad:You can just hop in and be like, hey. I'm thinking about getting started. Maybe try to use it, the whole thing. I don't know.
Jonathan Hall:So have a question, a very different question. I probably should have asked this question at the beginning of the interview, but I didn't think of it till now, because I think this is the kind of question that I'm imagining that our audience falls into two general camps. Those who think observability is boring and probably aren't listening anymore, and those who think observability is really, really interesting or or it's it's an enabler. It's something you know, that's sort of the camp I'm in. Like, as a consumer of observability tools to enable me to be more effective.
Jonathan Hall:But you're one level deeper than that. You're building these observability tools, and I'm curious to hear what excites you about that, because you clearly are passionate about this topic. What is it about that stuff that gets you excited? Because it's like three levels removed from what most people work on, if that makes sense. Right?
Jonathan Hall:Most people are building things that users see. You're you're not even building the stuff that the engineers see. You're you're building the stuff that feeds the stuff that engineers see. So what excites you about this?
Kemal Akkoyun:Yeah. We should have definitely talked about this, like, in earlier. For for example, I
Shay Nehmad:am We might we might put this as the hook. Right? We're we're trying a new format where we put a hook. So it's like, some people think it's boring. That would be
Kemal Akkoyun:a good hook. For example, like, as you described, I am one of the worst engineers to ask about, like, product questions about, like, Datadog because I rarely open open the product and poke around. Right? Like, I'm really deep and working with the run times and maybe the hard hardware to collect these data. Exactly those aspects are, like, challenging for me to understand how hardware works, find the performance bottlenecks, like, learn about this, like, as a threaded Linux APIs, learn about, like, how Go runtime works, how even these, like, data structures are implemented and integrated details on how we can actually improve the performance.
Kemal Akkoyun:When you change something, all of a sudden, like, your program is slower, but like, what the heck? I just like added a single line, like what's going on? Like all these things are like super interesting to me. Since the college days, I was interested in these like details of the runtime, the system programming, low level bits, compilers, whatnot. I guess it's hard, but it's also fun.
Kemal Akkoyun:And like, this is a funnel, There are like so few engineers also working on this area, and like, it could also pay a lot of dividends, right? Like you are not doing the same web application that a lot of other people does, but then it's a double edged sword, right? But right now you have a competitive job market because you can maybe find a lot of employers to basically hire you because not everyone actually needs these skills. So yeah, it's a lot of trade offs, but like there's a more general question in there, like why do we need observability in the first place, right? This is my journey started from like the service level, right?
Kemal Akkoyun:I was a back end engineer, then I was a reliability engineer, and it when you run the service and be on call, would like to know about like is the service healthy and what is going on? Are, like, the users are actually using that? What they are experiencing? Is are there an outage, whatnot? All these questions can only be answered by the observability understanding what you deployed in production and know what's going on.
Kemal Akkoyun:Right? I can't imagine that you are running a service without any observability. Sometimes people come and they tell that, oh, no. We don't have any metrics. Like, there are some logs or, like, what is distributed tracing?
Kemal Akkoyun:That is really interesting to me. And, like, you cannot optimize what you can measure. That's but, again, it's the secondary effect, it's just the performance. But like, you need to know like what is going on if your service is up, if it's like fast enough, if your like database is saturated, whatnot. So you can only answer these questions by, like, having observability in your stack.
Jonathan Hall:Right. Yep. Very good. Well, I'm glad you're doing the work because You say you
Shay Nehmad:say a lot lot of not a lot a lot of people work on it, but there are a lot of people in the CNCF working on this stuff. Right?
Kemal Akkoyun:Yeah. But CNCF is bigger than the observability. Right? That that's like a lot of infrastructure work. Observability, yes, it's big.
Kemal Akkoyun:There are a lot of observability companies and vendors, but that's just a small part of what CNCF does, right? That's the whole cloud infrastructure and all these projects solving some other aspect of the cloud infrastructure problems, security, runtime, a lot of things. Because of all the agentic work nowadays, sandboxing is a huge topic, for example, Kubernetes and CNCF. And there are a lot of projects and a lot of movement around that, to how to serve models, how to basically provide sandboxes for these constantly running agents, whatnot. CNCF provides a solution for all of that.
Shay Nehmad:Well, we're coming up to close here, and we have a very classic, like, end of interview. First of all, what do you wanna plug? So you you mentioned all these projects, all these links, maybe about yourself, maybe your LinkedIn, maybe, I don't know, the best Turkish coffee in Germany. I don't know. What's the what do you wanna plug?
Shay Nehmad:We really appreciate your time, you know, coming to talk to all the people. So where should they find you and your work?
Kemal Akkoyun:First of all, thanks for the opportunity. Yeah. I think best Turkish coffee is only in Turkey, so you need to visit there, unfortunately.
Shay Nehmad:I don't know if I specifically should go
Kemal Akkoyun:visit there, my friend. Yeah.
Shay Nehmad:That Maybe in a few years.
Kemal Akkoyun:Maybe in a few years, yes. Maybe it's not the best
Jonathan Hall:for you.
Shay Nehmad:Go for Con Turkey. I would be down to do Go for Con Turkey. That would be great.
Kemal Akkoyun:There was a Go for Con Turkey once. I think they did it twice twice in a row, but it's been a while. So what I would like to plug is we have these special interest groups in OpenTelemetry, right? OBI, eBPF based instrumentation, we have Go Compile Time instrumentation, we have profiling, it's also based on eBPF, and these are hard work but these projects need to sustain and for sustainability they always need more help. So if you're interested, try to engage with the community, help us.
Kemal Akkoyun:Like this is mutually beneficial, we can teach you how to get into these nitty gritty details and you can contribute back. That's one thing. I guess the other one is like I'm trying to write more blog posts, so I have a blog. I will be writing more about Go, so I would appreciate if you can follow me.
Jonathan Hall:For sure.
Kemal Akkoyun:I guess you can provide the links.
Shay Nehmad:Yeah. Yeah. We'll put the links in the in the show notes, of course. The special interest group is in Slack. Right?
Shay Nehmad:I'm seeing, like, a CNCF cloudnativedot, Slack, hotel go compile and instrumentation.
Kemal Akkoyun:There there are there is, like, Cloud native Slack, and in that Slack, each special interest group or or each CNCF project has a has multiple maybe Slack channels you can engage with. Yeah.
Shay Nehmad:I I found that link.
Kemal Akkoyun:Yeah. These are all, like, documented in GitHub OpenTelemetry slash community, and you can also, like, find all the information from that page. Cool.
Jonathan Hall:Alright. One last question prompted you before the show started, you were be prepared. We asked this to all of our guests lately, which is what is your favorite third party tool or library related to Go?
Kemal Akkoyun:Yeah. I worked on a lot of compile time stuff. That's also I mean, like, I like a lot of, like, static checkers and whatnot. I would like to pluck one of these static checkers. It's called Checklocks.
Kemal Akkoyun:It's kind of niche one. It's part of, like, gVisor project. It's part of their tooling. It is basically you are adding these annotations. Basically, you are adding your expectations around the locks and the fields of a struct, then this lock should be locked or not, and should this field incremented atomically or not.
Kemal Akkoyun:And the static checker tool using the SSA step output of the Go build tool chain and checks if these assertions hold and like catch deadlocks, whatnot, like or the lock ordering problems. This is not something that you reach out for or day to day, I guess. But if you are using a lot of logs, if you have a comp like, complex code base, and you if you wanna provide some static assertions, this is an amazing tool.
Jonathan Hall:Very cool.
Shay Nehmad:Cool. Wouldn't, like, the race detector do that? Oh, that's on test time. You're saying it's compiled time. Yes.
Kemal Akkoyun:This is static. It's
Jonathan Hall:on Cool. The
Shay Nehmad:I didn't know about this library at all. What
Kemal Akkoyun:the hell? Check it out. This is yeah. This is very, like, unknown. I guess this is also, like, written by a core Go team member that used to work for gWiser.
Kemal Akkoyun:So it's maybe it should be part of GoVet. I don't know. Maybe they they probably they have a reason for this, but maybe it will someday.
Jonathan Hall:I will check this out. Interesting.
Shay Nehmad:GVisor. Cool.
Kemal Akkoyun:Yeah. It's part of gVisor under tools, check locks. They also have other, like, esoteric static checkers, whatnot, like, against Golink names, whatnot. Like, check check the the tools directory. It's it's it's kind of a gold mine.
Jonathan Hall:Wow. There's, that's a cool Two or three dozen Yeah.
Shay Nehmad:Haven't I haven't tried it.
Jonathan Hall:Yeah. Well, Kamal, thank you so much for coming on and helping us trace Shay's bicycle.
Shay Nehmad:Unfortunately, mystery didn't solve it off. No. Of course, I didn't find it. It's San Francisco. I went ahead and walked to the train station and bought a new one the next day.
Kemal Akkoyun:It sounds like Yarden as well.
Shay Nehmad:If if we wanna if we really wanna close the book on that, what happened was I went I walked to the train station because I live in San Jose. So I I took the train back to San Jose, like, two hours late than I actually wanted to. And I did file a police report and everything. And because it was so late, I asked my my wife to pick me up from the train station, and she came with my daughter. She was already, like, ready for sleep, but she didn't fall asleep because she heard my bike was stolen.
Shay Nehmad:She got excited. And I was like, I was super pissed off. Like, I was so angry. And then I got in the car, and I I talked to them about it, and my my daughter was like, well, you know what? Did did someone steal your bike, dad?
Shay Nehmad:And she's, like, five for context. Yeah. She's very small and cute. And I was like, yeah. And she was like, well, you know what?
Shay Nehmad:Maybe he needs it more than you. I was like
Kemal Akkoyun:Oh, so cute.
Shay Nehmad:Yeah. I was like, okay. I'm not gonna explain the moral place of is it okay to steal if you need it? I was like, oh, that's such a cute thought, and I've been okay with that theft since then. Just imagined, like
Jonathan Hall:Alright.
Shay Nehmad:Someone carrying water up a mountain with it or something.
Jonathan Hall:Alright. Well, I Alright. Yeah. Thanks a lot. I think that's a wrap.
Shay Nehmad:Oh, do you wanna do the outro? You mentioned you listened to
Jonathan Hall:the show
Shay Nehmad:for a long time. You can do the program exited part.
Kemal Akkoyun:Oh oh, I got okay. Program exited.
Shay Nehmad:Goodbye. Program exited. Goodbye.
Creators and Guests
