The s in "golang" stands for security and an interview with George Adams from Microsoft
This
Shay Nehmad:is Cup o' Go for 08/08/2025. Stay up to date with the important happenings of the Go community in about fifteen minutes per week. I'm Shay Nehmad.
Jonathan Hall:Oh, and I'm Jonathan Hall.
Jonathan Hall:to a slow start here, man.
Shay Nehmad:I'm actually I just drank really good coffee. I don't know if I wanna shout out the shop because they don't sponsor us. It's like very local to San Jose, it's not a chain. I'll do it. Nirvana Soul.
Shay Nehmad:You know these coffee shops where you walk in and they're like, oh, this place is too trendy for me. I'm not not cool enough to drink coffee here.
Jonathan Hall:I don't think I've had that experience.
Shay Nehmad:You should you should drink coffee in the Bay Area, and you will definitely have that experience.
Jonathan Hall:Yeah. Well, I've been to Starbucks store number one, but it it that that was not that vibe there at all.
Shay Nehmad:Anyway, I'm after coffee, and I'm, kinda tired. But let's, let's push through because we have a really, good interview today, and I wanna get to that.
Jonathan Hall:Yes, we do.
Shay Nehmad:Let's start with the new release.
Jonathan Hall:Oh, yeah. It's one point twenty five, right?
Shay Nehmad:No, not yet.
Jonathan Hall:No, not yet. Next week.
Shay Nehmad:Probably.
Jonathan Hall:Probably. So for now we have one point twenty four point six and one point twenty three point twelve and one point twenty five point RC Next, whatever it is, three or four or something. They all include two security releases. Tell us about the first one.
Shay Nehmad:So the first one is one we've mentioned in the past. It's OS exec look path may return unexpected paths. This is not like a hugely important security fix. I think it's important, but, you know, it's just a normal security fix, but it became a staple of the show just because a listener told us about it, then I got involved with the code review and I've been checking up on that. Like, I've been getting emails about it every day since, about the comments and I've been replying and when it got merged, was happy.
Shay Nehmad:We told about it last week and now it's in. Thank you, Olivier Mengui for reporting this issue. This is CV202547906.
Jonathan Hall:I have to say it's a little bit funny, at least from my perspective here, that I've been reading about this for the last few days because it was it was pre announced as a secret security patch. It's it's funny that one of the secret security patches is something we've been talking about for three weeks already.
Shay Nehmad:Yeah. Yeah. Still. There is another fix though that looks super interesting and I didn't do the code review on. So so what's up there?
Jonathan Hall:Yeah. So this is not the usual suspects at all.
Shay Nehmad:Kaiser Soze. Got the cripple in there from New York. Yeah. Did he mention Kaiser Soze? Who?
Shay Nehmad:Just bear with me here.
Jonathan Hall:Who's Kaiser Soze? Oh, So, yeah, this is not the usual suspects. A bug, a security bug in database SQL.
Shay Nehmad:That doesn't sound important.
Jonathan Hall:No. Of course not. Nobody uses that. So it's it's a it's really a very corner case. I'll just read the description.
Jonathan Hall:I think it's a pretty good description. Canceling a query, such as by canceling the context passed to one of the query methods, during a call to to the scan method of the returned rows can result in unexpected results if other queries are being made in parallel. So you have to be like doing multiple queries on the same database in parallel and cancel one of them by canceling a context. If you do that under the right circumstances, then you may get unexpected results or you may even get results from the wrong query into your scan result.
Shay Nehmad:Woah. Damn, that sounds actually super reasonable. Like, I'm thinking about the very reasonable case you have a DB that has like row level security to requests coming to your web app at once, one from org a, one from org b, then the connection gets closed on org a because they closed their web whatever, connection closed. You cancel org a and then org b gets the results for org a data leak. Hot damn.
Jonathan Hall:Pretty
Shay Nehmad:This seems like
Jonathan Hall:severe. Pretty serious. I mean, I think I think it's very difficult to trigger the condition, but if you do, the results could be very serious.
Shay Nehmad:So Is it like a race condition sort of thing? Like
Jonathan Hall:I haven't looked at the code, but that's what it sounds like to me, based on the description. So I don't wanna take chances. I'm upgrading.
Shay Nehmad:Yeah. You should I mean, it's easy. Right? You should upgrade anyway. These things are are usually, like, not breaking.
Shay Nehmad:And thank you for thank you to Spike Curtis from Coder for reporting this issue.
Jonathan Hall:Yeah. So go upgrade your code, and then next week, probably, you can upgrade to 1.25 and get all the other cool new things that we'll be talking about probably next week.
Shay Nehmad:I wonder how many times people ran into this bug and then they refreshed the query and it didn't happen again. They're like, whatever. Actually wondering how Spike ran into this, if it was like on purpose or not.
Jonathan Hall:Oh, that's a good question. Well, if
Shay Nehmad:only there was some way to, you know, see all the commands he ever ran. Could you think of such an option?
Jonathan Hall:I can't think of any way to do that.
Shay Nehmad:Well, how about telemetry? We'll force everybody to, you know, send us what they they build. The community is gonna be fine with that. Right?
Jonathan Hall:Sure. Why not?
Shay Nehmad:I think I'm gonna be using some flashbacks.
Jonathan Hall:To wait a second. Yeah. So Microsoft and telemetry, they decided to take a slightly different approach than Google. What's that about?
Shay Nehmad:So if you remember, actually, I think it was one of the first things we Like one of the first overarching topics we covered on the show because it happened like a year and a half ago when we started. The Google team wanted to add telemetry to better direct the direction of the Go team and whatever. And they made it opt- Opt in.
Jonathan Hall:After some debate. It was originally gonna be opt out.
Shay Nehmad:Was opt out and people were really pissed off about it, and then it they turned it into opt in after community feedback. Microsoft's build of Go adds another telemetry layer for the Go team at Microsoft. And if you wanna hear more about that and like, it's part of they they published it on the dev blog, but I implore you to just stick around because you could hear a lot more about it when we interviewed George Adams, who I just read this blog post and I was like, that seems interesting. I wonder why they do that. And I just contacted him and was like, of course, I'd love to come on.
Shay Nehmad:Yeah. It's a bit of a cliffhanger, Stick around for the interview, but just if you don't wanna listen to the interview, you know, the data collected is anonymized and they have full you have full control of it, and you can disable it by MS Go tool chain telemetry enabled, setting it to zero. But honestly, if you're using the Go the Microsoft Go version, you probably wanna send them this telemetry. And this is separate from the Google telemetry. It's just not the same thing.
Shay Nehmad:So you can even send both if you
Jonathan Hall:don't I love to send data.
Shay Nehmad:I mean, I have opted into the Go telemetry I
Jonathan Hall:have I have too. Yes.
Shay Nehmad:And, yeah, stick around for the interview. We we actually go into detail about that and just go in Microsoft in general, which is interesting.
Jonathan Hall:So, something we talk about in that interview, a lot is security since that's one of the reasons that, Microsoft has their own fork. So on that topic, Filipe Valsorto, who's been on the show before and does a lot of work with the security and crypto in Go, has a new blog post out this about a week ago, I guess, which I think is fascinating. It's about so essentially, it's about assembly mutation testing in Go. But I think
Shay Nehmad:What does that mean?
Jonathan Hall:Yeah. I'll talk what that means. What's what's actually a little bit more interesting than that on and of itself is the reason it's so useful in the sort of work he does with crypto. But one step at a time. So let's dissect that title Go Assembly Mutation Testing.
Jonathan Hall:We already talked about this last week about the pure Go tag, That some Go code is written in assembly or Go's sort of pseudo assembly. And that's very true for a lot of the crypto packages, low level stuff that do bite or bit manipulation and so on and so forth. It's just a lot more efficient to do that sort of stuff in assembly than it is in Go in many cases. So a lot of the crypto work is done in assembly. So that's why Go assembly.
Jonathan Hall:What's mutation testing? Are you familiar with mutation testing, Shay?
Shay Nehmad:I'm familiar with the term.
Jonathan Hall:Okay.
Shay Nehmad:But it's basically like you run your thing, you have like an expected output and then you change your binary in some way or your environment in some way and you try to fuzz, like, what changes will cause your expected result to break? Is that the
Jonathan Hall:So, yeah, you're you're pretty close. So we had John Arundel on way back when, probably a year and a half ago, who talked about fuzz testing when that was new in Go. Fuzz testing is where you sort of feed random inputs to your existing program and see which ones cause breakage. And this is a way to sort of build a more robust test suite. Mutation testing is kind of the opposite or the other side of that, where you use maybe the same inputs, but you actually mutate the program.
Jonathan Hall:You don't mutate the inputs. And the idea is if you can change the program and it doesn't break your tests, then your tests probably aren't sophisticated enough or they aren't covering enough, right? If you take So a simple if statement, you have if X equals Y and you mutate that randomly, so now it says if X equals X, but it doesn't cause any tests to fail, then you probably don't have sufficient test coverage, right? Obviously, mutations will generate an invalid program and those are easy to filter out. But assuming that you do a mutation that generates a valid program, but an incorrect program, you want it to cause your test to fail.
Jonathan Hall:So we're talking about doing mutation testing for Go assembly. So that's topic of the blog post. Now, why would this be a particularly interesting topic? He goes into some background here, which I'm not going go through all of it because if you want that, you can go read the blog post. But he goes into some of the reasons why this is particularly interesting for the sorts of work that we do with crypto and Go.
Jonathan Hall:So one simple example is something I don't usually deal a lot with, but it's really important in crypto, that is to have code that executes in constant time, meaning that given different inputs, it still takes the same amount of time, the same operations are executed regardless of input. You don't take shortcuts. Know, if you're doing certain decoding, might know that, oh, this means that I can I don't have to decode the next 16 bytes because of the way this is laid out? I could skip that and be faster. Normally, want to do that with crypto.
Jonathan Hall:We don't want to do that because that can allow for certain types of forensic attacks, side channel attacks. You can if you have access to the machine that goes running on, you can see, oh, this ran faster in this case than with this case so that I can deduce certain things about security, which I'm sure you know a lot more about Shy than I do.
Shay Nehmad:I mean, you don't even have to have access to the machine. Like, let's say you try to decode a password, right? Sure. You can do the side channel thing of, oh, if I pass a password that starts with the correct letter, then it takes longer. This is like so many layers deep.
Shay Nehmad:I feel like that the level of coverage and the level of precision the Go team is talking about is like way more than even the above layman understanding of security. Yeah, yeah. I mean, is like standard standard. But mutation testing is like, itself seems like a fragile thing that would bring up like random edge cases anyway. I mean, That's kind of the point, right?
Shay Nehmad:The machine flips random bits here and there, things are not gonna work.
Jonathan Hall:So, yeah, I suppose that's true. I think we mostly trust our machines not to flip random bits in most cases, but it certainly does happen in times, right? But getting back to this constant time, one of the So I sort of knew the concept of constant time and why it was important. I had never considered the implications on testing and he lays that out here. One of simple ways to do constant time is to, whenever you have an if branch, for example, execute both branches all the time and then discard one result, the one you don't need, depending on the thing.
Jonathan Hall:That way you always do the same amount of execution regardless of your inputs. Well, that would mean that your normal tests look like no matter what you do, you covered all branches, right? Because they all executed, but you didn't necessarily assert on all of So this is where the mutation testing starts to come in. It helps you it helps sort of bring to the surface the areas where you may have executed the code because of this constant time constraint, but you aren't necessarily testing all of the conditionals that might have been triggered. So mutation testing is one tool that they can use to help cover these sorts of cases.
Jonathan Hall:And then he goes into, he actually shows some examples of assembly code, which I haven't bothered to read in-depth enough to fully understand it. But he's basically, he's building a test framework to do mutation testing in GoAssembly, which I think is pretty cool. I'll probably never use it since I don't write GoAssembly. And if I ever do, it'll it won't be a lot. I don't imagine.
Jonathan Hall:But I think it's a pretty cool concept. So I like this post. Thanks, Filippo, for sharing this with us.
Shay Nehmad:The question I have, if what you wanna do is uncover these problems, why do you have to do it in the assembly layer and not just mutate source code? Can't you do this thing and just change the Go code instead of changing, like, the assembly commands?
Jonathan Hall:I think this is where they're writing assembly directly to be more efficient. So this isn't so so, yes, you have Go code that's essentially transpiled to the Go assembly, right? I think that's what you're talking about. Here is where they're actually writing the assembly raw, I believe.
Shay Nehmad:Okay. Okay.
Jonathan Hall:Because they want they want the the precise control over the Bitwise operations, and they want to make sure that the GoCompiler doesn't do any path optimizations to to, clean up dead code and stuff like that, that would ruin the constant time.
Shay Nehmad:Generally, unless you But the general policy around like writing assembly, you absolutely need it, it's better for you to write Go. Sure. Right? It's more portable, etcetera, etcetera.
Jonathan Hall:Right, right, Definitely.
Shay Nehmad:I mean, assembly is fun. I I enjoy writing assembly and disassembly and reverse engineering, but it's I think the the general thing is
Jonathan Hall:It's like doing a Sudoku. You do it for fun, not because it's productive.
Shay Nehmad:Right? Totally. It is very Sudoku ish in a sense. You have to like remember what you have in each register, what you've pushed to the stack. Yeah.
Shay Nehmad:It is Sudoku e in nature for sure. Cool. I wanted to mention one other blog post and I feel bad about breaking it up because I put it in the backlog and then every week I was like, Oh, we'll get to it next week, we'll get to it next week. And it's been like a month and a half. So, if you read this already, I'm sorry.
Shay Nehmad:But GitLab caught another typosquatting supply chain attack thing, and I like their technique. So I'll quickly explain what the typosquatting thing is. When you import a package in Go, you usually do like, go get, and then you pass in the, like, path, right?
Jonathan Hall:Mhmm.
Shay Nehmad:The github.com/something/whatever for third party libraries. With, you know, copy pasted snippets and AI code generation and blog posts and whatever, it's very rare that I just sit out and type out the package name. Usually I just go on GitHub, I find the package I need, and if I don't already know what I need. Typical squatting is where people make their packages look like the package you actually want to import. Maybe the name is like one character over or something like that.
Shay Nehmad:And then you import their package and theoretically it does the same, maybe it's even a fork, but somewhere inside the code they have their malicious intent, like installing a key logger or whatever, just something malicious. CryptoMiner, many other malicious options, remain, right? CNC server, CNC client, or a botnet, blah blah blah. There is one. If you use a QMGO, which is a MongoDB model, double check your import because the real one is github.com/qiniu/qmgo.
Shay Nehmad:Title squatting one is github.com/qiniu/qmgo. So you won't even be able to see it when you look at the package name, only the package path. And it's just like one extra I in a name that I think most people won't be able to realize is because it's not a word, it's kind of hard to even understand, you know, you imported the wrong Go driver.
Jonathan Hall:You know how easy it is to make a mistake with this spelling? The authors of the article made a mistake with the spelling.
Shay Nehmad:What do mean?
Jonathan Hall:That's how easy it is. In one place, they say Q I N I I u. In another place, they say q I I n I u. So they they put the double I in different places depending on where you read in the article.
Shay Nehmad:Oh, no. No. No. You should read the article very carefully. The first type of squatting
Jonathan Hall:Two was type of smuttings.
Shay Nehmad:They GitHub took it down, and then they did another one with the double I in a different place.
Jonathan Hall:Got it.
Shay Nehmad:But the fact that you thought that there was, both of them the same thing just shows it goes to show how easy it is. Yeah. And again, it's not a word, so it's not very easy to like understand that it's the wrong thing. First of all, like double check which package you imported, especially if it's a third party thing and especially if you added it recently, generally. What I liked about this blog post other than the fact that they did it all seriously and analyzed the code, and the code by the way is like the malicious payload is actually downloading like a file and then executing thing, and then it downloads an MP3 and then it puts it somewhere and it's a Go binary, and then it gives you like a persistent remote access to the machine, tries to connect to a command and control server, and then you could do whatever you want.
Shay Nehmad:Screenshots, SOX proxy, shell access. So it's like it basically completely owns your machine. Just, like, installs a Trojan horse, opens it to the to the web. Pretty pretty, like, hardcore. The way they discovered it is pretty cool.
Shay Nehmad:Like, how would you discover type of squatting, a type of squatting attack in the wild?
Jonathan Hall:Probably by something weird happening on my machine and investigating to see what happened.
Shay Nehmad:But that's a bit too late, right? You want to discover it before it happens.
Jonathan Hall:I'm trying to do it at a time, like you're asking, like, what would be my like, if I was implementing security policies, I don't know, a white list of known good packages or something that
Shay Nehmad:has So that's be one way to do it. But then when you wanna add a new package, you gotta somehow add it to the thing.
Jonathan Hall:Yeah, yeah.
Shay Nehmad:Actually, the Vollum research team at GitLab developed a whole automated detection system. Nice. So they do automated type of squatting detection, like just look at suspicious naming patterns. So I guess everything that's like leverage staying distance of X or suspicious existing patterns already, and semantic code analysis. So they look at Go code in the packages that might be typosquatting and filter it by like there's a network request or command execution, things that shouldn't happen in libraries basically.
Shay Nehmad:And finally, they do AI assisted initial screening for like trying to understand if there's like advanced payloads or obfuscation detection, which I think is a great use of like LLMs. You're like, Oh, just look at this code and tell me if it looks suspicious or not. Here are a few examples of normal looking code, and here's a few examples of payload cyber y looking code. And given enough examples, you could get a pretty high confidence vote here. When trying to look at all packages, this is like a great way to filter and prioritize what to look at.
Shay Nehmad:So I really liked it. It's a systematic approach to solving the problem. And I was like, oh, let's see what happens after someone gets owned, which is cool. Great stuff for GitLab security team. Like, And you know, if they find it and they find it on GitHub, take it down, they're helping everybody, like every Go developer, just GitLab.
Shay Nehmad:So that's awesome as well.
Jonathan Hall:I like this episode. We have a theme going here that GitLab helps GitHub and Microsoft helps Go. We'll have more about that in the interview.
Shay Nehmad:Yeah. Kumbaya, everybody together. And only the only the cyber attackers are in the corner,
Jonathan Hall:like, stop taking all our vectors.
Shay Nehmad:Talking about stop, let's stop for a brief moment here before we continue with our episode. What do you say? Okay. Like I mentioned at the top of the episode, this show is supported by you. This is a hobby, Jonathan and I do it for fun and to stay on top of Go, but it also costs us some money.
Shay Nehmad:If you wanna help us recoup some of that costs, if the show is helpful for you, you can support us directly on Patreon, where you can kick us 3 or $8 a month and helps cover the cost of the show. Last week I mentioned that I couldn't pull the funds, I fixed it, I got the money. All good. If you want to find the Patreon link or the Swag Store link or our Gopher Slack channel
Jonathan Hall:or our
Shay Nehmad:email or all past episodes, including transcripts, you can find all of that at kapogo.dev. And if you want to help the show in other ways, spreading the word, leaving reviews, rating it, sharing what you've learned, and then kicking us back and say, oh, I learned it on that show. Just getting more people to listen, would be great. I recently opened our, analytics and I was very happy to see, like, some milestones we reached and seems like the show is doing really well. That's mostly thanks to you sharing it with other people.
Shay Nehmad:So we really appreciate listening, sharing the show and just talking about it. One tiny programming, well, it's not a programming note, I guess it's just an ask. If you're a gopher in the Bay Area, I'm thinking about hosting another meetup somewhere around October. If you think you'd like to come, join the San Francisco channel and go for a second, let me know. I'll I'll try to arrange it.
Shay Nehmad:This is not related to the show, but probably we'll do another live episode because that was super fun.
Jonathan Hall:That was fun, even though I wasn't there.
Shay Nehmad:Yeah. Maybe we'll fly you out this time. Probably not.
Jonathan Hall:With all that all that Patreon money we've got sitting in your
Shay Nehmad:account now. Yeah. Exactly. Private jet. So, yeah, San Francisco meetup, might happen in October.
Shay Nehmad:A good chance to see me live. Are there any good chances to see you live, Jonathan, and if we if we won't end up flying you out?
Jonathan Hall:Yeah. Well, there there are. So I don't know if you remember. I used to livestream coding in Go back before I moved, and I'm finally at the point where I'm ready to start doing that again. So next week, Friday the fifteenth, 03:00PM local time for me, which is I think 7PM UTC.
Jonathan Hall:I'm going to start live streaming. I'm going to try to do it every Friday or almost every Friday. So if you'd like to watch me code and go, I'm going to be working on my open source project. Some of the comments I've gotten before are, wow, I never saw anybody do TDD on a real project. Or I get commentary asking, why do you do it this way instead of that way?
Jonathan Hall:So it's fun. Sometimes I make a fool of myself because I don't know the answer to how to do something. But yeah, I'll make a fool of myself in public, that's fine.
Shay Nehmad:Usually we have a lighting around here, but we just really want you all to hear what George Adams has to say about Go at Microsoft. So we're just gonna move to that interview. Thanks a lot for listening. Alright, Jonathan. Today, we're doing an interview with, George Adams from Microsoft.
Jonathan Hall:George Adams from Microsoft. I I never signed up for that. How do I opt out?
Shay Nehmad:No. No. We're doing an interview right now. No questions.
Jonathan Hall:Fine. Fine. I'll do it. Hi, George.
George Adams:Hey. Nice to meet you. Nice to exciting to be on the show.
Jonathan Hall:Yeah. Glad to have you here. So, George, I believe you are our first official Microsoft employee on the show. Welcome and thank you for being the first one to respond to our request. We've we've reached out to a couple.
George Adams:Well, I mean, it's it's obviously an honor to be here, and and hopefully first, but, certainly not last. I'm sure there's a whole load of, people at Microsoft that would love to talk to you.
Shay Nehmad:Well, depends how it goes. You know what I mean? If if we if we totally roast you, maybe they wouldn't wanna come. So we'd we'd better be nice, actually.
Jonathan Hall:If I'm
George Adams:in the headlines of Hacker News tomorrow, you'll never hear from me again.
Jonathan Hall:Alright. Well, tell us, George, a little bit about what you do at Microsoft, and and in general. Tell us about who you are.
George Adams:Yeah. Sure. So, yeah, I guess hey, folks. My name is George Adams. I'm a software engineering manager at Microsoft, where I lead the team behind the Microsoft Builder Go.
George Adams:Our kind of primary purpose is that we maintain a downstream distribution of Go that powers major services like AKS, Azure Kubernetes Service, GitHub, many other first party customers. And I guess the kind of key focus that my team has is focusing on security, compliance, And also crucially, we have a really kind of strict upstream first contribution policy where everything we do, we try and put it upstream first so that the community gets goodness from it as well.
Shay Nehmad:Really cool. The value of, doing it upstream first is like, it's great that the community gets it, but also you get some like testing and RC and whatever before you deploy it to like critical Microsoft resources like AKS, right, that run like the biggest enterprise customers in the world. Like if you have downtime, perhaps it's better to run it through the normal process anyways, right?
George Adams:Yeah, think you often have it going both ways. Like there's times where we'll have patches that we float first and actually we see putting that through a team like AKS as a good practice before upstreaming it. You know, if it breaks AKS, it's probably going to have broken 10,000 other people in the community. So sometimes that way works as a great way of testing a change. But equally, yeah, we also love trying to get things up into the Google code base as quickly as possible so that they get the goodness of testing it.
George Adams:Obviously, by going into the primary Go distribution, you get the largest number of people testing it.
Shay Nehmad:I'll start with a really basic question before we dive into the technical details of how the Microsoft Teams works and what do you do there and whatever. It's not really a question for me, but it's a question I've heard. So so let me know what you think. Go is a Google language. Right?
Shay Nehmad:It's out of Google. And Microsoft notably is not Google. So how comes, Microsoft uses Go? While Microsoft does have, like, internal language programming languages that it developed. The first thing that comes to my mind is C, but I assume there are, like, a a million others.
George Adams:Yes. I mean, that's a great point. Right? It's true. Go started at Google, but it's always been open source.
George Adams:And I think that's been one of the biggest strengths of Go, right? Microsoft deals with dozens of different runtimes in other languages. Yeah, C sharp, .net, Java, Rust, Python, you name it, Microsoft's dealing with that. I think the kind of thing that really stuck out with Go firstly is Azure Kubernetes is just a massive part of the Azure platform and AKS and all of the Kubernetes stuff is written in Go fundamentally, right? So Microsoft is almost forced into adopting Go for AKS.
George Adams:But equally, Go is also completely awesome for CLI tooling. It's great for tools that we have in Azure. It's great for things in Defender, for example. We also dozens of SDKs that developers are using that are all written in Go. And so I think, yeah, okay, Go is written by Google.
George Adams:It's open source. Microsoft is in Google. But equally, we can benefit a huge amount from that. And we have a lot of industry expertise and also expertise in shaping the future direction of languages and runtimes in general that we can contribute back to Google. Hopefully that makes Go a better place, I guess.
Shay Nehmad:Cool. Well, among the projects, there's also TypeScript, right? This is like a Microsoft language and the engine is being rewritten to Go. We reported on this a few months ago. I'm like, I haven't I haven't checked up on that project.
Shay Nehmad:Is that like part of your team or is that like someplace else? I assume Microsoft is actually a huge place, so that's probably a dumb question.
George Adams:Microsoft is one of those places where you've got to go digging deep to find certain teams. But yeah, interestingly, the TypeScript team owned this for obvious reasons. They reached out to us a little while ago, and I remember at the time, they were assessing a whole load of languages as well as Rust, looking at what the best options were. And I think it became really clear from almost day one that they wanted to go with Go, which was fantastic. Yeah, the performance increases, I think it's like 10x is what they're reporting, is pretty cool.
George Adams:And I think a lot of people in the TypeScript community are looking forward to that GA ing and being able to build their massive TypeScript monster dependent projects in in a fraction of the time. But, yeah, I think it's it's a great example of Go being used outside of its, like, traditional back end space. Right? So, yeah, it was kind of an interesting one and and one I love to talk about as well.
Jonathan Hall:So you manage remind me your title or the team that you work on.
George Adams:So I'm an engineering manager basically. I look after the, we'll call it the Go toolset team in Microsoft.
Jonathan Hall:Okay.
George Adams:So there are other parts of Microsoft that have other Go teams as SDK teams, there's other teams that maintain things like AKS, but anything that's to do with the Go toolset, essentially Microsoft's fork of Go, that comes down to me.
Jonathan Hall:So talk to me about what Microsoft does. I mean, I don't know, as somebody who's not familiar with what Microsoft does for Go, I don't know what that is because the Go toolset's already pretty robust. It has a lot of things in it. I can imagine you're just for rubber stamping things, you know, making sure that they pass security audits or something. But I'm sure there's more to it than that.
Jonathan Hall:Tell me a little bit more about what you do. What kind of features have you added other than turning on the opt out, turning off opt in? I don't know the telemetry thing. What are the kinds features or changes that you make to go from the fork?
George Adams:Yeah, that's a good question. And to be perfectly honest with you, like if you look under the hood, 99% of the code base is identical. The key thing here is that Microsoft builds and distributes a Go tool chain for our internal teams. Microsoft, like just about every other large corp has a whole load of security governed policies and crypto policies that we have to comply with. And a lot of that boils down to things like SBOM, secure supply chain.
George Adams:And frankly, when you're looking at a team the size of AKS, they want to be able to pick up the phone in the middle of the night and say, I need a build and I need it yesterday. And that's the sheer reality, right? We can't ask Google or expect Google to provide that to us. And so we're primarily there to essentially act as a stop gap there. Beyond that, the other reason that the team was put together at the time was that Go had no support for FIPS, which is the government crypto Yeah.
George Adams:I've messed that up. FIPS, which is the that acronym that I never remember.
Jonathan Hall:Yeah. Me either.
Shay Nehmad:We we had someone on the show literally talk about FIPS. What what was what who was that? Alex Schiele or some? For maybe for forty five minutes, and I still don't remember what the acronym it's just like, yeah, the government wants a strong encryption standards, post quantum, very Federal I'm just looking here.
George Adams:Federal Information Processing Standards. There you go. FIPS. You see, I say FIPS daily, but my, my team will, my team will tease me for not knowing what that stands for.
Shay Nehmad:You know what? I'll tell you, George, actually, like, behind the curtain, that's the whole reason we do this show, just to learn things so our coworkers can't hold them over our heads. We can hold them over their heads.
George Adams:Absolutely. Absolutely.
Jonathan Hall:Sean, you're saying the quiet part out loud. Stop it.
George Adams:Yeah. We can't see what's going on behind the scene, surely. But, yeah, I guess so that the kind of key thing around Phipps is the crypto board doesn't essentially allow Go Native cryptography. So the crypto board is a Microsoft thing. This is internal policies.
George Adams:Any first party customer that wants to use Microsoft's crypto board says, well, you can't go and use Go Native cryptography. And the reason is it follows the same suit with every other language and runtime in Microsoft that says you need to go and use the OS provided cryptography layer. So like OpenSSL or on macOS, we've got CommonCrypto, on Windows we have CNG. So we essentially provide this kind of wrapper shim which allows people when they need to, to shell out of Go cryptography and into system provided cryptography. And that was really why the team was put together.
George Adams:I guess one of the interesting things beyond that is that now Filippo and his team have gone and implemented a FIPS validated crypto layer in Go, which I think is the first time this has been done. But unfortunately, at least for now, Microsoft isn't able to use that. And so the reason that we're still here doing this FIPS port is for that exact reason.
Shay Nehmad:And just to, you know, again, it's sometimes it's hard to understand how big the when you say, oh, there's the FIPS compliance thing sometimes, you know, I think of it as like a more of a startup guy. Oh, maybe I have a customer that has like a FIPS requirement because of a subcontractor or something. Microsoft has dedicated government Azure data centers and like a $10,000,000,000 contract with the Department of Defense. It's like the actual people who need to have that working.
George Adams:Absolutely. Yeah. It's all about just basically meeting stricter compliance in cryptographic standards, right? So you've listed government, finance, healthcare, any regulated industry you can think of, and Microsoft has a lot of customers in that space. And so, yeah, for that exact reason, we have to build FIPS validated builds.
George Adams:For example, you can run Kubernetes in Azure for government, and because you can run Kubernetes, you need to be able to run with a, a FIPS validated GoPinary.
Shay Nehmad:So is the I guess a follow-up on that. If I want to be a really good, team member on your team, do I need to be more of a, like, go person or do I actually need to be more of like a DevOps person? Is my day spent, you know, just on average? Obviously, I assume the team is diverse and has a lot of diverse, tasks, but I'm just trying to understand the vibe. Is it like trying to untangle build processes because you need to build Kubernetes from scratch?
Shay Nehmad:Or is it like changing Go code at, you know, the standard library, level and having to both of them sound super hard. I mean, the team sounds super hardcore. Just trying to understand if it's a hardcore like DevOps team or is it hardcore dev team or is it sort of both?
George Adams:Yeah, it's kind of a hybrid. There's obviously a DevOps team that has to produce the binaries and that in its own right, as I'm sure Google will tell you, is a pretty serious job in its own right, just the amount of builds, the amount of testing, all of that stuff. But equally, a lot of what my team is focusing on is, A, sort of changes to the standard library. So we're making a lot of changes to Go and also changes in the cryptographic layer. So again, we've got a lot of crypto expertise in our group, which often is not necessarily Go specific.
George Adams:A lot of it's written in C plus plus for example. So Go and C Go are definitely kind of key skills that we're looking for in the team. And equally just DevOps skills are very useful.
Shay Nehmad:One question I had about the sort of the place your team has at Microsoft. I don't know, but let me ask Jonathan first. Jonathan, have you ever worked at a company before where you were like the go person, but the company wasn't like a go company yet. Yeah. You know, as a contractor, and then you get some technical tasks, but you end up with, oh, I need to Actually, people come for me for questions and training and they come for me for guidance sort of situation.
Jonathan Hall:Oh, yeah. That happens. That's that's normal for me.
Shay Nehmad:Yeah. So I'm wondering if, George, you you are just that person for, Microsoft where they ping you on Teams, I assume, and
Jonathan Hall:are It
George Adams:is on Teams.
Shay Nehmad:Off the record, I'm sort of jealous that you actually get to use Teams. I'm I'm done with Slack. I wanna move to Teams. Don't tell anybody I said that because people love to hate on Teams. Never mind.
Shay Nehmad:Anyway, I assume people ping you on the internal chat, and are like, hey, can you, we want to move this service to Go. Or is there like a Go community already inside Microsoft, like an internal Go channel? Is there a guild? Like how, if I'm a Microsoft person who's currently not writing Go and for some business reason or even like a personal reason, you know, I wanna learn, I wanna move to Go. Do I end up talking to you?
George Adams:Yeah. That's a it's a good question. Actually like full disclosure, so I came into the team, I'm trying to think three, four years ago now. And at the time, I can say this on a podcast because people can't throw drinks at me. At the time I was in the Java team.
George Adams:I kind of come from a Java background anyway. But at the time I came into this Go team that was reasonably young in its existence. And the first thing we had to do was just map out Go usage across the company. And one of the things we really quickly realized was Go is being used in loads of places, but no one really has this centralized place to talk about it. And so that was one of the kind of key goals when I came into the group, which was map out company usage and try and make my team, not necessarily me, but certainly my team, the Go experts so that when someone in the company let's pick on TypeScript because we spoke about them earlier when TypeScript is evaluating Go for this particular component of their products, and they're all TypeScript developers with no idea about Go, they know that they can come to us and get our guidance.
George Adams:And so, yeah, spent a lot of time building out these wonderful Teams channels and we've built a whole list of internal dev pages where people can find us. And equally, this is kind of one of the other areas where we wanted to enable telemetry so that we can actually try and help track down some of our first party customers and and provide a better layer of support to them.
Shay Nehmad:By the way, the people are not gonna throw drinks at you for the you know, being a Java developer. I I assume we have some Java recovery people in the in the listeners. You know what I mean?
Jonathan Hall:Well, we probably have people who still like Java in our listenership, and that's fine too. I don't understand them, but it's fine that they're there.
George Adams:I mean, we like Java. I like Java. Yeah, no, it was it was fun. When I first got when I first got brought into the group, I remember a thread on Reddit where I was referred to as a Java heavyweight taking over Go. So I think the the initial concern was, are we are we going to have a whole lot of horrible Java crust in the in the tool chain?
George Adams:But I can assure you we're not.
Shay Nehmad:Oh, you you don't have a FIPS, factory implementation factory, singleton factory sort of thing going on?
George Adams:No. Not yet. But
Shay Nehmad:now that we have generic, cool. So other than, you know, is there like a formal, something more formal, like a training thing inside Microsoft for for Go? Is there a movement towards Go? Are you more like, it sounds like you're letting the customers come to you, you're giving the telemetry, you're answering things. Are you actively pushing, you know, like evangelizing Go in Microsoft or do you not think that's actually the point?
George Adams:I think it absolutely is the point, and I think it's worth doing. And I think one of the challenges is, as I say, the team is still reasonably young in its existence and hasn't had a huge amount of time to go beyond that. But one of the areas we're definitely trying to improve is just developer learning across the company for Go, right? I think there's a lot of resources for you want to become a Python developer, you want to become a dot net developer, you want to go and learn more about these things. And so, I think one of the areas that we're definitely trying to improve is for people inside the company, A, how can they find Go and how can they connect with the community?
George Adams:And B, how can they, if they want to shift career or they want to up shift what they're doing into a more Go focused space, how can we support them in doing that? And Microsoft is a good company for this. Full transparency, you can move around really easily in the company and retrain, relearn new skills. And I see lots of people doing this in Developer Division where you come from a particular language and you go to another. And to be perfectly honest with you, I'm a great example.
George Adams:I came from Java. And so, I think there is the existence of some of the support there, but I'd like to see more. And it's one of the areas that I'm hoping we can build out more. Part of that involves having a more formal developer advocacy set up for Go, which we just don't have today.
Jonathan Hall:All right, George, I have a couple of questions, I think before we try to wrap this up. So of course, we led this interview with a silly joke about opting out, which is kind of how we stumbled upon your contact details in the first place that Microsoft has decided for opt out telemetry, whereas the Go team famously decided for opt in telemetry sort of against protest. What made the Microsoft team make that decision? And secondarily, who does it affect? Does it only affect internal Microsoft teams or who else is this going to be affecting?
George Adams:Yeah, that's a great question. And I think the first disclaimer is, yeah, absolutely. Microsoft has gone and enabled telemetry to just sort of make it clear the telemetry we've enabled is completely separate from the upstream Google telemetry that's there. And yes, Google's is opt in, ours is opt out. The key thing here, and I guess the kind of headline figures are, we want to understand real world usage of our toolchain.
George Adams:Right now, we just don't have that. We want to be able to prioritize the optimizations or fixes that will have high impact. So we want to understand which teams are using which particular components, and that then helps us prioritize these things. And the other area is we can then make better upstream contributions informed by that data, which was one of the key reasons I wanted to do this in the first place. We wanted to be able to say, Well, here's a whole load of Windows users, and they're all using this API that uses this really old Cisco.
George Adams:That's slow and we can do that better. And that then helps us prioritize making the Go community better for everyone. So yeah, I guess to answer your question, why opt out rather than opt in? Most of the languages in Microsoft are that way. So like .net, for example, is the same.
George Adams:And the reason we did that really was nine times out of 10, the use cases that we're encouraging people to use Microsoft Builder Go for are: A, you're an internal customer, so you're a first party or B, you're probably using it in some sort of CI environment. And both of those scenarios, we think it's valuable enough to be able to gather that telemetry. And the key thing here is the telemetry helps us understand whether people are using our crypto backend or Google's crypto backend. That's kind of relevant to us because that then allows us to kind of say, Well, how many users are using our backend and how do we then prioritize what we're doing So, yeah, what are we collecting? It's basically command and usage, flags.
George Adams:We look at some environment context, but sort of crucially, whenever tracking your code, your identifiers, yeah, you're safe. It's anonymized. And we really don't want it to ever become something more than that, just as Google have the same kind of guidelines. Right?
Jonathan Hall:Mhmm. Is it possible to opt into both Google's telemetry and yours and send that to both places?
George Adams:Yeah. Absolutely.
Shay Nehmad:That's a good
George Adams:question. You can opt into both Google's and Microsoft's. So telemetry works in a slightly different way, which is it sends the telemetry on the fly. Google's telemetry creates a local telemetry store, I think it's weekly pushes an update up, and you enable that with the GoTelemetry command. Ours, because it's opt in by default sorry, opt out by default this is, see, even I'm confused with opt in, opt out now.
George Adams:Because ours is opt out by default, it essentially just pushes it whenever you run certain subcommands. So yes, you can absolutely send your telemetry in both ways. And that was one of the key things we wanted to do here. We didn't want to take away from the telemetry that Google get and then use in their annual reports.
Shay Nehmad:Yeah. Nice. How can I opt into Metas as well? I need I need Amazon and Metas to know what I'm doing
Jonathan Hall:with Go. You're opted into that just by being born.
George Adams:Yes. There's the, export Mark Zuckerberg is a, is a robot equal one, and that will start sending them telemetry.
Shay Nehmad:You know what? I do have a Facebook account, and I really wanted to delete it. And I didn't delete it for two reasons. First one was, my Stack Overflow account was related to that and Uh-huh. At the time I was like, oh, I really care about the points.
Shay Nehmad:It was I used it as a social login. And then my mom doesn't allow me to delete it because she sells like secondhand stuff on Facebook Marketplace and wants me to reshare it. So I'm stuck with it. I might I might, do an integration, you know, every time I run go build, post the status on my wall. Shai just ran go build.
Shay Nehmad:It took him x y seconds. Nice.
George Adams:I would be would love to have a vote. That would be pretty cool. One
Jonathan Hall:one last question. I I think it I mean, I think this goes without saying, but I I think you could elaborate on my assumptions here. I'm sure that your team also contributes upstream to the Go project. Some of the things you do, maybe they make sense to merge upstream eventually. Maybe you're fixing bugs that affect you.
Jonathan Hall:What kind of what kind of contributions do you make to go that don't affect your fork?
George Adams:Yeah, that's a that's a good question. I think sort of fundamentally we we have a couple of people in my team that sit on the official Windows port maintainers group. And so that's kind of one of the areas that we've focused quite heavily on, on making Windows a first class platform in Go. And I think it's fair to say I'll hold my hands up and say the experience today on Windows is not as good for Go developers as it is on macOS and Linux. And that's an area that we want to improve.
George Adams:There's certain Cisco's, for example, that use very old, say, like Windows XP or Windows Vista Cisco's that we just want to kill. And that's kind of one of the areas that we've been working pretty hard on improving that. There's also more tricky decisions that we have to make. One of them, for example, being long path support, which was added into the upstream Go toolset, but that was using a whole load of undocumented APIs. And so we actually made the decision to remove long path support from our Microsoft build for compliance reasons.
George Adams:But the challenge that we now have is trying to obviously realign upstream Go and work together with them to find a new version of doing that. Other than that, I think the key thing is just working on the Windows ports in general. So like Windows ARM 64, we were very heavily involved in. I think that's definitely an exciting area, one that is becoming more widely used. I think even looking at our internal downloads, we're seeing more people using Windows ARM than ever before.
George Adams:So that's definitely an area that we want to we're working with Google to get Windows ARM to be a first class, first tier platform, I think it's called. And that will be an area where I expect that Microsoft will have to help sort of maintain that and keep that stable for the long term.
Jonathan Hall:Well, thank you. I'm not a Windows ARM user, at least not yet, but I appreciate the contributions on behalf of the community.
Shay Nehmad:Know that Yeah. I'm planning to run, like, on my Windows machine the moment I find someone who can give me a case to rebuild it because I had it back in Israel. And then I took it apart, and I can't rebuild it yet. But once it's up, man, I'm gonna run tons of Go projects on that thing. And it is like it it always has been a pain.
Shay Nehmad:It's it's become much easier since you have a WSL, but you don't wanna run everything through a container or, you know, a virtual environment. You just want to run things on your machine. And whenever the Go survey comes out, Jonathan and I are like incredibly surprised by how many people use Windows, for Go. It's always like 20 something percent, which is way more than I would expect.
George Adams:Yeah. In fairness, the numbers surprised me too, actually. And to be fair, like full disclosure, I'm sitting in front of my MacBook right now, which is always funny when people sort of joke about me being a Microsoft employee and not using Windows. But yeah, I think there definitely is a reasonable market share there. And they are obviously screaming for more features, especially looking at the IDE space.
George Adams:So Versus Code, making Windows Versus Code and Go work together. And also one of the areas that we've been working with more recently is working with GitHub on getting Copilot and particularly a GenSync mode of Copilot to work better with Go. And that's, I think going to be an area that will hopefully help millions of developers around the world, which would be pretty cool.
Shay Nehmad:Awesome. Damn. Cool stuff. Well, one thing we wanted to, mention is the Go blog from Microsoft, the Go Dev blog, which is how we met, so already an incredible resource. But, let's just, put it out there.
Shay Nehmad:First of all, if you're listening, the link is in the show notes. What can people expect to see there, Jordan?
George Adams:Yeah, it's good question. Thanks plugging our blog. I'm pretty proud that we've got it there. And I think it's our first sort of really public platform we can use to share more of the exciting things we do. A lot of the blogs right now are release notes.
George Adams:And so bear with those. They're obviously useful and we want people to see them. But we've also got a lot of other things that we're talking about, particularly Windows optimizations we're making in the community. Our plan is to try and use that and also share more company wide updates. So for example, TypeScript and TypeScript Go, that's gonna be an awesome opportunity to share some of the cool things that we're doing in that dev blog.
George Adams:So definitely join that. I think you can put in your email and hit subscribe, and then that automatically sends you sends you more blogs that I that I well, me and my team write.
Shay Nehmad:So, you know, we're obviously, we're happy to mention it. The link is in the show notes. And also, you open that link, you do have an RSS you could, follow, and you could even enter your email, to, like, sign up for the newsletter. So no no reason not to not to stay informed unless you want to stay informed only through our show because we'll only tell about the interesting, blogs, not all of them.
George Adams:Absolutely. And and if there's if there's blogs that that your show find interesting, would happily send you the contacts for some of the other people that are writing them. I'm sure there's other people in Microsoft that would love to talk more technically about what we're doing in the space.
Jonathan Hall:Cool. Cool. Cool. Well, George, thanks so much for giving us a glimpse behind the Microsoft curtain at the Go team there. As we talked before we started recording, we like to ask our guests a common question.
Jonathan Hall:This year we're asking, who has been most influential for you in learning Go? I know you came from the Java background, but you've been doing Go now for a few years. Who's who's been the biggest influence?
George Adams:Yeah. That's a that's a good question. And in fairness, I think that's a tough one to answer. My my team will all be sitting here saying, well, why don't you pick me? Think for me, there's a handful of people that have definitely influenced my journey, both Google side, particularly Filippo, I've followed pretty closely from the crypto side.
George Adams:I've also got some amazing engineers in my team, Kim Montal, Davis Gooden, to name a few that are working on some super interesting stuff and are very heavily involved in upstream Go and have really acted sort of as mentors to me coming in and asking them why I can't run Java minus version of my Go app. So they're definitely out there as some great people. But my team is continuing to grow. One of the greatest things about being an engineering manager at Microsoft is you can hire some of the most amazing people and work with them daily. So I've got a whole handful of people I'd love to name on this, but I'd be going for a long time.
Jonathan Hall:That's great. That's a great answer. I think it's hard for most of us to pick a single person. Kind of take my knowledge, as they say, right? Yeah, there you go.
Jonathan Hall:Easy. ChatGPT was my biggest influence.
Shay Nehmad:Absolutely. And ironically, Jonathan, what your behavior right now, like five years from now, gonna be considered like, oh, he's not cool enough yet to not be racist on the chatbots. I'd be careful what you put
Jonathan Hall:on record, man. I'm not worried. I'm not worried.
Shay Nehmad:George, thanks a lot for, coming on and shedding some light. I'll be watching the Microsoft Devlog a lot more closely.
George Adams:No. Thanks for having me. It's been, it's been awesome.
Jonathan Hall:Yeah. It's been wonderful.
Creators and Guests
