support dispatch: https://citadeldispatch.com/donate
EPISODE: 79
BLOCK: 762763
PRICE: 5945 sats per dollar
TOPICS: recent lightning bugs, impact, responsible disclosures, adversarial environment, tradeoffs, path forward
GUESTS: @brqgoo, @thebluematt, @benthecarman, @evankaloudis, @seardsalmon, and Tony G
nostr live chat: https://citadeldispatch.com/stream
nostr account: https://primal.net/odell
youtube: https://www.youtube.com/@citadeldispatch
podcast: https://www.podpage.com/citadeldispatch
stream sats to the show: https://www.fountain.fm/
(00:00:33) Introduction
(00:01:10) Supporting the podcast through podcasting 2.0
(00:03:11) Introduction to the panel and topic of discussion
(00:46:49) The default values and configuration options for HTLCs and commitment outputs
(00:48:49) The process of responsible disclosure and how to proceed when discovering a bug
(01:34:14) Responsible disclosure and user risk
(01:36:32) Vendor approach to disclosure
(01:39:47) Duty of care and disclosure
(01:49:01) The importance of responsible disclosure and communication in the lightning network
[00:00:33]
ODELL:
Happy Bitcoin Friday, freaks. It's your boy, Odell, here for another Citadel Dispatch. Citadel Dispatch is a interactive live show about Bitcoin and Freedom Tech. We have no ads, no sponsors. It is only possible thanks to our great listeners who continue to support the podcast, whether that's with stats through podcasting 2.0. Podcasting 2.0 allows you to just download an app. Big ones are fountain podcast, Breezewallet, podverse.fm, echoln.com, and others. You download the app. You load it up with stats. You search Siddle Dispatch. You press subscribe. Choose how many sats per minute you think this show is worth, and then those sats automatically, get streamed directly to my lightning node. It is extremely cool seeing all the sats come in. You can also support the pod, through podcasting 2.0 through something called boostograms.
Boostagrams are a single payment that include a comment or message with, the payment. I read the top 4 Boostagrams from the last episode. Every week, it's really great to see, your comments and your support. Before I read those Boostagrams, you can also support the show by going to sidildispatch.com/contribute. I have a BTC pay server there. You can support that using that either via via Onchain or Lightning. And I also have my BIP 47 payment code there, which you can support via Samurais or Sparrow wallets, and, that's attached to the associated pay name, Odell. Very easy to remember. I know it's a bear market. You can also support the show by simply sharing it with friends and family.
Citadel dispatch is available on all major platforms, every podcast app, Twitch, Twitter, YouTube, Rumble, Bitcoin TV. You just search Citadel Dispatch, press subscribe, reviews help as well. Yeah. So thank you for your support, Freaks. I really do appreciate it. The show would not be possible without it. I also wanna thank the Freaks who joined us in the live chat. You're able to join the live chat either via YouTube or Twitch or Twitter or via Matrix. We have a Matrix chat that runs 247, 365. Lot of great conversation there on and off the show. You can find links to all of these things at citadel dispatch.com.
So with all that said, we have a great panel lined up. It's been about a week and a half in the making. We've been trying to get it out there ASAP because it's a very important topic. Ironically enough, in classic Bitcoin fashion, a bunch of things have happened over the last week as many of you are known. And, you guys might forget that that we had 2 major, lightning bugs over the last month or so, maybe, like, last 2 months, with one of them happening last week where we all had to update our l and d notes. So this topic will be focused on that plus lightning development going forward and just the current state of lightning development. Before I introduce our guests, I just wanna read the boostograms, the top 4 boostograms from last week. Once again, thank you, Freak, for your support.
We have at Humble Pleb, gave 69 420. Nice. 69 420 sats. I don't know. It looks like he tried to put some kinda emoji there that I can't read on Android, a bunch of times. Appreciate you, Freak. We have Eric 99 with 50,000 sats saying, stay humble, stack sats. Great advice. We have Chad Farrell with 50,000 sats saying, thanks for all that you do, Matt. And we have 8, with 7,778 saying, I've been listening to a new privacy podcast, Watchmen Privacy. It could be interesting for you to appear on this pod or vice versa. He has a strong Bitcoin focus in his podcast when speaking on cryptocurrencies, which is rare for privacy gurus. His past guests include NVK, LOP, Zelco, s 21, Seth for privacy, Bitcoin q and a, and Eco. Wow. I'll go check that out. It was not on my radar.
Appreciate you, Freaks. And, also, we have Vic in here, honorary mention with 5,000 sats. Right or die freak. Appreciate you, Veek. So thank you, Freaks, for supporting the show. We have Tony jumping in and out trying to troubleshoot. Tony, can you hear me? We cannot hear you, Tony. So I'm gonna introduce our guests. We have a long time listener and often guest of the podcast, Ben Carmen. How's it going, Ben? Can't hear you, Ben. Classic. We have Barack who I had the pleasure of meeting in Miami on the open source stage, that I helped coordinate.
Barak was the one who discovered the last two lightning bugs. How's it going, Barak?
[00:05:51] Unknown:
Yo. Hey. I can perfectly hear you. Hopefully, you guys can hear me too.
[00:05:55] ODELL:
Yes, sir. We can hear you. You rejoined.
[00:05:58] Unknown:
Yo. Hey. So, yeah, my name is. So I'm a Bitcoin dev. I mostly do some stuff on Liquid, and recently discovering Lightning Space. So I'm also known as the LND slayer, and I'm this bug eating this burger eating guy on Twitter. And happy to be here, guys, and add some you know, maybe give some details on what have happened, over the last few weeks and, you know, give some inputs and, you know, I'm not here to tell, you know, to tell whether my actions were good or bad. I think it's up to community to tell and, but I I can't tell you guys, what did happen and what my motive was.
So, happy to be here. Hopefully, this is gonna be a a friendly conversation.
[00:06:43] ODELL:
Thank you for joining, Barak. We got Evan Kaludis, also rider die freak, common common guest on the podcast. How's it going, Evan?
[00:06:53] Unknown:
Hey. Great, Matt. How are you doing today? Doing doing great. Doing great. Yeah. Happy to be back on the show. Wish we were talking about a more positive topic,
[00:07:04] ODELL:
But, nonetheless, I think we're gonna have some, you know, great, very important discussion today. Well, compared to all the other podcast topics of insolvent exchanges and people losing their life savings, I feel like focusing on lightning development is a very positive topic.
[00:07:20] Unknown:
You know, when it's not getting attacked, unnecessarily,
[00:07:24] ODELL:
but, we'll get to that soon. Fair enough. Thank you, Evan. We have Matt Corallo. He's been on the podcast before and also has joined me with Marty on TFTC. How's it going, Matt?
[00:07:37] Unknown:
Pretty good. Pretty good. Thanks for allowing me on.
[00:07:40] ODELL:
And, Matt is, very, what's your what's your role at LDK? What would I call your role at LDK?
[00:07:50] Unknown:
I don't know. I work on it like everybody else.
[00:07:53] ODELL:
Matt works on LDK like everybody else. LDK is one of, one of the top lightning implementations. We have, Vivek here, also repeat guest on the podcast. How's it going, Vivek?
[00:08:08] Unknown:
It is going well. Just this and, yeah, explore is, and and what
[00:08:23] ODELL:
Yeah. So Vivek's Internet connection is horrible. It's probably because he stopped paying it because of the bear market. I don't know how we can fix that, Vivek. We have who did I not introduce? Tony. Can we hear you now? Can you? Yeah. We can. Long time since this is a podcast. How's it going?
[00:08:45] Unknown:
I'm good. I'm good. I'm excited to be here. I'm trying to refresh my memory on what happened, but other than that, I'm happy to be here and talk about this. So, Tony I mean, if if you have been listening to dispatch, you know what Tony's been up to. He's been working on basically
[00:09:00] ODELL:
attacking lightning, and he has this new tool called Ellen's Ploit, which makes it easier to attack lightning. So we will be talking all about that. And then I think Carmen didn't wasn't able to speak last time I introduced Carmen. Carmen, can you hear me? Carmen, can you speak? Okay. Well, we still cannot hear Carmen.
[00:09:28] Unknown:
Test. Test.
[00:09:29] ODELL:
We can hear Vivek.
[00:09:30] Unknown:
Cool.
[00:09:32] Unknown:
Okay. So we have Ben's, issue is that he's on Linux, and sound never works on Linux.
[00:09:39] ODELL:
I mean, I ran the show I'm on Linux. I ran the show from Linux for, like, 2 years. Ben is gonna switch switch computers. Okay, Ben. Switch computers and come back. Let's get started. I wanna thank the Freaks for for, this is a this is a big group, but it's a it's a great group, and I'm happy to have them here. Okay. And, so just I appreciate the freaks for sticking with us during connection issues and whatnot, considering we have so many people here. I guess I guess where to start is, Barack, you want to give us a basic overview of both bugs or Yeah. Or at least how you how you created what led to both bugs?
[00:10:33] Unknown:
Yeah. Sure. Yeah. I can give it a bit give a bit of a history what happened. So, obviously, it was around a month ago or something. Yeah. I did this first large multi tap script transaction on Mainnet. I did it on Testnet first, though, and then it was unintentionally broke LND due to a pricing issue in BTCD. And, I I, obviously, it made a lot of trouble at the time. I guess I feel sorry about that. And then in the following ways, I had this this this, this urge, this inner feeling to my gut telling me to break it more. And, like, like, I did once, when, or twice. I'm not sure, like, if that sounds weird, but, I guess, yeah, I'm discovering the dark side in me.
Yeah. Maybe I'm a bad actor. Who knows? But, yeah, I just I just I just wanted to break more, and I did it only for fun. Not well, for nothing. Just for fun. And then I dig into the consensus code of BTCD. 1st and foremost, I'm not a software engineer, although I did some programming in the past. I I think I'm not qualified for the title, but, I, I'm a Bitcoin dev. That's what I can tell. And I dig into consensus code and for weeks. Right? And then I couldn't really find find any exploit, like, any conflict, consensus conflict between BTCD and core.
And from what I can tell, BTCD is actually a very well, written implementation. It's it it it's it's it complies with the Bitcoin consensus rules bit by bits. I I couldn't really find any exploit. I need to dig more into it, to find more, but, yeah, I'm not sure if I can find a
[00:12:34] ODELL:
third. Also, the yeah. So the first bug the first bug, essentially, you used functionality that was enabled by Taproot that allowed you to do a massive multisig transaction. Right?
[00:12:48] Unknown:
Yeah. That's true.
[00:12:50] Unknown:
And
[00:12:51] ODELL:
Tap with yes. And then and so LND had some BTCD libraries in it. BTCD is a Bitcoin implementation that is written in Go, and those libraries made it so l and d nodes can no longer essentially, were no longer synced to the Bitcoin chain. They can no longer see what was going on in the Bitcoin chain. Is that a good overview?
[00:13:17] Unknown:
Yeah. That's correct. So layer 2's obvious landing is the only layer 2 in my book, and layer 2's, inherit the security properties of layer 1, and it depends on, it relies I mean, BTC in Lendly, in particular, it relies on the BTCD, which is an alternative implementation of Bitcoin. For the record, I'm not against alternative implementations. In fact, I'm in favor. I think diversity is overall good for Bitcoin, whether it's Lightning implementations or Bitcoin. Think more diversity is good for the long term, although it may be, scary for the short term.
So yeah. So the the the first part was unintentional. It was due to a parsing issue. We had it had a cons it had a parsing, check from SegWit SegWit scripts I mean, the scripts are limited to, 10 k, 10 kilobytes. Sorry. 10 kilobytes. And, I I I managed to make a large I did a large multi transaction, which consumed, like, 1 out of 8 block space or something. And and, yeah, it's it's it's couldn't pass this pre check. And, yeah, it's unintentionally broke LND the first time. And the second time, I I tried to look a second look for a second's conflicts, and I couldn't find any in the consensus the in the actual interrupt repo code, the TX script code.
And from what I can tell, yeah, I think b two c d is is a it's a good implementation. Then I when I couldn't find, any conflict, then I come back to then I go back to the wire code, the the the the TX parsing code, and I realized, okay, there is another conflict in the literally the above line. The the line above the previous pipeline, which was also a Taproot related issue. So in Bitcoin, we have a consensus rule in place that limits number of SEC elements in the row of a thousand. Right? You cannot exceed that. Although I'm not sure why it had why this limit was 500 k.
Not sure about that, but it's we, you know, we have another rule in template that says, okay. If you encounter an op success x op occurred in scripts, it's past the validation regardless. So it these rule does not does do no longer apply. So I I then I when I realized, I tried to first create a template output, a funding and ops success x, literally, script path. And maybe it's a bit of technical, but I then I tweak this output with my own inner key. I thought I did. But it apparently, I tweaked this output with an unspendable inner key. And then I found this, like, 3 weeks ago or something.
I discovered the bug myself, and and found this address, and then I reach out. I I build this transaction by hand and placing 500, 500,000 stake elements. And he said, yeah. He didn't question it. He said just pay extra fees for the manual work and then email us. And then yeah. I emailed them my rolled transaction. It was like it was paying overpaying, like, 3 x. And yeah. And then, obviously, it was invalid because my my my control block was incorrect because I thought I tweeted what my inner keypad. Then, you know, then they they said, okay. We need to patch. We couldn't broadcast it, whatever.
And then, like, on November 1st, 12 days ago, I was in need of some money, and then I I wanted to revert the money back to myself. But then I realized, okay, my transactions was wasn't correct. It was tweaked with was tweaked with an in unspendable inner key. Then, okay, let's, then let's let's make some chaos. Then I fix the evolved transaction and send back to f two pool in then in within an hour or something. Yeah. They it ended up in a Bitcoin block. So it was a bit of a mistake, but not not really. I think yeah. I did it by I did it with my. I I did it intentionally.
Then, yeah, then you all know the rest.
[00:17:48] ODELL:
So both were triggered by different on chain transactions. Both have the same end result that l and d nodes fell out of sync. Yeah. Both were based both of the issues stemmed from the use of the BTCD library. I it's interesting that you say you're for I one of the interesting things about this situation is depending on your bias, I've noticed people using it to argue against multiple implementations and for multiple implementations, which is just fun. It's just classic Bitcoin. So we have, well, first of all, Ben, can you communicate with us now? No. You can't.
Wow. They're trying to silence him. Okay. Well, we'll get back to Ben. We have Tony here. Tony, you wanna talk about how, how how someone could have used if if Barack was malicious, you did a you did a guide, basically, in reg test, which I guess so unlike Bitcoin test net one of the Bitcoin test nets, for how someone could exploit this attack to steal funds. You wanna talk about that real quick? Yeah.
[00:19:09] Unknown:
And and props props to Ben. He he's pretty much the one that wrote all these exploits. So Alan's point, I kinda made it you know, I've been doing these, like, you you know, lightning abusing projects, I guess, to to to say kind of being, along the same lines of, like, a bad actor in a way. But, so I decided, that it was time to, like, finally do Alan's point and write something that, like, actually tried to, you know, exploit some of these things that we keep talking about and that keep not getting fixed. And it was it it was with the intention of doing it at the Atlanta tab comp. There was, was in a workshop there, but I had you know, it takes a while to build, you know, some of the things that I've been building on, and I didn't have any exploits in there by the time. It was the week of the conference.
And then, week of the conference comes on Sunday, and and this this this bug breaks L and D. And, of course, like, I was like, Ben texted me and said, okay. Can we reproduce this, then put it into Ellen's point? And I was like, yeah. Let's let's do it. So we we did that. And over the course of that week, I was like, okay. Well, how do we actually, like, exploit this? Like, I got a reproducing that works on regs test. Of course, you can't reproduce it on main net, like, once it's already, you know, once it's already out there and being fixed. It just takes that one transaction to, like, break it. But, essentially, I broadcast you you basically if if you're in the same block as this transaction that, like, can't be, you know, one of Barak's transactions, You know, the node can't process it.
And if you're not running LND and bit, with with Bitcoin 0 mq, then then LND actually doesn't process any blocks after that either. So, which is something I found out by actually reproducing this, and and figuring out that, it wasn't just that block that it wasn't processing. If you didn't run-in 0MQ mode, it wouldn't it it thought there was a chain split, and it wouldn't process any other blocks beyond that. But, essentially, you know, I just went ahead and just put it in the same block in my reg test scenario. And I have a whole guide on this. It's, you can go to Bytes journey.com. It's one of the top top blogs.
But, essentially, after that, after you after your channel peer can't process the block, the block with the with with the transaction and at that old channel state, it doesn't know how to do the justice transaction. So in Lightning, if if someone were to try to cheat you, your node this is why nodes has to be, you know, pretty much always online or have a watchtower. But, you know, there's problems with watchtowers too. But if if you're not online and you don't see that transaction, you can't broadcast justice transactions, which basically take all of the other person's funds even if, you know, even if you weren't rightfully owed all of them. You do get all of the cheaters funds, that they have locked up in that channel. So if they're not online, they don't get to see that, and then I demonstrate. It was actually kinda cool too because I got to demonstrate, like, how an LND node you know, there's part of my tutorial walks through, like, okay. Here's a a patched LND node, and here's a non patched LND node. Let's, like, see what each of them do under this scenario. And the, you know, the patched one, you know, says, you know, stuff like, oh, your your peer is being fishy in the logs, and it's, like, broadcasting a transaction.
But the other one doesn't. It just says, oh, there's a chain reorg, and it's none the wiser. And then eventually, you might have enough blocks, and and you should get you know, you should be able to redeem those funds on chain. But if the other peer patches in time, even if we did even if someone did exploit this in the same block as as Barak's, All it takes is you updating your l and d note in time. Now there there's some questions there on, like, how soon you need to actually update it under all the scenarios. But if you do update in time, you do get to submit that justice transaction. But if you don't, then then you're kinda shit out of luck, and there's nothing once once enough blocks pass, you don't you don't get those funds back.
[00:23:25] Unknown:
It it may be useful to to point out that, in order to exploit this bug, you don't have to take that risk of broadcasting a a stale state where someone can steal all of your funds. But in fact, you can do it using HTLCs as well. So you can send an HTLC through the target node through the victim and then force close. And that victim, if you claim that HTLC on chain, the victim won't learn the preimage from that HTLC or at least in this class of bugs in general, the victim won't learn that preimage from that HTLC. And then after the HTLC times out, you can just take funds.
So that not only allows you to do it, allows you to exploit this this class of bugs risk free, but it also allows you to exploit this class of bugs, without, what was the point I was gonna make?
[00:24:16] ODELL:
And Without exposing yourself to the justice transaction?
[00:24:20] Unknown:
Oh, no. Sorry. I was gonna point, and and also in a much shorter time horizon. So that that's where you see the difference between 40 blocks, which is the default in most lightning nodes for the time that you have to claim an HDLC that's been routed through you versus, I think the default for doing justice transactions and broadcasting in old state is usually, like, 144. Some nodes set it to a full week. But when you're talking about HTLCs, you can exploit this bug much faster, and the victim has a much shorter time around time, to turn around where they need to upgrade their node and and catch it in time.
[00:24:58] ODELL:
And what was I was looking into this. What was the reason again, Matt, for why, it defaults to a low value? Like, people routing through you want that value to be lower in a normal situation. Right?
[00:25:10] Unknown:
Yeah. Yeah. So that that's basically the stuckness. So, like, if if you have an HTLC that gets stuck, that's how that's the total number of these, you know, it's normally 40 blocks per hop. So if you send a payment that's, you know, 3 hops, then it's 3 times 40, a 120 blocks that you have to wait before you're sure that that payment's gonna come back and get unstuck. Luckily, stuck payments aren't as big a deal as they used to be. We don't see them as much, but a node can always make a payment, get stuck for that amount of time. So if you're sending a payment, you care about that number being really low. Now the lightning protocol, and I don't know what settings different nodes expose, allows someone, opening a channel to say, you know, I only want to expose myself to up to this amount of funds in HTLCs at any given time, and that would allow you to limit your exposure to these kinds of attacks that are exploiting against HTLCs rather than exploiting against, the the the stale closing or the the stale commitment transaction.
But I think, most people don't bother sending that. Most people set that to, the full channel value. You should also not do this because it's bad for everyone else's privacy. And in fact, LDK, and thus Cash App use, prefer to route over channels or very slightly prefer to route over channels that do not set their HTLC max to the total value of the or the their total HTLC max in flight to the value of the channel, because it's actually better for the sender's privacy, in order for you to do that.
[00:26:52] ODELL:
Thanks, Matt. While we have you, LDK was also affected but only by the second bug. Right?
[00:26:59] Unknown:
For old versions? Yeah. So, I mean, the the this class of issue has bitten a number of, various, Bitcoin library deserialization libraries. Right? This this this kind of you know, when you're deserializing untrusted input, you have to make sure you're not gonna over allocate memory and crash yourself by by allocating gigabytes. You know, there there are better ways to design this code. There are worse ways to design this code, but a number of implementations, you know, BDCV has had more of these class of bugs than just these two bugs. You know, they've they've had some of these classes of bugs since before l and d. But yeah. So Rust so LDK uses Rust Bitcoin under the hood. Rust Bitcoin had a similar bug, that we caught internally, so we didn't we actually caught it, someone while one of the developers working on it was reading the code, not when it went to chain. And we fixed it at that time, by rewriting a bunch of the deserialization pipeline to just be more robust against this whole class of issues.
But old versions did have that issue, and so some people some users were using old versions of Rust Bitcoin. I don't know that any LDK LK users were using old versions of Rust Bitcoin, but, you saw some people running mempool and and running, Elect RS. We're using old versions of Rust Bitcoin, which did get bit by this bug when the second transaction was broadcasted.
[00:28:23] Unknown:
Right. Yeah. Actually, I I do have one project that's still on an old version of LDK, and I I found that out yesterday when, my hidden lighting network project, the project that does probes, for private channels. I launched it yesterday because Ben Ben, Ben had his first successful vortex channel open. So I wanted to see if I could figure out, which output was his that he opened the channel with because he didn't use sort channel ID aliases. So I launched up that LDK project, and then it's it's fetching the blocks, and then it crashes. So, but that's an unmaintained, you know, kind of like a throwaway project at this point, but that's,
[00:29:03] ODELL:
that that it was kind of funny that came back full circle on that. So, Tony, in either situation, an attacker had to essentially be set up as your channel peer before the transaction goes out. Correct? Like, they have to have a channel with you.
[00:29:17] Unknown:
Well, I think I think how Matt kinda described it earlier, I I think you can even just route through any node that has a problem, right, with the HDLs. I I was a little fuzzy on how you could exploit it under the HTLC path. But, Matt, for at least the way I have implemented the exploit, it does require the the the you'd have a channel with the peer. Well, call us Corallo and Odell.
[00:29:42] ODELL:
Yeah. Corallo. So is that correct? Like, you wouldn't to use the HTLC attack vector, you could just if you just had a channel open with someone I have a channel open, you could potentially exploit me on that?
[00:29:58] Unknown:
You know, I don't it it a lot depends on exact details of the bug and what what does or doesn't happen in the node. I think it it did. So in the general like, I could imagine in going both ways. I don't know the details of lnd and how it might have have operated in this case. I think for LDK, that might have been the case, if you were using a a really old version of LDK. But, you know, in general, that is a possible outcome depending on some of the the nuances how the note is implemented.
[00:30:30] ODELL:
Got it. Ben, are you here with us now?
[00:30:34] Unknown:
You hear me
[00:30:36] ODELL:
now? Yes. Yeah. We can. Alright. Do you wanna introduce yourself?
[00:30:41] Unknown:
A half hour late,
[00:30:43] Unknown:
but, I've been the car man.
[00:30:45] Unknown:
My company, but did kind of open source stuff on. And work with the.
[00:30:56] ODELL:
Awesome. Well, it's a pleasure to have you, and I'm glad you can finally speak. I guess, Kaludis, what are your thoughts here as the lead maintainer of Zeus, the number one way people interact with their node at home during all of this?
[00:31:12] Unknown:
So, when this second exploit dropped, I was sort of like a double take, like, on Twitter and in, like, a few chats that I'm in, including, like, you know, the Zeus Telegram. I'm like, what is everyone talking about? You know, this is the exploit from, like, we're still patching this up. And then it hit me that, you know, a second exploit had dropped. And, you know, I wasn't super surprised, but it was a huge huge inconvenience, I feel, for a whole lot of different parties. You know, from, you know, the BTCD and LND guys who had to patch up their code, to, the maintainers of, like, node projects and, you know, Lightning wallets, who had to update their releases, individual users who were asking, hey. What's going on? Why do I have to update my note again?
You know, it it really shook a lot of things up and, you know, ate up a lot of people's time.
[00:32:21] Unknown:
Matt, I wanted to add that Rust Elements was also affected, but the attack is obviously harder to do, because of Liquid's functionary model instead of, reaching out of band to a minor. Yeah. You wanna go into that?
[00:32:36] ODELL:
Like, what was the what was the impact on liquid?
[00:32:39] Unknown:
Well, it's it's the same bug, and it just needs to be patched. But Rust Elements also had it, and I wanted to add that because, you know, we said LDK had it, and LDK is using Rust Bitcoin and Matt mentioned for the older versions. And Greenlight also for its watch tower, was using rustic wood, so it also was impacted. But, I mean, these are remedied now, and, I guess we're good. But, just like Evan said, we were also a bit inconvenienced in trying to scramble to figure out, what exactly needs to be done. It wasn't just an LND bug.
[00:33:17] Unknown:
Yeah. It was, like, a general block parsing error that, you know, a lot of people had looked over in implementing a Taproot, it seems like. And and then
[00:33:26] Unknown:
It it wasn't a Taproot thing. No. It was a a segwit thing, a witness witness side. Yeah. I think it looks like and it just made it easier to to create that that large webinar.
[00:33:40] Unknown:
Yeah. That's what Barack was mentioning earlier, someone, where they're wondering why the original size limit. And I believe Laulu's explanation was, because when he originally added the SegWit code a long time ago for parsing, like, in v zero. That's what it was.
[00:33:59] Unknown:
Gotcha.
[00:34:01] ODELL:
But so what does that mean for liquid users at the time?
[00:34:06] Unknown:
Liquid users, everything was fine, but, because of the multisig and other things, you know, all of that needs to be updated, whether it's the functionaries or the on chain multisig. Yeah. And there was, like, also a scramble, by a certain amount of blocks, things that needed to be updated. So, yeah, I think I think we, Blockstream formally released a statement. So I can, post it in the elements chat, but I'll leave the exact details to that. But similar issue, witness size, parsing.
[00:34:46] Unknown:
Did it halt liquid, or is it just, like, the peg in, peg out stuff got broken?
[00:34:51] Unknown:
It it didn't halt liquid, but, yeah, it's the peg in, peg out. Exactly.
[00:34:57] Unknown:
What so the connection to the chain?
[00:35:00] Unknown:
Yeah. I saw a side swap had some issues. I didn't really see any other people complain too much. I think Barack would also know a lot about, the impact to Liquid because he's probably the most knowledgeable in Liquid script.
[00:35:17] ODELL:
Yeah. And his project bit matrix is runs on Liquid, so he kind of Barak, did you did you bork your own project when you sent out the transaction?
[00:35:26] Unknown:
No. Bitmetrics, 4 g, and, you know, other that like, TDEC are not they they were not affected. The I mean, the the liquid network functioned well. I think, yeah, it only apparently, it only affected Pagan and Paycard functionality, and, obviously, Rust Bitcoin and few other infrastructure projects in the space. And in fact, I mean, I wasn't I was only targeting BTCD and LND. I wasn't aware this would cost I mean, this could have caused, any other issues.
[00:35:58] Unknown:
So we got caught up in the lateral damage.
[00:36:01] Unknown:
So Yeah.
[00:36:05] ODELL:
Do you wanna talk about real quickly, the tweet you sent out when you when you sent out that transaction? What was it? In order to find the light, you need to touch the darkness?
[00:36:16] Unknown:
Yeah. That was basically a joke maybe, but I think it has also a truth in it. Right? Obviously, I did it for fun. I guess, yeah, I love making cows. I guess I crave for cows maybe. Yeah. It makes me feel good. I guess, that's you know, good or bad, that was the most meaningful thing I did in my life, regardless the outcome. But, yeah, basically, I did it for fun, but I think it also had, it will have also have good, you know, consequences or, like like, long long term benefits in in the long in the long run, I would say. Bringing into chaos into into software, I think, yeah, makes everything else more robust in the long run. Maybe causing some, damage in the short term, but, yeah, making everything more robust.
[00:37:14] Unknown:
Brock, did I, grok the transaction page right? Like, the transaction cost something like $750?
[00:37:23] Unknown:
Sorry. What what was that again?
[00:37:25] Unknown:
The fee for the transaction to broadcast it
[00:37:28] Unknown:
Yeah. I paid it $750
[00:37:30] Unknown:
out of your own pocket. Got you. I did it for fun. And, yeah, I was actually in order of money as I told. And I I was revert I was trying to revert that money back to my myself, but I think I could I could. Was an off success anyway. I could have done that, but I I realized I couldn't. I mean, I didn't realize that at the time. And then I just I I choose the cows.
[00:37:52] Unknown:
I I I want to make cows. Gotcha. I I just think it's funny because most people are receiving money when they're burning their reputation instead of having to spend out of their own pocket. It was a reverse rug pull. I'll yeah. I like that. Yeah. I guess so.
[00:38:07] Unknown:
I I would love to possibly kinda get into, because because I agree. Like like, what doesn't kill you makes you stronger. Right? But there's you know, I'm curious of the balance here between, you know, causing chaos and in the end goal, we're all I I think we can all agree that, you know, I don't know, code wise, you know, or or ecosystem wise, we're we're better for it probably. But, I mean, what you know, is there a balance there at all? Like like, doing stuff like this versus responsible disclosure, because, you know, I I don't know where I land, like, on it. But because it does I think fire drills do make us better.
In the end, we need fire drills. Anyone could have done this for $750. It's just the matter of, like, well, yeah, when to you know, at least going forward in the future, you know, when when to do this versus versus responsibility, the skill. So I wonder if we can, like, have a conversation around that. It's probably useful
[00:39:20] Unknown:
it's
[00:39:21] Unknown:
probably useful just to kind of high just to to quickly make sure we're on the same page about what responsible disclosure is. There's there's a lot of people who get confused and think, like, responsible disclosure is all about, like, protecting the vendor and making sure you you disclose appropriately to the vendor and whatever. And that's not at all what responsible disclosure is. Right? Responsible disclosure is all about highlighting that you have a duty of care to their users. It doesn't matter what the vendor wants or what the vendor thinks or, like, you know, if you wanna do something that makes a vendor look like shit or and look like an idiot because of some bug they had, you know, it'll probably backfire, but you're welcome to try. And it doesn't really matter. It's not it's not related to responsible disclosure. What responsible disclosure is is saying, like, look. There's some users out there. They're running the software.
You know, for whatever reason, they decided to run the software. They they decided that it was worth worth running because, you know, maybe there's some risk some risk to running any software, but they decided that that they needed to run the software and that, you know, you have to do right by them. So that means, you know, responsible disclosure can mean going public if the vendor refuses to fix an issue. You know, that is a part of responsible disclosure, but it does mean making sure that the vendor has a chance to fix the issue so that users don't get hurt and aren't, hurt more than they need to be. You know, another part of responsible disclosure is is, reporting the issue publicly after it has been fixed to encourage people to, upgrade. And so that that's that's something where you can say, like, okay. After it's been fixed, exploiting it to to force users to upgrade, you know, is is an option, and that that's a legitimate part of responsible disclosure. But it's never, you know, putting users at risk when when just by taking action, you know, before there's even a patch or before users have a chance to upgrade.
[00:41:14] Unknown:
Yeah. I think I I agree on that definition, and I would definitely pick, responsible disclosure if this was Bitcoin Core. And I would definitely pick, again, this approach if this if Lightning is in production. I mean, is if Lightning is used by a lot of people, I think, like, the way I see Lightning is it's really a beta software, and it has, like, this public capacity of 5,000 or something. Even Liquid has, like, 35100 Bitcoin. I think, I don't really take lightning seriously at this at this very stage, and I I kind of see this as a playground. And
[00:41:53] Unknown:
I don't think that's fair at all. I mean, there there are large companies who use lightning now, and and see material transaction volume. I mean, I think I I don't know about Cash App, but I know, like, River disclosed their transaction volume on lightning. And it's not a joke anymore. It's not it's not trivial. They have a lot of funds locked up in lightning that are put at risk in something like this. And that's something too where, like, you know, if lightning were were super alpha and everyone was on the same page there, it'd be one thing. But when there's large corporates who have money locked up or considering deploying Lightning, you know, because they see their peers deploying Lightning, and they're they're trying to decide whether it makes sense to deploy Lightning right now.
You see these kinds of things and it it it makes corporates think twice. It it it really hurts the deployment of lightning even if we assume that there's no no material funds at risk here. But that's not not accurate anymore either that there there are there is a relatively significant amount of money locked up in Lightning and especially, you know, I lot I know a lot of, individuals who who put a lot of money in their lightning node, who put a large amount of their Bitcoin stack in their lightning node. And, you we can agree or disagree about whether that's a great idea. I obviously don't think that's a great idea. I agree. So, like, yeah, there there is a lot of Yeah. I think the I don't know. Well, I mean,
[00:43:16] Unknown:
Yep. Yeah. So, yeah, I think there is a little relatively large on Bitcoin, yeah, in in Lightning. But, yeah, I think, yeah, the corporate's using Lightning infrastructure, I think, yeah, they can easily upgrade to a new version and, you know, and and and the other folks, perhaps, you know, can simply use watchtowers to avoid these kind of attacks. And, you know, when when I text right. Sorry. What? There's
[00:43:42] Unknown:
no watchtower implementation that exists that would have avoided this issue because the, a, the l watchtower is, uses the same code, but, b, all of the current watchtower implementations don't enforce HTLC, HTLC forwards. So the watchtower is don't, address the issue even if and and doing that would be a very non private watchtower. You can give up a lot of your privacy to the watchtower. So there's no code to do HTLC enforcements, even if you even if you wanted to. Not in the watchtower.
[00:44:15] Unknown:
Yeah. Okay. Good to know that. I didn't know this. So okay. But it's not more so easy. You can yeah. For HDLCs, I think, yeah, there is a tech point with HDLCs, which has a is a shorter abstract walk time in it. But I think when the HDLCs claim, the preimage is revealed on chain anyway. But, as for, you know, the actual commitment outputs, these outputs have a relatively larger relative walk time in it anyway. So no other runners can easily upgrade to the newer version and avoid these, attack vectors. So when I did attack this when I created this, the second, transaction, I was aware of this, and I I knew there wouldn't be any there wouldn't be any potential loss of funds, and it appeared, though, we haven't I I personally haven't seen any
[00:45:07] Unknown:
any loss of funds. That's that's not true. Right? That even though there wasn't loss of funds in the first one, there couldn't be loss of funds in the second one. I I would I would push back and say, in fact, that because there wasn't because there there was a recent bug of the same class, they're substantially more likely to be loss of funds, in the second case because people are aware of it. People are are aware of how to exploit it. In fact, there's sample code to exploit it, that the the folks here worked on because of the first bug. Because, you know, once the first bug's out there, of course, you just write sample code, whatever. It's, not not making the exploit all that much easier or harder.
But that because of the first one, it makes it substantially easier to exploit the second one, which which put a substantially higher risk on the second one. No?
[00:45:52] Unknown:
Yeah. I I totally agree that phone files are totally at risk due to the interactive model of lightning. And I still see these attacks as more of a denial of service type of attack
[00:46:03] Unknown:
than an than, like That's yeah. But, I mean, that's not how lightning works. Right? Lightning has always been denial of service attacks allow you to steal funds. That's that's the lightning security model for better or for worse. So there's there's basically no such thing as a denial of service attack against a lightning node.
[00:46:21] Unknown:
Yeah. But, again, you have still you see this as kinda, a type as long service attacks because of the large relative work time in commitment happens in front runners.
[00:46:31] Unknown:
Forty blocks. Right? Forty blocks is like, the time during which you're asleep.
[00:46:38] Unknown:
Is is that default?
[00:46:40] Unknown:
For a for HCLCs. Yeah. It's 41. It's default for LND.
[00:46:45] ODELL:
I think core lightning, the default is 144.
[00:46:50] Unknown:
It might be. The the default for I mean, 40, I think, is the the most common default is the most common value on the network. It may just be for LND.
[00:46:59] Unknown:
But that is for the HTLCs. Right? How about the commitment outputs?
[00:47:04] Unknown:
That's true. But the HTLCs are how you would choose to exploit this kind of bug, because there's no risk to the the x the attacker. It wouldn't make sense to to exploit via the commitment output. But you're right. The the commitment output the commitment output historical state of broadcasting is a much higher value. I'm not sure what the default is there. I think it might be a day. It might be a week. I I keep hearing 2 weeks. Is 2 weeks not true for the commitment outputs? I think it's configurable. I I heard I thought the most common was a week, but it it may be 2 weeks. I don't know. That's totally within would be within reason for for a value to choose.
[00:47:42] Unknown:
I've been hearing a week and 2 weeks also, Matt.
[00:47:46] ODELL:
Both, Matt. Which is it?
[00:47:50] Unknown:
I don't know. Pull up the source yourself.
[00:47:52] ODELL:
Yeah. I know. I know. I'm just saying, you know, I mean, there's definitely there's gotta be a bright side here on, you know, on on Barack's side, right, that we're actually having this conversation.
[00:48:06] Unknown:
Yeah. I mean, it's it's healthy for the community to have a conversation about what responsible disclosure is or isn't so that people are are aware of this kind of thing and so that we can be be more you know, have the community have a a more common understanding of what what responsible disclosure means and and how it should look, so that hopefully we as a community kind of are all on a on a similar page. I mean, I certainly through the course of this this exercise, it it became very clear that the the Bitcoin community has very different ideas of of both what responsible disclosure are and what people's responsibilities are. And I think having a a a more common understanding of what responsible disclosure is should get us hopefully to a a more common page.
[00:48:50] ODELL:
So what would you wanna see here? Let's say you're Barack and you discover the second exploit. Presumably, you wouldn't do what Barack did. What how would you proceed?
[00:49:03] Unknown:
I mean, I think what you do is certainly, the first thing you do is you reach out to the vendor and you make sure you give them a reasonable amount of time to to patch it. Right? Which vendor? To all the lightning implementations or just l and d? Or If it if it affects more than one that you're aware of, certainly, you let you know, for for security issues, you tell the minimal number of people required, in order for the the the issue to get patched. So you start with l and d because that's the only one you're aware of has this problem.
[00:49:32] Unknown:
And then And to be clear, it's like you would you would reach out to them privately. You wouldn't just, you know, go post, like, an issue on GitHub. Right? Right. Yeah. Yeah. Which is what Anthony Towns did, right, about 10 days before this happened?
[00:49:45] Unknown:
Yeah. I mean, I don't think I think he he also I I think he'd said that he'd screwed up. He he didn't realize that this was a a a major security issue. But, yeah, I mean, that's also not the correct Yeah. He hung up here. But then so so I think an important part of of understanding responsible disclosure is is that you coordinate with the vendor about release timeline. Right? So so if you're the one reporting a bug, you are allowed to say to the vendor, hey. I think this is really critical. I think I should go public with this in 2 weeks. Can you get a patch together and and make sure you get that shipped ASAP?
And then that's a conversation that you have with the vendor. It's not like the vendor gets to say, like, no. This has to stay private or here's the release timeline. Take it or leave it. You know, that's a conversation you have with them. And then I think once the the fix is released, it be that, at that point, it becomes a question of, like, what you think is the makes the most sense for for users. So, really, you know, patching a a an issue quietly does not get the most people to upgrade. And so, you know, you've seen this con we've had, you know, back when I worked a bunch on Bitcoin Core, we had these kinds of conversations a lot. You know, when you have an issue, do you tell people about it so that people go and upgrade really quickly, or do you not tell people about it because, you don't wanna give attackers enough information to go exploit the vulnerability?
And so it becomes a a balancing act. And, again, this is where, you know, you, the reporter, get to have a conversation with the vendor and where you, the reporter, gets to ultimately decide. And I think it would be I I don't know if I would have decided to do that, but it would have been totally reasonable or at least totally justified to go exploit the issue after the patch had been released or maybe maybe a few days after the patch had been released so the people had a chance to upgrade on their own, to highlight the issue and and to for kinda force people to upgrade, quote, unquote.
But, you know, prior to there being a patch and prior to people having at least a small window of time to to upgrade themselves, actually exploiting it just just seems like putting users at more risk than is necessary.
[00:52:01] Unknown:
Well, isn't that kinda just, like, the sad truth, the sad reality of the situation? Like, we talk about we're just just saying that, like, oh, because there are significant volumes on Lightning now, and there are big companies and corporations looking at Lightning. And then things like this overall hurt the perception of Lightning. I mean but isn't the perception of Lightning just wrong currently? Like, we live in this happy little lightning world that, like, all of our payments are fast and easy. We can do 0 conf atomic swaps. We can do all this other fancy shit, but, like, Lightning's pretty broken. I mean, Matt, your talk, Crowley, your talk at, Atlanta was, like, a really good talk, and it just kinda highlighted that lightning is kind of broken in its current state.
[00:52:49] Unknown:
Yeah. I mean, to add to you, as well, like, this is literally a valid transaction on chain. Right? I think that's the way certain people are framing it. So if your l two is affected by it, they feel like, in essence, it was your fault. Bitcoin Core thought this was valid, and I think that's where some people are drawing the line.
[00:53:10] Unknown:
Yeah. Totally. I mean, like, if you're if you're, a it was LND's bug. It was Rust Lightning's bug. It's not like the the the the foundational issue was that there was a bug. The foundational issue was not that it was exploited. The only the only question then is, like, okay. There is a bug. Given there is a bug, should you exploit it? That's a very different question from, like, whose fault is the fact that there is a bug. Right? And bugs happen. You know, that's not gonna not gonna change. That's the reality of software development. But but yeah. So, you know, I I think to the point about lightning being broken and the perception of lightning being wrong, I think it is important to to highlight the difference between, your counterparty and, like, you know, I always kinda make the point that you shouldn't open channels with a counterparty you don't know or at least don't trust enough that they're not gonna, like, try to steal your money and, like, put in a bunch of effort to steal your money. I think this is a very different world from and then there's denial of service attacks against Lightning, not in the, like, take your note offline, but in the, like, saturate your liquidity so that you can't send payments. So let me I guess I should rephrase my earlier statement about there not being denial of service attacks against Lightning. Just that the the the, saturating liquidity denial of service attacks.
So there's a lot of things that are broken with lightning and where the perception is maybe wrong, but I think those are different from you're going to lose funds maybe because your counterparty did something without them even being involved. So when we were talking about, you know, routing an HDL through multiple hops and then stealing funds, that's something where like your counterparty can be totally honest and they go to chain and then your node doesn't see it and then you still lose funds. Right? So I think that's the kind of thing where, you know, aside from these bugs, generally, I think there's not really much in the way of known issues, for lightning for why this might be why you might lose funds if your counterpart is, like, completely honest.
[00:55:10] Unknown:
And regarding previous bugs, Matt, it's not our first bug on lightning. Right? There's one in 2019, and then Antoine Riard and Gleb also found a dust one last year that, I guess, they responsibly disclosed. Like, do you remember how that process was, and maybe we can, like, touch on that?
[00:55:31] Unknown:
Yeah. That was across the whole that was basically on every lightning note. So they they reached out to everyone. They emailed, you know, I guess, all of the I don't know who all was on the email list. But, you know, basically, all the maintainers or the kind of lead maintainer or 2 of all the lightning implementations and kinda describe the issue. They describe the basically, the issue was that, because lightning has some concept dust outputs. If an if an output is dust, if an HTLC the value of the HTLC is dust below some threshold, then it just gets burned to feeds.
So, a counterparty could send a bunch of HTLCs that were just barely below this threshold. And in fact, they can even do things to, increase this threshold, that nodes will currently accept. And then if they do that, then they can burn a lot of your money to fees. And I guess if they're a minor, they get free money then because they burn the money to fees and then they get to claim the fees because they're a minor. So the they they reached out to all the lightning implementations. All the lightning implementations added a bunch of code to, patch it and then ship that code. And then I think after everyone had shipped or some period of time afterwards, they disclosed it publicly. And and, Antoine wrote up a a a change to the bolts and described this issue and and note that that implementers of the bolt spec need to, do something to to ameliorate this issue.
Unlike the kind of on chain stuff that we've been talking about, there's no way to go, like, exploit the issue to force people to upgrade. There's nothing like that. We can only, like, write code and and tell people that this this issue exists and you should upgrade. And then, like, if you were to exploit this issue, you have to actually go steal funds. You you couldn't just, like, cause the issue and then and then get no stuff.
[00:57:22] ODELL:
And then there was the there was the other bug where none of the lightning implementations
[00:57:27] Unknown:
were actually checking the chain for commitment transactions. Right? For funding, I believe. And that was the For funding. Yep. The funding transaction had the right script and the right value. Actually, that's all I we were. So before LDK was LDK, Rust Lightning existed and it was a joke. It was like a tiny project, and and no one used it, and it wasn't even ready for production. But but we were checking it. So Mhmm. But but yeah. I mean, that that was another again, that was another case where you couldn't do something to actively force everyone to upgrade, and you could only actually go steal funds, which obviously has a has a very different moral question.
[00:58:03] Unknown:
So it seems that the heart of all these attacks is a security assumption that your peer, ideally, will not be a lightning node peer as well as a miner. I know there's a lot of stuff being kinda shuffled around with the mempool and Bitcoin Core right now. People happy or unhappy about the RBF policies. How do you think this will affect, any other things, any other new potential dust issues, HCLCs, anything? Does anything come to your mind, Matt? Anything about, like, l two that's needed or penalty transactions?
[00:58:44] Unknown:
Yeah. I mean, we know a lot of stuff needs to change. And this isn't even so that there's questions about minors, but also just around mempool policy like you mentioned. We know a lot of stuff has to change. We have ideas for how that stuff changes. All of it to exploit it though, like I mentioned, requires some amount of effort on the part of your counterparty. They have to, like, do a bunch of work writing a bunch of software, coding stuff, to to actually go steal funds, you know, anchor outputs. As long as you have anchor outputs in your channels. So anchor outputs don't prevent don't protect you against an actively malicious counterparty, but they do protect you against, the just the fees going up in the network.
So the the kind of straightforward issues that existed prior to Anchor, solved by Anchor, you know, encourage you to make sure your opening channels that support Anchor outputs that have the Anchor flight turned on. And aside from that, you know, there there's things that need to happen, but they they do require some amount of effort to to exploit. So also just make sure you know who your channel counterparties are. You trust them not to to do a bunch of work to steal your money. Trust them to at least be kind of honest.
[00:59:57] Unknown:
Well and and then maybe that's just, like, the thing that should be highlighted more because I don't think that's, like, the current situation today. I mean, where people just, like, open channels with, pretty much anyone, really. Any random on the Internet, they kind of just, like, go and open channels, and they let their little Raspberry Pi sit in the closet for months that they don't touch. And then that's kind of the end of the story there. So, you know, hopefully I mean, I would say, you know, after, you know, some, you know, person that just had their lightning note in the closet for 2 months, and they had to go figure out how to update it and then having to do it again in the same month. I mean, you know, if they're not ready to sign up for that life, then maybe they they really should consider. Like, okay. Maybe I shouldn't shouldn't be running this lightning note, you know, after after all of this. Because that's kind of the reality. You don't just, like, throw it in a closet and, like, you know, don't come back to it again.
[01:00:58] Unknown:
And your your blog, Tony, you said it well. Don't open or if you're gonna go into a coma because, like, it's not just something you can like, you know, lighting a car that you can maintain personally, then it's just, like, you're running up cold yourself. Like, figure out how to either study lighting all of sucks or, like, something that's like like a analysis or something.
[01:01:30] Unknown:
Yeah. And that's also another question about, like, how do how do the developers communicate the safety of the the products they built? That's something I struggle with a lot. I don't know. There's no easy answer, but how do and this is, like, a very, very broad question too. Right? And and maybe this is something worth discussing, but, like, if you're writing software, you're writing a wallet, how do you communicate to users what the relative risks of all these different attacks are, especially when you don't know for sure? You know, you you have some idea of how to exploit it, but, like, will it work in practice or not? It's a whole other question. And then, you know, how do you how do you evaluate this and how do you communicate it to users? I mean, this is another thing, you know, I think I always say that the only two things the only thing in the entire kind of quote, unquote cryptocurrency space that's in competition with Bitcoin is stable coins in practice because people do wanna you know, the only thing that works that does payments is is stable coins and people use stable coins for payments.
And so how does how do you if you're, like, writing a wallet that supports Bitcoin and stablecoins, how do you communicate to users that, like, this stablecoin exposes you to risk of, you know, Tether going under and stealing all the funds. It exposes you to risk of, like, the blockchain it's on and it's like Tron and it might just, like, randomly implode and, like, you know, it also exposes you to the risk of of just US dollar and sanctions and, like, maybe the US government decides to, like, this this stable coin is not enforcing sanctions and suddenly all the money is taken. Or you can use Bitcoin and the the the risk here is the volatility. Like, how do you communicate that to users in a way that they understand that intuitively and then make a decision that makes sense based on that communication?
Or do you just say, like, here's stable coins and here's Bitcoin. Use what you want. And then users just assume that the stable coin is, like, gonna keep working and has no risk associated with it.
[01:03:25] Unknown:
I think that's too many options. They can only do swipe.
[01:03:33] ODELL:
So it kinda seems like the conclusion is, that at least Sovereign Lightning usage is not in the cards or should be dissuaded from end users right now. Or or at least
[01:03:49] Unknown:
make sure you know who your channel counterparty is.
[01:03:54] ODELL:
Yeah. I mean, that almost feels like like, I don't know who my channel counter parties are.
[01:04:00] Unknown:
Well, you started your node at a different time. You know, like, you were one of the earlier nodes, and it was, quote, unquote, reckless, have fun. Like, you're just trying to test the technology. I think now we're in a more mature state where, I guess, it is still reckless, but you should not you should do your, best efforts to know your peers and, update your software, all that. Like, it's no longer SpongeBob memes. You have Tony and Brock out here now.
[01:04:34] ODELL:
So I have a question, which I asked on the open source stage in Miami this year. So in my Bitcoin experience, one of the things that I learned early on is that unlike traditional proprietary software that people are comfortable with or or used to, I guess, we intentionally do not have auto updates on Bitcoin Core because it's an attack vector. If if if you just update blindly, then you might update, a malicious update. You might update an attack. So a lot of people with with Bitcoin Core specifically, they're they're they're slow to update. They don't update very quickly. On the Lightning side, there seems like I know it's still early days in Lightning, but it seems like there's been many, many, many times where, like, the community has been, like, pushed to to update ASAP. There's something critical you have to update ASAP.
Now, enthusiasts, like, hardcore Bitcoiners that are not necessarily, you know, fluid in code or whatnot, but they live and breathe Bitcoin, are running their lightning nodes on Umbrel and RaspiBlitz and Start 9 and Noddle. So they're they're running these, like, prepackaged node projects, and then they see that there has to be an update, and they just go to them, and they just quickly, like, blindly click the update button. Is there I'm 1 I'm curious if anyone here supports the idea of an opt in ability to auto update, Because they're practically doing it anyway. Like, what is the negative of giving them the option to auto update instead of, like, you know, being traveling, being away from your node, not being able to update for 2 weeks even if they're just gonna blindly up press the update button anyway.
[01:06:28] Unknown:
I mean, I I'll I'll go first. I mean, I think it's a super valid conversation. I mean, I also hate the idea about updates, but that is the reality of the situation is that they're just clicking update anyways. I mean, when you even think about mobile apps and and mobile phones and, like, no one, for 1, like, make it incredibly hard to actually verify any source code at all if you're going through app stores. And for 2, you know, some of the app stores even auto update anyways. So, like, there's there's there's nothing they they really do there either. So and and most people's lightning nodes are probably mobile phones, mobile phone apps anyway. So I I think it's a valid talking point. I don't know if I would go far go as far to say, like, there should be an auto update option in in apps like Umbrel, but I think it's a definitely a valid conversation to have. I'm curious what other people think on that.
[01:07:24] Unknown:
So I would say that's fine, Matt, but I would then also insist a contingency that there's an atomic rollback feature. I'm a big fan of Nix Bitcoin and NixOS for this reason, and I believe Bitcoin Core uses geeks also for reproducible and deterministic builds. So as long as there's a rollback option after that auto update, they can immediately go back if there's something wrong.
[01:07:51] ODELL:
They can rush to unupdate. Correct. So rush to update.
[01:07:55] Unknown:
Correct. The soft so the the rollbacks are hard too though because you might have, like, updated your state. Like, your local database might have gotten upgraded or something, and then, like, you're gonna try to roll back, and then it's not gonna be able to read the new the new stuff. That that's actually something that LDK supports that's maybe a little unique, but it's a lot of effort to support, and I wouldn't necessarily say that that all developers should support it. One other point I I would also highlight that, Bitcoin Core is very different in the the requirements here because there's not only a cost to you personally. If you like auto update and the the software is now, malicious and steals your keys, That's one thing.
But it's there's also issue with, if you auto update and there's a consensus rule change and all the nodes auto update, suddenly someone can for example, like, BSV pushed an update today to to be able to steal arbitrary coins so that Craig can go just steal your coins if he doesn't like you. You know, suddenly you could do something like that if there were, like, a real auto update mechanism. So there's additional social costs in Bitcoin Core for that kind of thing that maybe don't apply in lightning or in kind of other wallet software cases.
[01:09:17] Unknown:
Yeah. I don't think there's, like, a real clear, clean answer about what the best behavior here would be. Obviously, any opportunity you have to let the user verify, that they're running, like, a specific version, like, a deterministic build is always welcome, but, you know, much easier said than done. Like, an iOS app, like, you can't do that at all. Right? And, you know, Bitcoin wallets, like, only a handful are, you know, deterministically reproducible on Android. But I think there might be value in some of these node platforms giving users the ability to, manually upgrade to certain releases perhaps before, you know, like, a maintainer can, like, bundle it back up again.
I think that's a feature that, my node has, that I think is pretty underrated. Like, you could just pop in a version number, and it'll fetch and try to build it and install it on your machine. But, you know, depending on these architectures, it might not be so easy for a lot of these projects to transition to that. So Evan also has the room for user error too. So
[01:10:23] ODELL:
So, Evan, I mean, out of everyone in this roundtable right now, you have, the most exposure directly to end users of Lightning, because of Zeus.
[01:10:34] Unknown:
Yep.
[01:10:36] ODELL:
Like, what has what are your after these bugs and and the reaction among the Zeus community and how everything went there, how do you, I guess, you and Ben do because of, the Bitcoin company. How do you how do you how do you feel about Lightning going forward? Like, how do you think about actually developing for end users?
[01:11:02] Unknown:
Yeah. I mean, there's a lot of complexities here. You know, we're definitely seeing a lot of users that are frustrated by a lot of different things. Like, you know, 1st and foremost, this year, Tor has been really awful in providing a really bad user experience for people who want to sovereignly run their own node at home, able to, you know, test when they're out in a meat space. Right? Something like, you know, these l and d exploits are are sort of exacerbating, a lot of the frustrations these users have. You know, while some users may be technical enough to put together their own node and run something like Umbrell. They might not, you know, be following or have the understanding to see what's going on with, these exploits and and the real nuance about what's happening.
And that's unfortunate because, the alternative is, you know, these users putting more trust in other third parties or, you know, running, you know, just a prebuilt version of, like, a lightning wallet on their phone that they might not verify. A lot of users might just be falling back on LSPs as their only peers now. You know, there's there's a lot of complexity whether that's a good or a bad thing. But I think, ultimately, what we're seeing from exploits like this, we have people that are less willing to run Lightning and and are dissuaded from using it at all. And, I think that's ultimately a bad thing because we have a network that's less distributed where, you know, these big intermediaries are, you know, playing even bigger roles than we'd like them to.
[01:12:58] ODELL:
Yeah. I mean, I see that trend kind of happening as well. I just wonder how, like, how that can really be avoided.
[01:13:07] Unknown:
Right. Like, on on one hand, it's like, you don't want people running these setups that are uncapable of doing it, that are just gonna be neglecting it. But on the other, you just don't want, like, these centralizing factors. I mean, obviously, there's, like, a degree of centralization that's gonna happen. Like, you're gonna have, like, these big hubs on the network. They're gonna use the SSPs or, you know, maybe, you know, use them as introduction points on blinded past. They offer up a plethora of services. Like, that's unavoidable. But, like, we also want to make, you know, having, having the ability to, you know, be as sovereign as possible and and run this infrastructure out of your house possible and also, like, appealing. Like, we we didn't want this to be something that's that's commonplace.
And, you know, it's it's it's happening in a lot of ways. Like, something like Umbrell is a consumer facing product. It's not just corporations that are running this thing. You know? And, you know, a lot of users in some nation states like El Salvador are dependent on lightning to to live every day. Right? So when we have someone, like, go and exploit something like this and, you know, there's a risk of actual people who are dependent on this stuff losing savings, it's, really frustrating, especially considering all the effort that people are putting into this on the lightning side to make Bitcoin something viable that we could be using in our day to day lives. Was Chivo down during the lnd or block parsing bugs?
[01:14:41] Unknown:
I don't know. Jivo is actually just, like, public, is just powered by somebody's back end. It's not it's just it's it's a database, but it also somebody else does their lightning in and out. I don't know if it's public who does that, so I'm not gonna mention it.
[01:14:58] Unknown:
But I know I think they announced it. Do that. It's River.
[01:15:04] Unknown:
Yeah. It used to be a scene. Now now it's now River announce that they're doing it. Yeah. It used to be some other folks. Now it's River. I I I know River was able to update pretty quick. So
[01:15:14] Unknown:
Yeah. They're doing a pretty good job with your infrastructure, so that was good to see at least. I think, Chivo's problems really rely on Beyond chain stuff that they've been doing where people will receive a or send a transaction, and then Chivo says it's canceled despite it being confirmed on the blockchain.
[01:15:32] Unknown:
Wow. That's pretty bad. I was unaware.
[01:15:36] Unknown:
Yeah. It's, they're not really helping themselves out over there in El Salvador from, you know, governments and Elis. My understanding is they've gotten a lot better.
[01:15:44] Unknown:
That that it was a it was a trash fire for the first while after it launched, but it it had gotten a lot better. But I haven't tried it. I don't know. Yeah. I mean, I think I think people have just
[01:15:55] Unknown:
given up on it at that point, though. There's a good Bitcoin Magazine article. I know it's kinda getting off in the weeds, but there's a good Bitcoin Magazine article of people trying to use it now, like, 1 year later, and and people have just sort of given up on on Bitcoin because of the Chivo experiences. So, I mean, that I mean, that kinda relates to Lightning. Right? Like, we have so many, bugs and exploits and issues with lightning, that we're that we're all kinda aping into right now too early, it's gonna leave a bad taste in the mouth. So maybe, like, the more responsible thing to do is kind of have a slow roller a a slow, rollout of of lightning instead of just jumping into it.
[01:16:37] Unknown:
Yeah. I mean, it's also important to note that folks in El Salvador some parts of El Salvador had real demand for the features that Bitcoin can provide for people. Others not so much. You know, it is a US dollar based economy. There wasn't you know, inflation's now high with dollars, but especially when that first rolled out, it wasn't, and there wasn't, you know, a lot of of user demand for it. You know, I'm much more interested in in parts of the world where where there is real user demand because Bitcoin provides a much more compelling, service compared to the alternatives. You know, your places like Nigeria, your places, you know, with there there's real issue with especially financial seizure, but also hyperinflation.
And in those places, I think it's it's, you know, you you can't blame Chivo for all the problems is my point. It's also just that, like, for the most part, people in El Salvador didn't need Bitcoin. They didn't care about Bitcoin. Some people care about it now, and maybe with the the dollar inflation, they might care about it more. But in most parts of El Salvador, there wasn't you know, there are a lot of problems, but financial like, seizure of of financial assets was not not high on the list.
[01:17:53] Unknown:
Well well, what about us, you know, United States citizens? I mean, do we need do we need Bitcoin, and do we need lightning at all? I mean, you could almost, like, say the same thing right there.
[01:18:06] Unknown:
Yeah. That's true. You know, I think what what does need Bitcoin mean? You know, people care about different things. And if you care about something that Bitcoin provides, it's great. You should totally use it. I think my point was more that the average person isn't sitting there considering, you know, who their counterparty risk is when they're using the dollar. And so to them, Bitcoin's not compelling. It's not solving a problem they see in their daily lives. You know, to those of us Bitcoin is in the west, you know, we might see real you know, like, personally, I use lightning to pay for some server bills, not because, like, I don't like the dollar, but actually because it provides a better user experience and it's faster and more reliable than paying with a freaking credit card. And, like, that's actually solving a problem for me that I had, you know.
And and, you know, the to to folks in the US who are worried about all kinds of other issues, you know, Bitcoin is providing something that that is an issue to you. It's a little different if, you you know, someone's telling you to adopt Bitcoin to solve something that is not already an issue in your mind. And and so I I just I care more about places where, like, some people are already thinking about these issues and and the normal average person is already, like, this is a major issue. I need something to solve this issue. Oh, Bitcoin is a solution to this issue rather than this isn't a problem for me. Why are you telling me to use Bitcoin? And then maybe a handful of people are like, oh, yeah. This is something I was thinking about and concerned about.
[01:19:52] Unknown:
I guess another data point, you know, just because Matt brought up, Nigeria, There was that partnership, with M Pesa, that's now going to essentially open up Bitcoin to a lot more users through Bitnub, and I believe that opens it up to, like, 50,000,000 people. And then Cash App, previously, was probably the largest with 44,000,000. And then, just understanding those, I guess, aren't necessarily, like, daily active users or whatever VCs, wanna attribute them to, but there might not be addresses either. It's in a weird, metric they use where, essentially, they have the capability of buying it or doing something with it. And, that says something. Right? Like, I don't know. It's, like, for them or they need it, but someone wants to build a service for that market.
[01:20:50] ODELL:
But they couldn't I mean, they would experience downtime in the case of Barack's 2 bugs, but they wouldn't have they wouldn't lose money in that situation because, I mean, all the all the major players updated really quickly. And they all, like, limit who their counterparties are on Lightning.
[01:21:10] Unknown:
Yeah. And, I mean, if we wanna talk about, you know, positive outcomes here, I don't know if if LND has done it, but certainly one thing we've been thinking about more, to some extent as a result of this, but also just software maturing, is how how we better communicate quickly with people when, when something like this happens or when you people need to upgrade urgently. There's not to my knowledge, I don't know if if LND has, like, a a mailing list you can subscribe to or what. But, like, you know, how do you make sure engineers at Cash App and at Kraken and at River get paged when something like this happened and and and are are told to to move to a computer quickly. Right?
Versus just like, well, you gotta be following us on Twitter and and make sure you see it, which isn't really a a sustainable outcome. And and, you know, some companies have engineers who follow Bitcoin Twitter personalities, but but others don't. And they shouldn't necessarily need to in order to discover, issues that they need to to patch quickly.
[01:22:18] Unknown:
Yeah. I think that's a fair point. I mean, I'm I'm not on Twitter myself. And if that's the only way to hear about these things, then, you know, they they they do have a very extensive email list that they send out every other month to get feedback from people, but haven't heard anything from that about security patches.
[01:22:38] ODELL:
So, Tony, I mean, you're not on Twitter. How did you find out about the bug?
[01:22:42] Unknown:
I get people on Twitter to tell me things. Ben Ben Carm, it's always telling me about l and d bugs.
[01:22:49] Unknown:
So he's he's my source. And then I'm Ben's source, and shout out to Richard Saffer who's, DevOps at Zebedee. He's the one who filed the original issue for the first block parsing bug, on l and d.
[01:23:07] Unknown:
So it's all personal connections. And if you don't know enough, big partners, you're, you're gonna get your money stolen because you're not gonna know the upgrade.
[01:23:14] Unknown:
So you won't discover it as fast as you might have otherwise. But I'm sure if you hung out in the l and d Slack, you are fine.
[01:23:23] ODELL:
So there's wait. There's 2 there's 2 different when we're talking about bugs like this, there's there's basically 2 different classes of users. There's, like, the almost, like, institutionalized user or the corporate user, and then there is that end user. The end user, like, the sovereign node runner, like, we have obviously some issues there in communication. But on the on, like, the corporate side on the corporate side, is there is there something to be said about, like, what bottle pay was I I think Youse released it. I think he called it multiplexer where essentially you're running multiple nodes, of different implementations. Like, Corelightning didn't go down here.
Is there is there a strong argument that the corporate users should essentially just be running multiple lightning implementations at the same time?
[01:24:25] Unknown:
So I don't think use multi I think use multiplexer, supports running multiple LND nodes. So I don't think that's that helps here. I know some corporates have built their own solutions for running, multiple nodes or at least have talked to me about it. I I don't know that they've actually built it fully or rolled it out. I think that's a a valid a valid point to make sure you don't have downtime, but it doesn't protect you from funds loss. And in fact, it it makes your funds loss exposure worse. Right? Because then if any issue, any, implementation has has a bug, you can lose some money.
[01:25:02] Unknown:
At at at the very least, we need we desperately need, like, multi node implementation watchtowers. Like and and I don't think we talked about it here yet, but or maybe we hinted at it. But, you know, l and d nodes can only talk to l and d watchtowers currently. So, you know, you would be affected by this. And if you had your own watch tower running and you didn't update that in time either, then you would be you would you wouldn't be helped at all. Sure. If you outsource your watch tower to, like, a responsible node runner, that is, like, servicing a bunch of users, you you could probably expect that they're at least doing a good service, a good good service of running watchtowers, even if they are on the same node implementation. But, we we desperately need multi node watch towers, the very least. Because these bugs broke the watch towers as well.
Yeah. Well, the l and d watch tower was also broken, and l and d can only talk to l and d watchtowers.
[01:26:01] Unknown:
Yeah. So there's two points here. There is actually a third party watchtower, guy Guy Sergio, who who Spiral has funded for, several years has won. I believe it currently only supports Core Lightning because Core Lightning is the only node implementation that has enough hooks, for it to support. I'm kind of embarrassed that LDK doesn't, but LND certainly doesn't. So you you can use that. That does exist. However, it is, again, useful to note that in order for a watchtower to you know, we mentioned that there's 2 kinds of, ways you can exploit this class of vulnerability, both stealing funds from, by broadcasting an old revoked state, and there, a watchtower will get your money back. But if you steal funds using the HTLC, logic, if you have a a an HTLC that's routed through you and then someone steals funds using that, then, you can't, use a watchtower currently. And in order to use a watchtower, watchtowers would have to expose a lot of you don't have to expose a lot of information about your channels to that watchtower.
Losing, you know, one of the pitch of watchtowers today is that they're they're private, that they don't learn about your payment history, about your, all the details of your node, and you would have to actually expose all that information to a watchtower to enforce HTLC, claims correctly. So it would require a bit of a change to the watchtower model. Currently, I don't think anyone's working on that. But in order to, you know, I guess I guess the moral of the story is is set your HTLC maximum in flight configuration knobs down to a threshold you're okay with losing, which is good for everyone's privacy in the network. So do it anyway.
Make sure you change that value.
[01:27:50] Unknown:
Well, is it is it easy for people to change? That's always the the tough part for me. It's like I I have all these ideas of how to be how to have better privacy on lightning, and then it's like, well, none of these are easy to actually implement. I'm sure that Thank can do it. But yeah.
[01:28:07] Unknown:
Yeah. I I don't know about lnd. I assume at least Corelightning because it's very configurable, has the support for LDK does, if if you're using an LDK node, if you're writing code for an LDK node, you can do it. I would imagine LND does. I don't know for sure. If it doesn't, you should certainly nag them. But you should be able to configure this in your lightning node config or in your channel config to say, like, max in flight lower than the total amount. At least set it less than half. Ideally set it maybe to a quarter of your channel balance, or the maximum you're willing to lose.
[01:28:38] Unknown:
Or or maybe it should just be defaults even on a certain point.
[01:28:42] Unknown:
It it really should be. This is only a more recent discussion that we had in the Oakland lightning dev lightning dev summit that we should default this to below, below half the channel balance at least, if not less. I don't know which nodes, if any, have deployed that default change. I'm not even sure if we got around to it yet. But but if you do have that setting set, then LDK nodes, including Cash App, will slightly prefer to route through you. So maybe you'll actually make some some additional get some additional HTLCs and routing revenue, to pay for the the reduction in capacity that you that you have in the channel.
[01:29:21] Unknown:
You can, actually set these values in your LDconf, I believe, also with their API. So I think, the only one I'm unsure of is eclair, but, I I think everyone allows you to set those values.
[01:29:39] Unknown:
Yeah. But but, like, what Umbro user is, like, going into their their configs and, like, writing writing text into them? You know? That that's the only thing I have to say about that. Like, I I know defaults are very hot topic in Bitcoin, but maybe it's one of those situations where, like, this is just better for the whole network. Let let's just go ahead and do it. But, I'm glad to know that there's there's conversations around it now.
[01:30:12] ODELL:
Yeah. I mean, from that, like, that perspective, like, I still feel like I like, Carollo, I'm I'm kind of curious. Like, so you you think just unequivocally the way Barak proceeded with just executing these bugs by issuing a transaction was not the way of doing it. There was no benefit of doing it in that way rather than going through normal
[01:30:39] Unknown:
channels. Yeah. I I don't think there's any reason to exploit something like this unless, unless it's already been patched, unless people have had, you know, at least a little bit of a chance to upgrade. You know, you you can debate, I think, what the time period after a release with a patch, at what point it makes sense to exploit something like this, after that patch has been shipped. But I don't know what that number is. And I I I think it's definitely not negative. It's definitely not before a patch has been made available or at least made public that it makes sense to to do this kind of exploit.
You know, every exploit is unique. Every every vulnerability, every bug is unique in terms of what the specific outcomes are, what the specific, you know, ways it can be exploited, ways you can potentially mitigate, all that kind of good stuff. So, you know, it depends a lot on the context. It always depends on the context, but I think that's even more reason to make sure you reach out to the vendor before you before you do anything because the vendor you know, some vendors are shitty. I think in Bitcoin, we're pretty lucky. None of the vendors are are really shitty generally. At least to my knowledge, everyone takes these kinds of things seriously and appreciates, vulnerability reports even if it's from somebody they don't like or if it's from someone who's be doing it, you know, rudely or whatever.
Obviously, that that doesn't apply here, but just, you know, in general, vendors can be shitty when you report stuff to them and and some people are are shitty when they do report stuff to you. But I think everyone's pretty pretty chill in the Bitcoin world about this kind of thing. So it always makes sense to to reach out to the vendor first. Get their take. Get make sure you're on the same page in terms of how this can be exploited, what the the consequences of it, what you can do to prevent exploits, taking off, and then take action. And I don't I don't think it ever makes sense to take action before you've at least reached out to the vendor initially and and and flagged it to them.
[01:32:49] ODELL:
So, I mean, from my perspective, it kind of felt like enlightening development world or just like enlightening network land in general. It it kind of felt like it kind of felt
[01:33:05] Unknown:
like,
[01:33:07] ODELL:
assumptions were a bit naive, and a lot of it was glossed over with, like, hashtag reckless or whatnot, but, like, real funds are on the line, and they're all hot wallets. This kind of the way Barack did it to me felt like a shot across the bow, like, we are entering an adversarial environment. Consider the risks. And I think a lot of users, as a result, now have a different feeling. Particularly, node runners have a different, you know, perspective on lightning. They're more aware of the trade offs.
[01:33:42] Unknown:
But I think that that's separate. Right? That that's a separate question. Right? So, like, you know, should is the lightning development world not are lightning node vendors, accurately communicating the risks? I think the answer you you're right. They're not, and that that needs to change. And that's something that's that's frustrated me in a few contexts across the lightning space. And so, you know, I I think Lightning node vendors generally need to do a better job at that. Totally. It's a separate question as to whether or not to exploit something before there's a patch. And then, you know, if you if your goal is to have a similar outcome of, like, making sure it's clear that they, that users see this and feel urgency to upgrade, generating a similar feeling. You could say, you know, exploit it, you know, 8 hours after the patch is released or something. You know? Make sure people have a have a chance to upgrade, you know, even the people who were asleep at the time the patch was released, whatever, and and then do it. You know, I I I don't think that's I don't know that I would decide to to exploit it that quickly, but you you could make an argument for that and that becomes more of a a value judgment that we'd have to debate. And I don't think there's, like, a correct answer. There's never a correct answer or you know?
So but but doing it beforehand, I think, is a little different. And and certainly, you know, another part of responsible something that's not a part of responsible disclosure is how it gets communicated. Right? So, the vendor always vendors always wanna choose how something gets communicated, but that's ultimately not just their choice. That's also the reporter's choice. Right? So if the reporter is like, you know, I think this is a huge deal and you're not doing a good job of communicating it, the reporter is totally free to go on Twitter and say, like, vendor is moron.
Screw this vendor. They're not telling you the the real story, and this is a huge deal, and you need to upgrade ASAP and, like, your funds are at risk and look, here's why. That's totally legitimate. That's that's not you know, how an issue gets communicated is not a part of responsible disclosure. Your your duty is to the users to make sure you minimize the risk that users lose funds, But there's a lot that goes into that and, you know, there's no correct answer. Again, everything's unique. But I I think it it it I I think it's pretty clear that exploiting this before there's a patch does not minimize user risk. I think it it it's pretty clear that there are many different ways to go about this that might get similar responses in every way that would have minimized the risk a little more by just waiting to exploit it until after there was a path.
[01:36:33] Unknown:
What what about the scenario? So there there are multiple vendors here that that had the issue. Right? Now, of course, Brock was honing on a specific one, and and there's a lot more that that we found out. How from the vendor approach, what would have been what would have been, like, a reasonable thing to do? Because I I could be wrong, but I think I remember years back, Bitcoin Core had this scenario where there was a responsible disclosure to Bitcoin Core from from some shitcoin Bitcoin side chain thing. And Bitcoin Core fixed it and then released, you know, released it, but the other chains, were vulnerable to it. Now you could argue, you know, fuck those shit coins or whatever. But when there's an issue that could be affecting everyone, but only one party knows about it, is is that a problem at all?
[01:37:22] Unknown:
Yeah. Who who's who do you owe a duty of care to is is a huge question that has no answer. There you're right. There were a lot of discussions around that bug, around do Bitcoin Core developers have a duty of care to every random, fork of Bitcoin Core building any random coin. You know, I personally think they don't. That's just my view. You know, I don't think I think Bitcoin Core developers have a duty of care to Bitcoin users, and not a legal duty of care, mind you, but but, you know, a moral duty of care and and not to anyone else. And that, you know, insofar as you can do anything to to avoid other people getting screwed, you should.
But, certainly, you don't have a duty of care, where it might impact where it might negatively impact Bitcoin users. And that includes, you know, if you tell with any vulnerability the, you know, one thing that that you have to maximize, you have to make sure you don't tell any more people that need to know. Right? That that kind of information should always be need to know because, you know, any additional people you tell, they might tell their friend and their friend might exploit it or they might have it up on their screen at a conference and somebody might read it and exploit it or or what have you. And especially when you're talking about other coins, you know, that that can be a very adversarial relationship. So I do know, you know, when it was disclosed by Bitcoin Core or maybe kind of an hour before or something like that. You know, once once it was clear that Bitcoin was a little safer and then when it was going to go public anyway, a number of people were notified, you know, kind of immediately prior or maybe an hour before, to make sure they had at least a little bit of a chance to to catch up and and fix their own stuff.
I don't recall whether any other coins were. I think maybe, like, b cash was or something or, like, one coin or 2 where there was, like, you know, at least a little bit larger market cap, at least a little bit larger user base, and some of the developers had had a history of being reasonable, even if their community maybe wasn't, their their developers were. But but no. I mean, I I don't think Bitcoin Core developers have a duty of care to other clients. But yeah.
[01:39:48] Unknown:
Do CLN developers have any duty to to l and d developers?
[01:39:54] Unknown:
I actually was just about to bring that up. Matt brought up communications of, disclosures. Brock, why did you choose to write what you did in the off return?
[01:40:11] ODELL:
You still here, Brock?
[01:40:13] Unknown:
Oh, yeah. I'm here. Yeah. It does for trolling. I just I just wanted to troll. I I guess yeah. I have a troll personality. I just you know, I'm making this transaction and why not just put something in there? I just yeah. I just put some message.
[01:40:29] ODELL:
To to all the listeners tell the listeners you put in the op return, you'll run Core Lightning and be happy.
[01:40:36] Unknown:
Yep. It was, yeah, it was referenced to, World Economic Forum. Yeah. That was for fun and for trolling. Yeah. Yeah. I I'm not sponsored the message was for trolling. I have no problems whatsoever with Lining Labs. I love folks working at Lining Labs. In fact, in favor of multiple implementations, whether it's LND and CLN or other Bitcoin query implement other Bitcoin implementations, I don't have any problem. I I'm just trolling.
[01:41:21] Unknown:
Well, Tony, there is your answer. We have a duty of care, but that was just trolling. We have a duty of care to each other. You know? As anyone that's part of the Bolt compliance spec, we work together as much as possible. I consider Evan a good friend, Lalu as well, Con, and, Sueb. So, you know, we all, like, are in this community together, and I think I also am a bit guilty of this stirring the pot sometimes when, you know, tempers get flared up about decisions based on upgrades or what priorities should be. But at the end of the day, we're in this together, and I really value everyone contributing to the Bolt specs.
[01:42:15] ODELL:
I just wanted to say, you know, as a as a non dev, who's been running lightning, specifically l and d for 3 years, extremely well connected node, nearly 200 channels. I had more than that at one point. I have no idea who any of my counterparties are. Part of the reason why I was running it was to see if I would lose money, and I'm pretty sure I haven't lost money yet. So, yeah, cheers to that. Thank thank you for your service.
[01:42:52] Unknown:
I mean, Matt, you kinda too.
[01:42:55] Unknown:
Oh, yeah. Go ahead. Matt, you kinda brought it up earlier. We're saying, like, you know, we're not really in an adverse sale environment at all. Barak's is, like, quote, unquote attack is, like, signaling that. But, like, I mean, we definitely aren't. I I, like, when I went to Nashville, like, earlier this year, my lightning node went down on the plane over, and that was, you know, I never I don't lose any money either. So, like, people could have clearly stolen from me and didn't. You know, it's it's definitely not a non air Australian environment and I don't know. Hopefully hopefully is is bad, but, like, we need to get there. Like, if we're in there, that means it's like lightning is a lot more legit. It's not just like, you know, attacks or trolls anymore. It's like actually, like, people trying to steal money and potentially profiting from it.
But I don't know if we're there yet, but it'd be cool to actually see real feel like you know, because, like, in the DeFi space, something like this is like a full time job of people Bitcoin projects. So it'd be, you know, scary but also cool to see people doing the same thing in light.
[01:44:05] Unknown:
Yeah. I think we'll get there eventually, but I'm not in any rush.
[01:44:09] ODELL:
Thanks for the conversation, Paul. I I actually have to dip. I've gotta go to a dinner. Which is cool. Yeah. I was the Matt, I was about to wrap up. Thank you for joining us. You wanna just give us your final thoughts real quick before you wrap?
[01:44:23] Unknown:
You know, I don't I don't know if I had any specific final thoughts. No. These issues are tricky. There's there's a lot that goes into them. There's there's usually no no clear answer. There's there's better things. There's worse things. But I just, you know, keep keep in mind that that as humans, we wanna we wanna help each other, at least as as non you know, we we we should just, you know, do do what you think is right for the for the people, not for the vendor or anybody else. But thanks for having me on. Out. Thanks, Matt. Appreciate you. Evan, final thoughts.
[01:44:54] Unknown:
Yeah. So, you know, I think there's a great discussion today. Barak, I think, you got your taste of the dark side. I hope moving forward, you consider coming back to the light. I I know our community has a lot of values of, like, anarchism, but that still requires cooperation for, you know, the benefit of the community. And, you know, ultimately, I think you're a really talented guy. Just, we hate for you to keep going down on this path and, there's positives to this, but, you know, it's not something I would like to see happen again.
[01:45:33] ODELL:
Thanks, Evan. Vivek, final thoughts.
[01:45:40] Unknown:
Thanks for having us on to even discuss this, man. I know it's kind of a touchy subject. Wanted to thank everyone that actually joined. Felt it was great that, we had Evan and, Matt Corallo here, especially. Ben, fix your Internet, and thank you all freaks for listening to us. Thanks, Vivek. I hope,
[01:46:01] ODELL:
I really want dispatch to be a place where people are comfortable having open discussion, and I really like wrangling people for these these round tables. Ben, final thoughts.
[01:46:12] Unknown:
Thought, fuck Google Fiber, and, you know, that I'll I'll stick with that.
[01:46:20] ODELL:
Thanks, Ben. Tony, final thoughts.
[01:46:24] Unknown:
Great discussion, guys. Yeah. I I think I think all, you know, all things like this make Bitcoin better in the end. So, Barak, if you wanna coordinate on more dark side things, let me know.
[01:46:40] ODELL:
Thanks, Tony. Barak, final thoughts.
[01:46:42] Unknown:
Yeah. I I I personally see Lightning as a better software, and it has a long way to go. I personally think 5, 5,000 Bitcoin is a joke number. I still see lightning as a playground. And, yeah, I love breaking things and I besides, I believe, attacks help Bitcoin, helps, hardening the the software, the infrastructure for long term stability. I'm not saying this in defense of my actions. I'm just saying this as thought as my as as my thought thought consideration. My only motive was for fun, and I guess, I'll I'll dig more into codebase and see if there are any other exploits.
[01:47:24] ODELL:
Thanks, Barack. I appreciate you. You're always welcome on dispatch, and I I hope you and Tony continue to, attack me. I do appreciate it. To all the freaks, I wanna thank you for joining us for this conversation, particularly the freaks who joined us in the live chat. I know it's a Friday evening. I know it's a crazy week, but you guys make it special. Once again, dispatch has no ads or sponsors. It's funded purely by you guys. There's a lot of value for value haters out there, including some people I've recorded podcasts with in the past. I think we can prove them wrong. I think most people are just half in, half out and not actually trying to make value for value work. So I do appreciate you guys seeing seeing the stats flow and seeing dispatch consistently on the top of the fountain leaderboards really makes it all worth it. I appreciate you all. Once again, it is a bear market. If you can't contribute stats, sharing with your friends and family helps, subscribing in your favorite podcast app helps, Subscribing on YouTube, Twitch. I mean, don't use YouTube. But if you are gonna use YouTube, subscribe to the channel.
It it is appreciated. Every little bit helps, and, it keeps us going forward. I wanna say to all the freaks, I know it's been a long week. It's mostly Shitcoiners playing Shitcoin games. Fiat Maximal is playing Fiat games. Bitcoin remains unaffected. I love you all. We're gonna keep moving forward here. We can't be stopped if we if if we work together. So I appreciate everybody. Have a great weekend. Stay humble, stack sets.
Happy Bitcoin Friday, freaks. It's your boy, Odell, here for another Citadel Dispatch. Citadel Dispatch is a interactive live show about Bitcoin and Freedom Tech. We have no ads, no sponsors. It is only possible thanks to our great listeners who continue to support the podcast, whether that's with stats through podcasting 2.0. Podcasting 2.0 allows you to just download an app. Big ones are fountain podcast, Breezewallet, podverse.fm, echoln.com, and others. You download the app. You load it up with stats. You search Siddle Dispatch. You press subscribe. Choose how many sats per minute you think this show is worth, and then those sats automatically, get streamed directly to my lightning node. It is extremely cool seeing all the sats come in. You can also support the pod, through podcasting 2.0 through something called boostograms.
Boostagrams are a single payment that include a comment or message with, the payment. I read the top 4 Boostagrams from the last episode. Every week, it's really great to see, your comments and your support. Before I read those Boostagrams, you can also support the show by going to sidildispatch.com/contribute. I have a BTC pay server there. You can support that using that either via via Onchain or Lightning. And I also have my BIP 47 payment code there, which you can support via Samurais or Sparrow wallets, and, that's attached to the associated pay name, Odell. Very easy to remember. I know it's a bear market. You can also support the show by simply sharing it with friends and family.
Citadel dispatch is available on all major platforms, every podcast app, Twitch, Twitter, YouTube, Rumble, Bitcoin TV. You just search Citadel Dispatch, press subscribe, reviews help as well. Yeah. So thank you for your support, Freaks. I really do appreciate it. The show would not be possible without it. I also wanna thank the Freaks who joined us in the live chat. You're able to join the live chat either via YouTube or Twitch or Twitter or via Matrix. We have a Matrix chat that runs 247, 365. Lot of great conversation there on and off the show. You can find links to all of these things at citadel dispatch.com.
So with all that said, we have a great panel lined up. It's been about a week and a half in the making. We've been trying to get it out there ASAP because it's a very important topic. Ironically enough, in classic Bitcoin fashion, a bunch of things have happened over the last week as many of you are known. And, you guys might forget that that we had 2 major, lightning bugs over the last month or so, maybe, like, last 2 months, with one of them happening last week where we all had to update our l and d notes. So this topic will be focused on that plus lightning development going forward and just the current state of lightning development. Before I introduce our guests, I just wanna read the boostograms, the top 4 boostograms from last week. Once again, thank you, Freak, for your support.
We have at Humble Pleb, gave 69 420. Nice. 69 420 sats. I don't know. It looks like he tried to put some kinda emoji there that I can't read on Android, a bunch of times. Appreciate you, Freak. We have Eric 99 with 50,000 sats saying, stay humble, stack sats. Great advice. We have Chad Farrell with 50,000 sats saying, thanks for all that you do, Matt. And we have 8, with 7,778 saying, I've been listening to a new privacy podcast, Watchmen Privacy. It could be interesting for you to appear on this pod or vice versa. He has a strong Bitcoin focus in his podcast when speaking on cryptocurrencies, which is rare for privacy gurus. His past guests include NVK, LOP, Zelco, s 21, Seth for privacy, Bitcoin q and a, and Eco. Wow. I'll go check that out. It was not on my radar.
Appreciate you, Freaks. And, also, we have Vic in here, honorary mention with 5,000 sats. Right or die freak. Appreciate you, Veek. So thank you, Freaks, for supporting the show. We have Tony jumping in and out trying to troubleshoot. Tony, can you hear me? We cannot hear you, Tony. So I'm gonna introduce our guests. We have a long time listener and often guest of the podcast, Ben Carmen. How's it going, Ben? Can't hear you, Ben. Classic. We have Barack who I had the pleasure of meeting in Miami on the open source stage, that I helped coordinate.
Barak was the one who discovered the last two lightning bugs. How's it going, Barak?
[00:05:51] Unknown:
Yo. Hey. I can perfectly hear you. Hopefully, you guys can hear me too.
[00:05:55] ODELL:
Yes, sir. We can hear you. You rejoined.
[00:05:58] Unknown:
Yo. Hey. So, yeah, my name is. So I'm a Bitcoin dev. I mostly do some stuff on Liquid, and recently discovering Lightning Space. So I'm also known as the LND slayer, and I'm this bug eating this burger eating guy on Twitter. And happy to be here, guys, and add some you know, maybe give some details on what have happened, over the last few weeks and, you know, give some inputs and, you know, I'm not here to tell, you know, to tell whether my actions were good or bad. I think it's up to community to tell and, but I I can't tell you guys, what did happen and what my motive was.
So, happy to be here. Hopefully, this is gonna be a a friendly conversation.
[00:06:43] ODELL:
Thank you for joining, Barak. We got Evan Kaludis, also rider die freak, common common guest on the podcast. How's it going, Evan?
[00:06:53] Unknown:
Hey. Great, Matt. How are you doing today? Doing doing great. Doing great. Yeah. Happy to be back on the show. Wish we were talking about a more positive topic,
[00:07:04] ODELL:
But, nonetheless, I think we're gonna have some, you know, great, very important discussion today. Well, compared to all the other podcast topics of insolvent exchanges and people losing their life savings, I feel like focusing on lightning development is a very positive topic.
[00:07:20] Unknown:
You know, when it's not getting attacked, unnecessarily,
[00:07:24] ODELL:
but, we'll get to that soon. Fair enough. Thank you, Evan. We have Matt Corallo. He's been on the podcast before and also has joined me with Marty on TFTC. How's it going, Matt?
[00:07:37] Unknown:
Pretty good. Pretty good. Thanks for allowing me on.
[00:07:40] ODELL:
And, Matt is, very, what's your what's your role at LDK? What would I call your role at LDK?
[00:07:50] Unknown:
I don't know. I work on it like everybody else.
[00:07:53] ODELL:
Matt works on LDK like everybody else. LDK is one of, one of the top lightning implementations. We have, Vivek here, also repeat guest on the podcast. How's it going, Vivek?
[00:08:08] Unknown:
It is going well. Just this and, yeah, explore is, and and what
[00:08:23] ODELL:
Yeah. So Vivek's Internet connection is horrible. It's probably because he stopped paying it because of the bear market. I don't know how we can fix that, Vivek. We have who did I not introduce? Tony. Can we hear you now? Can you? Yeah. We can. Long time since this is a podcast. How's it going?
[00:08:45] Unknown:
I'm good. I'm good. I'm excited to be here. I'm trying to refresh my memory on what happened, but other than that, I'm happy to be here and talk about this. So, Tony I mean, if if you have been listening to dispatch, you know what Tony's been up to. He's been working on basically
[00:09:00] ODELL:
attacking lightning, and he has this new tool called Ellen's Ploit, which makes it easier to attack lightning. So we will be talking all about that. And then I think Carmen didn't wasn't able to speak last time I introduced Carmen. Carmen, can you hear me? Carmen, can you speak? Okay. Well, we still cannot hear Carmen.
[00:09:28] Unknown:
Test. Test.
[00:09:29] ODELL:
We can hear Vivek.
[00:09:30] Unknown:
Cool.
[00:09:32] Unknown:
Okay. So we have Ben's, issue is that he's on Linux, and sound never works on Linux.
[00:09:39] ODELL:
I mean, I ran the show I'm on Linux. I ran the show from Linux for, like, 2 years. Ben is gonna switch switch computers. Okay, Ben. Switch computers and come back. Let's get started. I wanna thank the Freaks for for, this is a this is a big group, but it's a it's a great group, and I'm happy to have them here. Okay. And, so just I appreciate the freaks for sticking with us during connection issues and whatnot, considering we have so many people here. I guess I guess where to start is, Barack, you want to give us a basic overview of both bugs or Yeah. Or at least how you how you created what led to both bugs?
[00:10:33] Unknown:
Yeah. Sure. Yeah. I can give it a bit give a bit of a history what happened. So, obviously, it was around a month ago or something. Yeah. I did this first large multi tap script transaction on Mainnet. I did it on Testnet first, though, and then it was unintentionally broke LND due to a pricing issue in BTCD. And, I I, obviously, it made a lot of trouble at the time. I guess I feel sorry about that. And then in the following ways, I had this this this, this urge, this inner feeling to my gut telling me to break it more. And, like, like, I did once, when, or twice. I'm not sure, like, if that sounds weird, but, I guess, yeah, I'm discovering the dark side in me.
Yeah. Maybe I'm a bad actor. Who knows? But, yeah, I just I just I just wanted to break more, and I did it only for fun. Not well, for nothing. Just for fun. And then I dig into the consensus code of BTCD. 1st and foremost, I'm not a software engineer, although I did some programming in the past. I I think I'm not qualified for the title, but, I, I'm a Bitcoin dev. That's what I can tell. And I dig into consensus code and for weeks. Right? And then I couldn't really find find any exploit, like, any conflict, consensus conflict between BTCD and core.
And from what I can tell, BTCD is actually a very well, written implementation. It's it it it's it's it complies with the Bitcoin consensus rules bit by bits. I I couldn't really find any exploit. I need to dig more into it, to find more, but, yeah, I'm not sure if I can find a
[00:12:34] ODELL:
third. Also, the yeah. So the first bug the first bug, essentially, you used functionality that was enabled by Taproot that allowed you to do a massive multisig transaction. Right?
[00:12:48] Unknown:
Yeah. That's true.
[00:12:50] Unknown:
And
[00:12:51] ODELL:
Tap with yes. And then and so LND had some BTCD libraries in it. BTCD is a Bitcoin implementation that is written in Go, and those libraries made it so l and d nodes can no longer essentially, were no longer synced to the Bitcoin chain. They can no longer see what was going on in the Bitcoin chain. Is that a good overview?
[00:13:17] Unknown:
Yeah. That's correct. So layer 2's obvious landing is the only layer 2 in my book, and layer 2's, inherit the security properties of layer 1, and it depends on, it relies I mean, BTC in Lendly, in particular, it relies on the BTCD, which is an alternative implementation of Bitcoin. For the record, I'm not against alternative implementations. In fact, I'm in favor. I think diversity is overall good for Bitcoin, whether it's Lightning implementations or Bitcoin. Think more diversity is good for the long term, although it may be, scary for the short term.
So yeah. So the the the first part was unintentional. It was due to a parsing issue. We had it had a cons it had a parsing, check from SegWit SegWit scripts I mean, the scripts are limited to, 10 k, 10 kilobytes. Sorry. 10 kilobytes. And, I I I managed to make a large I did a large multi transaction, which consumed, like, 1 out of 8 block space or something. And and, yeah, it's it's it's couldn't pass this pre check. And, yeah, it's unintentionally broke LND the first time. And the second time, I I tried to look a second look for a second's conflicts, and I couldn't find any in the consensus the in the actual interrupt repo code, the TX script code.
And from what I can tell, yeah, I think b two c d is is a it's a good implementation. Then I when I couldn't find, any conflict, then I come back to then I go back to the wire code, the the the the TX parsing code, and I realized, okay, there is another conflict in the literally the above line. The the line above the previous pipeline, which was also a Taproot related issue. So in Bitcoin, we have a consensus rule in place that limits number of SEC elements in the row of a thousand. Right? You cannot exceed that. Although I'm not sure why it had why this limit was 500 k.
Not sure about that, but it's we, you know, we have another rule in template that says, okay. If you encounter an op success x op occurred in scripts, it's past the validation regardless. So it these rule does not does do no longer apply. So I I then I when I realized, I tried to first create a template output, a funding and ops success x, literally, script path. And maybe it's a bit of technical, but I then I tweak this output with my own inner key. I thought I did. But it apparently, I tweaked this output with an unspendable inner key. And then I found this, like, 3 weeks ago or something.
I discovered the bug myself, and and found this address, and then I reach out. I I build this transaction by hand and placing 500, 500,000 stake elements. And he said, yeah. He didn't question it. He said just pay extra fees for the manual work and then email us. And then yeah. I emailed them my rolled transaction. It was like it was paying overpaying, like, 3 x. And yeah. And then, obviously, it was invalid because my my my control block was incorrect because I thought I tweeted what my inner keypad. Then, you know, then they they said, okay. We need to patch. We couldn't broadcast it, whatever.
And then, like, on November 1st, 12 days ago, I was in need of some money, and then I I wanted to revert the money back to myself. But then I realized, okay, my transactions was wasn't correct. It was tweaked with was tweaked with an in unspendable inner key. Then, okay, let's, then let's let's make some chaos. Then I fix the evolved transaction and send back to f two pool in then in within an hour or something. Yeah. They it ended up in a Bitcoin block. So it was a bit of a mistake, but not not really. I think yeah. I did it by I did it with my. I I did it intentionally.
Then, yeah, then you all know the rest.
[00:17:48] ODELL:
So both were triggered by different on chain transactions. Both have the same end result that l and d nodes fell out of sync. Yeah. Both were based both of the issues stemmed from the use of the BTCD library. I it's interesting that you say you're for I one of the interesting things about this situation is depending on your bias, I've noticed people using it to argue against multiple implementations and for multiple implementations, which is just fun. It's just classic Bitcoin. So we have, well, first of all, Ben, can you communicate with us now? No. You can't.
Wow. They're trying to silence him. Okay. Well, we'll get back to Ben. We have Tony here. Tony, you wanna talk about how, how how someone could have used if if Barack was malicious, you did a you did a guide, basically, in reg test, which I guess so unlike Bitcoin test net one of the Bitcoin test nets, for how someone could exploit this attack to steal funds. You wanna talk about that real quick? Yeah.
[00:19:09] Unknown:
And and props props to Ben. He he's pretty much the one that wrote all these exploits. So Alan's point, I kinda made it you know, I've been doing these, like, you you know, lightning abusing projects, I guess, to to to say kind of being, along the same lines of, like, a bad actor in a way. But, so I decided, that it was time to, like, finally do Alan's point and write something that, like, actually tried to, you know, exploit some of these things that we keep talking about and that keep not getting fixed. And it was it it was with the intention of doing it at the Atlanta tab comp. There was, was in a workshop there, but I had you know, it takes a while to build, you know, some of the things that I've been building on, and I didn't have any exploits in there by the time. It was the week of the conference.
And then, week of the conference comes on Sunday, and and this this this bug breaks L and D. And, of course, like, I was like, Ben texted me and said, okay. Can we reproduce this, then put it into Ellen's point? And I was like, yeah. Let's let's do it. So we we did that. And over the course of that week, I was like, okay. Well, how do we actually, like, exploit this? Like, I got a reproducing that works on regs test. Of course, you can't reproduce it on main net, like, once it's already, you know, once it's already out there and being fixed. It just takes that one transaction to, like, break it. But, essentially, I broadcast you you basically if if you're in the same block as this transaction that, like, can't be, you know, one of Barak's transactions, You know, the node can't process it.
And if you're not running LND and bit, with with Bitcoin 0 mq, then then LND actually doesn't process any blocks after that either. So, which is something I found out by actually reproducing this, and and figuring out that, it wasn't just that block that it wasn't processing. If you didn't run-in 0MQ mode, it wouldn't it it thought there was a chain split, and it wouldn't process any other blocks beyond that. But, essentially, you know, I just went ahead and just put it in the same block in my reg test scenario. And I have a whole guide on this. It's, you can go to Bytes journey.com. It's one of the top top blogs.
But, essentially, after that, after you after your channel peer can't process the block, the block with the with with the transaction and at that old channel state, it doesn't know how to do the justice transaction. So in Lightning, if if someone were to try to cheat you, your node this is why nodes has to be, you know, pretty much always online or have a watchtower. But, you know, there's problems with watchtowers too. But if if you're not online and you don't see that transaction, you can't broadcast justice transactions, which basically take all of the other person's funds even if, you know, even if you weren't rightfully owed all of them. You do get all of the cheaters funds, that they have locked up in that channel. So if they're not online, they don't get to see that, and then I demonstrate. It was actually kinda cool too because I got to demonstrate, like, how an LND node you know, there's part of my tutorial walks through, like, okay. Here's a a patched LND node, and here's a non patched LND node. Let's, like, see what each of them do under this scenario. And the, you know, the patched one, you know, says, you know, stuff like, oh, your your peer is being fishy in the logs, and it's, like, broadcasting a transaction.
But the other one doesn't. It just says, oh, there's a chain reorg, and it's none the wiser. And then eventually, you might have enough blocks, and and you should get you know, you should be able to redeem those funds on chain. But if the other peer patches in time, even if we did even if someone did exploit this in the same block as as Barak's, All it takes is you updating your l and d note in time. Now there there's some questions there on, like, how soon you need to actually update it under all the scenarios. But if you do update in time, you do get to submit that justice transaction. But if you don't, then then you're kinda shit out of luck, and there's nothing once once enough blocks pass, you don't you don't get those funds back.
[00:23:25] Unknown:
It it may be useful to to point out that, in order to exploit this bug, you don't have to take that risk of broadcasting a a stale state where someone can steal all of your funds. But in fact, you can do it using HTLCs as well. So you can send an HTLC through the target node through the victim and then force close. And that victim, if you claim that HTLC on chain, the victim won't learn the preimage from that HTLC or at least in this class of bugs in general, the victim won't learn that preimage from that HTLC. And then after the HTLC times out, you can just take funds.
So that not only allows you to do it, allows you to exploit this this class of bugs risk free, but it also allows you to exploit this class of bugs, without, what was the point I was gonna make?
[00:24:16] ODELL:
And Without exposing yourself to the justice transaction?
[00:24:20] Unknown:
Oh, no. Sorry. I was gonna point, and and also in a much shorter time horizon. So that that's where you see the difference between 40 blocks, which is the default in most lightning nodes for the time that you have to claim an HDLC that's been routed through you versus, I think the default for doing justice transactions and broadcasting in old state is usually, like, 144. Some nodes set it to a full week. But when you're talking about HTLCs, you can exploit this bug much faster, and the victim has a much shorter time around time, to turn around where they need to upgrade their node and and catch it in time.
[00:24:58] ODELL:
And what was I was looking into this. What was the reason again, Matt, for why, it defaults to a low value? Like, people routing through you want that value to be lower in a normal situation. Right?
[00:25:10] Unknown:
Yeah. Yeah. So that that's basically the stuckness. So, like, if if you have an HTLC that gets stuck, that's how that's the total number of these, you know, it's normally 40 blocks per hop. So if you send a payment that's, you know, 3 hops, then it's 3 times 40, a 120 blocks that you have to wait before you're sure that that payment's gonna come back and get unstuck. Luckily, stuck payments aren't as big a deal as they used to be. We don't see them as much, but a node can always make a payment, get stuck for that amount of time. So if you're sending a payment, you care about that number being really low. Now the lightning protocol, and I don't know what settings different nodes expose, allows someone, opening a channel to say, you know, I only want to expose myself to up to this amount of funds in HTLCs at any given time, and that would allow you to limit your exposure to these kinds of attacks that are exploiting against HTLCs rather than exploiting against, the the the stale closing or the the stale commitment transaction.
But I think, most people don't bother sending that. Most people set that to, the full channel value. You should also not do this because it's bad for everyone else's privacy. And in fact, LDK, and thus Cash App use, prefer to route over channels or very slightly prefer to route over channels that do not set their HTLC max to the total value of the or the their total HTLC max in flight to the value of the channel, because it's actually better for the sender's privacy, in order for you to do that.
[00:26:52] ODELL:
Thanks, Matt. While we have you, LDK was also affected but only by the second bug. Right?
[00:26:59] Unknown:
For old versions? Yeah. So, I mean, the the this class of issue has bitten a number of, various, Bitcoin library deserialization libraries. Right? This this this kind of you know, when you're deserializing untrusted input, you have to make sure you're not gonna over allocate memory and crash yourself by by allocating gigabytes. You know, there there are better ways to design this code. There are worse ways to design this code, but a number of implementations, you know, BDCV has had more of these class of bugs than just these two bugs. You know, they've they've had some of these classes of bugs since before l and d. But yeah. So Rust so LDK uses Rust Bitcoin under the hood. Rust Bitcoin had a similar bug, that we caught internally, so we didn't we actually caught it, someone while one of the developers working on it was reading the code, not when it went to chain. And we fixed it at that time, by rewriting a bunch of the deserialization pipeline to just be more robust against this whole class of issues.
But old versions did have that issue, and so some people some users were using old versions of Rust Bitcoin. I don't know that any LDK LK users were using old versions of Rust Bitcoin, but, you saw some people running mempool and and running, Elect RS. We're using old versions of Rust Bitcoin, which did get bit by this bug when the second transaction was broadcasted.
[00:28:23] Unknown:
Right. Yeah. Actually, I I do have one project that's still on an old version of LDK, and I I found that out yesterday when, my hidden lighting network project, the project that does probes, for private channels. I launched it yesterday because Ben Ben, Ben had his first successful vortex channel open. So I wanted to see if I could figure out, which output was his that he opened the channel with because he didn't use sort channel ID aliases. So I launched up that LDK project, and then it's it's fetching the blocks, and then it crashes. So, but that's an unmaintained, you know, kind of like a throwaway project at this point, but that's,
[00:29:03] ODELL:
that that it was kind of funny that came back full circle on that. So, Tony, in either situation, an attacker had to essentially be set up as your channel peer before the transaction goes out. Correct? Like, they have to have a channel with you.
[00:29:17] Unknown:
Well, I think I think how Matt kinda described it earlier, I I think you can even just route through any node that has a problem, right, with the HDLs. I I was a little fuzzy on how you could exploit it under the HTLC path. But, Matt, for at least the way I have implemented the exploit, it does require the the the you'd have a channel with the peer. Well, call us Corallo and Odell.
[00:29:42] ODELL:
Yeah. Corallo. So is that correct? Like, you wouldn't to use the HTLC attack vector, you could just if you just had a channel open with someone I have a channel open, you could potentially exploit me on that?
[00:29:58] Unknown:
You know, I don't it it a lot depends on exact details of the bug and what what does or doesn't happen in the node. I think it it did. So in the general like, I could imagine in going both ways. I don't know the details of lnd and how it might have have operated in this case. I think for LDK, that might have been the case, if you were using a a really old version of LDK. But, you know, in general, that is a possible outcome depending on some of the the nuances how the note is implemented.
[00:30:30] ODELL:
Got it. Ben, are you here with us now?
[00:30:34] Unknown:
You hear me
[00:30:36] ODELL:
now? Yes. Yeah. We can. Alright. Do you wanna introduce yourself?
[00:30:41] Unknown:
A half hour late,
[00:30:43] Unknown:
but, I've been the car man.
[00:30:45] Unknown:
My company, but did kind of open source stuff on. And work with the.
[00:30:56] ODELL:
Awesome. Well, it's a pleasure to have you, and I'm glad you can finally speak. I guess, Kaludis, what are your thoughts here as the lead maintainer of Zeus, the number one way people interact with their node at home during all of this?
[00:31:12] Unknown:
So, when this second exploit dropped, I was sort of like a double take, like, on Twitter and in, like, a few chats that I'm in, including, like, you know, the Zeus Telegram. I'm like, what is everyone talking about? You know, this is the exploit from, like, we're still patching this up. And then it hit me that, you know, a second exploit had dropped. And, you know, I wasn't super surprised, but it was a huge huge inconvenience, I feel, for a whole lot of different parties. You know, from, you know, the BTCD and LND guys who had to patch up their code, to, the maintainers of, like, node projects and, you know, Lightning wallets, who had to update their releases, individual users who were asking, hey. What's going on? Why do I have to update my note again?
You know, it it really shook a lot of things up and, you know, ate up a lot of people's time.
[00:32:21] Unknown:
Matt, I wanted to add that Rust Elements was also affected, but the attack is obviously harder to do, because of Liquid's functionary model instead of, reaching out of band to a minor. Yeah. You wanna go into that?
[00:32:36] ODELL:
Like, what was the what was the impact on liquid?
[00:32:39] Unknown:
Well, it's it's the same bug, and it just needs to be patched. But Rust Elements also had it, and I wanted to add that because, you know, we said LDK had it, and LDK is using Rust Bitcoin and Matt mentioned for the older versions. And Greenlight also for its watch tower, was using rustic wood, so it also was impacted. But, I mean, these are remedied now, and, I guess we're good. But, just like Evan said, we were also a bit inconvenienced in trying to scramble to figure out, what exactly needs to be done. It wasn't just an LND bug.
[00:33:17] Unknown:
Yeah. It was, like, a general block parsing error that, you know, a lot of people had looked over in implementing a Taproot, it seems like. And and then
[00:33:26] Unknown:
It it wasn't a Taproot thing. No. It was a a segwit thing, a witness witness side. Yeah. I think it looks like and it just made it easier to to create that that large webinar.
[00:33:40] Unknown:
Yeah. That's what Barack was mentioning earlier, someone, where they're wondering why the original size limit. And I believe Laulu's explanation was, because when he originally added the SegWit code a long time ago for parsing, like, in v zero. That's what it was.
[00:33:59] Unknown:
Gotcha.
[00:34:01] ODELL:
But so what does that mean for liquid users at the time?
[00:34:06] Unknown:
Liquid users, everything was fine, but, because of the multisig and other things, you know, all of that needs to be updated, whether it's the functionaries or the on chain multisig. Yeah. And there was, like, also a scramble, by a certain amount of blocks, things that needed to be updated. So, yeah, I think I think we, Blockstream formally released a statement. So I can, post it in the elements chat, but I'll leave the exact details to that. But similar issue, witness size, parsing.
[00:34:46] Unknown:
Did it halt liquid, or is it just, like, the peg in, peg out stuff got broken?
[00:34:51] Unknown:
It it didn't halt liquid, but, yeah, it's the peg in, peg out. Exactly.
[00:34:57] Unknown:
What so the connection to the chain?
[00:35:00] Unknown:
Yeah. I saw a side swap had some issues. I didn't really see any other people complain too much. I think Barack would also know a lot about, the impact to Liquid because he's probably the most knowledgeable in Liquid script.
[00:35:17] ODELL:
Yeah. And his project bit matrix is runs on Liquid, so he kind of Barak, did you did you bork your own project when you sent out the transaction?
[00:35:26] Unknown:
No. Bitmetrics, 4 g, and, you know, other that like, TDEC are not they they were not affected. The I mean, the the liquid network functioned well. I think, yeah, it only apparently, it only affected Pagan and Paycard functionality, and, obviously, Rust Bitcoin and few other infrastructure projects in the space. And in fact, I mean, I wasn't I was only targeting BTCD and LND. I wasn't aware this would cost I mean, this could have caused, any other issues.
[00:35:58] Unknown:
So we got caught up in the lateral damage.
[00:36:01] Unknown:
So Yeah.
[00:36:05] ODELL:
Do you wanna talk about real quickly, the tweet you sent out when you when you sent out that transaction? What was it? In order to find the light, you need to touch the darkness?
[00:36:16] Unknown:
Yeah. That was basically a joke maybe, but I think it has also a truth in it. Right? Obviously, I did it for fun. I guess, yeah, I love making cows. I guess I crave for cows maybe. Yeah. It makes me feel good. I guess, that's you know, good or bad, that was the most meaningful thing I did in my life, regardless the outcome. But, yeah, basically, I did it for fun, but I think it also had, it will have also have good, you know, consequences or, like like, long long term benefits in in the long in the long run, I would say. Bringing into chaos into into software, I think, yeah, makes everything else more robust in the long run. Maybe causing some, damage in the short term, but, yeah, making everything more robust.
[00:37:14] Unknown:
Brock, did I, grok the transaction page right? Like, the transaction cost something like $750?
[00:37:23] Unknown:
Sorry. What what was that again?
[00:37:25] Unknown:
The fee for the transaction to broadcast it
[00:37:28] Unknown:
Yeah. I paid it $750
[00:37:30] Unknown:
out of your own pocket. Got you. I did it for fun. And, yeah, I was actually in order of money as I told. And I I was revert I was trying to revert that money back to my myself, but I think I could I could. Was an off success anyway. I could have done that, but I I realized I couldn't. I mean, I didn't realize that at the time. And then I just I I choose the cows.
[00:37:52] Unknown:
I I I want to make cows. Gotcha. I I just think it's funny because most people are receiving money when they're burning their reputation instead of having to spend out of their own pocket. It was a reverse rug pull. I'll yeah. I like that. Yeah. I guess so.
[00:38:07] Unknown:
I I would love to possibly kinda get into, because because I agree. Like like, what doesn't kill you makes you stronger. Right? But there's you know, I'm curious of the balance here between, you know, causing chaos and in the end goal, we're all I I think we can all agree that, you know, I don't know, code wise, you know, or or ecosystem wise, we're we're better for it probably. But, I mean, what you know, is there a balance there at all? Like like, doing stuff like this versus responsible disclosure, because, you know, I I don't know where I land, like, on it. But because it does I think fire drills do make us better.
In the end, we need fire drills. Anyone could have done this for $750. It's just the matter of, like, well, yeah, when to you know, at least going forward in the future, you know, when when to do this versus versus responsibility, the skill. So I wonder if we can, like, have a conversation around that. It's probably useful
[00:39:20] Unknown:
it's
[00:39:21] Unknown:
probably useful just to kind of high just to to quickly make sure we're on the same page about what responsible disclosure is. There's there's a lot of people who get confused and think, like, responsible disclosure is all about, like, protecting the vendor and making sure you you disclose appropriately to the vendor and whatever. And that's not at all what responsible disclosure is. Right? Responsible disclosure is all about highlighting that you have a duty of care to their users. It doesn't matter what the vendor wants or what the vendor thinks or, like, you know, if you wanna do something that makes a vendor look like shit or and look like an idiot because of some bug they had, you know, it'll probably backfire, but you're welcome to try. And it doesn't really matter. It's not it's not related to responsible disclosure. What responsible disclosure is is saying, like, look. There's some users out there. They're running the software.
You know, for whatever reason, they decided to run the software. They they decided that it was worth worth running because, you know, maybe there's some risk some risk to running any software, but they decided that that they needed to run the software and that, you know, you have to do right by them. So that means, you know, responsible disclosure can mean going public if the vendor refuses to fix an issue. You know, that is a part of responsible disclosure, but it does mean making sure that the vendor has a chance to fix the issue so that users don't get hurt and aren't, hurt more than they need to be. You know, another part of responsible disclosure is is, reporting the issue publicly after it has been fixed to encourage people to, upgrade. And so that that's that's something where you can say, like, okay. After it's been fixed, exploiting it to to force users to upgrade, you know, is is an option, and that that's a legitimate part of responsible disclosure. But it's never, you know, putting users at risk when when just by taking action, you know, before there's even a patch or before users have a chance to upgrade.
[00:41:14] Unknown:
Yeah. I think I I agree on that definition, and I would definitely pick, responsible disclosure if this was Bitcoin Core. And I would definitely pick, again, this approach if this if Lightning is in production. I mean, is if Lightning is used by a lot of people, I think, like, the way I see Lightning is it's really a beta software, and it has, like, this public capacity of 5,000 or something. Even Liquid has, like, 35100 Bitcoin. I think, I don't really take lightning seriously at this at this very stage, and I I kind of see this as a playground. And
[00:41:53] Unknown:
I don't think that's fair at all. I mean, there there are large companies who use lightning now, and and see material transaction volume. I mean, I think I I don't know about Cash App, but I know, like, River disclosed their transaction volume on lightning. And it's not a joke anymore. It's not it's not trivial. They have a lot of funds locked up in lightning that are put at risk in something like this. And that's something too where, like, you know, if lightning were were super alpha and everyone was on the same page there, it'd be one thing. But when there's large corporates who have money locked up or considering deploying Lightning, you know, because they see their peers deploying Lightning, and they're they're trying to decide whether it makes sense to deploy Lightning right now.
You see these kinds of things and it it it makes corporates think twice. It it it really hurts the deployment of lightning even if we assume that there's no no material funds at risk here. But that's not not accurate anymore either that there there are there is a relatively significant amount of money locked up in Lightning and especially, you know, I lot I know a lot of, individuals who who put a lot of money in their lightning node, who put a large amount of their Bitcoin stack in their lightning node. And, you we can agree or disagree about whether that's a great idea. I obviously don't think that's a great idea. I agree. So, like, yeah, there there is a lot of Yeah. I think the I don't know. Well, I mean,
[00:43:16] Unknown:
Yep. Yeah. So, yeah, I think there is a little relatively large on Bitcoin, yeah, in in Lightning. But, yeah, I think, yeah, the corporate's using Lightning infrastructure, I think, yeah, they can easily upgrade to a new version and, you know, and and and the other folks, perhaps, you know, can simply use watchtowers to avoid these kind of attacks. And, you know, when when I text right. Sorry. What? There's
[00:43:42] Unknown:
no watchtower implementation that exists that would have avoided this issue because the, a, the l watchtower is, uses the same code, but, b, all of the current watchtower implementations don't enforce HTLC, HTLC forwards. So the watchtower is don't, address the issue even if and and doing that would be a very non private watchtower. You can give up a lot of your privacy to the watchtower. So there's no code to do HTLC enforcements, even if you even if you wanted to. Not in the watchtower.
[00:44:15] Unknown:
Yeah. Okay. Good to know that. I didn't know this. So okay. But it's not more so easy. You can yeah. For HDLCs, I think, yeah, there is a tech point with HDLCs, which has a is a shorter abstract walk time in it. But I think when the HDLCs claim, the preimage is revealed on chain anyway. But, as for, you know, the actual commitment outputs, these outputs have a relatively larger relative walk time in it anyway. So no other runners can easily upgrade to the newer version and avoid these, attack vectors. So when I did attack this when I created this, the second, transaction, I was aware of this, and I I knew there wouldn't be any there wouldn't be any potential loss of funds, and it appeared, though, we haven't I I personally haven't seen any
[00:45:07] Unknown:
any loss of funds. That's that's not true. Right? That even though there wasn't loss of funds in the first one, there couldn't be loss of funds in the second one. I I would I would push back and say, in fact, that because there wasn't because there there was a recent bug of the same class, they're substantially more likely to be loss of funds, in the second case because people are aware of it. People are are aware of how to exploit it. In fact, there's sample code to exploit it, that the the folks here worked on because of the first bug. Because, you know, once the first bug's out there, of course, you just write sample code, whatever. It's, not not making the exploit all that much easier or harder.
But that because of the first one, it makes it substantially easier to exploit the second one, which which put a substantially higher risk on the second one. No?
[00:45:52] Unknown:
Yeah. I I totally agree that phone files are totally at risk due to the interactive model of lightning. And I still see these attacks as more of a denial of service type of attack
[00:46:03] Unknown:
than an than, like That's yeah. But, I mean, that's not how lightning works. Right? Lightning has always been denial of service attacks allow you to steal funds. That's that's the lightning security model for better or for worse. So there's there's basically no such thing as a denial of service attack against a lightning node.
[00:46:21] Unknown:
Yeah. But, again, you have still you see this as kinda, a type as long service attacks because of the large relative work time in commitment happens in front runners.
[00:46:31] Unknown:
Forty blocks. Right? Forty blocks is like, the time during which you're asleep.
[00:46:38] Unknown:
Is is that default?
[00:46:40] Unknown:
For a for HCLCs. Yeah. It's 41. It's default for LND.
[00:46:45] ODELL:
I think core lightning, the default is 144.
[00:46:50] Unknown:
It might be. The the default for I mean, 40, I think, is the the most common default is the most common value on the network. It may just be for LND.
[00:46:59] Unknown:
But that is for the HTLCs. Right? How about the commitment outputs?
[00:47:04] Unknown:
That's true. But the HTLCs are how you would choose to exploit this kind of bug, because there's no risk to the the x the attacker. It wouldn't make sense to to exploit via the commitment output. But you're right. The the commitment output the commitment output historical state of broadcasting is a much higher value. I'm not sure what the default is there. I think it might be a day. It might be a week. I I keep hearing 2 weeks. Is 2 weeks not true for the commitment outputs? I think it's configurable. I I heard I thought the most common was a week, but it it may be 2 weeks. I don't know. That's totally within would be within reason for for a value to choose.
[00:47:42] Unknown:
I've been hearing a week and 2 weeks also, Matt.
[00:47:46] ODELL:
Both, Matt. Which is it?
[00:47:50] Unknown:
I don't know. Pull up the source yourself.
[00:47:52] ODELL:
Yeah. I know. I know. I'm just saying, you know, I mean, there's definitely there's gotta be a bright side here on, you know, on on Barack's side, right, that we're actually having this conversation.
[00:48:06] Unknown:
Yeah. I mean, it's it's healthy for the community to have a conversation about what responsible disclosure is or isn't so that people are are aware of this kind of thing and so that we can be be more you know, have the community have a a more common understanding of what what responsible disclosure means and and how it should look, so that hopefully we as a community kind of are all on a on a similar page. I mean, I certainly through the course of this this exercise, it it became very clear that the the Bitcoin community has very different ideas of of both what responsible disclosure are and what people's responsibilities are. And I think having a a a more common understanding of what responsible disclosure is should get us hopefully to a a more common page.
[00:48:50] ODELL:
So what would you wanna see here? Let's say you're Barack and you discover the second exploit. Presumably, you wouldn't do what Barack did. What how would you proceed?
[00:49:03] Unknown:
I mean, I think what you do is certainly, the first thing you do is you reach out to the vendor and you make sure you give them a reasonable amount of time to to patch it. Right? Which vendor? To all the lightning implementations or just l and d? Or If it if it affects more than one that you're aware of, certainly, you let you know, for for security issues, you tell the minimal number of people required, in order for the the the issue to get patched. So you start with l and d because that's the only one you're aware of has this problem.
[00:49:32] Unknown:
And then And to be clear, it's like you would you would reach out to them privately. You wouldn't just, you know, go post, like, an issue on GitHub. Right? Right. Yeah. Yeah. Which is what Anthony Towns did, right, about 10 days before this happened?
[00:49:45] Unknown:
Yeah. I mean, I don't think I think he he also I I think he'd said that he'd screwed up. He he didn't realize that this was a a a major security issue. But, yeah, I mean, that's also not the correct Yeah. He hung up here. But then so so I think an important part of of understanding responsible disclosure is is that you coordinate with the vendor about release timeline. Right? So so if you're the one reporting a bug, you are allowed to say to the vendor, hey. I think this is really critical. I think I should go public with this in 2 weeks. Can you get a patch together and and make sure you get that shipped ASAP?
And then that's a conversation that you have with the vendor. It's not like the vendor gets to say, like, no. This has to stay private or here's the release timeline. Take it or leave it. You know, that's a conversation you have with them. And then I think once the the fix is released, it be that, at that point, it becomes a question of, like, what you think is the makes the most sense for for users. So, really, you know, patching a a an issue quietly does not get the most people to upgrade. And so, you know, you've seen this con we've had, you know, back when I worked a bunch on Bitcoin Core, we had these kinds of conversations a lot. You know, when you have an issue, do you tell people about it so that people go and upgrade really quickly, or do you not tell people about it because, you don't wanna give attackers enough information to go exploit the vulnerability?
And so it becomes a a balancing act. And, again, this is where, you know, you, the reporter, get to have a conversation with the vendor and where you, the reporter, gets to ultimately decide. And I think it would be I I don't know if I would have decided to do that, but it would have been totally reasonable or at least totally justified to go exploit the issue after the patch had been released or maybe maybe a few days after the patch had been released so the people had a chance to upgrade on their own, to highlight the issue and and to for kinda force people to upgrade, quote, unquote.
But, you know, prior to there being a patch and prior to people having at least a small window of time to to upgrade themselves, actually exploiting it just just seems like putting users at more risk than is necessary.
[00:52:01] Unknown:
Well, isn't that kinda just, like, the sad truth, the sad reality of the situation? Like, we talk about we're just just saying that, like, oh, because there are significant volumes on Lightning now, and there are big companies and corporations looking at Lightning. And then things like this overall hurt the perception of Lightning. I mean but isn't the perception of Lightning just wrong currently? Like, we live in this happy little lightning world that, like, all of our payments are fast and easy. We can do 0 conf atomic swaps. We can do all this other fancy shit, but, like, Lightning's pretty broken. I mean, Matt, your talk, Crowley, your talk at, Atlanta was, like, a really good talk, and it just kinda highlighted that lightning is kind of broken in its current state.
[00:52:49] Unknown:
Yeah. I mean, to add to you, as well, like, this is literally a valid transaction on chain. Right? I think that's the way certain people are framing it. So if your l two is affected by it, they feel like, in essence, it was your fault. Bitcoin Core thought this was valid, and I think that's where some people are drawing the line.
[00:53:10] Unknown:
Yeah. Totally. I mean, like, if you're if you're, a it was LND's bug. It was Rust Lightning's bug. It's not like the the the the foundational issue was that there was a bug. The foundational issue was not that it was exploited. The only the only question then is, like, okay. There is a bug. Given there is a bug, should you exploit it? That's a very different question from, like, whose fault is the fact that there is a bug. Right? And bugs happen. You know, that's not gonna not gonna change. That's the reality of software development. But but yeah. So, you know, I I think to the point about lightning being broken and the perception of lightning being wrong, I think it is important to to highlight the difference between, your counterparty and, like, you know, I always kinda make the point that you shouldn't open channels with a counterparty you don't know or at least don't trust enough that they're not gonna, like, try to steal your money and, like, put in a bunch of effort to steal your money. I think this is a very different world from and then there's denial of service attacks against Lightning, not in the, like, take your note offline, but in the, like, saturate your liquidity so that you can't send payments. So let me I guess I should rephrase my earlier statement about there not being denial of service attacks against Lightning. Just that the the the, saturating liquidity denial of service attacks.
So there's a lot of things that are broken with lightning and where the perception is maybe wrong, but I think those are different from you're going to lose funds maybe because your counterparty did something without them even being involved. So when we were talking about, you know, routing an HDL through multiple hops and then stealing funds, that's something where like your counterparty can be totally honest and they go to chain and then your node doesn't see it and then you still lose funds. Right? So I think that's the kind of thing where, you know, aside from these bugs, generally, I think there's not really much in the way of known issues, for lightning for why this might be why you might lose funds if your counterpart is, like, completely honest.
[00:55:10] Unknown:
And regarding previous bugs, Matt, it's not our first bug on lightning. Right? There's one in 2019, and then Antoine Riard and Gleb also found a dust one last year that, I guess, they responsibly disclosed. Like, do you remember how that process was, and maybe we can, like, touch on that?
[00:55:31] Unknown:
Yeah. That was across the whole that was basically on every lightning note. So they they reached out to everyone. They emailed, you know, I guess, all of the I don't know who all was on the email list. But, you know, basically, all the maintainers or the kind of lead maintainer or 2 of all the lightning implementations and kinda describe the issue. They describe the basically, the issue was that, because lightning has some concept dust outputs. If an if an output is dust, if an HTLC the value of the HTLC is dust below some threshold, then it just gets burned to feeds.
So, a counterparty could send a bunch of HTLCs that were just barely below this threshold. And in fact, they can even do things to, increase this threshold, that nodes will currently accept. And then if they do that, then they can burn a lot of your money to fees. And I guess if they're a minor, they get free money then because they burn the money to fees and then they get to claim the fees because they're a minor. So the they they reached out to all the lightning implementations. All the lightning implementations added a bunch of code to, patch it and then ship that code. And then I think after everyone had shipped or some period of time afterwards, they disclosed it publicly. And and, Antoine wrote up a a a change to the bolts and described this issue and and note that that implementers of the bolt spec need to, do something to to ameliorate this issue.
Unlike the kind of on chain stuff that we've been talking about, there's no way to go, like, exploit the issue to force people to upgrade. There's nothing like that. We can only, like, write code and and tell people that this this issue exists and you should upgrade. And then, like, if you were to exploit this issue, you have to actually go steal funds. You you couldn't just, like, cause the issue and then and then get no stuff.
[00:57:22] ODELL:
And then there was the there was the other bug where none of the lightning implementations
[00:57:27] Unknown:
were actually checking the chain for commitment transactions. Right? For funding, I believe. And that was the For funding. Yep. The funding transaction had the right script and the right value. Actually, that's all I we were. So before LDK was LDK, Rust Lightning existed and it was a joke. It was like a tiny project, and and no one used it, and it wasn't even ready for production. But but we were checking it. So Mhmm. But but yeah. I mean, that that was another again, that was another case where you couldn't do something to actively force everyone to upgrade, and you could only actually go steal funds, which obviously has a has a very different moral question.
[00:58:03] Unknown:
So it seems that the heart of all these attacks is a security assumption that your peer, ideally, will not be a lightning node peer as well as a miner. I know there's a lot of stuff being kinda shuffled around with the mempool and Bitcoin Core right now. People happy or unhappy about the RBF policies. How do you think this will affect, any other things, any other new potential dust issues, HCLCs, anything? Does anything come to your mind, Matt? Anything about, like, l two that's needed or penalty transactions?
[00:58:44] Unknown:
Yeah. I mean, we know a lot of stuff needs to change. And this isn't even so that there's questions about minors, but also just around mempool policy like you mentioned. We know a lot of stuff has to change. We have ideas for how that stuff changes. All of it to exploit it though, like I mentioned, requires some amount of effort on the part of your counterparty. They have to, like, do a bunch of work writing a bunch of software, coding stuff, to to actually go steal funds, you know, anchor outputs. As long as you have anchor outputs in your channels. So anchor outputs don't prevent don't protect you against an actively malicious counterparty, but they do protect you against, the just the fees going up in the network.
So the the kind of straightforward issues that existed prior to Anchor, solved by Anchor, you know, encourage you to make sure your opening channels that support Anchor outputs that have the Anchor flight turned on. And aside from that, you know, there there's things that need to happen, but they they do require some amount of effort to to exploit. So also just make sure you know who your channel counterparties are. You trust them not to to do a bunch of work to steal your money. Trust them to at least be kind of honest.
[00:59:57] Unknown:
Well and and then maybe that's just, like, the thing that should be highlighted more because I don't think that's, like, the current situation today. I mean, where people just, like, open channels with, pretty much anyone, really. Any random on the Internet, they kind of just, like, go and open channels, and they let their little Raspberry Pi sit in the closet for months that they don't touch. And then that's kind of the end of the story there. So, you know, hopefully I mean, I would say, you know, after, you know, some, you know, person that just had their lightning note in the closet for 2 months, and they had to go figure out how to update it and then having to do it again in the same month. I mean, you know, if they're not ready to sign up for that life, then maybe they they really should consider. Like, okay. Maybe I shouldn't shouldn't be running this lightning note, you know, after after all of this. Because that's kind of the reality. You don't just, like, throw it in a closet and, like, you know, don't come back to it again.
[01:00:58] Unknown:
And your your blog, Tony, you said it well. Don't open or if you're gonna go into a coma because, like, it's not just something you can like, you know, lighting a car that you can maintain personally, then it's just, like, you're running up cold yourself. Like, figure out how to either study lighting all of sucks or, like, something that's like like a analysis or something.
[01:01:30] Unknown:
Yeah. And that's also another question about, like, how do how do the developers communicate the safety of the the products they built? That's something I struggle with a lot. I don't know. There's no easy answer, but how do and this is, like, a very, very broad question too. Right? And and maybe this is something worth discussing, but, like, if you're writing software, you're writing a wallet, how do you communicate to users what the relative risks of all these different attacks are, especially when you don't know for sure? You know, you you have some idea of how to exploit it, but, like, will it work in practice or not? It's a whole other question. And then, you know, how do you how do you evaluate this and how do you communicate it to users? I mean, this is another thing, you know, I think I always say that the only two things the only thing in the entire kind of quote, unquote cryptocurrency space that's in competition with Bitcoin is stable coins in practice because people do wanna you know, the only thing that works that does payments is is stable coins and people use stable coins for payments.
And so how does how do you if you're, like, writing a wallet that supports Bitcoin and stablecoins, how do you communicate to users that, like, this stablecoin exposes you to risk of, you know, Tether going under and stealing all the funds. It exposes you to risk of, like, the blockchain it's on and it's like Tron and it might just, like, randomly implode and, like, you know, it also exposes you to the risk of of just US dollar and sanctions and, like, maybe the US government decides to, like, this this stable coin is not enforcing sanctions and suddenly all the money is taken. Or you can use Bitcoin and the the the risk here is the volatility. Like, how do you communicate that to users in a way that they understand that intuitively and then make a decision that makes sense based on that communication?
Or do you just say, like, here's stable coins and here's Bitcoin. Use what you want. And then users just assume that the stable coin is, like, gonna keep working and has no risk associated with it.
[01:03:25] Unknown:
I think that's too many options. They can only do swipe.
[01:03:33] ODELL:
So it kinda seems like the conclusion is, that at least Sovereign Lightning usage is not in the cards or should be dissuaded from end users right now. Or or at least
[01:03:49] Unknown:
make sure you know who your channel counterparty is.
[01:03:54] ODELL:
Yeah. I mean, that almost feels like like, I don't know who my channel counter parties are.
[01:04:00] Unknown:
Well, you started your node at a different time. You know, like, you were one of the earlier nodes, and it was, quote, unquote, reckless, have fun. Like, you're just trying to test the technology. I think now we're in a more mature state where, I guess, it is still reckless, but you should not you should do your, best efforts to know your peers and, update your software, all that. Like, it's no longer SpongeBob memes. You have Tony and Brock out here now.
[01:04:34] ODELL:
So I have a question, which I asked on the open source stage in Miami this year. So in my Bitcoin experience, one of the things that I learned early on is that unlike traditional proprietary software that people are comfortable with or or used to, I guess, we intentionally do not have auto updates on Bitcoin Core because it's an attack vector. If if if you just update blindly, then you might update, a malicious update. You might update an attack. So a lot of people with with Bitcoin Core specifically, they're they're they're slow to update. They don't update very quickly. On the Lightning side, there seems like I know it's still early days in Lightning, but it seems like there's been many, many, many times where, like, the community has been, like, pushed to to update ASAP. There's something critical you have to update ASAP.
Now, enthusiasts, like, hardcore Bitcoiners that are not necessarily, you know, fluid in code or whatnot, but they live and breathe Bitcoin, are running their lightning nodes on Umbrel and RaspiBlitz and Start 9 and Noddle. So they're they're running these, like, prepackaged node projects, and then they see that there has to be an update, and they just go to them, and they just quickly, like, blindly click the update button. Is there I'm 1 I'm curious if anyone here supports the idea of an opt in ability to auto update, Because they're practically doing it anyway. Like, what is the negative of giving them the option to auto update instead of, like, you know, being traveling, being away from your node, not being able to update for 2 weeks even if they're just gonna blindly up press the update button anyway.
[01:06:28] Unknown:
I mean, I I'll I'll go first. I mean, I think it's a super valid conversation. I mean, I also hate the idea about updates, but that is the reality of the situation is that they're just clicking update anyways. I mean, when you even think about mobile apps and and mobile phones and, like, no one, for 1, like, make it incredibly hard to actually verify any source code at all if you're going through app stores. And for 2, you know, some of the app stores even auto update anyways. So, like, there's there's there's nothing they they really do there either. So and and most people's lightning nodes are probably mobile phones, mobile phone apps anyway. So I I think it's a valid talking point. I don't know if I would go far go as far to say, like, there should be an auto update option in in apps like Umbrel, but I think it's a definitely a valid conversation to have. I'm curious what other people think on that.
[01:07:24] Unknown:
So I would say that's fine, Matt, but I would then also insist a contingency that there's an atomic rollback feature. I'm a big fan of Nix Bitcoin and NixOS for this reason, and I believe Bitcoin Core uses geeks also for reproducible and deterministic builds. So as long as there's a rollback option after that auto update, they can immediately go back if there's something wrong.
[01:07:51] ODELL:
They can rush to unupdate. Correct. So rush to update.
[01:07:55] Unknown:
Correct. The soft so the the rollbacks are hard too though because you might have, like, updated your state. Like, your local database might have gotten upgraded or something, and then, like, you're gonna try to roll back, and then it's not gonna be able to read the new the new stuff. That that's actually something that LDK supports that's maybe a little unique, but it's a lot of effort to support, and I wouldn't necessarily say that that all developers should support it. One other point I I would also highlight that, Bitcoin Core is very different in the the requirements here because there's not only a cost to you personally. If you like auto update and the the software is now, malicious and steals your keys, That's one thing.
But it's there's also issue with, if you auto update and there's a consensus rule change and all the nodes auto update, suddenly someone can for example, like, BSV pushed an update today to to be able to steal arbitrary coins so that Craig can go just steal your coins if he doesn't like you. You know, suddenly you could do something like that if there were, like, a real auto update mechanism. So there's additional social costs in Bitcoin Core for that kind of thing that maybe don't apply in lightning or in kind of other wallet software cases.
[01:09:17] Unknown:
Yeah. I don't think there's, like, a real clear, clean answer about what the best behavior here would be. Obviously, any opportunity you have to let the user verify, that they're running, like, a specific version, like, a deterministic build is always welcome, but, you know, much easier said than done. Like, an iOS app, like, you can't do that at all. Right? And, you know, Bitcoin wallets, like, only a handful are, you know, deterministically reproducible on Android. But I think there might be value in some of these node platforms giving users the ability to, manually upgrade to certain releases perhaps before, you know, like, a maintainer can, like, bundle it back up again.
I think that's a feature that, my node has, that I think is pretty underrated. Like, you could just pop in a version number, and it'll fetch and try to build it and install it on your machine. But, you know, depending on these architectures, it might not be so easy for a lot of these projects to transition to that. So Evan also has the room for user error too. So
[01:10:23] ODELL:
So, Evan, I mean, out of everyone in this roundtable right now, you have, the most exposure directly to end users of Lightning, because of Zeus.
[01:10:34] Unknown:
Yep.
[01:10:36] ODELL:
Like, what has what are your after these bugs and and the reaction among the Zeus community and how everything went there, how do you, I guess, you and Ben do because of, the Bitcoin company. How do you how do you how do you feel about Lightning going forward? Like, how do you think about actually developing for end users?
[01:11:02] Unknown:
Yeah. I mean, there's a lot of complexities here. You know, we're definitely seeing a lot of users that are frustrated by a lot of different things. Like, you know, 1st and foremost, this year, Tor has been really awful in providing a really bad user experience for people who want to sovereignly run their own node at home, able to, you know, test when they're out in a meat space. Right? Something like, you know, these l and d exploits are are sort of exacerbating, a lot of the frustrations these users have. You know, while some users may be technical enough to put together their own node and run something like Umbrell. They might not, you know, be following or have the understanding to see what's going on with, these exploits and and the real nuance about what's happening.
And that's unfortunate because, the alternative is, you know, these users putting more trust in other third parties or, you know, running, you know, just a prebuilt version of, like, a lightning wallet on their phone that they might not verify. A lot of users might just be falling back on LSPs as their only peers now. You know, there's there's a lot of complexity whether that's a good or a bad thing. But I think, ultimately, what we're seeing from exploits like this, we have people that are less willing to run Lightning and and are dissuaded from using it at all. And, I think that's ultimately a bad thing because we have a network that's less distributed where, you know, these big intermediaries are, you know, playing even bigger roles than we'd like them to.
[01:12:58] ODELL:
Yeah. I mean, I see that trend kind of happening as well. I just wonder how, like, how that can really be avoided.
[01:13:07] Unknown:
Right. Like, on on one hand, it's like, you don't want people running these setups that are uncapable of doing it, that are just gonna be neglecting it. But on the other, you just don't want, like, these centralizing factors. I mean, obviously, there's, like, a degree of centralization that's gonna happen. Like, you're gonna have, like, these big hubs on the network. They're gonna use the SSPs or, you know, maybe, you know, use them as introduction points on blinded past. They offer up a plethora of services. Like, that's unavoidable. But, like, we also want to make, you know, having, having the ability to, you know, be as sovereign as possible and and run this infrastructure out of your house possible and also, like, appealing. Like, we we didn't want this to be something that's that's commonplace.
And, you know, it's it's it's happening in a lot of ways. Like, something like Umbrell is a consumer facing product. It's not just corporations that are running this thing. You know? And, you know, a lot of users in some nation states like El Salvador are dependent on lightning to to live every day. Right? So when we have someone, like, go and exploit something like this and, you know, there's a risk of actual people who are dependent on this stuff losing savings, it's, really frustrating, especially considering all the effort that people are putting into this on the lightning side to make Bitcoin something viable that we could be using in our day to day lives. Was Chivo down during the lnd or block parsing bugs?
[01:14:41] Unknown:
I don't know. Jivo is actually just, like, public, is just powered by somebody's back end. It's not it's just it's it's a database, but it also somebody else does their lightning in and out. I don't know if it's public who does that, so I'm not gonna mention it.
[01:14:58] Unknown:
But I know I think they announced it. Do that. It's River.
[01:15:04] Unknown:
Yeah. It used to be a scene. Now now it's now River announce that they're doing it. Yeah. It used to be some other folks. Now it's River. I I I know River was able to update pretty quick. So
[01:15:14] Unknown:
Yeah. They're doing a pretty good job with your infrastructure, so that was good to see at least. I think, Chivo's problems really rely on Beyond chain stuff that they've been doing where people will receive a or send a transaction, and then Chivo says it's canceled despite it being confirmed on the blockchain.
[01:15:32] Unknown:
Wow. That's pretty bad. I was unaware.
[01:15:36] Unknown:
Yeah. It's, they're not really helping themselves out over there in El Salvador from, you know, governments and Elis. My understanding is they've gotten a lot better.
[01:15:44] Unknown:
That that it was a it was a trash fire for the first while after it launched, but it it had gotten a lot better. But I haven't tried it. I don't know. Yeah. I mean, I think I think people have just
[01:15:55] Unknown:
given up on it at that point, though. There's a good Bitcoin Magazine article. I know it's kinda getting off in the weeds, but there's a good Bitcoin Magazine article of people trying to use it now, like, 1 year later, and and people have just sort of given up on on Bitcoin because of the Chivo experiences. So, I mean, that I mean, that kinda relates to Lightning. Right? Like, we have so many, bugs and exploits and issues with lightning, that we're that we're all kinda aping into right now too early, it's gonna leave a bad taste in the mouth. So maybe, like, the more responsible thing to do is kind of have a slow roller a a slow, rollout of of lightning instead of just jumping into it.
[01:16:37] Unknown:
Yeah. I mean, it's also important to note that folks in El Salvador some parts of El Salvador had real demand for the features that Bitcoin can provide for people. Others not so much. You know, it is a US dollar based economy. There wasn't you know, inflation's now high with dollars, but especially when that first rolled out, it wasn't, and there wasn't, you know, a lot of of user demand for it. You know, I'm much more interested in in parts of the world where where there is real user demand because Bitcoin provides a much more compelling, service compared to the alternatives. You know, your places like Nigeria, your places, you know, with there there's real issue with especially financial seizure, but also hyperinflation.
And in those places, I think it's it's, you know, you you can't blame Chivo for all the problems is my point. It's also just that, like, for the most part, people in El Salvador didn't need Bitcoin. They didn't care about Bitcoin. Some people care about it now, and maybe with the the dollar inflation, they might care about it more. But in most parts of El Salvador, there wasn't you know, there are a lot of problems, but financial like, seizure of of financial assets was not not high on the list.
[01:17:53] Unknown:
Well well, what about us, you know, United States citizens? I mean, do we need do we need Bitcoin, and do we need lightning at all? I mean, you could almost, like, say the same thing right there.
[01:18:06] Unknown:
Yeah. That's true. You know, I think what what does need Bitcoin mean? You know, people care about different things. And if you care about something that Bitcoin provides, it's great. You should totally use it. I think my point was more that the average person isn't sitting there considering, you know, who their counterparty risk is when they're using the dollar. And so to them, Bitcoin's not compelling. It's not solving a problem they see in their daily lives. You know, to those of us Bitcoin is in the west, you know, we might see real you know, like, personally, I use lightning to pay for some server bills, not because, like, I don't like the dollar, but actually because it provides a better user experience and it's faster and more reliable than paying with a freaking credit card. And, like, that's actually solving a problem for me that I had, you know.
And and, you know, the to to folks in the US who are worried about all kinds of other issues, you know, Bitcoin is providing something that that is an issue to you. It's a little different if, you you know, someone's telling you to adopt Bitcoin to solve something that is not already an issue in your mind. And and so I I just I care more about places where, like, some people are already thinking about these issues and and the normal average person is already, like, this is a major issue. I need something to solve this issue. Oh, Bitcoin is a solution to this issue rather than this isn't a problem for me. Why are you telling me to use Bitcoin? And then maybe a handful of people are like, oh, yeah. This is something I was thinking about and concerned about.
[01:19:52] Unknown:
I guess another data point, you know, just because Matt brought up, Nigeria, There was that partnership, with M Pesa, that's now going to essentially open up Bitcoin to a lot more users through Bitnub, and I believe that opens it up to, like, 50,000,000 people. And then Cash App, previously, was probably the largest with 44,000,000. And then, just understanding those, I guess, aren't necessarily, like, daily active users or whatever VCs, wanna attribute them to, but there might not be addresses either. It's in a weird, metric they use where, essentially, they have the capability of buying it or doing something with it. And, that says something. Right? Like, I don't know. It's, like, for them or they need it, but someone wants to build a service for that market.
[01:20:50] ODELL:
But they couldn't I mean, they would experience downtime in the case of Barack's 2 bugs, but they wouldn't have they wouldn't lose money in that situation because, I mean, all the all the major players updated really quickly. And they all, like, limit who their counterparties are on Lightning.
[01:21:10] Unknown:
Yeah. And, I mean, if we wanna talk about, you know, positive outcomes here, I don't know if if LND has done it, but certainly one thing we've been thinking about more, to some extent as a result of this, but also just software maturing, is how how we better communicate quickly with people when, when something like this happens or when you people need to upgrade urgently. There's not to my knowledge, I don't know if if LND has, like, a a mailing list you can subscribe to or what. But, like, you know, how do you make sure engineers at Cash App and at Kraken and at River get paged when something like this happened and and and are are told to to move to a computer quickly. Right?
Versus just like, well, you gotta be following us on Twitter and and make sure you see it, which isn't really a a sustainable outcome. And and, you know, some companies have engineers who follow Bitcoin Twitter personalities, but but others don't. And they shouldn't necessarily need to in order to discover, issues that they need to to patch quickly.
[01:22:18] Unknown:
Yeah. I think that's a fair point. I mean, I'm I'm not on Twitter myself. And if that's the only way to hear about these things, then, you know, they they they do have a very extensive email list that they send out every other month to get feedback from people, but haven't heard anything from that about security patches.
[01:22:38] ODELL:
So, Tony, I mean, you're not on Twitter. How did you find out about the bug?
[01:22:42] Unknown:
I get people on Twitter to tell me things. Ben Ben Carm, it's always telling me about l and d bugs.
[01:22:49] Unknown:
So he's he's my source. And then I'm Ben's source, and shout out to Richard Saffer who's, DevOps at Zebedee. He's the one who filed the original issue for the first block parsing bug, on l and d.
[01:23:07] Unknown:
So it's all personal connections. And if you don't know enough, big partners, you're, you're gonna get your money stolen because you're not gonna know the upgrade.
[01:23:14] Unknown:
So you won't discover it as fast as you might have otherwise. But I'm sure if you hung out in the l and d Slack, you are fine.
[01:23:23] ODELL:
So there's wait. There's 2 there's 2 different when we're talking about bugs like this, there's there's basically 2 different classes of users. There's, like, the almost, like, institutionalized user or the corporate user, and then there is that end user. The end user, like, the sovereign node runner, like, we have obviously some issues there in communication. But on the on, like, the corporate side on the corporate side, is there is there something to be said about, like, what bottle pay was I I think Youse released it. I think he called it multiplexer where essentially you're running multiple nodes, of different implementations. Like, Corelightning didn't go down here.
Is there is there a strong argument that the corporate users should essentially just be running multiple lightning implementations at the same time?
[01:24:25] Unknown:
So I don't think use multi I think use multiplexer, supports running multiple LND nodes. So I don't think that's that helps here. I know some corporates have built their own solutions for running, multiple nodes or at least have talked to me about it. I I don't know that they've actually built it fully or rolled it out. I think that's a a valid a valid point to make sure you don't have downtime, but it doesn't protect you from funds loss. And in fact, it it makes your funds loss exposure worse. Right? Because then if any issue, any, implementation has has a bug, you can lose some money.
[01:25:02] Unknown:
At at at the very least, we need we desperately need, like, multi node implementation watchtowers. Like and and I don't think we talked about it here yet, but or maybe we hinted at it. But, you know, l and d nodes can only talk to l and d watchtowers currently. So, you know, you would be affected by this. And if you had your own watch tower running and you didn't update that in time either, then you would be you would you wouldn't be helped at all. Sure. If you outsource your watch tower to, like, a responsible node runner, that is, like, servicing a bunch of users, you you could probably expect that they're at least doing a good service, a good good service of running watchtowers, even if they are on the same node implementation. But, we we desperately need multi node watch towers, the very least. Because these bugs broke the watch towers as well.
Yeah. Well, the l and d watch tower was also broken, and l and d can only talk to l and d watchtowers.
[01:26:01] Unknown:
Yeah. So there's two points here. There is actually a third party watchtower, guy Guy Sergio, who who Spiral has funded for, several years has won. I believe it currently only supports Core Lightning because Core Lightning is the only node implementation that has enough hooks, for it to support. I'm kind of embarrassed that LDK doesn't, but LND certainly doesn't. So you you can use that. That does exist. However, it is, again, useful to note that in order for a watchtower to you know, we mentioned that there's 2 kinds of, ways you can exploit this class of vulnerability, both stealing funds from, by broadcasting an old revoked state, and there, a watchtower will get your money back. But if you steal funds using the HTLC, logic, if you have a a an HTLC that's routed through you and then someone steals funds using that, then, you can't, use a watchtower currently. And in order to use a watchtower, watchtowers would have to expose a lot of you don't have to expose a lot of information about your channels to that watchtower.
Losing, you know, one of the pitch of watchtowers today is that they're they're private, that they don't learn about your payment history, about your, all the details of your node, and you would have to actually expose all that information to a watchtower to enforce HTLC, claims correctly. So it would require a bit of a change to the watchtower model. Currently, I don't think anyone's working on that. But in order to, you know, I guess I guess the moral of the story is is set your HTLC maximum in flight configuration knobs down to a threshold you're okay with losing, which is good for everyone's privacy in the network. So do it anyway.
Make sure you change that value.
[01:27:50] Unknown:
Well, is it is it easy for people to change? That's always the the tough part for me. It's like I I have all these ideas of how to be how to have better privacy on lightning, and then it's like, well, none of these are easy to actually implement. I'm sure that Thank can do it. But yeah.
[01:28:07] Unknown:
Yeah. I I don't know about lnd. I assume at least Corelightning because it's very configurable, has the support for LDK does, if if you're using an LDK node, if you're writing code for an LDK node, you can do it. I would imagine LND does. I don't know for sure. If it doesn't, you should certainly nag them. But you should be able to configure this in your lightning node config or in your channel config to say, like, max in flight lower than the total amount. At least set it less than half. Ideally set it maybe to a quarter of your channel balance, or the maximum you're willing to lose.
[01:28:38] Unknown:
Or or maybe it should just be defaults even on a certain point.
[01:28:42] Unknown:
It it really should be. This is only a more recent discussion that we had in the Oakland lightning dev lightning dev summit that we should default this to below, below half the channel balance at least, if not less. I don't know which nodes, if any, have deployed that default change. I'm not even sure if we got around to it yet. But but if you do have that setting set, then LDK nodes, including Cash App, will slightly prefer to route through you. So maybe you'll actually make some some additional get some additional HTLCs and routing revenue, to pay for the the reduction in capacity that you that you have in the channel.
[01:29:21] Unknown:
You can, actually set these values in your LDconf, I believe, also with their API. So I think, the only one I'm unsure of is eclair, but, I I think everyone allows you to set those values.
[01:29:39] Unknown:
Yeah. But but, like, what Umbro user is, like, going into their their configs and, like, writing writing text into them? You know? That that's the only thing I have to say about that. Like, I I know defaults are very hot topic in Bitcoin, but maybe it's one of those situations where, like, this is just better for the whole network. Let let's just go ahead and do it. But, I'm glad to know that there's there's conversations around it now.
[01:30:12] ODELL:
Yeah. I mean, from that, like, that perspective, like, I still feel like I like, Carollo, I'm I'm kind of curious. Like, so you you think just unequivocally the way Barak proceeded with just executing these bugs by issuing a transaction was not the way of doing it. There was no benefit of doing it in that way rather than going through normal
[01:30:39] Unknown:
channels. Yeah. I I don't think there's any reason to exploit something like this unless, unless it's already been patched, unless people have had, you know, at least a little bit of a chance to upgrade. You know, you you can debate, I think, what the time period after a release with a patch, at what point it makes sense to exploit something like this, after that patch has been shipped. But I don't know what that number is. And I I I think it's definitely not negative. It's definitely not before a patch has been made available or at least made public that it makes sense to to do this kind of exploit.
You know, every exploit is unique. Every every vulnerability, every bug is unique in terms of what the specific outcomes are, what the specific, you know, ways it can be exploited, ways you can potentially mitigate, all that kind of good stuff. So, you know, it depends a lot on the context. It always depends on the context, but I think that's even more reason to make sure you reach out to the vendor before you before you do anything because the vendor you know, some vendors are shitty. I think in Bitcoin, we're pretty lucky. None of the vendors are are really shitty generally. At least to my knowledge, everyone takes these kinds of things seriously and appreciates, vulnerability reports even if it's from somebody they don't like or if it's from someone who's be doing it, you know, rudely or whatever.
Obviously, that that doesn't apply here, but just, you know, in general, vendors can be shitty when you report stuff to them and and some people are are shitty when they do report stuff to you. But I think everyone's pretty pretty chill in the Bitcoin world about this kind of thing. So it always makes sense to to reach out to the vendor first. Get their take. Get make sure you're on the same page in terms of how this can be exploited, what the the consequences of it, what you can do to prevent exploits, taking off, and then take action. And I don't I don't think it ever makes sense to take action before you've at least reached out to the vendor initially and and and flagged it to them.
[01:32:49] ODELL:
So, I mean, from my perspective, it kind of felt like enlightening development world or just like enlightening network land in general. It it kind of felt like it kind of felt
[01:33:05] Unknown:
like,
[01:33:07] ODELL:
assumptions were a bit naive, and a lot of it was glossed over with, like, hashtag reckless or whatnot, but, like, real funds are on the line, and they're all hot wallets. This kind of the way Barack did it to me felt like a shot across the bow, like, we are entering an adversarial environment. Consider the risks. And I think a lot of users, as a result, now have a different feeling. Particularly, node runners have a different, you know, perspective on lightning. They're more aware of the trade offs.
[01:33:42] Unknown:
But I think that that's separate. Right? That that's a separate question. Right? So, like, you know, should is the lightning development world not are lightning node vendors, accurately communicating the risks? I think the answer you you're right. They're not, and that that needs to change. And that's something that's that's frustrated me in a few contexts across the lightning space. And so, you know, I I think Lightning node vendors generally need to do a better job at that. Totally. It's a separate question as to whether or not to exploit something before there's a patch. And then, you know, if you if your goal is to have a similar outcome of, like, making sure it's clear that they, that users see this and feel urgency to upgrade, generating a similar feeling. You could say, you know, exploit it, you know, 8 hours after the patch is released or something. You know? Make sure people have a have a chance to upgrade, you know, even the people who were asleep at the time the patch was released, whatever, and and then do it. You know, I I I don't think that's I don't know that I would decide to to exploit it that quickly, but you you could make an argument for that and that becomes more of a a value judgment that we'd have to debate. And I don't think there's, like, a correct answer. There's never a correct answer or you know?
So but but doing it beforehand, I think, is a little different. And and certainly, you know, another part of responsible something that's not a part of responsible disclosure is how it gets communicated. Right? So, the vendor always vendors always wanna choose how something gets communicated, but that's ultimately not just their choice. That's also the reporter's choice. Right? So if the reporter is like, you know, I think this is a huge deal and you're not doing a good job of communicating it, the reporter is totally free to go on Twitter and say, like, vendor is moron.
Screw this vendor. They're not telling you the the real story, and this is a huge deal, and you need to upgrade ASAP and, like, your funds are at risk and look, here's why. That's totally legitimate. That's that's not you know, how an issue gets communicated is not a part of responsible disclosure. Your your duty is to the users to make sure you minimize the risk that users lose funds, But there's a lot that goes into that and, you know, there's no correct answer. Again, everything's unique. But I I think it it it I I think it's pretty clear that exploiting this before there's a patch does not minimize user risk. I think it it it's pretty clear that there are many different ways to go about this that might get similar responses in every way that would have minimized the risk a little more by just waiting to exploit it until after there was a path.
[01:36:33] Unknown:
What what about the scenario? So there there are multiple vendors here that that had the issue. Right? Now, of course, Brock was honing on a specific one, and and there's a lot more that that we found out. How from the vendor approach, what would have been what would have been, like, a reasonable thing to do? Because I I could be wrong, but I think I remember years back, Bitcoin Core had this scenario where there was a responsible disclosure to Bitcoin Core from from some shitcoin Bitcoin side chain thing. And Bitcoin Core fixed it and then released, you know, released it, but the other chains, were vulnerable to it. Now you could argue, you know, fuck those shit coins or whatever. But when there's an issue that could be affecting everyone, but only one party knows about it, is is that a problem at all?
[01:37:22] Unknown:
Yeah. Who who's who do you owe a duty of care to is is a huge question that has no answer. There you're right. There were a lot of discussions around that bug, around do Bitcoin Core developers have a duty of care to every random, fork of Bitcoin Core building any random coin. You know, I personally think they don't. That's just my view. You know, I don't think I think Bitcoin Core developers have a duty of care to Bitcoin users, and not a legal duty of care, mind you, but but, you know, a moral duty of care and and not to anyone else. And that, you know, insofar as you can do anything to to avoid other people getting screwed, you should.
But, certainly, you don't have a duty of care, where it might impact where it might negatively impact Bitcoin users. And that includes, you know, if you tell with any vulnerability the, you know, one thing that that you have to maximize, you have to make sure you don't tell any more people that need to know. Right? That that kind of information should always be need to know because, you know, any additional people you tell, they might tell their friend and their friend might exploit it or they might have it up on their screen at a conference and somebody might read it and exploit it or or what have you. And especially when you're talking about other coins, you know, that that can be a very adversarial relationship. So I do know, you know, when it was disclosed by Bitcoin Core or maybe kind of an hour before or something like that. You know, once once it was clear that Bitcoin was a little safer and then when it was going to go public anyway, a number of people were notified, you know, kind of immediately prior or maybe an hour before, to make sure they had at least a little bit of a chance to to catch up and and fix their own stuff.
I don't recall whether any other coins were. I think maybe, like, b cash was or something or, like, one coin or 2 where there was, like, you know, at least a little bit larger market cap, at least a little bit larger user base, and some of the developers had had a history of being reasonable, even if their community maybe wasn't, their their developers were. But but no. I mean, I I don't think Bitcoin Core developers have a duty of care to other clients. But yeah.
[01:39:48] Unknown:
Do CLN developers have any duty to to l and d developers?
[01:39:54] Unknown:
I actually was just about to bring that up. Matt brought up communications of, disclosures. Brock, why did you choose to write what you did in the off return?
[01:40:11] ODELL:
You still here, Brock?
[01:40:13] Unknown:
Oh, yeah. I'm here. Yeah. It does for trolling. I just I just wanted to troll. I I guess yeah. I have a troll personality. I just you know, I'm making this transaction and why not just put something in there? I just yeah. I just put some message.
[01:40:29] ODELL:
To to all the listeners tell the listeners you put in the op return, you'll run Core Lightning and be happy.
[01:40:36] Unknown:
Yep. It was, yeah, it was referenced to, World Economic Forum. Yeah. That was for fun and for trolling. Yeah. Yeah. I I'm not sponsored the message was for trolling. I have no problems whatsoever with Lining Labs. I love folks working at Lining Labs. In fact, in favor of multiple implementations, whether it's LND and CLN or other Bitcoin query implement other Bitcoin implementations, I don't have any problem. I I'm just trolling.
[01:41:21] Unknown:
Well, Tony, there is your answer. We have a duty of care, but that was just trolling. We have a duty of care to each other. You know? As anyone that's part of the Bolt compliance spec, we work together as much as possible. I consider Evan a good friend, Lalu as well, Con, and, Sueb. So, you know, we all, like, are in this community together, and I think I also am a bit guilty of this stirring the pot sometimes when, you know, tempers get flared up about decisions based on upgrades or what priorities should be. But at the end of the day, we're in this together, and I really value everyone contributing to the Bolt specs.
[01:42:15] ODELL:
I just wanted to say, you know, as a as a non dev, who's been running lightning, specifically l and d for 3 years, extremely well connected node, nearly 200 channels. I had more than that at one point. I have no idea who any of my counterparties are. Part of the reason why I was running it was to see if I would lose money, and I'm pretty sure I haven't lost money yet. So, yeah, cheers to that. Thank thank you for your service.
[01:42:52] Unknown:
I mean, Matt, you kinda too.
[01:42:55] Unknown:
Oh, yeah. Go ahead. Matt, you kinda brought it up earlier. We're saying, like, you know, we're not really in an adverse sale environment at all. Barak's is, like, quote, unquote attack is, like, signaling that. But, like, I mean, we definitely aren't. I I, like, when I went to Nashville, like, earlier this year, my lightning node went down on the plane over, and that was, you know, I never I don't lose any money either. So, like, people could have clearly stolen from me and didn't. You know, it's it's definitely not a non air Australian environment and I don't know. Hopefully hopefully is is bad, but, like, we need to get there. Like, if we're in there, that means it's like lightning is a lot more legit. It's not just like, you know, attacks or trolls anymore. It's like actually, like, people trying to steal money and potentially profiting from it.
But I don't know if we're there yet, but it'd be cool to actually see real feel like you know, because, like, in the DeFi space, something like this is like a full time job of people Bitcoin projects. So it'd be, you know, scary but also cool to see people doing the same thing in light.
[01:44:05] Unknown:
Yeah. I think we'll get there eventually, but I'm not in any rush.
[01:44:09] ODELL:
Thanks for the conversation, Paul. I I actually have to dip. I've gotta go to a dinner. Which is cool. Yeah. I was the Matt, I was about to wrap up. Thank you for joining us. You wanna just give us your final thoughts real quick before you wrap?
[01:44:23] Unknown:
You know, I don't I don't know if I had any specific final thoughts. No. These issues are tricky. There's there's a lot that goes into them. There's there's usually no no clear answer. There's there's better things. There's worse things. But I just, you know, keep keep in mind that that as humans, we wanna we wanna help each other, at least as as non you know, we we we should just, you know, do do what you think is right for the for the people, not for the vendor or anybody else. But thanks for having me on. Out. Thanks, Matt. Appreciate you. Evan, final thoughts.
[01:44:54] Unknown:
Yeah. So, you know, I think there's a great discussion today. Barak, I think, you got your taste of the dark side. I hope moving forward, you consider coming back to the light. I I know our community has a lot of values of, like, anarchism, but that still requires cooperation for, you know, the benefit of the community. And, you know, ultimately, I think you're a really talented guy. Just, we hate for you to keep going down on this path and, there's positives to this, but, you know, it's not something I would like to see happen again.
[01:45:33] ODELL:
Thanks, Evan. Vivek, final thoughts.
[01:45:40] Unknown:
Thanks for having us on to even discuss this, man. I know it's kind of a touchy subject. Wanted to thank everyone that actually joined. Felt it was great that, we had Evan and, Matt Corallo here, especially. Ben, fix your Internet, and thank you all freaks for listening to us. Thanks, Vivek. I hope,
[01:46:01] ODELL:
I really want dispatch to be a place where people are comfortable having open discussion, and I really like wrangling people for these these round tables. Ben, final thoughts.
[01:46:12] Unknown:
Thought, fuck Google Fiber, and, you know, that I'll I'll stick with that.
[01:46:20] ODELL:
Thanks, Ben. Tony, final thoughts.
[01:46:24] Unknown:
Great discussion, guys. Yeah. I I think I think all, you know, all things like this make Bitcoin better in the end. So, Barak, if you wanna coordinate on more dark side things, let me know.
[01:46:40] ODELL:
Thanks, Tony. Barak, final thoughts.
[01:46:42] Unknown:
Yeah. I I I personally see Lightning as a better software, and it has a long way to go. I personally think 5, 5,000 Bitcoin is a joke number. I still see lightning as a playground. And, yeah, I love breaking things and I besides, I believe, attacks help Bitcoin, helps, hardening the the software, the infrastructure for long term stability. I'm not saying this in defense of my actions. I'm just saying this as thought as my as as my thought thought consideration. My only motive was for fun, and I guess, I'll I'll dig more into codebase and see if there are any other exploits.
[01:47:24] ODELL:
Thanks, Barack. I appreciate you. You're always welcome on dispatch, and I I hope you and Tony continue to, attack me. I do appreciate it. To all the freaks, I wanna thank you for joining us for this conversation, particularly the freaks who joined us in the live chat. I know it's a Friday evening. I know it's a crazy week, but you guys make it special. Once again, dispatch has no ads or sponsors. It's funded purely by you guys. There's a lot of value for value haters out there, including some people I've recorded podcasts with in the past. I think we can prove them wrong. I think most people are just half in, half out and not actually trying to make value for value work. So I do appreciate you guys seeing seeing the stats flow and seeing dispatch consistently on the top of the fountain leaderboards really makes it all worth it. I appreciate you all. Once again, it is a bear market. If you can't contribute stats, sharing with your friends and family helps, subscribing in your favorite podcast app helps, Subscribing on YouTube, Twitch. I mean, don't use YouTube. But if you are gonna use YouTube, subscribe to the channel.
It it is appreciated. Every little bit helps, and, it keeps us going forward. I wanna say to all the freaks, I know it's been a long week. It's mostly Shitcoiners playing Shitcoin games. Fiat Maximal is playing Fiat games. Bitcoin remains unaffected. I love you all. We're gonna keep moving forward here. We can't be stopped if we if if we work together. So I appreciate everybody. Have a great weekend. Stay humble, stack sets.
Introduction
Supporting the podcast through podcasting 2.0
Introduction to the panel and topic of discussion
The default values and configuration options for HTLCs and commitment outputs
The process of responsible disclosure and how to proceed when discovering a bug
Responsible disclosure and user risk
Vendor approach to disclosure
Duty of care and disclosure
The importance of responsible disclosure and communication in the lightning network
