Jamie on Software

December’s Solidity Fridays with transmissions11 is really excellent, and I think a very high-leverage way to learn how good Solidity actually gets written. When the guest is especially good, as with transmissions, I put Solidity Fridays in the same sort of category as Destroy All Software – teaching by communicating models and patterns of thought, rather than regurgitating tutorial content.

Anyway, here are my notes in case somebody else finds them useful:

• When emitting events, t11s emits instances of the contract rather than the address directly; the compiler will swap it out for an address anyway, but this approach gives you greater type safety

• Call CREATE2 to revert if the contract has already been deployed, by adding a salt of the underlying address to the deployment

• Addresses are 20 bytes, salts have to be 32 bytes. So we call fillLast12Bytes to add the remaining bytes (provided in Solmate’s bytes32 library)

• The gas cost of > is equivalent to != when the comparator is 0, more expensive otherwise.

• Illustration of his opinionated approach to smart contract development: “the performance of the code for users who aren’t stupid matters” - that’s why the [[ERC20]] implementation in [[Solmate]] doesn’t stop you from transferring into the contract’s address. “I’m not raising the cost for everyone else”

• fdiv is like division, but accounting for the bases. ‘Scale this down by the contract’s base’. Multiply the numerator by baseUnit and then divide the numerator * baseUnit by the denominator. Also checks for overflow on numerator * baseUnit, since overflow isn’t protected in assembly calls.

• In general, keep external calls all the way at the end of the function – including after any events are emitted – to make reentrancy more difficult.

• More important with eg ERC777 since eg safeTransferFrom might allow arbitrary code execution (https://eips.ethereum.org/EIPS/eip-777)

• Removing things from the end of an array is significantly cheaper than from the beginning, since in the latter case you have to move everything over.

• uint256 currentIndex = withdrawalQueue.length - 1; for (; ; currentIndex--) is better in this case than initialising the uint56 i = withdrawalQueue.length - 1 since we’re only doing effects (the tx will revert by underflow automatically), so we don’t need to check the length. Saves gas.

• Add a trusted boolean to the strategies, which is then checked on deposit and withdrawal. Makes it easier for EOAs to manage vaults without having to be wrapped in some other contract. Also makes it possible to disable withdrawal from strategies easily if they’re malicious in some way.

• Two reads to the same struct from getStrategyData[strategy] has no extra gas cost, since it gets optimised by the compiler into one single SLOAD. & makes it clearer to read by a dev where it’s coming from.

• Use unchecked when you know you won’t underflow or overflow, and so can therefore do without the safety. Saves gas.

• unchecked isn’t leaky - it won’t uncheck in nested function calls

• The implementation of Compound’s cToken is a little funky, since in a lot of places function calls return an error code rather than revert. So sometimes you need to require(cToken.blah() == 0) to ensure it succeeded.

A lot of spacey content this month, and a lot of crypto, as I left my old job at Pactio and moved into crypto full-time:

Finally got round to reading Values by Mark Carney. Seesaws from economic theory to memoir in a not-uninteresting way. Carney writes well, but sets things up in such a manner as to make his premises seem more interesting than this conclusions. A very safe book. I imagine he’s going to run for public office in Canada some time soon.

Also enjoyed The Power Law by Sebastian Mallaby. Clean writing, thoroughly researched.

Curricula for self-teaching maths and physics from Susan Rigetti.

I wrote a couple of posts on being an enthusiastic amateur.

The first test image from the James Webb Telescope, of the star HD 84406 is pretty spectacular (and even more so the more you learn about it.) You can clearly see the spiralling of the galaxies in the background, each one comprising on average 100 billion stars, and many of them billions of lightyears away. The scale of space is very hard to comprehend.

Nadia Eghbal is writing again, which is always a joyous event, this first new essay a gesture toward a broader project on philanthropy and the tech industry. Her prose is both incisive and imagistic, twisting and deforming ideas in the best way possible, finding their veins, snapping them like kindling.

An essay on infinite ethics, an approach to ethics that takes the existence of infinites seriously, and how infinity fits into the logical structure of existing mainstream ethical theories.

All of physics in nine lines. I’m surprised the basic theoretical scheme of physics is so parsimonious. (Although it might not actually be that parsimonious and this is expository slight-of-hand. What, for instance, explains why there are 27 constants?)

A fun collection of weird ERC-20 contracts, mostly exploits or incompatibilities with conventions.

I enjoyed watching this episode of Solidity Fridays with transmissions11. He articulates trade-offs very well. I took voluminous notes that I’ll type up soon.

On top of my normal reading, I listened to three audiobooks this month. The first, Spacefarers by Christopher Wanjek, is freely available to Audible subscribers, and a smart and deeply technical look about the next thousand years of spaceflight.

The second, The Planets by Andrew Cohen and Brian Cox, is a book about the history and physics of the Solar System, a companion to the 2019 BBC television series (which is itself really excellent.) Samuel West’s narration is extremely good, and Cohen is a talented science writer.

The third, also available for free on Audible, was a collection of Scientific American articles about Exoplanets. The article format is helpful, and the narrator’s voice is just monotonal enough to fall asleep to.

Emily St. John Mandel wrote a series of notes on GoodReads, discussing various passages from her excellent novel Station Eleven.

Vitalik on the roads not taken.

The user experience problems of quadratic voting. It’s easy to evaluate an approach to some problem in terms of its technical feasibility, or how attractive it is with respect to various theoretical constraints. A lot of the time, its success hinges simply on whether people can understand it.

A very, very good blog post on NHS performance. We need more LessWrong-style analyses of British government policy.

The Safari Web Extension mechanism is an important step forward for iOS apps – and a significant part of my current project, on which more soon – but the documentation is a little fragmented and the tooling isn’t nearly solid enough just yet. Developing these new extensions is challenging.

A small contribution to changing that:

By default, extension files that aren’t explicitly whitelisted in the manifest.json file are inaccessible from the browser. One common use case of the content.js script is to inject scripts into the same execution environment of the active web page. However, no such injection is possible unless the loaded file is whitelisted.

The manifest.json spec defines a web_accessible_resources parameter, which allows extensions to whitelist resources for access from the browser.

Thus, the following whitelist:

"web_accessible_resources": ["images/foo.png"]

Allows you to generate the resource’s URL with the following code:

browser.runtime.getURL("images/foo.png")

Which gives you the following sort of URL:

safari-web-extension://2AB33852-D69B-4ED6-99AD-A4839DFEC7ED/images/foo.png

The ID directly after the protocol in that URL is generated on a per-browser-instance basis, so you can’t guess it ahead of time. You have to call getURL to generate it.

There are two important things that the API documentation doesn’t specify, which took me some time to figure out:

• browser.runtime.getURL is only defined inside the extension’s execution contexts. So the content.js and background.js files are fine, but the webpage itself is not. If you want to use it in the webpage context, you’ll need to generate the URL in the extension somewhere and communicate it via the window.postMessage event trigger.

• Any values used in the web_accessible_resources parameter must be nested under a subdirectory. If you try to call a top-level file (such as getURL("foo.png")), the URL will generate fine, but the file itself won’t be loadable. The browser will simply report it as inaccessible.

Hopefully this saves somebody else some time.

Blockchain ‘wallets’ are generally just pairs of public and private keys with some UI wrapped around them.¹ We take the private key, and use it to derive the public key, which we then use to derive the wallet’s address.

What’s important is that the process of derivation is very difficult to reverse, in the same way that a hashing function is difficult to reverse: the chance of you guessing the private key correctly at random is about the same as selecting one atom from all the atoms in the universe – and there’s no better way than guessing at random.² We can therefore use the wallet address publicly, being able to prove mathematically that we own it, without ever leaking information about the private key we used to generate it.

This works great, until you need more than one wallet. You might be concerned about privacy, or you might want to keep certain types of transactions separated for tax or other organisational reasons. If you have more than one wallet, you need to manage more than one set of private keys, back each key up separately, store each key separately, restore each key separately, etc. This presents a user experience problem: it is inconvenient, and clunky, and pushes a lot of the infosec responsibility onto the user. That might be acceptable for a bunch of nerds or anarcho-libertarians, but isn’t going to cut it for the median user.

The agreed-upon solution to these UX problems is Hierarchical Deterministic (HD) wallets, proposed in the Bitcoin BIP-32/44 standards and used by most other chains. This post considers this standard, how we’re not meeting it, and why it matters.

The plan, in three sections:

• A short overview of what HD wallets are. Feel free to skip over this if you’re familiar with the spec already.
• A discussion of how common wallets are not meeting this standard
• A discussion of why that matters, and what we could do about it.

HD Wallets

Hierarchical Deterministic (HD) wallets take the basic derivation mechanism and encode structure into it. We take a master password – a single thing for the user to remember, to back up, etc. – and combine it with a path, a string following an a priori agreed-upon schema that allows us to generate multiple private keys from the same master password.

But it needn’t actually have much structure at all. You could simply take a master password and append 1, 2, 3, and so on, to generate different wallet addresses. This strategy would generate perfectly usable wallets with no obvious link between them. And since the generation process follows the same general sort of process as it does for the single-key case, the generation process produces hashed values that are similarly difficult to reverse.

We therefore only really need two pieces of information to calculate our wallet address:

• Our master password
• Some sort of seed

The master password is the user’s responsibility; it’s her input, her secret. What seed should we use?

One option is to let the user specify whatever sort of seed she wishes. But this doesn’t really solve our problem: instead of multiple private keys, we instead have to deal with a single password plus multiple paths. We’ve just given ourselves more passwords to remember.

Another is to do what I suggested above: append an incrementing integer to the end of it to generate different wallets. This is equivalent to giving ourselves more passwords, but at least there’s some rationale to it: our first wallet has a 1 at the end, our second wallet a 2, etc. It gives us some psychological safety: it means that our wallet is recoverable (assuming we can remember which number we used to generate it, or assuming we don’t mind iterating through a few guesses). This approach is fine, as far as it goes, but this is crypto, so, given the opportunity, we should make it more complicated.

A third approach is to develop a common standard for generating our seeds with more variables than just an incrementing number. This way, we can describe a tree structure independent of its values, embedding multiple values with which we might want to generate differing wallets. The benefit to this approach is that we can encode information about the purpose of the wallet into the seed itself, and then recover it later using our knowledge of those purposes without having to remember many arbitrary numbers. The standard gives us the template, and the purposes give us the values of the variables; all we have to do is fill them in. The other benefit to using a common standard is that wallet software can implement the standards too, so you don’t need to generate the wallets off-site somewhere.

This standard is called BIP-44 (it was originally a Bitcoin standard), and it presents this exactly this sort of predictable tree structure that we’ve been discussing. The goal here is minimises user input and maximise the number of wallets that can be generated with a single master password.

The standard calls the seed a derivation path, since it’s a path in a tree that we append to a master password and use the resulting string to derive a public address. The standard gives derivation paths the following structure:

m/purpose'/coin'/account'/change/index

And here’s the trick: most of these values are knowable by the wallet software, based on what sort of wallet you’re using:

• purpose is always 44'.³ They gave it a value to allow them to upgrade the standard if they wanted to.
• coin varies depending on the crypto network. For instance, coin = 60' is Ethereum mainnet, and coin = 966' is Polygon.
• account gives the wallet a degree of freedom to support multiple user accounts (c.f. to the /Users/username directory on your OS)
• change will generally be 0; it refers to whether the wallet should be used externally, or whether it should be use internal to the wallet for Bitcoin-based transaction change reasons. I’ve read somewhere that Ethereans sometimes use it, though for what I’m not sure.

The only non-guessable input value is index, which gives the user a degree of freedom to generate multiple wallets for under the same tree. This parameter is why the user can generate many wallets for a single password: she can keep incrementing index to generate more! It’s also exactly the same as my much simpler idea discussed previously.

These parameters then get put into the structure, like so:

m/44'/60'/0'/0/2

The structure then gets combined with the master password (or, more precisely, with a key generated from the master password), and users (or wallets) can vary coin, account and index to generate various wallet addresses.

Existing UIs and a Subtle Incompatibility

This isn’t a huge, bombshell-dropped discovery, I’ll admit it, but I’ve noticed that most wallets with support for both HD wallets and network switching don’t actually implement the BIP-44 correctly, or, at least, there is a tension between the model used for network switching and the model used for wallet generation.

Generally, what happens is:

• Users add a master password (or its equivalent in the form of a mnemonic phrase) from which the wallet derives a single keypair
• As far as I can make out, the ‘default wallet’ generated through this mechanism still uses the HD standard, it just relies implicitly upon the m/44'/60'/0'/0/0 derivation path (i.e. “give me external index 0 at account 0 for the Ethereum chain”).
• When the user switches between compatible chains – from Mainnet to Arbitrum, for instance – the wallet software uses the same wallet address and private key to sign new transactions. It just switches the RPC endpoint it uses to make the request.

If wallets were to follow the standard correctly, they would be varying the coin value when switching networks, generating different wallet addresses for use depending on the network being used. In other words, according to BIP-44 at least, there’s no such thing as a ‘cross-network address’ – and existing wallets ignore this subtle fact entirely.

I’ve been looking at how various different wallets handle this, and they all seem to do the same thing:

• Metamask’s network switcher is entirely independent from the wallet list, allowing the user to switch networks on the current wallet, even if that wallet was generated through a derivation path
• MyEtherWallet do the same thing, switching the network URL used for chain interactions and not (as far as I can see) adjusting the corresponding wallets.
• Similarly, there is nothing in the WalletConnect spec preventing this behaviour, meaning that any HD-compatible wallet software using the protocol facilitates wallet-independent network switching

The problem is not so much that nobody’s trying to follow the spec. The problem is that the spec is ambiguous with respect to the UI in which it’s being implemented. The community therefore has implicitly converged on this non-standard behaviour because of the ostensible UI benefits. This has created an implicit standard incompatible with the original BIP-32/44 proposals.

It gets even more confusing when you notice that there is a third, Ethereum-specific standard, EIP-601, designed to modify the BIP-44 standard for Ethereum use cases. From a brief google, I can’t see any mentions of 601 that aren’t merely links to the spec itself. But this ambiguity – what should happen to the valid wallet list when the user switches networks? – isn’t resolved by EIP-601 either.

This ambiguity is born because the BIP-32/44 standards were built around the assumption that the different networks a user might switch between were mutually incompatible. It didn’t foresee the rise of EVM-compatible layer 2s, and a range of dapps built to run on several of them concurrently, and therefore the capacity for the user to switch between them easily, in-app.

Why this matters, and what to do

Of course, this doesn’t seem like a critical problem – there are bigger problems we could be tackling, for sure. Indeed, there’s even something comforting about going from Polygon to Ethereum Mainnet and taking your address with you. It’s certainly convenient. But this isn’t what the BIP-32/44 specs say, and I think there actually are good reasons to obey them more precisely:

1. It makes it possible to upgrade the spec in the future. The standard can evolve safely, and those implementing it correctly are able to evolve without having to hack in workarounds for backward compatibility, and keep track of previous fringe behaviours.

2. It makes interoperability with other wallets easier. Wallet onboarding and offboarding isn’t a light matter; the more activation energy required to move to one wallet from another, or from no wallet at all, the more intimidating crypto as a whole will become to the marginal user. Problems at the tail-end often get publicised more than problems at the mean.

3. Not doing so undermines one of the main reasons to use HD wallets in the first place: HD wallets allow you to keep public references to different addresses separated, increasing privacy. A wallet address that comes with you cross-network just makes your transactions that much easier to track.

Fortunately, I don’t believe that the UI concessions made by existing wallet implementations need to be locked in. There are some steps that wallets could take today, such as triggering a confirmation model when changing networks, that would enable users to opt-out of the spec. Many users don’t knowingly use HD wallets at all; in these cases, the default behaviour could just clear the wallet list and regenerate using the standard specs on network change.

Or, alternatively, we could develop a new, more parsimonious standard to capture the semantics of cross-chain wallets, compatible with the current UI approach. One simple method would be to amend the current spec such that network = 0 means ‘no specific network’, allowing cross-chain wallets to be represented in the existing spec. If a network changes while a user is connected with a wallet known to be generated with network = 0, the wallet persists.

Either way, this is the exactly the sort of subtle incompatibility that could prove to be an increasing nuisance, compounded by the ongoing growth in usage of layer 2s. Our standards for network switching were designed at a time when the only networks we would switch between were testnets. Today, the UI implications of network switching are a lot more important. And, today, that is incompatible with one of the most useful standards we have for managing multiple wallets.

Multiple wallets, multiple networks, good UX. We don’t need to pick only two.