btcwallet and btcgui: Wallet handling for btcd

Today Conformal is announcing alpha releases of btcwallet and btcgui, the wallet components of btcd, written in Go. We announced in a recent blog post that btcd, our full node Bitcoin implementation, was ready for public testing. We also announced that wallet functionality was being implemented separately and would be coming in the near future. Although our wallet daemon and GUI interface are not yet ready for production use, we feel they have progressed to the point where early adopters can begin testing their functionality on the Bitcoin Testnet network.

Our btcd blog post briefly discussed why wallet functionality is not a part of btcd. It highlighted various reasons why we believe separating wallet handling from blockchain handling improves on the integrated wallet design used by bitcoind and bitcoin-qt. This post will continue on that topic, further exploring the details of why a multiprocess wallet design was chosen, how such a design is beneficial to the Bitcoin community as a whole, and the implementation details this design.

The Wallet Daemon: btcwallet

To recap what was mentioned in our btcd blog post, all wallet handling for btcd is done by a separate process, btcwallet, rather than being integrated into the same binary as btcd. btcwallet acts as both a client to btcd and a server to any number of user-facing frontends. Separating wallet in this way provides far greater multi-user and multi-device support than the integrated wallet design used by bitcoind and bitcoin-qt. Although this design does come with its disavantages, such as increased development difficulity and user complexity, Conformal believes the benefits far outweigh any of these issues. A multi-process design allows chain handling and computing resources (such as disk space, memory, CPU cycles, etc.) to be shared between many individual users, each with their own wallet and devices.

Conformal also sees this design as being much more scalable and future proof, as well as providing a potential economic benefit as well. As the Bitcoin blockchain grows with every new block, it may be neither desirable nor feasible for individuals or small groups of users to maintain their own blockchain service with btcd. Instead, those with enough computing resources could rent out btcd connections to btcwallet users, charging a small fee for maintaining the blockchain for them. In the long run, as Bitcoin becomes more prevalent and both building and maintaining the chain becomes increasingly difficult, this design will prove to be much more scalable. Bitcoin users will maintain their own wallets on their own hardware, without requiring every user to purchase and run massive machines to maintain their own copy of the blockchain. To put it another way, separating wallet from chain provides the convenience and speed of Simplified Payment Verification (SPV) clients, while keeping most of the benefits and security of running a full-node Bitcoin implementation.

The Wallet Interface: btcgui

Along with btcwallet, Conformal is also releasing an alpha release of btcgui, our graphical wallet frontend. Just as btcd runs as a daemon, btcwallet also runs without any user interface and provides no direct method for a user to interact with their wallet. To provide users with a means of managing a wallet, a third process, called a frontend, connects to and provides a user interface for btcwallet. Frontends can take many forms, providing graphical, command line, or even web application interfaces for wallet access. Individual users are therefore able to choose their favorite frontends for whatever situation they find themselves in, without ever switching their wallet.

Separating the wallet from the user interface also has the practical benefit of allowing a user to share their wallet between multiple interfaces simultaneously. Rather than two or more processes opening and sharing a wallet.dat file, practically guaranteeing race conditions without ugly solutions like file locks, concurrent requests from multiple frontends can be executed by btcwallet safely. Changes made by one frontend are also sent as notifications to other frontends, keeping each frontend in sync with the wallet’s current state.

btcgui is the first btcwallet frontend to be developed, providing a graphical interface (based on GoTK3, our GTK3 bindings for Go) to interface with wallet, and is currently under heavy development to bring it up to (and beyond) feature parity with bitcoin-qt. Although it has been designed as a Unix client, it is possible to run btcgui on Windows. However, users of non-Unix platforms users may expect a native solution to be written sometime in the future.

Implementation Details

btcd, btcwallet, and btcgui all use websocket connections for inter-process communication (IPC). Some readers may be aware that bitcoind already runs an HTTP server to provide remote procedure calls (RPC) as a method of IPC, and includes its own API for JSON-RPC requests and replies. btcd includes this same RPC interface, leveraging our btcjson package to handle marshalling and unmarshalling of these API calls and responses. In addition to standard bitcoind RPC, websockets connections are run off the same HTTP server.

Why use websockets? Unlike Unix sockets, which are not portable, and standard HTTP, which is synchronous, Websockets provide platform-independent asynchronous IPC with little overhead (running off the same HTTP server as for standard RPC). As websocket communication is asynchronous, notification systems can be implemented. These notifications are used by btcd to notify wallets of (for example) new transactions sent to a watched address, or for btcwallet to notify all frontends when an account balance changes.

Wherever possible, standard bitcoind RPC methods are reused for communication, but in situations where this API is lacking, extensions (only available for websocket connections) have been added. These extensions (both requests and notifications) are implemented using our btcws package.

To save and open wallets to and from disk, btcwallet borrows the simple-yet-effective file format used by Armory. Bitcoin accounts are also implemented by btcwallet, but as Armory’s wallet format has no notion of accounts, one wallet file is saved for each account. In addition to the wallet file, two additional files (with a custom binary file format) are used to record transactions and unspent transaction outputs for account addresses.

Using Armory’s wallet file format also allowed us to implement a deterministic wallet. While the full details and benefits of this design are out of scope of this blog post, in short, this allows a single backup to contain every piece of information needed to recover any private keys generated in the future, so long as the wallet passphrase is known.

What’s Next?

While neither btcwallet nor btcgui are to be considered production ready yet, Conformal hopes technical-savvy early adopters will still take a look through the code, run some tests, and provide feedback. If you are interested, follow the installation and execution instructions for btcd, btcwallet, and btcgui.

Please be aware that as btcwallet and btcgui are both still under heavy development, using either on the Mainnet Bitcoin network is highly discouraged and currently disabled. Early adopters are requested to keep all testing on the Testnet network, reporting any bugs that are found. Once wallet handling has been throughly tested, Mainnet support will be enabled.

As of today, the following features are implemented and working:

• Creating encrypted wallets
• Locking and unlocking wallets
• Creating addresses
• Receiving transactions
• Creating transactions
• Updating account balances
• Rescaning for missed transactions

Several missing features are also planned to be implemented in the near future:

• Transaction lists showing all transactions to and from addresses in a wallet
• Address book for saving frequently used addresses
• Changing wallet passphrases
• Updating the unconfirmed balance based on transactions in btcd’s mempool
• Using and requiring TLS for all websocket communication

Testers are encouraged to verify implemented features work as expected on Testnet and report any unexpected issues using the Github issue trackers. Additional features will come in time, but for now the most important components have been written and must be throughly tested.