The Bitcoin Consensus Red Herring

Red Herring

Red Herring

In a recent post on bitcointalk, I stated that focusing solely on the forking risk created by alternative implementations is a red herring because, while there is certainly risk involved, it is merely a symptom of a more fundamental issue that all implementations, including Bitcoin Core itself, suffer. This statement seems to have confused a few people, so, in this blog post, I’d like to delve a little deeper into why this is the case, identify what the real underlying issue is, and offer a potential solution.

First, let’s establish a few baseline facts that, to my knowledge, everyone agrees with:

  • Every fully-validating node on the network must follow the exact same consensus rules or a fork will occur
  • The consensus rules are complex and contain various non-intuitive corner cases, some of which probably still have not been identified
  • Chain forks can be abused to create double spends and generally wreak havoc
  • Every new version of Bitcoin Core carries some level of forking risk (if there is any doubt about this, see the March 2013 fork that already happened)
  • Alternative implementations carry some level of forking risk

Continue reading

Monetas brings colored coins to btcd

This article is syndicated from the Monetas blog with the approval of the author, Justus Ranvier.

As a company with current and planned blockchain-synergistic products, Monetas is strongly invested in Bitcoin wallet development. After choosing btcwallet from Conformal Systems as our technical foundation for blockchain integration, Monetas is currently devoting four full-time software engineers to adding features and improvements to btcwallet.

One of those features is colored coin support. This article will explain colored coins, and how we are implementing them in btcwallet, and what capabilities these changes entail for both Monetas and Conformal.

colored-1

Continue reading

btcsim: simulating the rise of Bitcoin

btcsim: to the moon!
Over the past 4 months we have had 2 interns, Javed Khan and Michalis Kargakis, work on creating a system for simulating high transaction volumes with Bitcoin called btcsim. Since there has been and is currently a great deal of attention being paid to the issues surrounding mining incentives and block propagation, we figured that it would be interesting to investigate the less-discussed topic of high transaction volumes. There are a variety of claims about how the Bitcoin network will behave at transaction volumes approaching those of major credit card companies, e.g. 3,000 transactions per second (“tps”), so we used btcsim to put btcd and btcwallet to the test. After simulating the creation of blocks up to 32 MB in size, we have arrived at some interesting conclusions:

  • a 32 MB block, when filled with simple P2PKH transactions, can hold approximately 167,000 transactions, which, assuming a block is mined every 10 minutes, translates to approximately 270 tps
  • a single machine acting as a full node takes approximately 10 minutes to verify and process a 32 MB block, meaning that a 32 MB block size is near the maximum one could expect to handle with 1 machine acting as a full node
  • a CPU profile of the time spent processing a 32 MB block by a full node is dominated by ECDSA signature verification, meaning that with the current infrastructure and computer hardware, scaling above 300 tps would require a clustered full node where ECDSA signature checking is load balanced across multiple machines.

While the current mainnet rate of transactions (0.9 tps) are much lower than those tested using btcsim, this paints a relatively rosy picture for Bitcoin should the block size be allowed to increase.

Continue reading

BIP0064 – Not yet.

The Bitcoin Core team recently committed BIP0064, which adds two new commands to the protocol, getutxos and utxos. The getutxos command is used to request unspent transaction information based on the given outpoints, while the utxos command is the response.

The BIP was authored by Mike Hearn who also provided the implementation for Bitcoin Core.

Now, since this affects the Bitcoin wire protocol, btcwire needs to add support for this change, even though btcd will not implement these new commands. btcwire is the bitcoin wire protocol package that btcd uses. It has 100% test coverage, meaning it can find bugs before implementing the new API into btcd.

Why you do dat?

I wrote some code to test against Bitcoin Core. This code issues a single getutxos command for outpoint bd1f9401a9c284a04353f925276af62f23f452d297eb2cc582d037064b2a795f:1 on TestNet3. However, I made the command contain 50,000 requests for this outpoint, since I prefer test-to-fail instead of test-to-pass type development.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
package main

import (
        "fmt"
        "net"
        "sync"

        "github.com/conformal/btcwire"
        "github.com/davecgh/go-spew/spew"
)

const (
        pver   = btcwire.ProtocolVersion
        btcnet = btcwire.TestNet3
)

var (
        conn net.Conn
        wg   sync.WaitGroup
)

func readMessage() {
        defer wg.Done()
        for {
                msg, _, err := btcwire.ReadMessage(conn, pver, btcnet)
                if err != nil {
                        fmt.Printf("ReadMessage error: %v\n", err)
                        return
                }

                switch m := msg.(type) {
                case *btcwire.MsgPing:
                        pong := btcwire.NewMsgPong(m.Nonce)
                        err := btcwire.WriteMessage(conn, pong, pver, btcnet)
                        if err != nil {
                                fmt.Printf("WriteMessage error: %v\n", err)
                                return
                        }
                case *btcwire.MsgVerAck:
                // Ignore
                case *btcwire.MsgVersion:
                        verack := btcwire.NewMsgVerAck()
                        err = btcwire.WriteMessage(conn, verack, pver, btcnet)
                        if err != nil {
                                fmt.Printf("WriteMessage error: %v\n", err)
                                return
                        }
                default:
                        spew.Dump(m)
                }
        }
}

func main() {
        var err error

        host := "127.0.0.1:18333"
        fmt.Printf("Attempting connection to %s\n", host)

        conn, err = net.Dial("tcp", host)
        if err != nil {
                fmt.Printf("dial error: %v\n", err)
                return
        }
        defer conn.Close()

        wg.Add(1)
        go readMessage()

        nonce, err := btcwire.RandomUint64()
        if err != nil {
                fmt.Printf("RandomUint64 error: %v\n", err)
                return
        }

        // Build version message
        version, err := btcwire.NewMsgVersionFromConn(conn, nonce, 0)
        if err != nil {
                fmt.Printf("NewMsgVersionFromConn error: %v\n", err)
                return
        }
        err = version.AddUserAgent("test", "0.0.1", "")
        if err != nil {
                fmt.Printf("AddUserAgent error: %v\n", err)
                return
        }

        // Send version message
        err = btcwire.WriteMessage(conn, version, pver, btcnet)
        if err != nil {
                fmt.Printf("WriteMessage error: %v\n", err)
                return
        }

        // Build getutxos command
        get := btcwire.NewMsgGetUtxos()
        sha, err := btcwire.NewShaHashFromStr("bd1f9401a9c284a04353f925276af62f23f452d297eb2cc582d037064b2a795f")
        if err != nil {
                fmt.Printf("NewShaHashFromStr error: %v\n", err)
                return
        }

        // Add outpoint index1 50000 times.
        for i := 0; i < btcwire.MaxOutPointsPerMsg; i++ {
                op := btcwire.NewOutPoint(sha, 1)
                err = get.AddOutPoint(op)
                if err != nil {
                        fmt.Printf("AddOutPoint error: %v\n", err)
                        return
                }
        }

        // Send getutxos message
        err = btcwire.WriteMessage(conn, get, pver, btcnet)
        if err != nil {
                fmt.Printf("WriteMessage error: %v\n", err)
                return
        }
        wg.Wait()
}

When run, btcwire rejects the response due to the payload being larger than the bitcoin specification, which is 32MB.

ReadMessage error: ReadMessage: message payload is too large - header indicates 202306292 bytes, but max message payload is 33554432 bytes.

Add a few loops and extra connections in the code, and you can crash Bitcoin Core with out of memory errors:

************************
EXCEPTION: St9bad_alloc       
std::bad_alloc       
bitcoin in ProcessMessages()       

2014-08-26 21:02:48 ProcessMessage(getutxos, 1800004 bytes) FAILED peer=271
2014-08-26 21:02:49 

************************
EXCEPTION: St9bad_alloc       
std::bad_alloc       
bitcoin in ProcessMessages()       

2014-08-26 21:02:49 ProcessMessage(getutxos, 1800004 bytes) FAILED peer=275

So, this shows that Bitcoin Core is trying to send us a little over 202MB. The bug is that Bitcoin Core does not follow its own specification of sending messages that do not exceed the maximum message size of 32MB. Secondly, I also wonder if Bitcore Core should skip duplicate outpoints in the getutxos request.

Another issue this brings up is the amount of testing, or the lack thereof, that goes into protocol changes. It is worrisome that protocol changes get merged without accompanying extensive test coverage. This is why the Conformal team puts such a strong emphasis on complete test coverage to help catch such issues that easily go unnoticed by developers.

btcd won’t be supporting the command because it is completely unauthenticated and insecure as pointed out numerous times on the initial pull request. In response to these concerns, a section entitled “Authentication” was added to the BIP which attempts to address the concerns by simply calling them out along with some potential modifications that could happen in the future. However, we prefer to wait until the things necessary to implement this functionality in a secure fashion are already in place, which Bitcoin Core should have done as well.

Introducing btcrpcclient — Bitcoin RPC Made Easy

Bitcoin JSON-RPC

Bitcoin JSON-RPC

Have you been looking for a robust and easy to use way to interface with Bitcoin through the JSON-RPC API? We’ve got you covered!

We’re excited to announce btcrpcclient, a new Websocket-enabled Bitcoin JSON-RPC client package written in Go. This package allows you to quickly create robust Bitcoin RPC clients in just a few minutes.

Major Features of the btcrpcclient Package

  • Supports Websockets (btcd and btcwallet) and HTTP POST mode (Bitcoin Core)
  • Provides callback and registration functions for btcd and btcwallet notifications
  • Supports btcd and btcwallet extensions
  • Translates to and from high-level statically-typed Go types
  • Offers a synchronous (blocking) and asynchronous (non-blocking) API
  • When running in Websockets mode (the default):
    • Provides automatic reconnect handling (can be disabled if desired)
    • Outstanding commands are automatically reissued on reconnect
    • Registered notifications are automatically re-registered on reconnect
    • Back-off support on reconnect attempts

Continue reading

Btcd + getwork + cgminer = profit

Bitcoins

Bitcoins

We are pleased to announce that btcd, our full-node bitcoind (bitcoin core) alternative written in Go, now has support for the getwork RPC which allows it to function with cgminer!

We have extensively tested the code on testnet. The following is a sampling of the latest blocks we’ve generated on testnet using btcd + cgminer: 226713 226830 227228 227390 227393.

If you are simply interested in learning how to configure btcd to work with cgminer, you can view the setup instructions on the btcd wiki, however if you’re interested to learn about some of the nuts and bolts that make it all work, read on.

Isn’t getwork deprecated?

Before I dig into some of the details, I would like to touch on a topic that I’m sure some astute readers will undoubtedly be asking, which is “Why implement getwork when it has been deprecated in favor of getblocktemplate (BIP0022 and BIP0023)?”

There are a few reasons why we chose to implement it first:

  • Supporting getwork was a lot less time consuming than getblocktemplate proposals and software such as cgminer still functions properly with getwork. This means we were able to release support for mining more quickly by supporting getwork first.
  • The bulk of the work involved was the code to create and manipulate block templates which is used by both getwork and getblocktemplate. Thus it helps pave the way to supporting getblocktemplate proposals while allowing getwork to function in the mean time.
  • The current reference implementation (bitcoin core) still supports getwork. Even though it will likely be removed in the next version, one of the goals of btcd is to be a drop-in alternative that can be used with existing infrastructures.

Continue reading

Heart-not-so-bleed!

Executive overview

Cyphertite was not vulnerable to the Heartbleed attack.

Management overview

Disclaimer: These findings are true for the Cyphertite service ONLY. We are not making any statement about other people’s results and/or attack surface.

We did not receive an advance warning of the Heartbleed vulnerability. As soon as the news broke the Cyphertite team shut down all services and began investigating its exposure. We quickly established that the website was not at all vulnerable to Heartbleed (it uses OpenSSL 1.0.0f) but we used a vulnerable version of OpenSSL on the service itself (OpenSSL 1.0.1c). The service was patched and brought back online with 3 hours of the Heartbleed announcement, however, at that time, we did not know the extent of Cyphertite’s exposure.

Continue reading

2014 Summer Internships: hacking on Bitcoin with Go

We are pleased to announce our 2014 summer internship program at Conformal Systems. A short summary of the program is as follows:
hacking Bitcoin with Go

  • 2 full-time intern positions are available for the summer of 2014.
  • Interns will be expected to work on remote.
  • Interns can be from any country or timezone, but timezone may be a factor in intern selection due to synchronization issues.
  • Interns will be expected to speak and write in English at a conversant but not fluent level.
  • Interns will work on open source Bitcoin-related projects the entire summer.
  • Interns will be expected to develop exclusively in the Go programming language.
  • Interns are expected to have the “Minimum Recommended Skillset” from the code contribution guidelines for the btcd suite.
  • Interns can either (A) work on projects assigned by Conformal Systems staff or (B) “bring your own project” (BYOP) which we approve as part of the internship.
  • The internships are open to anyone currently attending undergraduate or graduate school, or anyone of similar age. Broadly speaking, we consider anyone age 15 and up to be eligible.
  • Pay will be roughly a “graduate student” level, approximately USD 2,000 (pre-tax) per month.

Continue reading

Transaction Malleability: No Shortcuts Allowed

The past couple days have been a roller coaster of events for the bitcoin world. First, MtGox suspended bitcoin withdrawals for a bitcoin “design issue that has been largely ignored”. Attackers took note of this and begin running malicious nodes to increase the frequency of the attack. To handle the network attacks, other big exchanges like Bitstamp temporarily suspended bitcoin withdrawals as well, while some businesses remain unaffected and continue operations as normal.

So, what have we learned?

Continue reading