“Blockchain” is one of the software industry’s hottest topics. As such, it’s an automatic discussion subject between distributed systems engineers from across the industry, and especially amongst those who build cloud-scale (or planet-scale) platforms. While most platform folks I’ve talked to about it are perfectly happy to help with frameworks that help selling their respective product or services to customers who are committed using blockchains, and some of those engineers even rode the crypto wave to make some hay, I have a hard time finding people in the distributed systems platform community who believe that blockchain as a universally applicable data structure is even remotely as significant as the hype wants to make us believe.
Blockchain in an architectural nutshell
Up front, allow a brief summary of blockchain untainted by the tribal lingo of the cryptocurrency community. A blockchain is an append-only log structure where each new log record (block) refers to the prior record as well as to its own contents with a “hash”.
A “hash” is a cryptographic function that can compute a “summary” number from a large block of data such that if anything in that large block of data changes, the “summary” number will be different. With a hash, you can therefore verify that data hasn’t changed by computing the “summary” number again over the data block using the same algorithm. A special quality of a cryptographic hash algorithm is that you can’t guess the content of the large data block from just looking at the hash.
Any blockchain block is signed with a hash that is computed over the prior block’s hash and the contents of the current block. Including the prior block’s hash ties the current block to that prior block, and, in consequence, to all blocks before that, which is how the chain is formed. Since the chain is shared, everyone can compute the hashes along the chain and can trivially detect blocks that were tampered with. No data can be removed from the chain because that would invalidate the entire chain. Therefore, if a fact is recorded on a chain, the recorded fact can’t ever be later denied, because the chain itself protects that fact from tampering. This is called “non-repudiation”.
For a blockchain to progress, new records are appended. For many uses of the data structure, there’s competition on who gets to append the next record. Bitcoin, for instance, maintains a singular ledger (one global log) that tracks Bitcoin ownership by ways of recording transactions. Transactions are recorded in the log by third parties. Those get paid for doing so.
Some infrastructures even allow constrained code execution in the context of a transaction to enforce certain policies (“smart contracts”).
The process by which consensus is formed about who gets to process and record a (block of) transaction(s) is called “mining”.The currently most common mining model is to solve a cryptographic puzzle that involves guessing the value of an element of the current block (the “nonce”) such that the resulting hash falls into a certain numeric range.
Since the hash algorithm is cryptographic, it is expressedly designed to make prediction of that value impossible, meaning that a random guess is as good as any: it’s a lottery. Whoever comes up with a fitting solution first and gets a majority of the network to proceed with their block as the latest log entry “wins” the associated reward. This is called “proof-of-work” (PoW); the miner proves to have actively participated in the lottery by doing hash computations.
The consensus about who gets to append the next log record can also be formed in many others ways, but for the Bitcoin use-case, the PoW model is architecturally a very clever choice for getting the log to converge to a single global structure without any central decision making or trust relationships.
The downside is that it’s tremendously wasteful, because solving the cryptographic puzzle is very hard and requires many (think: many trillion) of random guesses and computations. An emerging alternate model is “proof-of-stake” (PoS), where consensus is formed by a deterministic algorithm that picks the next block appender from a set of candidates, and the candidates nominate themselves by putting a “stake” (funds) into a pool and get punished by having their stake forfeited if they’re caught cheating. The reality of that theory is a bit tricky, because this consensus concept is itself based on having to settle transactions or it requires at least partial trust.
Many characteristics of the blockchain model explained above are obviously quite attractive, and many people who first come across concepts like a replicable append-only log, cryptographic non-repudation, and multi-writer consensus may believe that these are unique blockchain characteristics. They are not.
Distributed Systems View
Software engineers at the center of the distributed systems platform community understand the qualities of append-only logs, understand signatures and non-repudiation, and understand consensus protocols. They understand that those are a few building blocks from a large toolbox and that there are choices for each.
Append-only logs, to pick one of these building blocks, have become extremely popular over the last decade, in the form of event-ingestion infrastructure and architectural patterns like “event sourcing”. The great advantage is that all records in an append-only log are immutable after they have been written. You can make unlimited identical copies of a log without having having any consistency concerns except transfer latency imposed delays. Append-only logs scale limitlessly.
A “blockchain” is a specific composition of specific choices for these building blocks that is suitable for a fairly narrow set of use-cases; but for those use-cases it’s a brilliant composition.
The blockchain-specific consensus models are tailored to:
- all-around lack of trust where it is expected that everyone has an incentive to cheat and will cheat if given the opportunity.
- convergence to a singular global log.
Convergence to a singular global log is required when ownership of funds from a singular global pool of funds needs to be tracked and all-around lack of trust rules out partitioning of the pool.
Convergence to such a singular global log with many candidate writers and many replicas makes it hard to reach consensus about the next writer and for that consensus to be propagated such that any subsequent writer appends to the “current” consensus top of the log. “Proof-of-Work” or composite “Proof-of-Work plus Proof-of-Stake” solve that very cleverly: by being intentionally slow.
The Proof-of-Work lottery is probabilistically set up so that a singular mining winner can emerge and its winning block can be propagated throughout the network before another miner finds a competing solution (there can be many valid ones). The enabler for that is a time window.
The tradeoff the “nobody is trusted” blockchain model makes is that it literally trades trust for time. Because nobody is trusted, it runs a fair lottery game from which a winner emerges eventually after solving a hard problem, and solving that hard problem takes non-trivial time. With Bitcoin, that is the cumulative time all active miners spend doing compute work on that block, and the problem is hard enough for the absolute time spent, with all parallelism of the competing miners, still being long enough to allow for the chain to converge regularly. The average blocks per hour stats on bitinfocharts.com show a rather leisurely pace for many popular blockchains.
Some blockchain implementations, for example Ripple (PDF), choose a different set of tradeoffs and introduce partitioning and partial trust for their consensus model. That, of course, changes the game completely:
Trust
Once parties trust eachother to faithfully collaborate, the existing consensus models that we all use to build hyper-scale cloud systems become applicable and those can mostly resolve even complex and contested consensus problems in a few milliseconds at negligible compute cost, and largely gated by network latency.
Even if there’s no all-around trust, there’s often a neutral party that can supervise a transaction of two parties that don’t trust eachother, but who each trust that party.
“Sidechains” and “Private permissioned blockchains” are playing that trick.
However, once there’s a trusted neutral party, that trusted neutral party can already establish non-repudiation by forcing authentication/authorization along with a content signature and only ever allowing appends. That can be done with any regular current-technology database that the neutral party maintains and protects. A signature chain like described above can be done on any existing database with common crypto alogorithms.
The world’s economy today is built on the very principle that accounting records are both safe from deletion and immutable in digital accounting systems. There’s a clear sense of order. They are written to audit logs and are shared in statements for non-repudiation proof. The neutral party is a bank, or a gaming company that manages your online game world’s virtual diamonds stash.
The cryptographic chaining of records is a good idea to strengthen the immutability of the ledger as a whole, but it’s not really superior to holding a ongoing full copy of the ledger in safe escrow, e.g. at a neutral party continuously auditing the bank.
The single global log requirement and therefore the global consensus problem completely falls apart in all cases where the problem is partitionable. That is indeed true for most problems, otherwise nothing would ever scale.
Once you can use partitions, the consensus scope shrinks to the scope of the partition. When you transfer money from your bank account, the initial scope is just your bank, with the initial transaction to a clearing account. An account transfer at that scope can be processed in milliseconds with only local coordination. Subsequent transfer hops need at most peer-coodination.
A “sidechain”, by the way, is a partition of a global chain that acts as a clearing account.
For those familiar with (shall I say “classic”?) distributed systems architecture, it’s amusing to see things like “sidechains” with local trust relationships emerge, because they are nothing but a reaffirmation of the partitioning principles foundational for a functional economy.
Decentralization
Yes, the combination of decentralized operation, variable trust, and non-repudiation is very attractive. Decentralization is a major argument brought forward in favor of blockchain in general, and the “centralization of the banking system” is a major argument for digital currencies.
There are easily 15,000+ banks globally even by conservative counts, and countless more businesses that maintain various ledgers. Those banks and businesses are distributed across economic regions, operating under different rules and currencies. That’s hardly “centralized”.
“Centralization” is not when 15,000 organizations are federated such that you can transfer funds between them.
Yes, you need a banking license and audit in the local jurisdiction to be a bank. Because, as we see in the wild-west of crypto, some people are very happy to separate other people from their funds and dash. Regulations largely exist because of historic precedent and “banks” terminally separating people from their money has happened before.
“Centralization” is also not when people who have other things to do than studying and tracking the details of financial transactions and markets are relying on trusted pools of experts on such matters (“banks”). It turns out that market forces do a good job at decentralizing, for instance, the task of automobile repair and maintenance such that it scales to hundreds of millions of vehicles on the streets globally, but that decentralization is not so total that everyone repairs their own vehicle.
Total decentralization is impractical. The market already does a decent job at generating decentralized structures where needed.
Digital currencies
The specific combination of well-understood architectural building blocks that make up “blockchain” is very well applicable, but nearly exclusively applicable to all-around trustless global ledger accounting problem (e.g. “coins”).
It’s worth spending a moment thinking about how and if that solution improves the world. Crypto currency proponents take a technical position in comparisons with “fiat” currency, saying that the value of state-issued currencies or gold is just as imaginary as that of digital coins or tokens. Even if one accepts that being correct, the value of “fiat” currency or gold or tangible assets has emerged from broad societal consensus and those values are intertwined.
The value of crypto currencies is, not surprisingly, always expressed in reference to that established value system. That value system is quite elastic, politically partitioned into economically coherent regions with their own currencies, which can fluctate against each other to even out the value system balance across the regions. Central bank notes are, by far, not as perfectly secure as a digital transaction, but cash transactions are instant, anonymous, and cheap. The cash circulation even expects and is robust against some level of forgery. Forgery is just hard enough and penalties are drastic enough to not make it a worthwhile or sustainable at scale.
People delegate the handling of their money and money transfers to institutions (and people at those institutions) who they trust to do that right. The level of perfection aspired to by digital currencies is merely interesting geekery; the world is doing fine handling imperfections and resolving disputes and penalizing misbehavior through rules and courts. The foundational assumption of all-around distrust is not how the world works, and arguably not how it should work.
The reality of Bitcoin is that it is the application-specific token of the program code in this repo. It’s as meaningless outside the context of that code as “Age of Empires: Castle Siege” gold coins that I can use to build fortresses faster or “Star Wars: Battlefront II” crystals that I can trade for better weapons are meaningless outside the context of the respective game.
Even if alternate implementations exist, the reference implementation defines what “Bitcoin” is. The committers to that repo are the Bitcoin central bank. The same applies equivalently to other blockchain based currencies and their many clones.
Beyond digital currencies
Any set of requirements for a distributed systems solution that does not look exactly like an all-around trustless global ledger accounting problem is quite likely better addressed using a combination of elements from the broad toolset that exists across the distributed systems platform landscape today that looks quite different from the digital currency composition.
That solution might include an append-only log. It might include digital signatures. It might include strategies to enforce non-repudiation. It might include consensus building and leader election. And it will likely be much less complex and use more robust and proven technology building blocks than a pre-built “blockchain” framework.
I do recognize the attraction of micropayments transfers in context of a transaction such that the submission of some job to be performed and the transfer of the funds paying for the performance of that job become one. I also recognize that there’s a marketplace opportunity allowing parties to compete for execution of that job in a network. An oft-cited example are blockchain-driven storage services.
Being someone who runs platform, I do know, however, that cost is just one of very many aspects that my customers consider, and most of them rank quality of service concerns like security, availablity, reliability, recoverability, and compliance far higher, so a bidding contest falls short of their needs. Customers are looking for trustworthy capabilities they can bet their business on. One there’s trust and a relationship, the game changes. My employer’s platform has transaction-level billing and so do our competitors. Each of those platforms record many millions of billing-relevant transactions per second, into append-only logs, with data attributable to subscribers via signed tokens based on keys tied to subscription and resource scopes. If you trust AWS, Azure, IBM, and your favorite storage start-up to keep your data according to a set of quality of service criteria, you can get transaction- and per-resource level billing from each and can opportunistically store data where you get the best price this month if that’s what you need. That works today, without a blockchain.
PS: I’m not an ideolog on this matter. If you’re convinced that you need a blockchain, I’ll surely help with the communication path if asked. I have the finest shovels.
PPS: I’ve been told a few times that tech XYZ must obviously be the right path for the future, because thousands of intelligent people and billions of dollars of investments stand behind it. Engineering history doesn’t support that claim. Exhibits A and B: The civilian Vertical Take-Off and Landing (VTOL) and Supersonic Transport (SST) hypes of the 1960s. Hence the picture.
PPPS: This is personal architectural analysis unrelated to my place of employment.