What Problems Do Blockchain Sharding Need to Solve?

Challenges of Public Chain

As we all know, the biggest problem that public blockchain platforms encounter is scalability. All the mainstream platforms are trying to increase transaction per second. Bitcoin network can process seven transactions per second and 15 for that of Ethereum. Slow transaction speed creates a major problem: since a vast amount of incomplete transaction has blocked the network, that leaves little room for any applications that support real-time transactions to come into fruition on blockchain. As the time for confirming payment lengthens, users also experience a great deal of inconvenience, which is precisely why Visa and Paypal credit card payment method are still more preferred than cryptocurrencies. Impressively, Visa can process up to 65000 transactions per second.

Horizontal Expansion and Vertical Expansion

In the early days, many public chains use vertical expansion as an attempt to overcome the challenge. For example, adjusting block intervals or its block size, all of which is to increase the functionality of a single chain by adjusting different parameters. This is similar to expansion for centralized system whereby one would need to use the fastest CPU or a bigger hard disk. Vertical expansion has one Achilles heel however, which is limited increase in functionality because there are only that much wiggle room within which one can play with the parameters. One instance for illustration is the limitation in the speed of global P2P web broadcasting. For blockchain, increasing or decreasing the interval between blocks or adjusting the block size will heighten the possibility in producing orphan blocks and thus wasting hash power. Another similar method is to use PoS or DPoS, such as the DPoS used by EOS, which allows only 21 nodes to produce blocks and hurt the level of decentralization of the network.

The idea of horizontal expansion of blockchain is to divide and conquer, which is based upon the sharding concept of traditional database. In the original sense, sharding divides up the database into shards and host different shards on different servers. In the scenario for public chain, shards take the data or throughput and then divide into shards whereby processing of these data or transactions are carried out. Through increasing the number of shards of the network, the throughput of the entire blockchain network can increase linearly. According to many, sharding is a highly viable solution to scale while not curbing on the level of decentralization and security of the network.

Types of Shards

Sharding is one of the solutions allows expansion on the chain. Different kinds of blockchain sharding can be ranked according to level of functionalities. Here is the rank starting with the weakest to the strongest: network sharding, transaction sharding, computational sharding, and state sharding.

Network Sharding

Network sharding is the most basic kind of sharding, which is to divide the entire blockchain network into several subnetworks of which each constitutes one shard. All shards within the network process different transactions in parallel. Developers need to invent a mechanism to confirm which node can be kept on which shard securely, so to avoid attacks from those who are in the procession of a majority of shards.

Transaction Sharding

Transaction sharding is based upon network sharding and take one step further to classify all transactions and route to different shards for authentication and packaging. The problems ighag arise are for example what kind of characters and which kind of transactions should be mapped to which shards.

Computational Sharding

Only a subnetwork within a network executes the computation which can complete the task with high performance.

State Sharding

State sharding takes the entire ledger information to store in different shards, which means to divide up the state of the blockchain into multiple states, each of which can process rather independently and also to interact with other shards seamlessly.

In fact, for state sharding, the node of each shard is building a subchain, of which transactions will only affect the local state of the subchain. Therefore, the authenticator on one particular shard only need to keep a copy of the local state and only need to execute and interrupt transactions that would affect the local state. This kind of division reduces the demand of the node for computational power, storage capacity, and bandwidth of the network. At the same time, such design leads to new problems, for example, usability of the data and the procedure for cross-shard transactions.

Comparison of different types of shardings

Network sharding is the foundation that transaction, computational, and state sharding build upon. Network sharding and transaction sharding can solve the bottleneck for computational power while state sharding can solve the bottleneck for bandwidth and storage capacity. Additional, state sharding only lowers the number of nodes required for authentication, therefore state sharding the most ideal format for sharding.

Implementation of state sharding implies implementing transaction and network sharding as well. In terms of difficulty in implementation, network sharding is the easiest, then comes transaction sharding, and then the most difficult is state sharding.

Challenges Encountered By Blockchain Sharding

The most pressing problem at hand is to maintain the atomicity while running a cross-shard operation. For example, different processes on different shards need to be run according to a certain timeline like to confirm the time sequence between operations, which is indeed a daunting task.

Secondly, how to counter low-cost, malicious attacks is another problem. After sharding the blockchain system, the computational power and rewards on the node shrinks proportionally according to the number of shards. If the network has one hundred shards, each shard will only have 1% of the resources of the entire net. Then, using 1% of the computational power of the entire network can fully control one shard.

Thirdly, how to prevent overheating of the shard. If one single shard becomes the target for many cross-shard transactions, then the target shards will rank these transactions in order to prevent overloading of block producers (BP) on the target shard.

Comparisons of projects related to blockchain sharding

At present, there are a number of projects that implement sharding technology. We have made a comparison table to differentiate sharding/cross-chain public chain platforms.

We can see that sharding projects has higher TPS, which can already satisfy the needs of the majority. In terms of compatibility, besides QuarkChain, other sharding projects provide limited support for consensus mechanism, virtual machines, ledger model, and token economics.

There are some other questions raised to us recently. For instance, what is the difference between QuarkChain, Polkadot, and Cosmos? All the three projects are related to interaction between chains. Including Ethereum 2.0, these four projects can be compared in terms of the following aspects:

Flexibility

Shard chains of both Cosmos and QuarkChain are most flexible and support multiple consensus mechanisms.

Interaction

All four projects use an intermediary chain to coordinate the interaction among shard chains, Beacon Chain for Ethereum 2.0, Roo Chain for QuarkChain, Relay Chain for Polkadot, and Hub for Cosmos.

Development Schedule

Both QuarkChain and Cosmos has launched their mainnet while Ethereum 2.0 and Polkadot are still under development.

Boson Consensus

Consensus mechanism is a basic component of blockchain. After introducing sharding to blockchain, consensus mechanism is adapting to such design correspondingly. Here we will introduce an important sharding consensus mechanism, the Boson Consensus. The Boson Consensus was first proposed by QuarkChain as a framework of consensus mechanism specifically catering for sharding. It can be applied to QuarkChain, Polkadot, and even ETH2.0 (still in the process of design). The Boson Consensus has been operating smoothly in the QuarkChain mainnet production environment.

Overall speaking, the Boson Consens is a multi-shard consensus mechanism that builds upon single chain consensus mechanism. Its structure consists of two layers, the root chain and shard chains. Shard chains is the same as single chain, which is used for keeping transaction data, increasing TPS and expandability. Root chain only contains the block headers and the root block header, which is used for validating the shard chains and for protecting the entire network.

The consensus mechanism used by root chain can be tailored to specific needs, such as PoW, PoS, or dPoS. The consensus mechanism of the shards follows the principle of rootchain-first consensus, coupling with consensus mechanism of that particular shard, be it PoW, PoS, or dPoS. Different shards can deploy different consensus mechanism. For example, shard 1 can use PoW and shard 2 can use dPoS.

What is the principle of rootchain-first consensus? When there exists forks in the shard chain, the system will pick the best fork by comparing the root chain of each fork (which is related to the consensus mechanism of the root chain). The fork with the best root chain will win regardless of whether the shard chain of that fork is better than the other fork. When the two forks have the same root chain, then the system would use the consensus mechanism of the shard chain to determine which fork is better and pick the winner.

From this description, we can see that in order to attack the system that uses the Boson consensus, one would need to attack the root chain and the shards simultaneously in order to take over the system, which dramatically increase the hashpower required for the attack and thus increase the difficulty to attack. We can also introduce incentive mechanism (for public chain) or root chain protection mechanism (that has limited nodes like alliance chain) to increase the security of the root chain, which further enhances the security of the entire system.

The Boson consensus has many advantages:

1. High compatibility, supporting customization of shard chains
2. Support cross-chain, cross-shard transactions
3. High level of security for all shards. When a shard is added to the network, all shards are protected by the root chain, which allows each shard to enjoy high level of security immediately
4. Dynamic addition of shards

For those who are interested in reading more about the Boson consensus, please read our article here:

http://kuaibao.qq.com/s/20180927G1JM1P00?refer=spider

To know more about how the Boson consensus can defend malicious attacks, you can refer to this particular article:

https://zhuanlan.zhihu.com/p/55842744

For enthusiasts who enjoy reading research papers, you can take a look at our white paper on the Boson consensus:

https://github.com/QuarkChain/pyQuarkChain/blob/master/papers/boson.pdf

This paper uses mathematical language to describe the Boson consensus and the existing single-chain consensus, in which you will find stimulating ideas!

The Advantages Of Developing Blockchain Using Go

Blockchain requires high-performance operations for computing hashing and encryption algorithms, processing large amount of transaction data, broadcasting, and storing data. Blockchain has high level of coding complexity and requires network programming and multi-platform support. Blockchain also demands a high level of maintainability and guarantee for its security. Using Go for development satisfies most of the technical requirements that we just mentioned above.

Firstly, in terms of execution efficiency, Go has much higher efficiency than the interpretive languages like Python or JavaScript. Go can run directly on OS, which is much faster than compiled language like Java that needs to be run on virtual machine. Because of that, Go is used for developing Docker and EVM. Go is designed for distributed computing, which means that microservices that is built using Go can easily scale to millions of requests. Goroutine supports development using Go. Goroutine has an initial stack of 2k, which has a much lighter weight than threads of other languages. Its stack can also dynamically expand or shrink based on current running situation. Go’s runtime uses cooperative adjustment, which maximizes the power of a multi-core machine.

Secondly, in terms of characteristics of the language, as Go is a compiled language, errors will appear during compilation and will be handled during the process which avoid errors from occuring during execution; such feature makes multi-person development and maintenance more secure and reliable. Having compiled into executable files, it is very easy to run the files and support cross-platform compilation and execution. On the other hand, Go also bears advantages of interpretive language such as short compilation time, short initializing time, easy to use, test, plan, and develop other scripts. Go is extremely expressive: it supports pointers and garbage collection, it includes closure, and it is object-oriented. Compared to C which is also an interpretive language, Go can realize the same functionality with fewer lines of code which makes it easier to maintain.

Lastly, from an ecosystem perspective, its open source and its Google heritage attracts increasingly more developers. Go has a concise syntax, easy for developers to learn and master. It is gradually accumulating more standard libraries especially for network connection, like rpc and grpc. Go comes with an array of powerful tools, including go fmt, go get, go build, and go doc; in particular, go fmt standardizes the formatting of the code, thus streamlines exchanges and maintenance. Go supports C; inside Go code, one can insert C code and makes use of the existing rich collection of C modules.

Several blockchain projects have developed in Go such as hyperledger Fabric, go-ethereum, both have provided frameworks and infrastructure modules (like rlp, rlpx, libp2p) for future developers. These open-source modules have been implemented and code-reviewed and simplify the process to develop future blockchain projects using Go.

Other blockchain projects that are developed in Go include Cosmos, Polkadot, Loom Network, Dero, GoChain, as Hyperledger Fabric chaincode which uses Go as the language for smart contracts.

Implementation of blockchain sharding using Go

There are several sharding projects that use Go for development. Here we will use the Go version of Quarkchain as an example to illustrate how QuarkChain implements sharding with this language. Here is a short bio for QuarkChain:

QuarkChain is the first public chain to have implemented state sharding. Thanks to its great design of sharding function, QuarkChian has realized a high TPS. At the same time, it makes customization available for four factors: consensus, token economy, virtual machines, and ledger model. QuarkChain has become a flexible, scalable and user-friendly blockchain underlying architecture with high security, decentralization, and high efficiency.

QuarkChain uses the Boson Consensus that we discussed above to allow each shard to enjoy the protection provided by the root chain and introduce a taxation mechanism and attract over 50% hash power to shield the root chain from attacks. QuarkChain also uses its proprietary PoSW consensus mechanism, which combines the best of both PoW and PoS. PoSW requires staking of tokens proportional to hash power, which increases the cost of attackers and prevent shards from being attacked early on when it has relatively weaker hash power.

The shard layer which is responsible for carrying out transactions has infinite expandability and ability to dynamically add shards. New shards can immediately enjoy protection by the hash power of the root chain. Each chain can has its own mechanism and can also issue its own native coin on the shard chain and define its own smart contracts. This also means that what we currently see in the space such as cross-chain, PoS, Defi, all these new concepts are compatible to the design framework of QuarkChain since the birth of the project.

Structure of QuarkChain

QuarkChain borrows the concept of cluster, which is often used in solutions for centralized large-scale distributed systems. Each node is one cluster. Each cluster has a master and one or multiple slave. Master includes root chain and provides Json-PRC interface and P2P interface for inter-node communications. Slave includes one or more shards. Between master and slave(s), and between slaves, communications are transferred using the GRPC interface.

When we take a look at the code, we can see that under the cluster folder, there are master folder, slave folder, and modules that both need which are config, service, and rpc.

Inter-Shard Communication

Different nodes in blockchain communicate using P2P protocol to reach consensus. Within QuarkChain, shards from different nodes would not establish P2P connection directly because as the connection of existing P2P protocol has not reached bottleneck yet, doing so would introduce a higher level of complexity to the code. Currently, the communication between shards are routed through root chain P2P layer. Here is the structure of root chain P2P message:

Inside the structure of the message, we have added the metadata attribute to indicate the chain to which the message is forwarded (root chain or some other shard). When root chain receives the message and confirms with the shard who is sending the message (the messenger), it will then forward to the receiving shard (the receiver). After receiving a response from the receiver, then the root chain would forward the response to the cluster of the messenger who originated the P2P request. During the whole process, root chain would not encode, decode, or authenticate the messages. Such design signifies that the exchanging process requires little computational power and thus can handle a large amount of messaging jobs between shards.

To allow the synchronization between corresponding shards, we have added some codes for shards to enhance such functionality.

Support For Heterogeneous Shards

For blockchain, the most important objects are transaction, header, block, and block chain. For heterogeneous shards, this means that the definition of these objects and its processing logic would be different for each shard. At the same time, blockchain has standard processing flow and thus we have created corresponding interface for each of these objects.

Here is the code snippet that defines IHeader, IBlock, IHashable

Here is the definition of transaction. Inside its definition, one can use TxType section to specify whether the type of transaction is EVM or UTXO and the code will compile differently based on the specification.

We have built interface for sync, process, verification, consensus objects. These interfaces will take in IHeader or IBlock as parameters. Such design modularizes addition of heterogeneous shards, transaction type (account, utxo), consensus algorithm, VM. This structure makes deployment of the Go code flexible with large scalability.

Cross-shard Transactions

QuarkChain has designed for each user to have an address on each shard. Therefore the flow for cross-shard transactions is divided into two simple steps.

1. Extracting deposit: this function extracts tokens from User A’s deposit enough for transferring amount and transaction fee expected (calculated by multiplying gas price with the price of start gas. This process takes place within shard X.
2. Execution and return funds: For extracting deposit, the process will first increase the deposit in user B. Or for visiting the smart contract signed and execute the code for smart contract C. After finishing transferring of token or deployment of a smart contract, the system will calculate the actual transaction cost and return any leftover tokens to user A’s address on shard Y. Such process will take place on shard Y.

In terms of code, after the part of “From” shard finishes execution, the cross-shard transaction will be sent to cluster that lives the “To” shard and will kept there. When root chain confirms the block that includes the transaction created by the “From” chain, then the “To” shard will transfer the deposit or deploy smart contracts.

Introduction of the Competition:

During last year’s competition, the winning team reached throughput that surpassed 55000 TPS. We have recently open-sourced the Go version of QuarkChain which has even better functionalities. We would like to test the functionality of the Go version through hosting this TPS competition with 3 BTC reward. It is easy to join the competition. You will only need 5 minutes to initialize the node. During the competition period, you can submit as many submissions as you would like and we will rank according to your best submission. First place for the TPS Peak Challenge will receive 0.5 BTC and 1 million QKC. The first to reach 100000+ TPS will receive 0.1BTC + 0.2 million QKC. There are other awards including participation bonus and referral bonus and we welcome all of you to join!

Regulation of the Competition

Before 12/24/2019 11:59pm PST, build a QuarkChain network and carry out load testing. We will rank the teams according to the peak TPS value and reward the teams respectively. Please read the details of the competition and submission of results by visiting this page: https://QuarkChaintps.typeform.com/to/Ve2FKs

Here is a video to initialize a cluster within 5 minutes: https://v.qq.com/x/page/v3024712r2f.html

If you encounter any problem anytime during your process, do not be afraid! Please go to our technical forum https://community.QuarkChain.io/ to notify us of your question or participate in the discussion. We will regularly reply to questions on the forum.