Dynamic-EC: an efficient dynamic erasure coding method for permissioned blockchain systems

doi:10.1007/s11704-023-3209-3

Front. Comput. Sci.

2025, Vol. 19

Issue (1) : 191101 https://doi.org/10.1007/s11704-023-3209-3

Architecture

Dynamic-EC: an efficient dynamic erasure coding method for permissioned blockchain systems

Mizhipeng ZHANG¹, Chentao WU^1,²(

), Jie LI^1,², Minyi GUO¹

¹. Department of Computer Science and Engineering, Shanghai Jiao Tong University, Shanghai 200200, China
². Yancheng Blockchain Research Institute, Hengyang 421200, China

Download: PDF(9962 KB) HTML
Export: BibTeX | EndNote | Reference Manager | ProCite | RefWorks

Abstract

Blockchain as a decentralized storage technology is widely used in many fields. It has extremely strict requirements for reliability because there are many potentially malicious nodes. Generally, blockchain is a chain storage structure formed by interconnecting blocks

A block is consist of a block header and a block body. The metadata is stored in block header and data is stored in block body.

, which are stored by full replication method, where each node stores a replica of all blocks and the data consistency is maintained by the consensus protocol. To decrease the storage overhead, previous approaches such as BFT-Store and Partition Chain store blocks via erasure codes. However, existing erasure coding based methods utilize static encoding schema to tolerant f malicious nodes, but in the typical cases, the number of malicious nodes is much smaller than f as described in previous literatures. Using redundant parities to tolerate excessive malicious nodes introduces unnecessary storage overhead.

To solve the above problem, we propose Dynamic-EC, which is a Dynamic Erasure Coding method in permissioned blockchain systems. The key idea of Dynamic-EC is to reduce the storage overhead by dynamically adjusting the total number of parities according to the risk level of the whole system, which is determined by the number of perceived malicious nodes, while ensuring the system reliability. To demonstrate the effectiveness of Dynamic-EC, we conduct several experiments on an open source blockchain software Tendermint. The results show that, compared to the state-of-the-art erasure coding methods, Dynamic-EC reduces the storage overhead by up to 42%, and decreases the average write latency of blocks by up to 25%, respectively.

Keywords blockchain Byzantine Fault Tolerance (BFT) erasure coding consensus reputation evaluation

Corresponding Author(s): Chentao WU

Just Accepted Date: 28 September 2023 Issue Date: 12 March 2024

Cite this article:

Mizhipeng ZHANG,Chentao WU,Jie LI, et al. Dynamic-EC: an efficient dynamic erasure coding method for permissioned blockchain systems[J]. Front. Comput. Sci., 2025, 19(1): 191101.

URL:

https://academic.hep.com.cn/fcs/EN/10.1007/s11704-023-3209-3
https://academic.hep.com.cn/fcs/EN/Y2025/V19/I1/191101

Symbols	Description
$n$	the number of nodes in a permissioned blockchain
$f$	the maximum number of malicious nodes
$k$	the number of predicated risk nodes
$m$	the number of parity fragments to store blocks
$Φ 1$	the threshold of reputation value for integrity nodes
$Φ 2$	the threshold of reputation value for risk nodes
$N i$	the $i$ th node in the blockchain system
$B h$	a block with height $h$
$C h$	the checksum of $B h$
$f i h$	the $i$ th fragment of block $B h$
$T i$	the classification of $N i$
$p i j$	the personal reputation value from $N i$ to $N j$
$g i$	the global reputation value of $N i$
$α$	the penalty for silent nodes
$β$	the penalty for dishonest nodes
$γ$	the reward for correct nodes

Tab.1 Symbols used in this paper

Fig.1 An example of full replication approach in a blockchain. Each node stores a replica of blocks, thus the storage redundancy is 4×

Fig.2 The encoding process of RS(2,2) code. The leftmost matrix is called encoding matrix, which encodes data nodes (

D 0, D 1

) into a stripe (

D 0, D 1, P 0, P 1

)

Fig.3 An example of BFT-Store in a blockchain. Multiple blocks are organized into a stripe, where only

2 ×

storage space is required

Fig.4 An example of Partition Chain in a blockchain. Each block is encoded into fragments, which are organized into a stripe, where only

2 ×

storage space is required

Methods	Storage cost	Redundancy (n = 10, f = 3)	Network Cost	Punishment
Full replication	$n$	$n ×$	$n 2$	$×$
Partition chain	$n n ? 2 f$	$3 ×$	$n 2 n ? 2 f$	$√$
BFT-Store	$n n ? 2 f$	$3 ×$	$n 2 n ? 2 f$	$×$
Dynamic-EC (Optimal Case)	$n n ? ? 4 f 3 ?$	$1.67 ×$	$n 2 n ? ? 4 f 3 ?$	$√$
Dynamic-EC (Worst Case)	$n n ? 2 f$	$3 ×$	$n 2 n ? 2 f$	$√$

Tab.2 Comparison of various redundant methods for BFT

Fig.5 The overview of Dynamic-EC. The Dynamic-EC can be divided into three modules: the Node Classification module, the Dynamic Erasure Coding module and the Adaptive Fragment Placement module

Fig.6 An example of node classification process. Each node maintains a personal reputation list and updates corresponding personal reputation value as it interacts with other nodes. For instance, if

N j

receives wrong message from

N i

, it decreases

p j i

. When

N j

receives correct message from

N k

, it increases

p j k

. The personal reputation lists of each node are periodically aggregated to produce a global reputation list by the EigenTrust algorithm. Finally, each node is classified into: honest node, risk node, and malicious node according to its global reputation value

Fig.7 An example of block encoding process via RS(

3, 1

) code. The leader node first splits the block body into three data fragments and encodes them into four fragments with a parity fragment. After the encoding process, it calculates the block’s checksum, which is an array of hashes of the four fragments

Level	Risk nodes( $k$ )	Parities ( $m$ )	Malicious attacks tolerated
Low	$0 ≤ k ≤ f 3$	$? 4 f 3 ?$	$? 4 f 3 ? ? f$
Middle	$f 3 < k ≤ 2 f 3$	$? 5 f 3 ?$	$? 5 f 3 ? ? f$
High	$2 f 3 < k ≤ f$	$2 f$	$f$

Tab.3 Parameters for different risk levels

Fig.8 An example of different encoding schema with different risk levels in Dynamic-EC. There are total ten nodes in the system (

n = 10

) and at most three of them are malicious nodes (

f = 3

). When the number of risk nodes is one (

k = 1

), the risk level is low and the number of parity is four (

m = 4

), which means the encoding schema is RS(

6, 4

) and the storage redundancy is

1.67 ×

. If the number of risk nodes is two (

k = 2

) and the risk level is middle, the number of parity is five (

m = 5

) and the encoding schema is RS(

5, 5

) and the storage redundancy is

2 ×

. If the number of risk nodes is three (

k = 3

) and the risk level is high, the number of parity is six (

m = 6

) and the encoding schema is RS(

4, 6

) and the storage redundancy is

2.5 ×

Fig.9 An example of adaptive fragment placement in dynamic-EC. There are total ten nodes in the system (

n = 10

) and at most three of them are malicious nodes (

f = 3

). The current number of risk nodes are one, thus the leader node replicates blocks via a RS(6,4) code. The node

N 4

is the leader node. 1) For the block

B 0

, all nodes store a fragment of it, thus

B 0

is securely stored in the system. 2) For the block

B 1

N 9

fails to store a fragment of it. In this case, the number of fragments stored in the system is nine, which is greater than or equal to

n ? m + f = 9

. Thus,

B 1

is safely stored in the system as well. 3) For the block

B 2

N 8

and

N 9

are failed to store a fragment of it. At this time, the number of fragments stored in the system is eight, which is smaller than

n ? m + f = 9

. Therefore, each healthy node needs to store an additional fragment to ensure that the block is not lost

Description	Value
CPU	Intel Xeon 2.4 GHz
NIC	1000 Mbps
Memory	32 GB
Disk	12 TB HDD
Platform	Tendermint v0.35.0
# of nodes $n$	$4 ? 20$

Tab.4 Details of the evaluation platform

Fig.10 Comparisons on storage overhead per block under different data redundant methods. (a) BSO under different number of nodes; (b) BSO under different number of malicious nodes

Fig.11 Comparisons on storage capability under different data redundant methods. (a) BSC under different number of nodes; (b) BSC under different number of malicious nodes

Fig.12 Comparisons on storage cost of Dynamic-EC under different risk level. (a) BSO under different number of nodes; (b) BSC under different number of nodes

Fig.13 Comparisons on storage cost of Dynamic-EC under different Reward Penalty Ratios. (a) BSO under different Reward Penalty Ratio; (b) BSC under different Reward Penalty Ratio

Fig.14 Comparisons on read/write latency under different data redundant methods. (a) Block write latency against the number of nodes; (b) block write latency against the number of malicious nodes; (c) block read latency against the number of nodes; (d) block read latency against the number of malicious nodes

Fig.15 Comparisons on I/O throughput under different data redundant methods. (a) Block write throughput against the number of nodes; (b) block write throughput against the number of malicious nodes; (c) block read throughput against the number of nodes; (d) block read throughput against the number of malicious nodes

Fig.16 Comparisons on read/write latency with/without the reputation evaluation. (a) Write latency against the number of nodes; (b) write latency against the number of malicious nodes; (c) read latency against the number of nodes; (d) read latency against the number of malicious nodes

1	Wood G. Ethereum: a secure decentralised generalised transaction ledger. Ethereum Project Yellow Paper, 2014, 151(2014): 1
2	F, Armknecht G O, Karame A, Mandal F, Youssef E Zenner . Ripple: overview and outlook. In: Proceedings of the 8th International Conference on Trust and Trustworthy Computing. 2015, 163−180
3	Qi X, Zhang Z, Jin C, Zhou A. BFT-Store: storage partition for permissioned blockchain via erasure coding. In: Proceedings of the 36th IEEE International Conference on Data Engineering (ICDE). 2020, 1926−1929
4	M, Barborak A, Dahbura M Malek . The consensus problem in fault-tolerant computing. ACM Computing Surveys, 1993, 25( 2): 171–220
5	Zheng Z, Xie S, Dai H, Chen X, Wang H. An overview of blockchain technology: architecture, consensus, and future trends. In: Proceedings of 2017 IEEE International Congress on Big Data (BigData Congress). 2017, 557−564
6	Castro M, Liskov B. Practical byzantine fault tolerance. In: Proceedings of the 3rd Symposium on Operating Systems Design and Implementation. 1999, 173−186
7	Z, Du X, Pang H Qian . PartitionChain: a scalable and reliable data storage strategy for permissioned blockchain. IEEE Transactions on Knowledge and Data Engineering, 2023, 35( 4): 4124–4136
8	I S, Reed G Solomon . Polynomial codes over certain finite fields. Journal of the Society for Industrial and Applied Mathematics, 1960, 8( 2): 300–304
9	L, Alvisi D, Malkhi E, Pierce M K, Reiter R N Wright . Dynamic Byzantine quorum systems. In: Proceedings of Proceeding International Conference on Dependable Systems and Networks. 2000, 283−292
10	M, Xia M, Saxena M, Blaum D A Pease . A tale of two erasure codes in HDFS. In: Proceedings of the 13th USENIX Conference on File and Storage Technologies. 2015, 213−226
11	H, Qiu C, Wu J, Li M, Guo T, Liu X, He Y, Dong Y Zhao . EC-fusion: An efficient hybrid erasure coding framework to improve both application and recovery performance in cloud storage systems. In: Proceedings of 2020 IEEE International Parallel and Distributed Processing Symposium (IPDPS). 2020, 191−201
12	C, Stathakopoulou S, Rüsch M, Brandenburger M Vukolić . Adding fairness to order: preventing front-running attacks in BFT protocols using TEEs. In: Proceedings of the 40th International Symposium on Reliable Distributed Systems (SRDS). 2021, 34−45
13	Guo B, Lu Z, Tang Q, Xu J, Zhang Z. Dumbo: faster asynchronous BFT protocols. In: Proceedings of 2020 ACM SIGSAC Conference on Computer and Communications Security. 2020, 803−818
14	Miller A, Xia Y, Croman K, Shi E, Song D. The honey badger of BFT protocols. In: Proceedings of 2016 ACM SIGSAC Conference on Computer and Communications Security. 2016, 31−42
15	Duan S, Reiter M K, Zhang H. BEAT: asynchronous BFT made practical. In: Proceedings of 2018 ACM SIGSAC Conference on Computer and Communications Security. 2018, 2028−2041
16	Androulaki E, Barger A, Bortnikov V, Cachin C, Christidis K, De Caro A, Enyeart D, Ferris C, Laventman G, Manevich Y, Muralidharan S, Murthy C, Nguyen B, Sethi M, Singh G, Smith K, Sorniotti A, Stathakopoulou C, Vukolić M, Cocco S W, Yellick J. Hyperledger fabric: a distributed operating system for permissioned blockchains. In: Proceedings of the 13th EuroSys Conference. 2018, 30
17	E Buchman . Tendermint: Byzantine fault tolerance in the age of blockchains. University of Guelph, Dissertation, 2016
18	Ghemawat S, Gobioff H, Leung S T. The Google file system. In: Proceedings of the 19th ACM Symposium on Operating Systems Principles. 2003, 29−43
19	Shvachko K, Kuang H, Radia S, Chansler R. The hadoop distributed file system. In: Proceedings of the 26th IEEE Symposium on Mass Storage Systems and Technologies (MSST). 2010, 1−10
20	Dai X, Xiao J, Yang W, Wang C, Jin H. Jidar: a jigsaw-like data reduction approach without trust assumptions for bitcoin system. In: Proceedings of the 39th IEEE International Conference on Distributed Computing Systems (ICDCS). 2019, 1317−1326
21	Sukhwani H, Wang N, Trivedi K S, Rindos A. Performance modeling of hyperledger fabric (permissioned blockchain network). In: Proceedings of the 17th IEEE International Symposium on Network Computing and Applications (NCA). 2018, 1−8
22	Weatherspoon H, Kubiatowicz J D. Erasure coding vs. replication: a quantitative comparison. In: Proceedings of the 1st International Workshop on Peer-to-Peer Systems. 2002, 328−337
23	Yang G, Xue H, Gu Y, Wu C, Li J, Guo M, Li S, Xie X, Dong Y, Zhao Y. XHR-code: an efficient wide stripe erasure code to reduce cross-rack overhead in cloud storage systems. In: Proceedings of the 41st International Symposium on Reliable Distributed Systems (SRDS). 2022, 273−283
24	Y, Hu L, Cheng Q, Yao P P C, Lee W, Wang W Chen . Exploiting combined locality for Wide-Stripe erasure coding in distributed storage. In: Proceedings of the 19th USENIX Conference on File and Storage Technologies. 2021, 233−248
25	Ye L, Feng D, Hu Y, Liu Q. Hybrid-RC: Flexible erasure codes with optimized recovery performance and low storage overhead. In: Proceedings of the 36th IEEE Symposium on Reliable Distributed Systems (SRDS). 2017, 124–133
26	Wu C, Wan S, He X, Cao Q, Xie C. H-Code: a hybrid MDS array code to optimize partial stripe writes in RAID-6. In: Proceedings of 2011 IEEE International Parallel & Distributed Processing Symposium. 2011, 782−793
27	Kong L, Manohar D J, Subbiah A, Sun M, Ahamad M, Blough D M. Agile store: Experience with quorum-based data replication techniques for adaptive Byzantine fault tolerance. In: Proceedings of the 24th IEEE Symposium on Reliable Distributed Systems (SRDS’05). 2005, 143−154
28	J P, Bahsoun R, Guerraoui A Shoker . Making BFT protocols really adaptive. In: Proceedings of 2015 IEEE International Parallel and Distributed Processing Symposium. 2015, 904−913
29	Silva D S, Graczyk R, Decouchant J, Völp M, Esteves-Verissimo P. Threat adaptive byzantine fault tolerant state-machine replication. In: Proceedings of the 40th International Symposium on Reliable Distributed Systems (SRDS). 2021, 78−87
30	A Shostack . Threat Modeling: Designing for Security. Indianapolis: John Wiley & Sons, 2014
31	Zhang J, Huang Y, Ye F, Yang Y. A novel proof-of-reputation consensus for storage allocation in edge blockchain systems. In: Proceedings of the 29th IEEE/ACM International Symposium on Quality of Service (IWQOS). 2021, 1−10
32	S D, Kamvar M T, Schlosser H Garcia-Molina . The eigentrust algorithm for reputation management in P2P networks. In: Proceedings of the 12th International Conference on World Wide Web. 2003, 640−651
33	Naehrig M, Niederhagen R, Schwabe P. New software speed records for cryptographic pairings. In: Proceedings of the 1st International Conference on Cryptology and Information Security in Latin America. 2010, 109−123
34	Hossain M A, Islam M K, Das S K. Cryptanalyzing of message digest algorithms MD4 and MD5. International Journal on Cryptography and Information Security, 2012, 2 (1): 1-13.
35	Luu L, Narayanan V, Zheng C, Baweja K, Gilbert S, Saxena P. A secure sharding protocol for open blockchains. In: Proceedings of 2016 ACM SIGSAC Conference on Computer and Communications Security. 2016, 17−30
36	J, Yang Y, Yue K V Rashmi . A large-scale analysis of hundreds of in-memory key-value cache clusters at twitter. ACM Transactions on Storage, 2021, 17( 3): 17
37	B S, Egala A K, Pradhan V, Badarla S P Mohanty . Fortified-chain: a blockchain-based framework for security and privacy-assured internet of medical things with effective access control. IEEE Internet of Things Journal, 2021, 8( 14): 11717–11731
38	El Houda Z A, Hafid A, Khoukhi L. Co-IoT: a collaborative DDoS mitigation scheme in IoT environment based on blockchain using SDN. In: Proceedings of 2019 IEEE Global Communications Conference (GLOBECOM). 2019, 1−6

[1]

FCS-23209-OF-MZ_suppl_1

Download

[1]	Nan SUN, Wei WANG, Yongxin TONG, Kexin LIU. Blockchain based federated learning for intrusion detection for Internet of Things[J]. Front. Comput. Sci., 2024, 18(5): 185328-.
[2]	Xingxing CHEN, Qingfeng CHENG, Weidong YANG, Xiangyang LUO. An anonymous authentication and secure data transmission scheme for the Internet of Things based on blockchain[J]. Front. Comput. Sci., 2024, 18(3): 183807-.
[3]	Tiezheng GUO, Zhiwei ZHANG, Ye YUAN, Xiaochun YANG, Guoren WANG. Hybrid concurrency control protocol for data sharing among heterogeneous blockchains[J]. Front. Comput. Sci., 2024, 18(3): 183104-.
[4]	Guocheng ZHU, Debiao HE, Haoyang AN, Min LUO, Cong PENG. The governance technology for blockchain systems: a survey[J]. Front. Comput. Sci., 2024, 18(2): 182813-.
[5]	B Swaroopa REDDY, T Uday Kiran REDDY. CompactChain: an efficient stateless chain for UTXO-model blockchain[J]. Front. Comput. Sci., 2024, 18(2): 182806-.
[6]	Jian AN, Siyuan WU, Xiaolin GUI, Xin HE, Xuejun ZHANG. A blockchain-based framework for data quality in edge-computing-enabled crowdsensing[J]. Front. Comput. Sci., 2023, 17(4): 174503-.
[7]	Peng LI, Junzuo LAI, Yongdong WU. Accountable attribute-based authentication with fine-grained access control and its application to crowdsourcing[J]. Front. Comput. Sci., 2023, 17(1): 171802-.
[8]	Donghui WANG, Peng CAI, Weining QIAN, Aoying ZHOU. Efficient and stable quorum-based log replication and replay for modern cluster-databases[J]. Front. Comput. Sci., 2022, 16(5): 165612-.
[9]	Chaofan WANG, Xiaohai DAI, Jiang XIAO, Chenchen LI, Ming WEN, Bingbing ZHOU, Hai JIN. Demystifying Ethereum account diversity: observations, models and analysis[J]. Front. Comput. Sci., 2022, 16(4): 164505-.
[10]	Zeli WANG, Hai JIN, Weiqi DAI, Kim-Kwang Raymond CHOO, Deqing ZOU. Ethereum smart contract security research: survey and future research opportunities[J]. Front. Comput. Sci., 2021, 15(2): 152802-.
[11]	Yan ZHU, Khaled RIAD, Ruiqi GUO, Guohua GAN, Rongquan FENG. New instant confirmation mechanism based on interactive incontestable signature in consortium blockchain[J]. Front. Comput. Sci., 2019, 13(6): 1182-1197.
[12]	Lian YU, Wei-Tek TSAI. State synchronization in process-oriented chaincode[J]. Front. Comput. Sci., 2019, 13(6): 1166-1181.
[13]	Yu ZHANG, Yuxing HAN, Jiangtao WEN. SMER: a secure method of exchanging resources in heterogeneous internet of things[J]. Front. Comput. Sci., 2019, 13(6): 1198-1209.
[14]	Libo FENG, Hui ZHANG, Wei-Tek TSAI, Simeng SUN. System architecture for high-performance permissioned blockchains[J]. Front. Comput. Sci., 2019, 13(6): 1151-1165.

Viewed

Full text

Abstract

Cited

Shared

Discussed