Decentralized Storage Systems
Last updated
Last updated
Storage is an integral part of modern computer systems, and the same is true for distributed and decentralized systems like a blockchain. When interacting with the Polkadot ecosystem, it will be helpful if you familiarize yourself with the current WEB3 approach to decentralized storage systems, and how these systems can support your endeavors.
The key attribute that characterizes centralized cloud storage is the location of data.
In decentralized cloud storage, the key attribute becomes the data itself instead of the data's location.
This can be viewed as the shift from the centralized location-centric storage approach to the decentralized content-centric approach.
IPFS is a peer-to-peer distributed file system that seeks to connect all computing devices with the same system of files, by utilizing features such as content-addressing, content-signing, and enhanced security methods through encryption. IPFS aims to address the current hurdles of the HTTP-based Internet.
Brief comparison of IPFS & HTTP:
| IPFS | HTTP |
|---|---|
Like IPFS, Filecoin is a protocol developed by Protocol Labs that offers a decentralized storage network. Filecoin's main focus is the storage itself and uses IPFS as a [complementary] back-end protocol.
Substrate takes a layered approach to storage by using a key-value data store that is implemented as a database-backed, modified Merkle tree. Substrate's higher-layer storage abstractions are built on the key-value store.
The key-value data store is backed by RocksDB, and it also supports an experimental Parity database.
NOTE
The database is used for components that require persistent storage These are components like Substrate clients, Substrate light-clients & off-chain workers. For more information, check out Substrate Developer Hub's Storage Page.
When building on Substrate, runtime developers can utilize of Substrate's FRAME Storage pallet which gives access to Substrate's storage APIs. These storage items support values that are encoded by Parity's SCALE (Simple Concatenated Aggregate Little-Endian) Codec.
There is a Storage Value API that is used to store single values, a Storage Map API that is used to a key-value hash map, a Storage Double Map API that creates a storage map with two keys to provide the ability to efficiently remove all entries that have a common first key, and a Storage N Map API that can be used to store a hash map with any arbitrary number of keys.
These layered APIs act as runtime storage that allows you to store data in your blockchain. More information can be found at the Runtime Storage Page on Substrate Developer Hub.
Crust Network provides a Web3.0 decentralized storage network for the Metaverse. It is designed to realize core values of decentralization, privacy, and assurance. Crust supports multiple storage-layer protocols such as IPFS, and exposes instant accessible on-chain storage functions to users. Crustʼs technical stack is also capable of supporting data manipulating and computing.
Crust provides a native cross-chain communnication communication pallet based on XCMP, called xStorage.
The protocol also supports most smart contract platforms, including Ethereum, with its cross-chain dStorage solution.
LEARN MORE ABOUT CRUST
To learn more about Crust, check out the Crust Network Wiki. Try integrating with Crust by following their Crust Storage 101 guide.
The PolkadotJS API offers storage methods that are part of the default Substrate runtime. They are exposed via api.query.<module>.<method>. See the official docs for more details.
PolkadotJS Apps includes a decentralized storage module that allows Substrate-based chain users to upload their files to IPFS W3Auth Gateway and use the IPFS W3Auth Pinning Service to pin their files on Crust Network.
Start by uploading a single file or folder:
Choose a Gateway:
Sign the message:
You should be able to view the file info, as follows:
As well as the file status:
NOTE
The whole files module is decentralized, so your file directory is only cached in browser. The file directory info will not be visible if you switch to a new browser or clear the browser cache. The storage module allows you to export file directory info from the current browser and import it to the new browser.
NOTE
These above images are taken from this pull request
network: peer-to-peer model (decentralized)
network: client-server model (centralized)
requests: use a cryptographic hash of that data
requests: use the address on which data is hosted
accessibility: data is distributed to multiple nodes and can be accessed at any time. Bandwidth is high: nearest peer can serve the data
accessibility: data can only be accessed if the server is live and there are no interruptions in transmission. Bandwidth is limited: clients send requests to the same server
