<?xml version="1.0" encoding="utf-8" standalone="yes"?><rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:content="http://purl.org/rss/1.0/modules/content/"><channel><title>Erasure Coding on Mia Heidenstedt</title><link>https://heidenstedt.org/tags/erasure-coding/</link><description>Recent
content
in Erasure Coding on Mia Heidenstedt</description><generator>
Hugo</generator><language>en</language><lastBuildDate>Fri, 01 Nov 2024 11:17:46 +0000</lastBuildDate><atom:link href="https://heidenstedt.org/tags/erasure-coding/index.xml" rel="self" type="application/rss+xml"/><item><title>Release: OuroborosDB Data Storage Calculator</title><link>https://heidenstedt.org/posts/2024/release-ouroborosdb-data-storage-calculator/</link><pubDate>Fri, 01 Nov 2024 11:17:10 +0200</pubDate><guid>https://heidenstedt.org/posts/2024/release-ouroborosdb-data-storage-calculator/</guid><description><![CDATA[<p>
      <em>Best viewed on the <a href="https://heidenstedt.org/posts/2024/release-ouroborosdb-data-storage-calculator/">original page</a>, where extended functionality like the
    footnote helper is available.</em>
    </p><p>I forgot to mention that i released a new tool for my project <a href="https://github.com/i5heu/ouroboros-db">ouroboros-db</a>, so here it is:</p>
<p>The &ldquo;OuroborosDB Overhead Calculator&rdquo; is a little tool which helps you to calculate the overhead of erasure coding systems while also considering the overhead of indexes and the blocks themselves. It is kinda fun to play around with so give it a try!</p>
<p><a href="https://i5heu.github.io/ouroboros-db-overhead-calculator/">https://i5heu.github.io/ouroboros-db-overhead-calculator/</a></p>
<video width="100%" controls>
  <source src="https://heidenstedt.org/posts/2024/release-ouroborosdb-data-storage-calculator/demo.webm" type="video/webm">
  Your browser does not support the video tag.
</video>]]></description><content:encoded><![CDATA[<p>
      <em>Best viewed on the <a href="https://heidenstedt.org/posts/2024/release-ouroborosdb-data-storage-calculator/">original page</a>, where extended functionality like the
    footnote helper is available.</em>
    </p><p>I forgot to mention that i released a new tool for my project <a href="https://github.com/i5heu/ouroboros-db">ouroboros-db</a>, so here it is:</p>
<p>The &ldquo;OuroborosDB Overhead Calculator&rdquo; is a little tool which helps you to calculate the overhead of erasure coding systems while also considering the overhead of indexes and the blocks themselves. It is kinda fun to play around with so give it a try!</p>
<p><a href="https://i5heu.github.io/ouroboros-db-overhead-calculator/">https://i5heu.github.io/ouroboros-db-overhead-calculator/</a></p>
<video width="100%" controls>
  <source src="https://heidenstedt.org/posts/2024/release-ouroborosdb-data-storage-calculator/demo.webm" type="video/webm">
  Your browser does not support the video tag.
</video>]]></content:encoded></item><item><title>Ouroboros DB Dev Journal: Erasure Coding</title><link>https://heidenstedt.org/posts/2024/ouroboros-db-dev-journal-erasure-coding/</link><pubDate>Tue, 27 Aug 2024 17:55:05 +0200</pubDate><guid>https://heidenstedt.org/posts/2024/ouroboros-db-dev-journal-erasure-coding/</guid><description><![CDATA[<p>
      <em>Best viewed on the <a href="https://heidenstedt.org/posts/2024/ouroboros-db-dev-journal-erasure-coding/">original page</a>, where extended functionality like the
    footnote helper is available.</em>
    </p><p>This is a Dev Journal for the Ouroboros DB Project.<br>
I try to write down my thoughts and ideas Somewhat structured to have it as a reference for later and i publish it to give this information a chance to help other and to get feedback from the community.<br>
If you have feedback you can write me on my <a href="https://mastodon.social/@heidenstedt">Mastodon</a>.</p>
<p>Pls note that this is only a Dev Journal and not a Blog Post, so it may be a bit unstructured and not as polished as a Blog Post, including typos and other errors.</p>
<h2 id="tldr"><a href="https://heidenstedt.org/posts/2024/ouroboros-db-dev-journal-erasure-coding/#tldr">TL;DR:</a></h2><p>Today i worked on a refactoring of the architecture and feasibility of erasure coding for <a href="https://github.com/i5heu/ouroboros-db">Ouroboros DB</a>. There are some concerns i had regarding the potential index size and overhead that will result from it, and as it turns out, it is not as bad as i thought but i need a DHT for the index.</p>
<h2 id="architecture"><a href="https://heidenstedt.org/posts/2024/ouroboros-db-dev-journal-erasure-coding/#architecture">Architecture</a></h2><p>I think it would be super handy to be able to run parts of data pipelines (eg. storing files into chunked,compressed, encrypted and then erasure coded blocks) in a distributed manner. for safety and because i want to be able to run trustless nodes that can&rsquo;t see the raw data, it would be necessary to have the file chunking, compression and encryption part on the server the client currently speaks too, although it would be possible to pre-chunk the file on the client and upload it to different nodes, although here is the question if it is easy to port the chunking algorithm to browser JS or WASM.</p>
<h3 id="modules"><a href="https://heidenstedt.org/posts/2024/ouroboros-db-dev-journal-erasure-coding/#modules">Modules</a></h3><p>Refactoring of the code is needed to implement <code>erasure coding</code>, maybe we can even get rid of the <code>chunk</code> entirely as a stored thing since we have the data in the <code>parity block</code>s already.</p>
<p>Maybe best to add the <code>erasure coding</code> to the <code>StoreDataPipeline</code> and add the needed <code>erasure coding</code> metadata to <code>ChunkData</code>.</p>
<p>ASCII Art of the new architecture (click to get the .txt file):</p>
<a href="https://heidenstedt.org/posts/2024/ouroboros-db-dev-journal-erasure-coding/architecture.txt" target="_blank">
<p><div class="imageLoadingWrap"><img
          alt="ASCII Art of the new architecture"
          title="ASCII Art of the new architecture"
          loading="lazy"
          src="https://heidenstedt.org/posts/2024/ouroboros-db-dev-journal-erasure-coding/Architecture_hu_96d4787568ab57f9.webp"
          width="3030"
          height="871"
          srcset='/posts/2024/ouroboros-db-dev-journal-erasure-coding/Architecture_hu_2cb9192ee549f6f7.avif 100w,/posts/2024/ouroboros-db-dev-journal-erasure-coding/Architecture_hu_b05e3863fdd4ce9a.webp 100w,/posts/2024/ouroboros-db-dev-journal-erasure-coding/Architecture_hu_ab2ece513e436730.avif 200w,/posts/2024/ouroboros-db-dev-journal-erasure-coding/Architecture_hu_b80b28c6ea927abb.webp 200w,/posts/2024/ouroboros-db-dev-journal-erasure-coding/Architecture_hu_9a6a5eab6370057.avif 300w,/posts/2024/ouroboros-db-dev-journal-erasure-coding/Architecture_hu_f8a619137d8295b3.webp 300w,/posts/2024/ouroboros-db-dev-journal-erasure-coding/Architecture_hu_b9e944c8f507ee53.avif 400w,/posts/2024/ouroboros-db-dev-journal-erasure-coding/Architecture_hu_9b1c897b51eca118.webp 400w,/posts/2024/ouroboros-db-dev-journal-erasure-coding/Architecture_hu_97dc6486f99e7f3c.avif 500w,/posts/2024/ouroboros-db-dev-journal-erasure-coding/Architecture_hu_1048466be74d0ae7.webp 500w,/posts/2024/ouroboros-db-dev-journal-erasure-coding/Architecture_hu_12dba54ade38bf4a.avif 600w,/posts/2024/ouroboros-db-dev-journal-erasure-coding/Architecture_hu_4fb7e4485e6d9ba8.webp 600w,/posts/2024/ouroboros-db-dev-journal-erasure-coding/Architecture_hu_cc0d65b877a694cc.avif 700w,/posts/2024/ouroboros-db-dev-journal-erasure-coding/Architecture_hu_863c0c2a40d204e.webp 700w,/posts/2024/ouroboros-db-dev-journal-erasure-coding/Architecture_hu_a0b76a1673bd8ce1.avif 800w,/posts/2024/ouroboros-db-dev-journal-erasure-coding/Architecture_hu_48b438099fcc934f.webp 800w,/posts/2024/ouroboros-db-dev-journal-erasure-coding/Architecture_hu_5c003cfa191ab46a.avif 900w,/posts/2024/ouroboros-db-dev-journal-erasure-coding/Architecture_hu_f03d0655d1a1a4da.webp 900w,/posts/2024/ouroboros-db-dev-journal-erasure-coding/Architecture_hu_42356eb882d4c7df.avif 1000w,/posts/2024/ouroboros-db-dev-journal-erasure-coding/Architecture_hu_ea45c55f65bd9414.webp 1000w,/posts/2024/ouroboros-db-dev-journal-erasure-coding/Architecture_hu_a794c31c2c196144.avif 1100w,/posts/2024/ouroboros-db-dev-journal-erasure-coding/Architecture_hu_f8ecf047dd3ef009.webp 1100w,/posts/2024/ouroboros-db-dev-journal-erasure-coding/Architecture_hu_797db1b158ee89ac.avif 1200w,/posts/2024/ouroboros-db-dev-journal-erasure-coding/Architecture_hu_5900068a81c61786.webp 1200w,/posts/2024/ouroboros-db-dev-journal-erasure-coding/Architecture_hu_aba964185e50c26.avif 1300w,/posts/2024/ouroboros-db-dev-journal-erasure-coding/Architecture_hu_b9393aa6e5960f12.webp 1300w,/posts/2024/ouroboros-db-dev-journal-erasure-coding/Architecture_hu_311d76129c9977b3.avif 1400w,/posts/2024/ouroboros-db-dev-journal-erasure-coding/Architecture_hu_50cf36bdf8cc20a4.webp 1400w,/posts/2024/ouroboros-db-dev-journal-erasure-coding/Architecture_hu_35fb69dfc8065d4d.avif 1500w,/posts/2024/ouroboros-db-dev-journal-erasure-coding/Architecture_hu_d03d70c6373baf8d.webp 1500w,/posts/2024/ouroboros-db-dev-journal-erasure-coding/Architecture_hu_f793111d3415434c.avif 1600w,/posts/2024/ouroboros-db-dev-journal-erasure-coding/Architecture_hu_50de16a06f6c909.webp 1600w,/posts/2024/ouroboros-db-dev-journal-erasure-coding/Architecture_hu_f44d13cbf9438f06.avif 1700w,/posts/2024/ouroboros-db-dev-journal-erasure-coding/Architecture_hu_a01af93ff7648eb3.webp 1700w,/posts/2024/ouroboros-db-dev-journal-erasure-coding/Architecture_hu_1c062ad7c62ae03a.avif 1800w,/posts/2024/ouroboros-db-dev-journal-erasure-coding/Architecture_hu_9fa6b205fb52c14d.webp 1800w,/posts/2024/ouroboros-db-dev-journal-erasure-coding/Architecture_hu_e28bcab6f174e919.avif 1900w,/posts/2024/ouroboros-db-dev-journal-erasure-coding/Architecture_hu_2eb74c1265d3c7b0.webp 1900w,/posts/2024/ouroboros-db-dev-journal-erasure-coding/Architecture_hu_54f2d99662709774.avif 2000w,/posts/2024/ouroboros-db-dev-journal-erasure-coding/Architecture_hu_1e31b1da25e1eb32.webp 2000w,/posts/2024/ouroboros-db-dev-journal-erasure-coding/Architecture_hu_646abb2a2bff3fa3.avif 2100w,/posts/2024/ouroboros-db-dev-journal-erasure-coding/Architecture_hu_f089d367a3a280fc.webp 2100w,/posts/2024/ouroboros-db-dev-journal-erasure-coding/Architecture_hu_1978db329ec16d6d.avif 2200w,/posts/2024/ouroboros-db-dev-journal-erasure-coding/Architecture_hu_6ee5a59dc7098578.webp 2200w,/posts/2024/ouroboros-db-dev-journal-erasure-coding/Architecture_hu_7e06f488ab5ffa79.avif 2300w,/posts/2024/ouroboros-db-dev-journal-erasure-coding/Architecture_hu_3e2c0136633d8f0.webp 2300w,/posts/2024/ouroboros-db-dev-journal-erasure-coding/Architecture_hu_8e267fbf50cb3a52.avif 2400w,/posts/2024/ouroboros-db-dev-journal-erasure-coding/Architecture_hu_5fa6ff521302409f.webp 2400w,/posts/2024/ouroboros-db-dev-journal-erasure-coding/Architecture_hu_d1ab2af2aaec0825.avif 2500w,/posts/2024/ouroboros-db-dev-journal-erasure-coding/Architecture_hu_7fb22e5fa281cb76.webp 2500w,/posts/2024/ouroboros-db-dev-journal-erasure-coding/Architecture_hu_4569bdffbcc0329b.avif 2600w,/posts/2024/ouroboros-db-dev-journal-erasure-coding/Architecture_hu_c30c96df17374589.webp 2600w,/posts/2024/ouroboros-db-dev-journal-erasure-coding/Architecture_hu_aa217e5aa81c2f2c.avif 2700w,/posts/2024/ouroboros-db-dev-journal-erasure-coding/Architecture_hu_a6644da4d424ff65.webp 2700w,/posts/2024/ouroboros-db-dev-journal-erasure-coding/Architecture_hu_d128f208c0c6ba4.avif 2800w,/posts/2024/ouroboros-db-dev-journal-erasure-coding/Architecture_hu_bd4e57588a16519a.webp 2800w,/posts/2024/ouroboros-db-dev-journal-erasure-coding/Architecture_hu_6b01e3577982cafa.avif 2900w,/posts/2024/ouroboros-db-dev-journal-erasure-coding/Architecture_hu_c5dbd5148973c5ad.webp 2900w,/posts/2024/ouroboros-db-dev-journal-erasure-coding/Architecture_hu_5d1962d1966460a.avif 3000w,/posts/2024/ouroboros-db-dev-journal-erasure-coding/Architecture_hu_3abd1e7ef89d2c43.webp 3000w,/posts/2024/ouroboros-db-dev-journal-erasure-coding/Architecture_hu_e80e0bc829f354e7.avif 3030w,/posts/2024/ouroboros-db-dev-journal-erasure-coding/Architecture_hu_80249ad413f32588.webp 3030w
          '
          sizes="(max-width: 672px) calc(85vw - 32px), (max-width: 736px) 640px, 657px"
        ><div class="imageLoading"></div>
</div></p>
</a>
<h2 id="erasure-coding"><a href="https://heidenstedt.org/posts/2024/ouroboros-db-dev-journal-erasure-coding/#erasure-coding">Erasure Coding</a></h2><p>For now i think it is okay yo make the erasure coding a simple n6 k3, which would result in a <code>parity block</code> size of 327,680 Bytes or 0.31MB as average. This would result in about 1.875MB per chunk that is 1.25MB big in average. This would result in a 50% overhead for the erasure coding, which i think is quite okay.</p>
<p>For the goal of storing 100TB in it, which are about 400M Chunks, we would need 2,400,000,000 <code>parity block</code>s, aka 2,4 Billion.</p>
<p>If we consider following overhead for the erasure coding:</p>
<details>
  <summary>Parity Meta Data</summary>
    This is the Meta Data for a `parity block`:
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-json" data-lang="json"><span style="display:flex;"><span>{
</span></span><span style="display:flex;"><span><span style="color:#f92672">&#34;parityHash&#34;</span>: <span style="color:#e6db74">&#34;aeae379a6e857728e44164267fdb7a0e27b205d757cc19899586c89dbb221930f1813d02ff93a661859bc17065eac4d6edf3c38a034e6283a84754d52917e5b0&#34;</span>,
</span></span><span style="display:flex;"><span><span style="color:#f92672">&#34;chunkHash&#34;</span>: <span style="color:#e6db74">&#34;aeae379a6e857728e44164267fdb7a0e27b205d757cc19899586c89dbb221930f1813d02ff93a661859bc17065eac4d6edf3c38a034e6283a84754d52917e5b0&#34;</span>,
</span></span><span style="display:flex;"><span><span style="color:#f92672">&#34;sizeByte&#34;</span>: <span style="color:#ae81ff">433000</span>,
</span></span><span style="display:flex;"><span><span style="color:#f92672">&#34;lastChecked&#34;</span>: <span style="color:#ae81ff">1724760126282</span>,
</span></span><span style="display:flex;"><span><span style="color:#f92672">&#34;reblanceLog&#34;</span>: [
</span></span><span style="display:flex;"><span>    {
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">&#34;time&#34;</span>: <span style="color:#ae81ff">1724760126282</span>,
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">&#34;from&#34;</span>: <span style="color:#e6db74">&#34;2deb000b57bfac9d72c14d4ed967b572&#34;</span>
</span></span><span style="display:flex;"><span>    },
</span></span><span style="display:flex;"><span>    {
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">&#34;time&#34;</span>: <span style="color:#ae81ff">1724760126282</span>,
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">&#34;from&#34;</span>: <span style="color:#e6db74">&#34;2deb000b57bfac9d72c14d4ed967b572&#34;</span>
</span></span><span style="display:flex;"><span>    },
</span></span><span style="display:flex;"><span>    {
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">&#34;time&#34;</span>: <span style="color:#ae81ff">1724760126282</span>,
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">&#34;from&#34;</span>: <span style="color:#e6db74">&#34;2deb000b57bfac9d72c14d4ed967b572&#34;</span>
</span></span><span style="display:flex;"><span>    },
</span></span><span style="display:flex;"><span>    {
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">&#34;time&#34;</span>: <span style="color:#ae81ff">1724760126282</span>,
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">&#34;from&#34;</span>: <span style="color:#e6db74">&#34;2deb000b57bfac9d72c14d4ed967b572&#34;</span>
</span></span><span style="display:flex;"><span>    },
</span></span><span style="display:flex;"><span>    {
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">&#34;time&#34;</span>: <span style="color:#ae81ff">1724760126282</span>,
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">&#34;from&#34;</span>: <span style="color:#e6db74">&#34;2deb000b57bfac9d72c14d4ed967b572&#34;</span>
</span></span><span style="display:flex;"><span>    },
</span></span><span style="display:flex;"><span>    {
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">&#34;time&#34;</span>: <span style="color:#ae81ff">1724760126282</span>,
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">&#34;from&#34;</span>: <span style="color:#e6db74">&#34;2deb000b57bfac9d72c14d4ed967b572&#34;</span>
</span></span><span style="display:flex;"><span>    },
</span></span><span style="display:flex;"><span>    {
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">&#34;time&#34;</span>: <span style="color:#ae81ff">1724760126282</span>,
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">&#34;from&#34;</span>: <span style="color:#e6db74">&#34;2deb000b57bfac9d72c14d4ed967b572&#34;</span>
</span></span><span style="display:flex;"><span>    },
</span></span><span style="display:flex;"><span>    {
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">&#34;time&#34;</span>: <span style="color:#ae81ff">1724760126282</span>,
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">&#34;from&#34;</span>: <span style="color:#e6db74">&#34;2deb000b57bfac9d72c14d4ed967b572&#34;</span>
</span></span><span style="display:flex;"><span>    },
</span></span><span style="display:flex;"><span>    {
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">&#34;time&#34;</span>: <span style="color:#ae81ff">1724760126282</span>,
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">&#34;from&#34;</span>: <span style="color:#e6db74">&#34;2deb000b57bfac9d72c14d4ed967b572&#34;</span>
</span></span><span style="display:flex;"><span>    },
</span></span><span style="display:flex;"><span>    {
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">&#34;time&#34;</span>: <span style="color:#ae81ff">1724760126282</span>,
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">&#34;from&#34;</span>: <span style="color:#e6db74">&#34;2deb000b57bfac9d72c14d4ed967b572&#34;</span>
</span></span><span style="display:flex;"><span>    }
</span></span><span style="display:flex;"><span>],
</span></span><span style="display:flex;"><span><span style="color:#f92672">&#34;userLog&#34;</span>: [
</span></span><span style="display:flex;"><span>    {
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">&#34;time&#34;</span>: <span style="color:#ae81ff">1724760126282</span>,
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">&#34;from&#34;</span>: <span style="color:#e6db74">&#34;2deb000b57bfac9d72c14d4ed967b572&#34;</span>
</span></span><span style="display:flex;"><span>    },
</span></span><span style="display:flex;"><span>    {
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">&#34;time&#34;</span>: <span style="color:#ae81ff">1724760126282</span>,
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">&#34;from&#34;</span>: <span style="color:#e6db74">&#34;2deb000b57bfac9d72c14d4ed967b572&#34;</span>
</span></span><span style="display:flex;"><span>    },
</span></span><span style="display:flex;"><span>    {
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">&#34;time&#34;</span>: <span style="color:#ae81ff">1724760126282</span>,
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">&#34;from&#34;</span>: <span style="color:#e6db74">&#34;2deb000b57bfac9d72c14d4ed967b572&#34;</span>
</span></span><span style="display:flex;"><span>    },
</span></span><span style="display:flex;"><span>    {
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">&#34;time&#34;</span>: <span style="color:#ae81ff">1724760126282</span>,
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">&#34;from&#34;</span>: <span style="color:#e6db74">&#34;2deb000b57bfac9d72c14d4ed967b572&#34;</span>
</span></span><span style="display:flex;"><span>    },
</span></span><span style="display:flex;"><span>    {
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">&#34;time&#34;</span>: <span style="color:#ae81ff">1724760126282</span>,
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">&#34;from&#34;</span>: <span style="color:#e6db74">&#34;2deb000b57bfac9d72c14d4ed967b572&#34;</span>
</span></span><span style="display:flex;"><span>    },
</span></span><span style="display:flex;"><span>    {
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">&#34;time&#34;</span>: <span style="color:#ae81ff">1724760126282</span>,
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">&#34;from&#34;</span>: <span style="color:#e6db74">&#34;2deb000b57bfac9d72c14d4ed967b572&#34;</span>
</span></span><span style="display:flex;"><span>    },
</span></span><span style="display:flex;"><span>    {
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">&#34;time&#34;</span>: <span style="color:#ae81ff">1724760126282</span>,
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">&#34;from&#34;</span>: <span style="color:#e6db74">&#34;2deb000b57bfac9d72c14d4ed967b572&#34;</span>
</span></span><span style="display:flex;"><span>    },
</span></span><span style="display:flex;"><span>    {
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">&#34;time&#34;</span>: <span style="color:#ae81ff">1724760126282</span>,
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">&#34;from&#34;</span>: <span style="color:#e6db74">&#34;2deb000b57bfac9d72c14d4ed967b572&#34;</span>
</span></span><span style="display:flex;"><span>    },
</span></span><span style="display:flex;"><span>    {
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">&#34;time&#34;</span>: <span style="color:#ae81ff">1724760126282</span>,
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">&#34;from&#34;</span>: <span style="color:#e6db74">&#34;2deb000b57bfac9d72c14d4ed967b572&#34;</span>
</span></span><span style="display:flex;"><span>    },
</span></span><span style="display:flex;"><span>    {
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">&#34;time&#34;</span>: <span style="color:#ae81ff">1724760126282</span>,
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">&#34;from&#34;</span>: <span style="color:#e6db74">&#34;2deb000b57bfac9d72c14d4ed967b572&#34;</span>
</span></span><span style="display:flex;"><span>    }
</span></span><span style="display:flex;"><span>],
</span></span><span style="display:flex;"><span><span style="color:#f92672">&#34;this is for the KV Key&#34;</span>: <span style="color:#e6db74">&#34;aeae379a6e857728e44164267fdb7a0e27b205d757cc19899586c89dbb221930f1813d02ff93a661859bc17065eac4d6edf3c38a034e6283a84754d52917e5b0&#34;</span>
</span></span><span style="display:flex;"><span>}
</span></span></code></pre></div></details>
<p>We see that <code>parity block metadata</code> has under 2373 Bytes of overhead, which is about 0.72%% of the <code>parity block</code> size (very good). If one node would need to store the entire hash table we look at a size of 5.18TB, which means that we need a DHT for this to work. This also is the case for the Events, which introduce more storage overhead (more in <a href="https://heidenstedt.org/posts/2024/ouroboros-db-dev-journal-erasure-coding/#dht">DHT</a> )</p>
<h3 id="dht"><a href="https://heidenstedt.org/posts/2024/ouroboros-db-dev-journal-erasure-coding/#dht">DHT</a></h3><p>We need additional DHT Metadata like this:</p>
<details>
  <summary>DHT Meta Data</summary>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-json" data-lang="json"><span style="display:flex;"><span>{
</span></span><span style="display:flex;"><span>  <span style="color:#f92672">&#34;parityHash&#34;</span>: <span style="color:#e6db74">&#34;aeae379a6e857728e44164267fdb7a0e27b205d757cc19899586c89dbb221930f1813d02ff93a661859bc17065eac4d6edf3c38a034e6283a84754d52917e5b0&#34;</span>,
</span></span><span style="display:flex;"><span>  <span style="color:#f92672">&#34;storingNodes&#34;</span>: [
</span></span><span style="display:flex;"><span>    {
</span></span><span style="display:flex;"><span>      <span style="color:#f92672">&#34;node&#34;</span>: <span style="color:#e6db74">&#34;2deb000b57bfac9d72c14d4ed967b572&#34;</span>,
</span></span><span style="display:flex;"><span>      <span style="color:#f92672">&#34;lastValidated&#34;</span>: <span style="color:#ae81ff">1724760126282</span>
</span></span><span style="display:flex;"><span>    },
</span></span><span style="display:flex;"><span>    {
</span></span><span style="display:flex;"><span>      <span style="color:#f92672">&#34;node&#34;</span>: <span style="color:#e6db74">&#34;2deb000b57bfac9d72c14d4ed967b572&#34;</span>,
</span></span><span style="display:flex;"><span>      <span style="color:#f92672">&#34;lastValidated&#34;</span>: <span style="color:#ae81ff">1724760126282</span>
</span></span><span style="display:flex;"><span>    },
</span></span><span style="display:flex;"><span>    {
</span></span><span style="display:flex;"><span>      <span style="color:#f92672">&#34;node&#34;</span>: <span style="color:#e6db74">&#34;2deb000b57bfac9d72c14d4ed967b572&#34;</span>,
</span></span><span style="display:flex;"><span>      <span style="color:#f92672">&#34;lastValidated&#34;</span>: <span style="color:#ae81ff">1724760126282</span>
</span></span><span style="display:flex;"><span>    },
</span></span><span style="display:flex;"><span>    {
</span></span><span style="display:flex;"><span>      <span style="color:#f92672">&#34;node&#34;</span>: <span style="color:#e6db74">&#34;2deb000b57bfac9d72c14d4ed967b572&#34;</span>,
</span></span><span style="display:flex;"><span>      <span style="color:#f92672">&#34;lastValidated&#34;</span>: <span style="color:#ae81ff">1724760126282</span>
</span></span><span style="display:flex;"><span>    },
</span></span><span style="display:flex;"><span>    {
</span></span><span style="display:flex;"><span>      <span style="color:#f92672">&#34;node&#34;</span>: <span style="color:#e6db74">&#34;2deb000b57bfac9d72c14d4ed967b572&#34;</span>,
</span></span><span style="display:flex;"><span>      <span style="color:#f92672">&#34;lastValidated&#34;</span>: <span style="color:#ae81ff">1724760126282</span>
</span></span><span style="display:flex;"><span>    }
</span></span><span style="display:flex;"><span>  ]
</span></span><span style="display:flex;"><span>}
</span></span></code></pre></div></details>
<p>This example meta data has a size of 674 Bytes, which is about 0.21% of the <code>parity block</code> size. The <code>DHT meta data</code> would require an additional 1.47TB of storage.</p>
<h3 id="overhead"><a href="https://heidenstedt.org/posts/2024/ouroboros-db-dev-journal-erasure-coding/#overhead">Overhead</a></h3><p><code>DHT meta data</code> in combination with the <code>parity block meta data</code> would have a total overhead of 0.93%% of the <code>erasure coding metadata</code> relative to the <code>erasure coding parity block size</code>, the entire metadata overhead at 100TB and 2,4 Billion <code>parity block</code>s would be 6.65TB.</p>
<p>Adding this to the overhead from the <code>erasure coding</code> which are 50% we would have a total overhead of 56.65% relative to the raw data size. If we have a utilization of each <code>chunk</code> of 1310720 Bits which has been achieved with 10MB files with random binary data. This would result in a HDD to raw data ratio of 43.35% (which is pretty good for a k6n3 config and having self healing capabilities) for following configuration:</p>
<ul>
<li>100TB of raw data</li>
<li>data deduplication via BuzzHash into chunks</li>
<li>chunk compression via Zstd</li>
<li>chunk encryption via AES256</li>
<li>each chunk becomes 6 parity blocks of which 3 can be lost without data loss</li>
</ul>
<h2 id="conclusion"><a href="https://heidenstedt.org/posts/2024/ouroboros-db-dev-journal-erasure-coding/#conclusion">Conclusion</a></h2><p>I have written a Google Calc Spreadsheet for it, which you can find here: <a href="https://docs.google.com/spreadsheets/d/12Ad4vvA0dLSOffDLz6gMkkIJkYNZE70gY0wI1Qqkg8c/edit?usp=sharing">ouroboros-db Overhead Calculator</a>.</p>
<p>Though this Spreadsheet is neat, i discoverd that i need some surface plotting to find the best configuration for the erasure coding.<br>
Sadly there is no way to do this in Google Calc, so i will need to write a small Browser App for it&hellip; maybe i use Svelte for it, i haven&rsquo;t used it in a while now.</p>
<h2 id="further-reading"><a href="https://heidenstedt.org/posts/2024/ouroboros-db-dev-journal-erasure-coding/#further-reading">Further Reading</a></h2><p>A good overview over what erasure coding is about: <a href="https://transactional.blog/blog/2024-erasure-coding">Erasure Coding for Distributed Systems</a><br>
Also See <a href="https://news.ycombinator.com/item?id=41361281">HN comments</a></p>
]]></description><content:encoded><![CDATA[<p>
      <em>Best viewed on the <a href="https://heidenstedt.org/posts/2024/ouroboros-db-dev-journal-erasure-coding/">original page</a>, where extended functionality like the
    footnote helper is available.</em>
    </p><p>This is a Dev Journal for the Ouroboros DB Project.<br>
I try to write down my thoughts and ideas Somewhat structured to have it as a reference for later and i publish it to give this information a chance to help other and to get feedback from the community.<br>
If you have feedback you can write me on my <a href="https://mastodon.social/@heidenstedt">Mastodon</a>.</p>
<p>Pls note that this is only a Dev Journal and not a Blog Post, so it may be a bit unstructured and not as polished as a Blog Post, including typos and other errors.</p>
<h2 id="tldr"><a href="https://heidenstedt.org/posts/2024/ouroboros-db-dev-journal-erasure-coding/#tldr">TL;DR:</a></h2><p>Today i worked on a refactoring of the architecture and feasibility of erasure coding for <a href="https://github.com/i5heu/ouroboros-db">Ouroboros DB</a>. There are some concerns i had regarding the potential index size and overhead that will result from it, and as it turns out, it is not as bad as i thought but i need a DHT for the index.</p>
<h2 id="architecture"><a href="https://heidenstedt.org/posts/2024/ouroboros-db-dev-journal-erasure-coding/#architecture">Architecture</a></h2><p>I think it would be super handy to be able to run parts of data pipelines (eg. storing files into chunked,compressed, encrypted and then erasure coded blocks) in a distributed manner. for safety and because i want to be able to run trustless nodes that can&rsquo;t see the raw data, it would be necessary to have the file chunking, compression and encryption part on the server the client currently speaks too, although it would be possible to pre-chunk the file on the client and upload it to different nodes, although here is the question if it is easy to port the chunking algorithm to browser JS or WASM.</p>
<h3 id="modules"><a href="https://heidenstedt.org/posts/2024/ouroboros-db-dev-journal-erasure-coding/#modules">Modules</a></h3><p>Refactoring of the code is needed to implement <code>erasure coding</code>, maybe we can even get rid of the <code>chunk</code> entirely as a stored thing since we have the data in the <code>parity block</code>s already.</p>
<p>Maybe best to add the <code>erasure coding</code> to the <code>StoreDataPipeline</code> and add the needed <code>erasure coding</code> metadata to <code>ChunkData</code>.</p>
<p>ASCII Art of the new architecture (click to get the .txt file):</p>
<a href="https://heidenstedt.org/posts/2024/ouroboros-db-dev-journal-erasure-coding/architecture.txt" target="_blank">
<p><div class="imageLoadingWrap"><img
          alt="ASCII Art of the new architecture"
          title="ASCII Art of the new architecture"
          loading="lazy"
          src="https://heidenstedt.org/posts/2024/ouroboros-db-dev-journal-erasure-coding/Architecture_hu_96d4787568ab57f9.webp"
          width="3030"
          height="871"
          srcset='/posts/2024/ouroboros-db-dev-journal-erasure-coding/Architecture_hu_2cb9192ee549f6f7.avif 100w,/posts/2024/ouroboros-db-dev-journal-erasure-coding/Architecture_hu_b05e3863fdd4ce9a.webp 100w,/posts/2024/ouroboros-db-dev-journal-erasure-coding/Architecture_hu_ab2ece513e436730.avif 200w,/posts/2024/ouroboros-db-dev-journal-erasure-coding/Architecture_hu_b80b28c6ea927abb.webp 200w,/posts/2024/ouroboros-db-dev-journal-erasure-coding/Architecture_hu_9a6a5eab6370057.avif 300w,/posts/2024/ouroboros-db-dev-journal-erasure-coding/Architecture_hu_f8a619137d8295b3.webp 300w,/posts/2024/ouroboros-db-dev-journal-erasure-coding/Architecture_hu_b9e944c8f507ee53.avif 400w,/posts/2024/ouroboros-db-dev-journal-erasure-coding/Architecture_hu_9b1c897b51eca118.webp 400w,/posts/2024/ouroboros-db-dev-journal-erasure-coding/Architecture_hu_97dc6486f99e7f3c.avif 500w,/posts/2024/ouroboros-db-dev-journal-erasure-coding/Architecture_hu_1048466be74d0ae7.webp 500w,/posts/2024/ouroboros-db-dev-journal-erasure-coding/Architecture_hu_12dba54ade38bf4a.avif 600w,/posts/2024/ouroboros-db-dev-journal-erasure-coding/Architecture_hu_4fb7e4485e6d9ba8.webp 600w,/posts/2024/ouroboros-db-dev-journal-erasure-coding/Architecture_hu_cc0d65b877a694cc.avif 700w,/posts/2024/ouroboros-db-dev-journal-erasure-coding/Architecture_hu_863c0c2a40d204e.webp 700w,/posts/2024/ouroboros-db-dev-journal-erasure-coding/Architecture_hu_a0b76a1673bd8ce1.avif 800w,/posts/2024/ouroboros-db-dev-journal-erasure-coding/Architecture_hu_48b438099fcc934f.webp 800w,/posts/2024/ouroboros-db-dev-journal-erasure-coding/Architecture_hu_5c003cfa191ab46a.avif 900w,/posts/2024/ouroboros-db-dev-journal-erasure-coding/Architecture_hu_f03d0655d1a1a4da.webp 900w,/posts/2024/ouroboros-db-dev-journal-erasure-coding/Architecture_hu_42356eb882d4c7df.avif 1000w,/posts/2024/ouroboros-db-dev-journal-erasure-coding/Architecture_hu_ea45c55f65bd9414.webp 1000w,/posts/2024/ouroboros-db-dev-journal-erasure-coding/Architecture_hu_a794c31c2c196144.avif 1100w,/posts/2024/ouroboros-db-dev-journal-erasure-coding/Architecture_hu_f8ecf047dd3ef009.webp 1100w,/posts/2024/ouroboros-db-dev-journal-erasure-coding/Architecture_hu_797db1b158ee89ac.avif 1200w,/posts/2024/ouroboros-db-dev-journal-erasure-coding/Architecture_hu_5900068a81c61786.webp 1200w,/posts/2024/ouroboros-db-dev-journal-erasure-coding/Architecture_hu_aba964185e50c26.avif 1300w,/posts/2024/ouroboros-db-dev-journal-erasure-coding/Architecture_hu_b9393aa6e5960f12.webp 1300w,/posts/2024/ouroboros-db-dev-journal-erasure-coding/Architecture_hu_311d76129c9977b3.avif 1400w,/posts/2024/ouroboros-db-dev-journal-erasure-coding/Architecture_hu_50cf36bdf8cc20a4.webp 1400w,/posts/2024/ouroboros-db-dev-journal-erasure-coding/Architecture_hu_35fb69dfc8065d4d.avif 1500w,/posts/2024/ouroboros-db-dev-journal-erasure-coding/Architecture_hu_d03d70c6373baf8d.webp 1500w,/posts/2024/ouroboros-db-dev-journal-erasure-coding/Architecture_hu_f793111d3415434c.avif 1600w,/posts/2024/ouroboros-db-dev-journal-erasure-coding/Architecture_hu_50de16a06f6c909.webp 1600w,/posts/2024/ouroboros-db-dev-journal-erasure-coding/Architecture_hu_f44d13cbf9438f06.avif 1700w,/posts/2024/ouroboros-db-dev-journal-erasure-coding/Architecture_hu_a01af93ff7648eb3.webp 1700w,/posts/2024/ouroboros-db-dev-journal-erasure-coding/Architecture_hu_1c062ad7c62ae03a.avif 1800w,/posts/2024/ouroboros-db-dev-journal-erasure-coding/Architecture_hu_9fa6b205fb52c14d.webp 1800w,/posts/2024/ouroboros-db-dev-journal-erasure-coding/Architecture_hu_e28bcab6f174e919.avif 1900w,/posts/2024/ouroboros-db-dev-journal-erasure-coding/Architecture_hu_2eb74c1265d3c7b0.webp 1900w,/posts/2024/ouroboros-db-dev-journal-erasure-coding/Architecture_hu_54f2d99662709774.avif 2000w,/posts/2024/ouroboros-db-dev-journal-erasure-coding/Architecture_hu_1e31b1da25e1eb32.webp 2000w,/posts/2024/ouroboros-db-dev-journal-erasure-coding/Architecture_hu_646abb2a2bff3fa3.avif 2100w,/posts/2024/ouroboros-db-dev-journal-erasure-coding/Architecture_hu_f089d367a3a280fc.webp 2100w,/posts/2024/ouroboros-db-dev-journal-erasure-coding/Architecture_hu_1978db329ec16d6d.avif 2200w,/posts/2024/ouroboros-db-dev-journal-erasure-coding/Architecture_hu_6ee5a59dc7098578.webp 2200w,/posts/2024/ouroboros-db-dev-journal-erasure-coding/Architecture_hu_7e06f488ab5ffa79.avif 2300w,/posts/2024/ouroboros-db-dev-journal-erasure-coding/Architecture_hu_3e2c0136633d8f0.webp 2300w,/posts/2024/ouroboros-db-dev-journal-erasure-coding/Architecture_hu_8e267fbf50cb3a52.avif 2400w,/posts/2024/ouroboros-db-dev-journal-erasure-coding/Architecture_hu_5fa6ff521302409f.webp 2400w,/posts/2024/ouroboros-db-dev-journal-erasure-coding/Architecture_hu_d1ab2af2aaec0825.avif 2500w,/posts/2024/ouroboros-db-dev-journal-erasure-coding/Architecture_hu_7fb22e5fa281cb76.webp 2500w,/posts/2024/ouroboros-db-dev-journal-erasure-coding/Architecture_hu_4569bdffbcc0329b.avif 2600w,/posts/2024/ouroboros-db-dev-journal-erasure-coding/Architecture_hu_c30c96df17374589.webp 2600w,/posts/2024/ouroboros-db-dev-journal-erasure-coding/Architecture_hu_aa217e5aa81c2f2c.avif 2700w,/posts/2024/ouroboros-db-dev-journal-erasure-coding/Architecture_hu_a6644da4d424ff65.webp 2700w,/posts/2024/ouroboros-db-dev-journal-erasure-coding/Architecture_hu_d128f208c0c6ba4.avif 2800w,/posts/2024/ouroboros-db-dev-journal-erasure-coding/Architecture_hu_bd4e57588a16519a.webp 2800w,/posts/2024/ouroboros-db-dev-journal-erasure-coding/Architecture_hu_6b01e3577982cafa.avif 2900w,/posts/2024/ouroboros-db-dev-journal-erasure-coding/Architecture_hu_c5dbd5148973c5ad.webp 2900w,/posts/2024/ouroboros-db-dev-journal-erasure-coding/Architecture_hu_5d1962d1966460a.avif 3000w,/posts/2024/ouroboros-db-dev-journal-erasure-coding/Architecture_hu_3abd1e7ef89d2c43.webp 3000w,/posts/2024/ouroboros-db-dev-journal-erasure-coding/Architecture_hu_e80e0bc829f354e7.avif 3030w,/posts/2024/ouroboros-db-dev-journal-erasure-coding/Architecture_hu_80249ad413f32588.webp 3030w
          '
          sizes="(max-width: 672px) calc(85vw - 32px), (max-width: 736px) 640px, 657px"
        ><div class="imageLoading"></div>
</div></p>
</a>
<h2 id="erasure-coding"><a href="https://heidenstedt.org/posts/2024/ouroboros-db-dev-journal-erasure-coding/#erasure-coding">Erasure Coding</a></h2><p>For now i think it is okay yo make the erasure coding a simple n6 k3, which would result in a <code>parity block</code> size of 327,680 Bytes or 0.31MB as average. This would result in about 1.875MB per chunk that is 1.25MB big in average. This would result in a 50% overhead for the erasure coding, which i think is quite okay.</p>
<p>For the goal of storing 100TB in it, which are about 400M Chunks, we would need 2,400,000,000 <code>parity block</code>s, aka 2,4 Billion.</p>
<p>If we consider following overhead for the erasure coding:</p>
<details>
  <summary>Parity Meta Data</summary>
    This is the Meta Data for a `parity block`:
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-json" data-lang="json"><span style="display:flex;"><span>{
</span></span><span style="display:flex;"><span><span style="color:#f92672">&#34;parityHash&#34;</span>: <span style="color:#e6db74">&#34;aeae379a6e857728e44164267fdb7a0e27b205d757cc19899586c89dbb221930f1813d02ff93a661859bc17065eac4d6edf3c38a034e6283a84754d52917e5b0&#34;</span>,
</span></span><span style="display:flex;"><span><span style="color:#f92672">&#34;chunkHash&#34;</span>: <span style="color:#e6db74">&#34;aeae379a6e857728e44164267fdb7a0e27b205d757cc19899586c89dbb221930f1813d02ff93a661859bc17065eac4d6edf3c38a034e6283a84754d52917e5b0&#34;</span>,
</span></span><span style="display:flex;"><span><span style="color:#f92672">&#34;sizeByte&#34;</span>: <span style="color:#ae81ff">433000</span>,
</span></span><span style="display:flex;"><span><span style="color:#f92672">&#34;lastChecked&#34;</span>: <span style="color:#ae81ff">1724760126282</span>,
</span></span><span style="display:flex;"><span><span style="color:#f92672">&#34;reblanceLog&#34;</span>: [
</span></span><span style="display:flex;"><span>    {
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">&#34;time&#34;</span>: <span style="color:#ae81ff">1724760126282</span>,
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">&#34;from&#34;</span>: <span style="color:#e6db74">&#34;2deb000b57bfac9d72c14d4ed967b572&#34;</span>
</span></span><span style="display:flex;"><span>    },
</span></span><span style="display:flex;"><span>    {
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">&#34;time&#34;</span>: <span style="color:#ae81ff">1724760126282</span>,
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">&#34;from&#34;</span>: <span style="color:#e6db74">&#34;2deb000b57bfac9d72c14d4ed967b572&#34;</span>
</span></span><span style="display:flex;"><span>    },
</span></span><span style="display:flex;"><span>    {
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">&#34;time&#34;</span>: <span style="color:#ae81ff">1724760126282</span>,
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">&#34;from&#34;</span>: <span style="color:#e6db74">&#34;2deb000b57bfac9d72c14d4ed967b572&#34;</span>
</span></span><span style="display:flex;"><span>    },
</span></span><span style="display:flex;"><span>    {
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">&#34;time&#34;</span>: <span style="color:#ae81ff">1724760126282</span>,
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">&#34;from&#34;</span>: <span style="color:#e6db74">&#34;2deb000b57bfac9d72c14d4ed967b572&#34;</span>
</span></span><span style="display:flex;"><span>    },
</span></span><span style="display:flex;"><span>    {
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">&#34;time&#34;</span>: <span style="color:#ae81ff">1724760126282</span>,
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">&#34;from&#34;</span>: <span style="color:#e6db74">&#34;2deb000b57bfac9d72c14d4ed967b572&#34;</span>
</span></span><span style="display:flex;"><span>    },
</span></span><span style="display:flex;"><span>    {
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">&#34;time&#34;</span>: <span style="color:#ae81ff">1724760126282</span>,
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">&#34;from&#34;</span>: <span style="color:#e6db74">&#34;2deb000b57bfac9d72c14d4ed967b572&#34;</span>
</span></span><span style="display:flex;"><span>    },
</span></span><span style="display:flex;"><span>    {
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">&#34;time&#34;</span>: <span style="color:#ae81ff">1724760126282</span>,
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">&#34;from&#34;</span>: <span style="color:#e6db74">&#34;2deb000b57bfac9d72c14d4ed967b572&#34;</span>
</span></span><span style="display:flex;"><span>    },
</span></span><span style="display:flex;"><span>    {
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">&#34;time&#34;</span>: <span style="color:#ae81ff">1724760126282</span>,
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">&#34;from&#34;</span>: <span style="color:#e6db74">&#34;2deb000b57bfac9d72c14d4ed967b572&#34;</span>
</span></span><span style="display:flex;"><span>    },
</span></span><span style="display:flex;"><span>    {
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">&#34;time&#34;</span>: <span style="color:#ae81ff">1724760126282</span>,
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">&#34;from&#34;</span>: <span style="color:#e6db74">&#34;2deb000b57bfac9d72c14d4ed967b572&#34;</span>
</span></span><span style="display:flex;"><span>    },
</span></span><span style="display:flex;"><span>    {
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">&#34;time&#34;</span>: <span style="color:#ae81ff">1724760126282</span>,
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">&#34;from&#34;</span>: <span style="color:#e6db74">&#34;2deb000b57bfac9d72c14d4ed967b572&#34;</span>
</span></span><span style="display:flex;"><span>    }
</span></span><span style="display:flex;"><span>],
</span></span><span style="display:flex;"><span><span style="color:#f92672">&#34;userLog&#34;</span>: [
</span></span><span style="display:flex;"><span>    {
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">&#34;time&#34;</span>: <span style="color:#ae81ff">1724760126282</span>,
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">&#34;from&#34;</span>: <span style="color:#e6db74">&#34;2deb000b57bfac9d72c14d4ed967b572&#34;</span>
</span></span><span style="display:flex;"><span>    },
</span></span><span style="display:flex;"><span>    {
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">&#34;time&#34;</span>: <span style="color:#ae81ff">1724760126282</span>,
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">&#34;from&#34;</span>: <span style="color:#e6db74">&#34;2deb000b57bfac9d72c14d4ed967b572&#34;</span>
</span></span><span style="display:flex;"><span>    },
</span></span><span style="display:flex;"><span>    {
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">&#34;time&#34;</span>: <span style="color:#ae81ff">1724760126282</span>,
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">&#34;from&#34;</span>: <span style="color:#e6db74">&#34;2deb000b57bfac9d72c14d4ed967b572&#34;</span>
</span></span><span style="display:flex;"><span>    },
</span></span><span style="display:flex;"><span>    {
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">&#34;time&#34;</span>: <span style="color:#ae81ff">1724760126282</span>,
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">&#34;from&#34;</span>: <span style="color:#e6db74">&#34;2deb000b57bfac9d72c14d4ed967b572&#34;</span>
</span></span><span style="display:flex;"><span>    },
</span></span><span style="display:flex;"><span>    {
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">&#34;time&#34;</span>: <span style="color:#ae81ff">1724760126282</span>,
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">&#34;from&#34;</span>: <span style="color:#e6db74">&#34;2deb000b57bfac9d72c14d4ed967b572&#34;</span>
</span></span><span style="display:flex;"><span>    },
</span></span><span style="display:flex;"><span>    {
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">&#34;time&#34;</span>: <span style="color:#ae81ff">1724760126282</span>,
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">&#34;from&#34;</span>: <span style="color:#e6db74">&#34;2deb000b57bfac9d72c14d4ed967b572&#34;</span>
</span></span><span style="display:flex;"><span>    },
</span></span><span style="display:flex;"><span>    {
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">&#34;time&#34;</span>: <span style="color:#ae81ff">1724760126282</span>,
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">&#34;from&#34;</span>: <span style="color:#e6db74">&#34;2deb000b57bfac9d72c14d4ed967b572&#34;</span>
</span></span><span style="display:flex;"><span>    },
</span></span><span style="display:flex;"><span>    {
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">&#34;time&#34;</span>: <span style="color:#ae81ff">1724760126282</span>,
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">&#34;from&#34;</span>: <span style="color:#e6db74">&#34;2deb000b57bfac9d72c14d4ed967b572&#34;</span>
</span></span><span style="display:flex;"><span>    },
</span></span><span style="display:flex;"><span>    {
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">&#34;time&#34;</span>: <span style="color:#ae81ff">1724760126282</span>,
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">&#34;from&#34;</span>: <span style="color:#e6db74">&#34;2deb000b57bfac9d72c14d4ed967b572&#34;</span>
</span></span><span style="display:flex;"><span>    },
</span></span><span style="display:flex;"><span>    {
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">&#34;time&#34;</span>: <span style="color:#ae81ff">1724760126282</span>,
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">&#34;from&#34;</span>: <span style="color:#e6db74">&#34;2deb000b57bfac9d72c14d4ed967b572&#34;</span>
</span></span><span style="display:flex;"><span>    }
</span></span><span style="display:flex;"><span>],
</span></span><span style="display:flex;"><span><span style="color:#f92672">&#34;this is for the KV Key&#34;</span>: <span style="color:#e6db74">&#34;aeae379a6e857728e44164267fdb7a0e27b205d757cc19899586c89dbb221930f1813d02ff93a661859bc17065eac4d6edf3c38a034e6283a84754d52917e5b0&#34;</span>
</span></span><span style="display:flex;"><span>}
</span></span></code></pre></div></details>
<p>We see that <code>parity block metadata</code> has under 2373 Bytes of overhead, which is about 0.72%% of the <code>parity block</code> size (very good). If one node would need to store the entire hash table we look at a size of 5.18TB, which means that we need a DHT for this to work. This also is the case for the Events, which introduce more storage overhead (more in <a href="https://heidenstedt.org/posts/2024/ouroboros-db-dev-journal-erasure-coding/#dht">DHT</a> )</p>
<h3 id="dht"><a href="https://heidenstedt.org/posts/2024/ouroboros-db-dev-journal-erasure-coding/#dht">DHT</a></h3><p>We need additional DHT Metadata like this:</p>
<details>
  <summary>DHT Meta Data</summary>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-json" data-lang="json"><span style="display:flex;"><span>{
</span></span><span style="display:flex;"><span>  <span style="color:#f92672">&#34;parityHash&#34;</span>: <span style="color:#e6db74">&#34;aeae379a6e857728e44164267fdb7a0e27b205d757cc19899586c89dbb221930f1813d02ff93a661859bc17065eac4d6edf3c38a034e6283a84754d52917e5b0&#34;</span>,
</span></span><span style="display:flex;"><span>  <span style="color:#f92672">&#34;storingNodes&#34;</span>: [
</span></span><span style="display:flex;"><span>    {
</span></span><span style="display:flex;"><span>      <span style="color:#f92672">&#34;node&#34;</span>: <span style="color:#e6db74">&#34;2deb000b57bfac9d72c14d4ed967b572&#34;</span>,
</span></span><span style="display:flex;"><span>      <span style="color:#f92672">&#34;lastValidated&#34;</span>: <span style="color:#ae81ff">1724760126282</span>
</span></span><span style="display:flex;"><span>    },
</span></span><span style="display:flex;"><span>    {
</span></span><span style="display:flex;"><span>      <span style="color:#f92672">&#34;node&#34;</span>: <span style="color:#e6db74">&#34;2deb000b57bfac9d72c14d4ed967b572&#34;</span>,
</span></span><span style="display:flex;"><span>      <span style="color:#f92672">&#34;lastValidated&#34;</span>: <span style="color:#ae81ff">1724760126282</span>
</span></span><span style="display:flex;"><span>    },
</span></span><span style="display:flex;"><span>    {
</span></span><span style="display:flex;"><span>      <span style="color:#f92672">&#34;node&#34;</span>: <span style="color:#e6db74">&#34;2deb000b57bfac9d72c14d4ed967b572&#34;</span>,
</span></span><span style="display:flex;"><span>      <span style="color:#f92672">&#34;lastValidated&#34;</span>: <span style="color:#ae81ff">1724760126282</span>
</span></span><span style="display:flex;"><span>    },
</span></span><span style="display:flex;"><span>    {
</span></span><span style="display:flex;"><span>      <span style="color:#f92672">&#34;node&#34;</span>: <span style="color:#e6db74">&#34;2deb000b57bfac9d72c14d4ed967b572&#34;</span>,
</span></span><span style="display:flex;"><span>      <span style="color:#f92672">&#34;lastValidated&#34;</span>: <span style="color:#ae81ff">1724760126282</span>
</span></span><span style="display:flex;"><span>    },
</span></span><span style="display:flex;"><span>    {
</span></span><span style="display:flex;"><span>      <span style="color:#f92672">&#34;node&#34;</span>: <span style="color:#e6db74">&#34;2deb000b57bfac9d72c14d4ed967b572&#34;</span>,
</span></span><span style="display:flex;"><span>      <span style="color:#f92672">&#34;lastValidated&#34;</span>: <span style="color:#ae81ff">1724760126282</span>
</span></span><span style="display:flex;"><span>    }
</span></span><span style="display:flex;"><span>  ]
</span></span><span style="display:flex;"><span>}
</span></span></code></pre></div></details>
<p>This example meta data has a size of 674 Bytes, which is about 0.21% of the <code>parity block</code> size. The <code>DHT meta data</code> would require an additional 1.47TB of storage.</p>
<h3 id="overhead"><a href="https://heidenstedt.org/posts/2024/ouroboros-db-dev-journal-erasure-coding/#overhead">Overhead</a></h3><p><code>DHT meta data</code> in combination with the <code>parity block meta data</code> would have a total overhead of 0.93%% of the <code>erasure coding metadata</code> relative to the <code>erasure coding parity block size</code>, the entire metadata overhead at 100TB and 2,4 Billion <code>parity block</code>s would be 6.65TB.</p>
<p>Adding this to the overhead from the <code>erasure coding</code> which are 50% we would have a total overhead of 56.65% relative to the raw data size. If we have a utilization of each <code>chunk</code> of 1310720 Bits which has been achieved with 10MB files with random binary data. This would result in a HDD to raw data ratio of 43.35% (which is pretty good for a k6n3 config and having self healing capabilities) for following configuration:</p>
<ul>
<li>100TB of raw data</li>
<li>data deduplication via BuzzHash into chunks</li>
<li>chunk compression via Zstd</li>
<li>chunk encryption via AES256</li>
<li>each chunk becomes 6 parity blocks of which 3 can be lost without data loss</li>
</ul>
<h2 id="conclusion"><a href="https://heidenstedt.org/posts/2024/ouroboros-db-dev-journal-erasure-coding/#conclusion">Conclusion</a></h2><p>I have written a Google Calc Spreadsheet for it, which you can find here: <a href="https://docs.google.com/spreadsheets/d/12Ad4vvA0dLSOffDLz6gMkkIJkYNZE70gY0wI1Qqkg8c/edit?usp=sharing">ouroboros-db Overhead Calculator</a>.</p>
<p>Though this Spreadsheet is neat, i discoverd that i need some surface plotting to find the best configuration for the erasure coding.<br>
Sadly there is no way to do this in Google Calc, so i will need to write a small Browser App for it&hellip; maybe i use Svelte for it, i haven&rsquo;t used it in a while now.</p>
<h2 id="further-reading"><a href="https://heidenstedt.org/posts/2024/ouroboros-db-dev-journal-erasure-coding/#further-reading">Further Reading</a></h2><p>A good overview over what erasure coding is about: <a href="https://transactional.blog/blog/2024-erasure-coding">Erasure Coding for Distributed Systems</a><br>
Also See <a href="https://news.ycombinator.com/item?id=41361281">HN comments</a></p>
]]></content:encoded></item></channel></rss>