moved shared-file to RAM - better performance, healthier disk

2 years ago

Hi Guys!

This post is a technical one for whoever is running or considering to run a seed/witness for smoke.

Since i've been running a smoke witness, I've noticed that the disk is heavily used. Ceph reported +300kb/sec writes and over 100 ops/sec. This is HIGHER of what the smoke.io network is actually writing to the blockchain.

Digging into it I found out the shared file under the /blockchain folder is the issue. (This is the file that uses the shared-file-size in config.ini (for seed, right now its about 8GB recommended).

This file is used as a "cache" or "swap" for smoked.

i'm running on NVMe as the blockchain storage but this can also apply to those that uses SSD or those who want to improve system performance.

NVMe and SSDs can get into corruptions faster then normal disks as they have "limits" on their writes. there is so much they can write to the same place until this place is no longer reliable and will be marked as bad. This makes both NVMe and SSDs suitable to store blockchain as blockchain has many reads but very few writes as it writes only new blocks. Not to mention the performance benefits compare to spin disks.

The issue with using such of storage devices as cache or swap, in my experience, is that it is dramatically reducing their lifespan and reliability over time.

The resolution for this was to move the shared file to be written directly to RAM via /opt/shm.

Please note that the system RAM requirements are higher and should be more then the shared-file-size in order for smoked to run.

As this file is volatile, if you restart the computer or if running in docker, restart the container, the file will not exist...so you need to run the smoked with --replay-blockchain command. a small fixed to contrib/smoked.run did the trick.

if [[ -s $HOME/blockchain/block_log ]]; then
    if [ ! -f SHARED_FILE_DIR ]; then
        ARGS+=" --replay-blockchain"
    fi
elif [[ -e /var/cache/smoked/blocks.tbz2 ]]; then
    # init with blockchain cached in image
    ARGS+=" --replay-blockchain"
    mkdir -p $HOME/blockchain/database
    cd $HOME/blockchain/database
    tar xvjpf /var/cache/smoked/blocks.tbz2
    chown -R smoked:smoked $HOME/blockchain
fi

Here is a snapshot of the two Ceph cluster I have with the performance diferences.


The first cluster does not have the fix. You can see the high writes and ops/sec. Compare to the second cluster who does have the fix and as such has lower writes and lower ops/sec.

as I run everything under kubernetes, here is the change:

         volumeMounts:
        - name: data
          mountPath: /var/lib/smoked/blockchain/
        - name: dshm
          mountPath: /opt/shm/
      volumes:
      - name: dshm
        emptyDir:
          medium: "Memory"
  volumeClaimTemplates:
  - metadata:
      name: data
    spec:
      accessModes: [ "ReadWriteOnce" ]
      storageClassName: "ceph-smoked"
      resources:
        requests:
          storage: 50Gi

#seed #witness #technical #superclutch



Get 4.2 Free Smoke Power On Sign Up To Start Your Journey On Smoke! The First Cannabis Community That Pays You To Post And Curate Content You Love..
Sort Order:  Trending

I forgot to mention the fix to the config.ini

shared-file-dir = /opt/shm/

Back in the day, the days before dual cores, I used to do this for my windows machines. Talk about a flashback lol.

·

:-) the days of RAM Disk