This post is a technical one for whoever is running or considering to run a seed/witness for smoke.
Since i've been running a smoke witness, I've noticed that the disk is heavily used. Ceph reported +300kb/sec writes and over 100 ops/sec. This is HIGHER of what the smoke.io network is actually writing to the blockchain.
Digging into it I found out the shared file under the /blockchain folder is the issue. (This is the file that uses the shared-file-size in config.ini (for seed, right now its about 8GB recommended).
This file is used as a "cache" or "swap" for smoked.
i'm running on NVMe as the blockchain storage but this can also apply to those that uses SSD or those who want to improve system performance.
NVMe and SSDs can get into corruptions faster then normal disks as they have "limits" on their writes. there is so much they can write to the same place until this place is no longer reliable and will be marked as bad. This makes both NVMe and SSDs suitable to store blockchain as blockchain has many reads but very few writes as it writes only new blocks. Not to mention the performance benefits compare to spin disks.
The issue with using such of storage devices as cache or swap, in my experience, is that it is dramatically reducing their lifespan and reliability over time.
The resolution for this was to move the shared file to be written directly to RAM via /opt/shm.
Please note that the system RAM requirements are higher and should be more then the shared-file-size in order for smoked to run.
As this file is volatile, if you restart the computer or if running in docker, restart the container, the file will not exist...so you need to run the smoked with --replay-blockchain command. a small fixed to contrib/smoked.run did the trick.
if [[ -s $HOME/blockchain/block_log ]]; then if [ ! -f SHARED_FILE_DIR ]; then ARGS+=" --replay-blockchain" fi elif [[ -e /var/cache/smoked/blocks.tbz2 ]]; then # init with blockchain cached in image ARGS+=" --replay-blockchain" mkdir -p $HOME/blockchain/database cd $HOME/blockchain/database tar xvjpf /var/cache/smoked/blocks.tbz2 chown -R smoked:smoked $HOME/blockchain fi
Here is a snapshot of the two Ceph cluster I have with the performance diferences.
The first cluster does not have the fix. You can see the high writes and ops/sec. Compare to the second cluster who does have the fix and as such has lower writes and lower ops/sec.
as I run everything under kubernetes, here is the change:
volumeMounts: - name: data mountPath: /var/lib/smoked/blockchain/ - name: dshm mountPath: /opt/shm/ volumes: - name: dshm emptyDir: medium: "Memory" volumeClaimTemplates: - metadata: name: data spec: accessModes: [ "ReadWriteOnce" ] storageClassName: "ceph-smoked" resources: requests: storage: 50Gi