📄 What is 'State of the Witnesses' ?
State of the witnesses will be a regular update provided to the community on the status of the Smoke.io witnesses. The intent is to keep members informed of what is happening on the platform and what witnesses are doing. It is important that members of the community vote for witnesses that represent them and are driving the platform in the right direction. These updates are not intended to be all inclusive, but should only be a portion of the information used for your decision making.
📰 Current Update
Well I started on an update to the last "special edition" and then all hell broke loose here on smoke. I will get into that in a minute, but just a quick follow up from last time and want to give a shout out to @cannaweedness and @cryptosmokers whom both immediately address and fixed the instability of their witness nodes. Unfortunately @trees did not and the witness continued missing blocks daily until the chain stalled on Tuesday.
What happened to Smoke.io
Short answer is that the chain fell below the 21 active witness needed and stalled.
About 3:30PM EST on Tuesday I noticed that initminers where showing in the missed block reporting. I found this strange since initminers were used during testnet but where decommissioned prior to mainnet launch. Mainnet was completely community run.
I immediately reach out the the lead developer but within 10 minutes the chain had stalled.
Why has it taken so long to fix?
Short answer is for any changes to be made consensus needs to be reached between 75% of the 21 active witnesses. So although a hot fix was quickly produced to restart the chain we needed 16 witness to load the hot fix and coordinate a restarting of the chain.
This is no easy task as witness are from all parts of the world and some have remained anonymous. We have two main methods of communicating with witness, Telegram and Discord. I immediately started the coordination effort between the two groups and asked witness that knew other witnesses to reach out to them.
@Smoke-Witnesses IMPORTANT - PLEASE READ ENTIRE MESSAGE
You are probably aware by now that the chain halted yesterday due to less than 21 witnesses. @baabeetaa#0782 has released a hot fix to get the chain back up. It requires a coordinated effort on our part. I will try to head up the coordination but ask each of you to do your best to make this happen. We need 16 active witnesses to install the hot fix before we can restart the chain. The hot fix (v0.0.7) is available here:
If you are an experienced admin and want to build from source you can. Otherwise there are prebuilt binaries available. For witness & seed nodes the lm (low-memory) version is recommended. --replay-blockchain is not needed unless you change versions or plugins in config.ini
I have also updated my witness guide (v1.5) with the new level.
DO NOT START YOUR WITNESS AFTER YOU UPGRADE TO v0.0.7
There is only about an 8 hour window for 16 witnesses to upgrade and become active before the chain will halt again. I will track feedback from witnesses as they upgrade and when we hit 16 we can they give the OK to start witnesses again (and the 8 hour window) and will pick one witness to force a block production to kick the chain off again.
Once the chain gets restored There are ~4 days for adding new witnesses to make the network stronger (> 30 witnesses). Eg., bring back initminers, create new witnesses or if you are running active/backup nodes then you could run 2 separated witnesses on 2 nodes instead. We can dig more into this after we get the chain back up.
Here was the witness status when chain hung. I will use this to track witnesses that updated to v0.0.7 and when we hit the needed 16 updated witness we can attempt to restart the chain. PLEASE keep us updated either in Discord or Telegram of your status and if you have ways to contact any witnesses that are not active in either of these groups, please reach out to them and be a conduit to us of their status. We need a coordinated effort to get the chain back up. Also, when providing status please indicate your witness name as it may be different than your Discord/Telegram name.
It took a few days to get enough witness updated to the hot fix and coordinated to relaunch the chain. This morning we finally reached it.
[email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] _v0.0.[email protected] [email protected] [email protected] (DISABLED) [email protected] [email protected] [email protected] [email protected]
To give everyone a little heads up, I announced that we would attempt to restart the chain at 5PM UTC.
So everything is back to normal?
The fact that you are reading this is a good sign and the chain is back up. We are not out of the woods yet though. The hot fix gives us 8 hours for 16 witnesses to reach consensus on v0.0.7 otherwise the chain will stall again. Once consensus is reached we have about 4 days to get up to 30 witnesses. After about 10 minutes of the chain restart we have 5 witnesses producing blocks.
How do we prevent this in the future
In the interim initminers will be spun up to get us to 30 witnesses. This is not the ideal situation as it effects the decentralization of the platform. It is imperative that we build the witness pool to 30 or above. I have recruited 2 DPOS veterans that will be starting up witnesses once their smoke accounts are approved and I know others have been recruiting new witnesses as well.
What can members do to help
It is real simple, support the witnesses. Running a witness cost real dollars and is not a profitable adventure, that is why you have seen so many witnesses come and go. They think they can make some money and quickly find out that is not the case. I would suggest each witness publish a weekly post and that the members support those posts with their votes. If we can support our witnesses we can have a strong platform. We do have quite a few witnesses that have been supporting the platform from early on and deserve your support.
I want to give a big THANK YOU to ALL the witnesses that made this restart happen. Many stepped up and helped others. Without this cooperation and help you would not be reading this post.
🎯 Get Involved!
- Interested in becoming a witness? Check out my Smoke Witness Guide .