State of the Witnesses #27 - ⭐️⭐️SPECIAL "HOT FIX" EDITION ⭐️⭐️

3 months ago

📄 What is 'State of the Witnesses' ?

State of the witnesses will be a regular update provided to the community on the status of the Smoke.io witnesses. The intent is to keep members informed of what is happening on the platform and what witnesses are doing. It is important that members of the community vote for witnesses that represent them and are driving the platform in the right direction. These updates are not intended to be all inclusive, but should only be a portion of the information used for your decision making.


📰 Current Update

Well I started on an update to the last "special edition" and then all hell broke loose here on smoke. I will get into that in a minute, but just a quick follow up from last time and want to give a shout out to @cannaweedness and @cryptosmokers whom both immediately address and fixed the instability of their witness nodes. Unfortunately @trees did not and the witness continued missing blocks daily until the chain stalled on Tuesday.

What happened to Smoke.io

Short answer is that the chain fell below the 21 active witness needed and stalled.

About 3:30PM EST on Tuesday I noticed that initminers where showing in the missed block reporting. I found this strange since initminers were used during testnet but where decommissioned prior to mainnet launch. Mainnet was completely community run.




I immediately reach out the the lead developer but within 10 minutes the chain had stalled.


Why has it taken so long to fix?

Short answer is for any changes to be made consensus needs to be reached between 75% of the 21 active witnesses. So although a hot fix was quickly produced to restart the chain we needed 16 witness to load the hot fix and coordinate a restarting of the chain.

This is no easy task as witness are from all parts of the world and some have remained anonymous. We have two main methods of communicating with witness, Telegram and Discord. I immediately started the coordination effort between the two groups and asked witness that knew other witnesses to reach out to them.

@Smoke-Witnesses IMPORTANT - PLEASE READ ENTIRE MESSAGE
You are probably aware by now that the chain halted yesterday due to less than 21 witnesses. @baabeetaa#0782 has released a hot fix to get the chain back up. It requires a coordinated effort on our part. I will try to head up the coordination but ask each of you to do your best to make this happen. We need 16 active witnesses to install the hot fix before we can restart the chain. The hot fix (v0.0.7) is available here:
https://github.com/smokenetwork/smoked/releases/tag/v0.0.7
If you are an experienced admin and want to build from source you can. Otherwise there are prebuilt binaries available. For witness & seed nodes the lm (low-memory) version is recommended. --replay-blockchain is not needed unless you change versions or plugins in config.ini
I have also updated my witness guide (v1.5) with the new level.
https://whaleshares.info/wls-download/Smoke.io%20Witness%20Guide%20v1.5.pdf
NOTE:
DO NOT START YOUR WITNESS AFTER YOU UPGRADE TO v0.0.7
There is only about an 8 hour window for 16 witnesses to upgrade and become active before the chain will halt again. I will track feedback from witnesses as they upgrade and when we hit 16 we can they give the OK to start witnesses again (and the 8 hour window) and will pick one witness to force a block production to kick the chain off again.
Once the chain gets restored There are ~4 days for adding new witnesses to make the network stronger (> 30 witnesses). Eg., bring back initminers, create new witnesses or if you are running active/backup nodes then you could run 2 separated witnesses on 2 nodes instead. We can dig more into this after we get the chain back up.
Here was the witness status when chain hung. I will use this to track witnesses that updated to v0.0.7 and when we hit the needed 16 updated witness we can attempt to restart the chain. PLEASE keep us updated either in Discord or Telegram of your status and if you have ways to contact any witnesses that are not active in either of these groups, please reach out to them and be a conduit to us of their status. We need a coordinated effort to get the chain back up. Also, when providing status please indicate your witness name as it may be different than your Discord/Telegram name.

It took a few days to get enough witness updated to the hot fix and coordinated to relaunch the chain. This morning we finally reached it.

[email protected]
[email protected]
[email protected]                
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected] (DISABLED)
[email protected]
[email protected]
[email protected]
[email protected]

To give everyone a little heads up, I announced that we would attempt to restart the chain at 5PM UTC.

So everything is back to normal?


The fact that you are reading this is a good sign and the chain is back up. We are not out of the woods yet though. The hot fix gives us 8 hours for 16 witnesses to reach consensus on v0.0.7 otherwise the chain will stall again. Once consensus is reached we have about 4 days to get up to 30 witnesses. After about 10 minutes of the chain restart we have 5 witnesses producing blocks.
Screen Shot 2019-11-16 at 12.09.29 PM.png

How do we prevent this in the future


In the interim initminers will be spun up to get us to 30 witnesses. This is not the ideal situation as it effects the decentralization of the platform. It is imperative that we build the witness pool to 30 or above. I have recruited 2 DPOS veterans that will be starting up witnesses once their smoke accounts are approved and I know others have been recruiting new witnesses as well.

What can members do to help


It is real simple, support the witnesses. Running a witness cost real dollars and is not a profitable adventure, that is why you have seen so many witnesses come and go. They think they can make some money and quickly find out that is not the case. I would suggest each witness publish a weekly post and that the members support those posts with their votes. If we can support our witnesses we can have a strong platform. We do have quite a few witnesses that have been supporting the platform from early on and deserve your support.

THANK YOU


I want to give a big THANK YOU to ALL the witnesses that made this restart happen. Many stepped up and helped others. Without this cooperation and help you would not be reading this post.


🎯 Get Involved!


✅ Support witnesses that support Smoke.io



Make your support count by going to Witness Voting and setting @bbq-iguana on 🔥!




#stateofthewitnesses #witness-update #witness #witness-guide


Get 4.2 Free Smoke Power On Sign Up To Start Your Journey On Smoke! The First Cannabis Community That Pays You To Post And Curate Content You Love..
Sort Order:  Trending

Thanks for all the effort you are putting in coordinating and solving this

Very well written as always my friend, glad to have you helping orchestrate and report on everything!

Damn hosting companies.

Thanks so much for the effort, update, and the information!

Ohh shit, I have been out for a while and you guys already breaking our awesome blockchain ?!?! :)))

Seriously though we should try to avoid such situation, we need a few more active witnesses...

Lets get Smoke.io a little boost of vitality ;)

Thanks for the coordination guys!

Thanks to you and other witnesses for the collective efforts to get the chain up and running... Bravo for the great job

Thank you for the job well done.

Glad to see the network back online. Would love to see more witnesses being recruited to smoke & would gladly upvote witness posts to supplement their witness rewards.

·

We have recruited a few new witnesses that are already running. Tomorrow, once things settle a bit I will post a list of new witnesses.

I really missed smoke.io 😢 Glad that all is fixed and working again 😊

That is great that the platform is running again. Good job guys! Thanks to all Devs and Witnesses!

So happy we are back. Voted.

I am using hotbox. Does hotbox upgrated?

·

I don't know, jrswab created and maintains hotbox. His witness has not updated, if that is any indication.

·

@jrswab how to fix the problem, will you publish a new version of hotbox?

·
·
·
·
·

Yes, it's ok now, I am producing block again. Thank you very much @jrswab

·
·

I don't think we can use hotbox / docker right now, we have to use screen instead to keep it going! I'm following chronocrypto's old guide, not sure if those seeds are good though because I just got another error on my config again. I need updated good seed nodes. Where do i look up bbq-iguana's seed node and port? I did a search and did not find it. Possibly post all the steps to get it up? Once I figure it out i will try to help.
here:
https://whaleshares.info/wls-download/Smoke.io%20Witness%20Guide%20v1.5.pdf

Thank you from the bottom of this girl's heart for explaining this. I am quite computer behind-the-scenes illiterate and this was a basic explanation that even I could understand!

LOL.

·

Sounds like the perfect time for groups of "New" witnesses to come "Help" the chain out. ;-)