A quick recap
The Light Hotspot activation was activated a few days later than expected. It should have begun on May 3rd, but it actually started on May 11th, 2022. The actual activation process went smoothly.
A lot of additional changes were also added. Like an increase to Challenges per block from 200 to 400. This will equate do ~1 beacon per Hotspot per day. The main goal is to reach 800 Challenges at a later time. During the Light Hotspot activation, the team also identified additional performance improvements for Validators and Hotspots.
A day later more issues were found. One of them was 0 Witness Receipts which was fixed.
“Some properly-configured Validators are unable to establish connections to Hotspots (this can affect Proof-of-Coverage as that’s the main communication channel to receive and send receipts). We’re still digging on this, but it will require a Validator release”, said the official core announcement.
Also, the team found that around 30% of all Validators are still with unopened ports and misconfigured. “An unopened port means this Validator cannot receive PoC receipts from Hotspots. This negatively affects rewards, and if your Hotspot tried to send a receipt to this type of Validator, the rewards will looks like a flatline”, the team added. This issue will slow down the ramp up of Challenges to avoid overloading Validators.
On May 13th, several bug fixes were released to Validators and Hotspots. It was intended to be the last update for the day, but something changed…
“All hands on deck”
Again on May 13th, the core team found out Proof-of-Coverage (PoC) Rates and Lower Rewards along with “stuck” Validators. The results were rejected PoC receipts and failed reconnections from the Hotspot to another Validator.
This triggered a multitude of updates of around every two hours. A lot of immediate fixes were added and implemented. All Validators were alerted for an immediate update. Steps were taken to develop methods to remove Validators from PoC.
The core team also increased the Challenges per block to 1000 to make up for lower PoC rates. A mandatory Validator fix was released. And a day later the Challenges were returned back to 400 as the updates seemed to be working.
All good, but not quite
On May 16th, the development team announced that PoC is still “not normal for many users” which results in “reduced hotspot activity or rewards”.
“No action is needed for hotspot owners. Any fixes or upgrades to improve hotspot Proof of Coverage will be automatically applied. Thank you for your continued patience”, said the team.
The team also released an official statement. “This is the biggest change to our network since introducing Validators, and a change of this size has never been attempted before”, it says. Several updates were released to fix Validators and the team then monitored the situation.
Close to a Light Hotspot roll back
On May 17th, the core team announced improvements were slow and a lot of Validators were offline, hopefully for the upgrade. Despite that, the team announced it is ready to roll back the entire Light Hotspot update if PoC doesn’t improve within a few hours.
Luckily, the update seemed to be working. Later during the day, the Core team announced “We are staying the course”. This means Light Hotspots will remain as is and the team found additional tasks and bugs to fix. They prove to be a bit more challenging as the developers need more time to fix them.
There’s Light in the tunnel
While all of this may seem a bit troubling, actually the situation is not that bad. Firstly, unless a Hotspot owner has messed with the sys.config file on their device, then they don’t need to do anything. All of the issues are handled by the core team. This also includes pushing all needed updates to the devices. All owners have to do is keep them on and connected to the Internet.
“The Light Hotspot transition was always meant to be a ramp-up. As a decentralized network, each party has their own priorities and motivations. Everyone, including Makers and Validators have kept pace with our improvements and we’re extremely thankful”, says the official core team statement.
“Since the Light Hotspot transition, there have been zero interruptions to device packet transfers and the blockchain continued moving; the Network is very much alive. Despite these wins, we recognize that Hotspots created this network and the Proof-of-Coverage issues since launch were not what we expected, and for that, we apologize”, it also adds.
The transition is not complete and we will see a few more bumps before it is smooth sailing, the team also notes.
All of this is generally par for the course for any big change and upgrade. It doesn’t mean there’s a fundamental issue with the Network. Closed testing always only reveals the major potential issues, but when any upgrade is “released in the wild”, there are other unforeseen bugs that arise. The good thing with Helium is that there are a lot of people working constantly so fixes come quick and updates are arriving to the devices within hours of release. So, a bit more patience will be enough to get through the challenges and enjoy the benefits of the upgrades in the long term.
