From fc940e5b62630137d9da1a3f41826d2d4fe55a40 Mon Sep 17 00:00:00 2001 From: Hein Dauven Date: Thu, 28 May 2026 10:43:27 +0200 Subject: [PATCH 1/2] docs: add operator recovery guides --- .../docs/learn/guides/staking-basics.md | 2 +- .../operator/guides/rollback-node-update.md | 89 +++++++++++++ .../docs/operator/guides/slashing-recovery.md | 117 ++++++++++++++++++ .../docs/operator/guides/upgrade-node.mdx | 8 ++ .../docs/operator/maintenance-monitoring.md | 2 + src/content/docs/operator/troubleshooting.md | 5 + src/sidebars/siteSidebar.js | 2 + 7 files changed, 224 insertions(+), 1 deletion(-) create mode 100644 src/content/docs/operator/guides/rollback-node-update.md create mode 100644 src/content/docs/operator/guides/slashing-recovery.md diff --git a/src/content/docs/learn/guides/staking-basics.md b/src/content/docs/learn/guides/staking-basics.md index 4e74ac2d..031a461d 100644 --- a/src/content/docs/learn/guides/staking-basics.md +++ b/src/content/docs/learn/guides/staking-basics.md @@ -55,7 +55,7 @@ For reward sources and distribution details, see: Dusk uses **soft slashing**: stake is not burned, but repeated faults or long downtime can suspend rewards and reduce effective stake. -See: [Slashing](/learn/tokenomics#slashing). +See: [Slashing](/learn/tokenomics#slashing). If you run a provisioner, also read [Slashing prevention and recovery](/operator/guides/slashing-recovery). ## Adding to an existing stake diff --git a/src/content/docs/operator/guides/rollback-node-update.md b/src/content/docs/operator/guides/rollback-node-update.md new file mode 100644 index 00000000..4bd39580 --- /dev/null +++ b/src/content/docs/operator/guides/rollback-node-update.md @@ -0,0 +1,89 @@ +--- +title: Roll back a node update +description: Learn how to reinstall a previous Dusk node installer release when an update needs to be rolled back. +--- + +Rolling back means reinstalling a previous `node-installer` release. This can be useful if a newly installed node release has a problem and the network has not yet activated a protocol change that requires it. + +:::caution +Do not roll back across a network upgrade or activation unless the Dusk team explicitly instructs operators to do so. If the chain already requires the newer node version, an older node may stop following the network. +::: + +## Before you roll back + +Check the version currently installed: + +```sh +ruskquery version +``` + +Check whether the node is syncing: + +```sh +ruskquery block-height +tail -n 50 /var/log/rusk.log +``` + +If the node is only stuck or behind, try [fast-sync](/operator/guides/fast-sync) or [manual resync](/operator/guides/manual-resync) before rolling back. + +## Roll back mainnet + +Replace `vX.Y.Z` with the installer release you want to roll back to. + +```sh +INSTALLER_VERSION="vX.Y.Z" + +sudo service rusk stop + +curl --proto '=https' --tlsv1.2 -sSfL \ + "https://github.com/dusk-network/node-installer/releases/download/${INSTALLER_VERSION}/node-installer.sh" \ + | sudo bash + +sudo service rusk start +``` + +## Roll back testnet + +Use the same pinned installer release, but pass the testnet flag. + +```sh +INSTALLER_VERSION="vX.Y.Z" + +sudo service rusk stop + +curl --proto '=https' --tlsv1.2 -sSfL \ + "https://github.com/dusk-network/node-installer/releases/download/${INSTALLER_VERSION}/node-installer.sh" \ + | sudo bash -s -- --network testnet + +sudo service rusk start +``` + +## Verify the rollback + +Confirm the installed version: + +```sh +ruskquery version +``` + +Check the service: + +```sh +service rusk status +``` + +Check whether the node is progressing: + +```sh +ruskquery block-height +tail -F /var/log/rusk.log +``` + +If block height does not progress, compare your height with the explorer and consider [fast-syncing the node](/operator/guides/fast-sync). + +## Notes + +- Use a pinned installer release URL. Do not use `latest` for rollback. +- Rollback changes the installed node software and service configuration. It does not automatically restore an older chain state. +- Archive nodes may have database migrations that are not safe to downgrade. If an archive node fails after rollback, update back to the supported version or resync the archive state. +- If you operate a provisioner, monitor the node after rollback. Downtime or running an incompatible version can affect consensus participation. See [Slashing prevention and recovery](/operator/guides/slashing-recovery). diff --git a/src/content/docs/operator/guides/slashing-recovery.md b/src/content/docs/operator/guides/slashing-recovery.md new file mode 100644 index 00000000..1fcc9ba5 --- /dev/null +++ b/src/content/docs/operator/guides/slashing-recovery.md @@ -0,0 +1,117 @@ +--- +title: Slashing prevention and recovery +description: Learn how Dusk soft slashing affects provisioners and what to check after a slashing event. +--- + +Dusk uses **soft slashing** to discourage repeated faults and long downtime. Stake is not burned, but a provisioner can lose eligibility or effective participation, which reduces rewards. + +Soft slashing can happen when a provisioner repeatedly fails to participate correctly in consensus. Common operational causes include: + +- Running an outdated or incompatible node version. +- Being offline for too long. +- Falling behind the network tip. +- Network or firewall problems that prevent consensus messages from being sent or received. +- Consensus key or node configuration issues. + +## If your provisioner was slashed + +Start by restoring healthy node operation. Slashing is a symptom; the first priority is to make sure the node is on the right chain, on the right version, and progressing. + +### 1. Check the installed version + +```sh +ruskquery version +``` + +If the network has a required release, upgrade to it: + +```sh +curl --proto '=https' --tlsv1.2 -sSfL https://github.com/dusk-network/node-installer/releases/latest/download/node-installer.sh | sudo bash +sudo service rusk start +``` + +For testnet: + +```sh +curl --proto '=https' --tlsv1.2 -sSfL https://github.com/dusk-network/node-installer/releases/latest/download/node-installer.sh | sudo bash -s -- --network testnet +sudo service rusk start +``` + +If a newly installed release is known to be problematic and the network can still run the previous version, follow [Rollback a node update](/operator/guides/rollback-node-update). + +### 2. Check sync status + +```sh +ruskquery block-height +``` + +Compare the height with the explorer. If the node is stuck or far behind, use [fast-sync](/operator/guides/fast-sync): + +```sh +download_state +sudo service rusk start +``` + +### 3. Check service status and logs + +```sh +service rusk status +tail -n 100 /var/log/rusk.log +``` + +Look for errors related to: + +- wrong network or chain mismatch +- consensus key loading +- peer discovery +- Kadcast address or UDP connectivity +- state/database errors + +### 4. Check staking status + +```sh +rusk-wallet stake-info +``` + +Confirm that the expected stake is still present and inspect the reported stake state. If the wallet cannot connect, fix node connectivity first. + +### 5. Monitor after recovery + +After the node is updated, synced, and running, keep the logs open and check the block height repeatedly: + +```sh +tail -F /var/log/rusk.log +ruskquery block-height +``` + +The height should continue to progress. If the node falls behind again, treat it as an unresolved infrastructure or networking issue. + +### 6. Unstake and restake if the node is healthy + +If the node is on the right version, fully synced, and operating normally, unstake and restake to restore normal provisioner participation. + +Slashing can be caused by temporary operational issues, such as cloud provider downtime, network disruption, or downtime during a live node update. If the underlying issue is gone, restaking is the recovery step. + +```sh +rusk-wallet unstake +rusk-wallet stake --amt +``` + +Replace `` with the amount you want to stake. The new stake must mature before it starts participating again. + +## Prevention checklist + +- Keep the node updated during announced network upgrades. +- Monitor `ruskquery block-height` against the explorer. +- Alert on service downtime and repeated restart loops. +- Keep UDP `9000` reachable for Kadcast. +- Keep consensus keys backed up and readable by the Rusk service. +- Avoid running experimental or mismatched binaries on a staked provisioner. +- Use [fast-sync](/operator/guides/fast-sync) when a node falls behind instead of waiting for a long resync from genesis. + +## Related guides + +- [Upgrade a node](/operator/guides/upgrade-node) +- [Roll back a node update](/operator/guides/rollback-node-update) +- [Fast-sync a node](/operator/guides/fast-sync) +- [Troubleshooting](/operator/troubleshooting) diff --git a/src/content/docs/operator/guides/upgrade-node.mdx b/src/content/docs/operator/guides/upgrade-node.mdx index f353dd72..168cbd21 100644 --- a/src/content/docs/operator/guides/upgrade-node.mdx +++ b/src/content/docs/operator/guides/upgrade-node.mdx @@ -65,6 +65,14 @@ Unable to figure it out yourself? Visit our [Node Runner Troubleshooting](https: If everything else fails, check out the [manual resync](/operator/guides/manual-resync) instructions. +## Roll back an update + +If a newly installed release causes problems and the network has not activated changes that require it, you can reinstall a previous `node-installer` release. + +Use a pinned installer release instead of `latest`, and only roll back across a network upgrade if the Dusk team instructs operators to do so. + +See: [Roll back a node update](/operator/guides/rollback-node-update). + ## Nocturne Reset :::note[Info] diff --git a/src/content/docs/operator/maintenance-monitoring.md b/src/content/docs/operator/maintenance-monitoring.md index 7280dc68..fb444a27 100644 --- a/src/content/docs/operator/maintenance-monitoring.md +++ b/src/content/docs/operator/maintenance-monitoring.md @@ -18,6 +18,8 @@ The recommended setup for network participants looking to stake and use the netw Effective monitoring and alerting systems are crucial to avoid slashing events. There are several tools available for real-time monitoring and alerting, which are particularly important for provisioners participating in consensus. Implementing these systems helps ensure continuous performance and timely responses to potential issues. +At minimum, monitor whether the Rusk service is running, whether `ruskquery block-height` is progressing, and whether your node is close to the network tip. If your provisioner was slashed or is at risk of being slashed, follow [Slashing prevention and recovery](/operator/guides/slashing-recovery). + ## Keys Management Proper management of your cryptographic keys is essential to ensure the security of your node. diff --git a/src/content/docs/operator/troubleshooting.md b/src/content/docs/operator/troubleshooting.md index 4b9ec231..8ab9f9bf 100644 --- a/src/content/docs/operator/troubleshooting.md +++ b/src/content/docs/operator/troubleshooting.md @@ -16,6 +16,11 @@ This means your node is receiving consensus messages for rounds much higher than Check your block height (`ruskquery block-height`) against the explorer. If you are stuck, consider fast-syncing (`download_state`) or a manual resync. +#### My provisioner was slashed +First, restore healthy node operation: check the installed version, sync height, service status, logs, and staking status. Then monitor the node to make sure it keeps progressing. + +Follow: [Slashing prevention and recovery](/operator/guides/slashing-recovery). + #### Unable to resolve domain: invalid socket address Such errors usually indicate DNS problems. Check your DNS settings. diff --git a/src/sidebars/siteSidebar.js b/src/sidebars/siteSidebar.js index 36f3ce38..fd2570a1 100644 --- a/src/sidebars/siteSidebar.js +++ b/src/sidebars/siteSidebar.js @@ -106,6 +106,8 @@ const siteSidebar = [ { label: "Fast-Sync a Node", link: "/operator/guides/fast-sync" }, { label: "Manually Re-Sync a Node", link: "/operator/guides/manual-resync" }, { label: "Upgrade a Node", link: "/operator/guides/upgrade-node" }, + { label: "Roll Back a Node Update", link: "/operator/guides/rollback-node-update" }, + { label: "Slashing Recovery", link: "/operator/guides/slashing-recovery" }, { label: "Choose a Network", link: "/operator/networks" }, { label: "Maintenance & Monitoring", link: "/operator/maintenance-monitoring" }, { label: "FAQ", link: "/operator/faq" }, From 4502f46c5f37869b10cd939b013257cd5d9b815a Mon Sep 17 00:00:00 2001 From: Hein Dauven Date: Thu, 28 May 2026 11:51:14 +0200 Subject: [PATCH 2/2] Update src/content/docs/operator/guides/rollback-node-update.md Co-authored-by: Neotamandua <107320179+Neotamandua@users.noreply.github.com> --- src/content/docs/operator/guides/rollback-node-update.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/src/content/docs/operator/guides/rollback-node-update.md b/src/content/docs/operator/guides/rollback-node-update.md index 4bd39580..77654704 100644 --- a/src/content/docs/operator/guides/rollback-node-update.md +++ b/src/content/docs/operator/guides/rollback-node-update.md @@ -30,6 +30,10 @@ If the node is only stuck or behind, try [fast-sync](/operator/guides/fast-sync) Replace `vX.Y.Z` with the installer release you want to roll back to. +For example, to roll back to installer release `v1.2.3`, set: + +```sh +INSTALLER_VERSION="v1.2.3" ```sh INSTALLER_VERSION="vX.Y.Z"