add retries to leaveEtcdcluster

[laverya]

What this PR does / why we need it:

        error: unable to leave etcd cluster: exit status 1: {"level":"warn","ts":"2025-04-05T12:39:27.465388Z","logger":"etcd-client","caller":"[email protected]/retry_interceptor.go:63","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000830000/127.0.0.1:2379","attempt":0,"error":"rpc error: code = Unavailable desc = etcdserver: server stopped"}
        time="2025-04-05 12:39:27" level=error msg="Failed to delete node from cluster" peerID=314ee8343a8b0c36 peerURL="https://172.17.0.4:2380"
        Error: etcdserver: server stopped

This isn't desirable.

Which issue(s) this PR fixes:

Does this PR require a test?

Does this PR require a release note?

Does this PR require documentation?

[github-actions]

This PR has been released (on staging) and is available for download with a embedded-cluster-smoke-test-staging-app license ID.

Online Installer:

curl "https://staging.replicated.app/embedded/embedded-cluster-smoke-test-staging-app/ci/appver-dev-7ffa674" -H "Authorization: $EC_SMOKE_TEST_LICENSE_ID" -o embedded-cluster-smoke-test-staging-app-ci.tgz

Airgap Installer (may take a few minutes before the airgap bundle is built):

curl "https://staging.replicated.app/embedded/embedded-cluster-smoke-test-staging-app/ci-airgap/appver-dev-7ffa674?airgap=true" -H "Authorization: $EC_SMOKE_TEST_LICENSE_ID" -o embedded-cluster-smoke-test-staging-app-ci.tgz

Happy debugging!

[sgalsaleh]

this used to return an error before, is this intentional?

[laverya]

Yes - we can still reset the actual node at that point and so we should