Reverb with AutoScaling DDOs when scaling down

[jleonardolemos]

I am running Reverb behind a network load balancer and AutoScaling it.
When a scaling down event happens or a spot instance is terminated, all connections to that Reverb instance are moved for other Reverb instances with a "client reconection". The client realizes that the connection has gone and tries to reconnect.

The problem is: To reconnect to private channels the client needs to authenticate then the instance death causes a bunch of re-connections/authentications at the same second.

It dont seems to be a bug but a dangerous behavior for high traffic applications that requires auto-scaling or spot instances.
See the problem is not with the reverb instances. The problem is with the application instance that receives the authentication spike.

Wondering if there is someone with any suggestion to deal with it.

[ghost]

From a logical perspective, can't you shut down the instance gradually?

I'll make a parallel with docker containers on scale down scenario.

You stop new http connections to the container from the load balancer.
You issue SIGTERM to the target container. It receives it, and gracefully shuts down after it finishes all jobs that are in progress (not possible with supervidord because it is unaware of signals and will restart the container before it is stopped).
After the jobs finished the container is stopped.

In your situation the graceful shutdown should close connections gradually so that you can spread the re connection requests through a wider period of time.

[jleonardolemos]

It is not possible to gradually restart the reverb server.
Following your suggestions What i can do is:

1. instead of having a instance holding 10000 thousand connections i can have small ECS containers holding 1000 connections. Then the blue green deployment will take more time but the authentications requests will spreed over time too.
2. Or it seems that it is possible to override the authentication handler for the pusher client to reconnect with a random setTimeOut to spread the authentication request over time based on the client behavior.

I will try the second one first and come back to tell what happened lol.

[jleonardolemos]

Just an update for this discussion. I stop auto scaling the reverb service. It does not handle very well scaling down events.
For now keeping a fixed number of horizontal instances is the mos resilient option. This should be enought for most of the cases too.