Long running ssh with network drops

[tasuki]

Hi, this is not a bug report, just something I'm very confused about.

My use case is (I have full control over both the laptop and the server.):

1. I ssh from my laptop into a server and run vim or whatever.
2. I put my laptop to sleep.
3. Wake up, I want to see my vim or whatever still open on the server.

I've used tmux for a long time. Kovid says tmux isn't necessary, so I try with kitty+kitten:

• When I run kitty +kitten ssh the session feels more persistent than a regular ssh session. It easily survives a couple minutes of sleep. Perhaps it does something similar to mosh?
• But when I put the laptop to sleep for the night, the session is gone.
• I'd like my sessions to survive sleep. I'm fine if they drop after 168 hours of inactivity.

It would be great if this was addressed in the docs, it's such a common use case! Or perhaps it is addressed and I just can't find it...

Anyways thank you for kitty, it's marvelous!

[kovidgoyal]

That cant be done, when you sleep your laptop, the remote side will eventually assume the connection is dead and drop the connection. You could theoretically configure the remote side to wait 10 hours before dropping connections, but this is slightly non-trivial to do as it means configuring the ssh daemon and the kernel on the remote machine and it can have unwanted side effects for other connections.

[tasuki]

Kovid, thank you for your response.

I know you've mentioned in the past that this hasn't been a problem for you. I wonder, do you not have any servers you sometimes ssh into? Or do you start everything from scratch every time you connect? Or, do you secretly use tmux? 😉

[tasuki]

Ah, another option I guess is to not put one's laptop to sleep. With new laptops not supporting S3 sleep, it doesn't matter much whether the laptop sleeps or not. Maybe I'm the weird one that I set my laptop to suspend on lid close...

[kovidgoyal]

I have been running servers with multi year uptimes for multiple decades, probably since before you were born. I have **never** used terminal multiplexers, they are a complete anti-pattern, and always were, even when terminals were far less capable than they are now. And I definitely do not appreciate the insinuation that I am lying about my usage of multiplexers, continue trying to make this personal and I will block you. As for your use case, if you really find yourself needing to stay logged into a remote server overnight, **and** you really want to sleep your laptop overnight, use one of the may non-multiplexing tools that preserve remote session across disconnects. Though personally, I would advise you to change your workflow to do actual work locally rather than when logged into a server and sync your changes to the server, that's a far more productive workflow in the long run. Good luck.

[kovidgoyal]

Once you impose the discipline of working that way you will find yourself making your environments less fiddly automatically, which will boost your long term productivity tremendously. It certainly did for me when I switched to that way of working many, many years ago.

And no I wasn't talking of mosh or eternalterminal. I was talking of tmux like tools that dont do multiplexing, just run remote programs in a single pty that they forward to the host terminal without actually having to implement a full terminal emulator themselves. That avoids the complexity cascade and all the associated problems of multiplexers. Though it retains the performance drawbacks, which to be fair, dont really matter in the remote case. For example abduco, though since I dont work this way I havent really kept up with this space. You should do your own research.

The other less bad alternative is to use a multiplexer that comes with the terminal so that it at least in theory should support everything the terminal does and work well. wezterm is an example of this approach. And someday I might get around to implementing such an approach in kitty, though since I wont have any use for it personally it's not a priority. My main motivation to do so would be to avoid having to repeat myself on this subject endlessly.

[tasuki]

I used to work with less fiddly environments. These days (some of) my environments are very fiddly, through – I hope – no fault of my own. The one I'm talking about depends on an amount of data that would hardly fit on my laptop. And requires GPUs that won't fit in my laptop. Plus it's extra fiddly as there's a lot of exploratory work going on.

Yes, preserving sessions is orthogonal to terminal. There are so many things that do both it took me a long time to notice they're two distinct things. I guess mosh will work for me, I don't care about scrollback very much. Thank you for taking the time to reply and explain.

[a3lem]

that's a far more productive workflow in the long run

Hi @kovidgoyal, I have a lot of respect for your judgement. However, I think your reasoning completely overlooks that many of us are doing most of our work on remote servers.

Take my use case, for instance: I write and run machine learning code. Everything from prototyping to training to inference depends on hardware that simply doesn't fit in a laptop. I treat my laptop as a thin client for my deep learning server. The local git repos on my server are the source of truth, not some copy synced from my laptop. Training processes sometimes run for several weeks. My tmux session has multiple windows, each with multiple panes. I can't imagine having to ssh in and reattach to abduco for each pane.

The last two years I've held out hope that someday I'll be able to ditch tmux for kitty entirely. I must admit that reading your response here, I feel slightly disheartened.

I don't wish to make any demands on your time. I'm grateful for everything you've built. This is just my honest feedback as a user with different needs than yours.

[kovidgoyal]

@a3lem Feel free to work however you like, but personally, if I needed to do tasks on a remote server (something I did a lot of when working on numerical simulations for my thesis that required a super computer), rather than subjecting myself to the unavoidable lag of working on the remote server, I would work locally, sync the files to the remote machine and run jobs on the remote server based on those synced files. Then the network lag only matters while, for example, running the job, not constantly while editing the files and running the jobs. There are many, many tools you can use to accomplish this, starting with rsync.

[a3lem]

Yeah, there's always some lag – that's true. I think I've either gotten used to it, or gigabit internet just makes it less noticeable.

I'd like to know more about your approach, if you're willing to indulge me. I want to believe that what you're saying could work for my use case, because I would prefer to build on top of Kitty as opposed to Tmux.

Questions (from a sincere place of curiosity):

• How did you launch your jobs? Was this managed by HPC tools à la SLURM or did you launch the process yourself?
• How did you monitor those processes? Were you able to reattach to them somehow? How about if you needed to debug them (e.g. fix a CUDA error)?
• Did you have multiple jobs running in parallel that you needed to babysit?

[kovidgoyal]

On Tue, May 13, 2025 at 07:00:09AM -0700, adriaan wrote: Yeah, there's always some lag – that's true. I think I've either gotten used to it, or gigabit internet just makes it less noticeable. I'd like to know more about your approach, if you're willing to indulge me. I *want* to believe that what you're saying could work for my use case, because I would prefer to build on top of Kitty as opposed to Tmux. Questions (from a sincere place of curiosity): - How did you launch your jobs? Was this managed by HPC tools à la SLURM or did you launch the process yourself?

I am a simple guy, I used ssh with a control master and just ran the command, the control master meant there is no connection setup latency, so it was basically like running locally minus unavoidable network lag. Essentially I had a tiny script that first built locally to make sure there were no compiler errors, then rsync to the remote machine then build on remote machine, then run whatever command(s) I specified. My development workflow was: Work on code locally with one window in which I ran vim. Execute the code in another window in which I was at a *local* shell, as described above. Basically just press up and enter on each iteration :) Have a third window open running a remote shell. This third window was used for any temporary debugging/editing I wanted to do. I didnt persist it.

- How did you monitor those processes? Were you able to reattach to them somehow? How about if you needed to debug them (e.g. fix a CUDA error)?

Cant help you there I am firmly in the debug with print/logging camp :) But, nothing preventing you ssh-ing in and attaching to a process, see my third window above. I am not familiar with CUDA so I cant comment on any nuances there.

- Did you have multiple jobs running in parallel?

Yes, of course, cant really do numerics with just a single job. I would typically have multiple "workspaces" each a group of three windows described above. Note that it has been over a decade since I did this, so I am likely forgetting some details.