When you run a command, in either the foreground or the background, on a Linux or Unix system and then either log out or get disconnected, the command you were running is normally killed.  However, there are ways to prevent this "hangup" situation from occurring.

nohup

The first and oldest method is to use the "nohup" (no hang up) command.   The "nohup" command instructs your process to ignore the SIGHUP signal that would normally shut it down at logout or when your session disconnects. This allows you to leave time-consuming processes to complete without your having to remain logged in. By default, the output of the command you are running will be written to a file named "nohup.out" in your working directory so that you can find it the next time you log in. You will generally want to change the name of this file once you log back in, not least of which is  that "nohup" appends
to this file.  Running multiple "nohup" commands in the same working directory can result in a nohup.out file that's a mixture or interleaving of different outputs.  You can send the output to any file by simply using file redirection  to a file of your naming choice.

Note that a process running via "nohup" is only immune to SIGHUP signals, which is enough to keep it from shutting down when you log off or get disconnected (in most cases).  If you want to stop a process running under "nohup", simply send a normal "kill" command (which sends a SIGTERM signal) or a "kill -9" (which sends a SIGKILL) to it.

The "nohup" command is best for non-interactive commands, or interactive commands that do all the interaction up-front and then run for a long time without any further interaction.

at and batch

In many cases, you can accomplish the same type of longevity by running your command as a "batch" or "at" job.  You can do this via the "batch" or "at" commands.  The "batch" command executes a job safely in the background, but prioritizes it to run only when the system load permits; it is therefore best for tasks that are not time sensitive.

The "at" command executes commands at a specified time, similarly to how "nohup" works.  Note that the specified time can be "now" to run it immediately.

Note that the "at" and "batch" commands work only with non-interactive jobs.

screen and tmux

If you want to be able to come back later and re-connect to your running job, then you want to look at applications like "screen" or "tmux".

When you first log in, run the "screen" command. You will get another shell and can run your commands in that new shell. If you're disconnected, the "screen" process keeps the terminal alive so that your shell and the processes it is running don't hang up. When you reconnect later, you can run 'screen -r' to resume the active session and commands.

The "tmux" command works in a way similar to "screen".

Both "screen" and "tmux" can work with interactive commands, since you can re-attach and continue to interact with it after being disconnected.

mosh

For those who have problems due to mobility (WiFi hopping, commuting, etc) and are at an advanced level, you might check out "mosh" (https://mosh.org/). Unlike the solutions above, which will work for any type of login to the machine, mosh will work only with remote (ssh-type) logins.  However, using "mosh" is a more advanced process, and left as an exercise for the reader.

The commands we've discussed all work to allow you to logout or survive being disconnected from a machine without having your jobs killed.  Note however, in most cases, these are only suitable for non-graphical commands.  For graphical commands you would probably need to look into an alternative such as starting an new graphical console on the local machine (for local access), or using a protocol such as VNC or RDP for remote graphical access.

If you need help with any of the above commands, contact the CNS Help Desk at https://cns.utexas.edu/help/.

Written by Eric Rostetter, Senior System Administrator
Questions or comments? The best and easiest way to contact us is via the CNS Help Desk form.