George Hubka

there is a great utility that will help to monitor and control processes on a linux box called monload2.  It can be found at http://www.dei.isep.ipp.pt/~andre/extern/monload2.html 

from the site…

The monitor program – how it works

We first address the high load mad process problem, that was easy, from time to time we run the ps command to check each user process CPU and memory usage, if it is above a predetermined level we register it as mad, if a process keeps mad for a number of iterations we put it on low priority, next we send it the TERM signal, if it still keeps alive we send it the KILL signal. To keep system processes away from this, root processes are ignored, also you may specify in the command line usernames to be ignored.

The second problem was much harder to solve, the infinite fork puts the system on such a load that all io will freeze, using the code that was created to solve the first problem we noticed the ps execution would freeze, so we had to find a way to have a timed out read of the output of the ps command, but nothing seems to work here, for example the select system calls does not work. The ultimate solution was to set an alarm just before the ps execution, if the reading was not done when the alarm was received then the ALARM handler sends a KILL to the ps process. This worked out ok, it seams signals still work ok on these conditions.

Next we need to find a way to shutdown the system, just calling shutdown or reboot is no solution, even if it works it will take some hours. First the infinite fork processes must be killed, but you can’t get their pids, and even if you could they are always forking, the solution is kind off hard, we first send TERM to all possible pids starting from 100, and then we do the same thing with the KILL signal. This kills all that bad processes (and others), but then we can shutdown and reboot the system, if this fails we still try to do a cool reboot.