README.bailout : How to recover the judge from a bailout

 $Log: README.bailout,v 



This document describes the procedures used to recover a judge from
a bailout. To determine if this has occured, you should have noticed the
following:

 - Judge sent an error message to the GAMES_MASTER address in dip.conf
 - There is a file 'zforward' in the judge's home directory
 - The judge is not responding to emails sent to it
 - There is probably a file called saved_mail in the judge's game directory
 - rdip and dip are not being invoked.

The procedure to restart the judge is the following:

   1) You should have a zforward file, new mail is going into the
      standard /usr/spool/mail (or equivalent) file and there are
      no more "dip"s running.

   2) Rename the saved_mail file to something like "t0" to prevent
      it from getting trashed.  Shouldn't be getting any more mail
      written to this file.

   3) Check in the log to determine why the bailout occured and fix
      the problem.

   4) Rename the zforward file to xforward.  The presence of the
      zforward causes all mail to be written to saved_mail.
      Note also that the presence of xforward will also cause all games 
      to be delayed 24 hours from the current time, to allow players time 
      to change orders/masters to alter deadlines after the judge comes 
      back on-line.

   5) Make sure there is no "dip.input" file.  If your system went
      down while the judge was running this file may get left lying
      around.  It may contain partially processed mail.  You should
      determine whether there is any unprocessed mail in there and
      save it off to a file called "t1".

   6) Process the t0 file with:

          rdip < t0 &

      It will print the "From" lines out as it processes them so you
      can keep an eye on its progress.  While it is running you can
      continue with the next step.

   7) Verify that it doesn't bailout again.  If it does, go back up
      to step 2 and try again.

   8) Process the t1 file (if you've got one) with: "rdip < t1".
      You should get a "dip.input exists, exiting".  This is okay,
      the mail has been appended to dip.mail and will be processed
      by the earlier rdip when it catches up with its backlog.

   9) Rename* the /usr/spool/mail/judge file to "t2" and process it
      with "rdip < t2".  Again you should get the "dip.input exists"
      message.

  10) Rename the xforward file to .forward.  If there are any new
      messages in /usr/spool/mail/judge you should process them with
      a rename* to t3 and a "rdip < t3".

*On some systems, it may not be possible to perform the rename operation 
specified as the judge user. In these cases, copy the mail file and use 
a mail client to delete the mails manually.


---- End of README.bailout ------------------------------------------