Help! Guide for SuperK Outer Detector Troubles

... always under construction !! Please, check this space again for updates later. Last update: 28 May 2003 [HGB]

If you feel you're too excited to handle the trouble yourself, and you can't locate any help onsite, simply contact the OD experts:

UW onsite representative (at "Settle" apt., Osawano):

0764-67-5517   -   at Osawano apartment "Settle"

NEW since Jan.'99: 0901-636-4087   -   UW cell phone in Japan (mostly in use at KEK, though)

Hans Berns (OD-DAQ electronics & software, radon hut, GPS system):

[0-001-010]-1-206-685-4725 (work)

[0-001-010]-1-206-365-3960 (home)

NEW since 5/13/99: +1-206-713-5919 (mobile phone)

berns@phys.washington.edu

Jeff Wilkes (expert to find Hans):

[0-001-010]-1-206-543-4232 (work)

[0-001-010]-1-206-282-5715 (home)

wilkes@phys.washington.edu

UW group's fax machine in Seattle:

[0-001-010]-1-206-685-9242

What to do when Sukant hangs up and refuses to start a new run?

1. Go to sukonh and exit the run control completely.
   • All of the large green panels should disappear and there should be none left on the screen of any type.

2. open a xterm window on sukonh

3. At the prompt type:

    cleanup

   • This will start a shellscript which has the ability to clean up every DAQ workstation, if desired. It will prompt the user 12 times with the question: "Clean up <workstation>. Ok ? [y/n]" for every workstation from sukon1 to sukon9, plus sukonh, kingfish, and sukant. It takes about 30 seconds to 1 minute for each host, so don't waste time answering "y" unless you believe the workstation in question really has a problem.

4. When the cleanup script claims that sukant is clean, type:

    initialize

   to restart the run control, just as you always do.

Important note about rebooting Sukant:

Please reboot sukant only if really necessary!

If you found that sukant rebooted during the power outage (e.g. by checking the time since last reboot via the unix command uptime) or if you rebooted sukant by hand, then do the following steps:

If sukant was rebooted (e.g. after a long power outage) then the OD-DAQ VME crate control is at a random state which can lead to crashing the anti-collector at a new run start. The crate controller needs to be properly initialized first before a new run is started:

1. log into sukant as shift, guest, or online,
2. turn off OD-DAQ VME crate (red power switch at bottom),
3. wait approx. 30 seconds,
4. turn OD-DAQ VME crate back on again,
5. wait approx. another 30 seconds,
6. execute VME initialization command

    initvme

7. if no error messages appear then everything is OK (a few status lines are ok),
   • if you see error messages (with words like "Error" in there), then try steps 2 - 6 again,
     • if that doesn't help, reboot sukant again (for password see instructions in "emergency" envelope at sukant's monitor) and then start with step 1 again.
     • if you still see trouble at the last step contact an OD-DAQ expert. See email addresses and phone numbers above.

How to disable/re-enable a OD PMT channel with the new OD HV paddle (relay) cards?

Here is the new shift instruction:

1. Identify the HV channel which powers the faulty OD PMT using tables here, or the wire_back program on kingfish
   • E.g. tube 408 = paddle no. 1.2.15.1 (hut.crate.card.channel) = HV channel 2.10 in hut 1.
2. DISABLE the selected HV channel using the LeCroy HV controller in the appropriate hut.
   • If you don't know how to operate the LeCroy HV supply, click here for a brief manual or ckeck the "Outer Detector High Voltage" shift manual.
3. Change the setting of the dip switch for desired channel on its paddle card
   • top = channel 1, bottom = channel 12;   left position = ON, right position = OFF
   • use a small screwdriver or toothpick to move the dip switch - if you can't move it with one of your fingers.
   • For confirmation, the red LED next to the LEMO monitor jack of that channel shows whether the relay is on or off.
     Note: The LEDs monitor the relay ON/OFF status only, not whether there is HV or not!
4. Finally, ENABLE the selected HV channel. Done!
5. (Don't forget to send Bill Kropp an email about the disabled channel, including base resistance measurement - see procedure below.)

1. Identify paddle/relay card corresponding to tripped HV channel.
   • E.g. use wire_back program on kingfish.
2. Verify that tripped HV channel is still off, using the LeCroy HV supply's display in the quadrant hut (see manual).
3. Make a note of the DIP switch settings on the appropriate paddle card.
4. Turn all DIP switch positions on the selected paddle card to OFF position.
5. Test the current draw of each individual channel (only the ones that were enabled before, of course). Best if you start from the top channel, i.e. move its DIP switch to ON position, and leave all other DIP switches to OFF, repeat for each PMT in use.
6. Re-ENABLE the tripped HV channel again. Observe the current values (Meas_uA) on the display of the LeCroy supply.
7. A healthy PMT has a base resistance of approx. 26 MOhms, i.e. the current should be approx. 59µA at 1600V (including the 1.2 MOhm resistor on the card in series), ~74µA at 2000V, ~88µA at 2400V, etc. (R=V/I, in case you forgot!).
   • If the current is far above the expected value then it must be the problem channel! E.g. a current of ~1600µA at 2000V means that the base is shorted.
8. DISABLE the HV channel again.
9. Repeat steps 5 - 8 until you find the problem channel.
10. After locating the problem channel, set its DIP switch to OFF, then set all others back to ON that were originally ON before.
11. Re-ENABLE the HV channel. Double-check that the current is approximately at the expected value (number_of_enabled_channels x target_voltage / 26 MOhms), e.g. 10 enabled channels at 2000V should draw approx. 770µA. If satisfied, then you're done.
12. (Don't forget to send Bill Kropp an email about the finding, including base resistance.)

How to handle continuous "The last XX events DIDN'T have anticounter data" error messages:

==> anti-sender and/or anti-sorter dead ?

The anti-sender (or any of the sukon1-9 senders) has been observed to die somewhat frequently(once or twice per week in average) mostly caused (with approx. 99% probability) by a network or communication problem between eventbuilder and sender(s). A dead anti-sender often causes the anti-sorter to run into a Nova buffer memory problem which usually results in a segmentation fault crash...

==> anti-collector dead ?

•   tail /home/online/log/anticollector.errorlog
  or
•   tail /home/online/log/anticoll.newlog

• ===> ERROR: (rd_mail) device read error ... I QUIT!: I/O error
  Status error 0x44:
          Remote bus error.
          Interface timeout.

• 20-06-1999 06:05:48 [7440]: ERROR VME/VSB: DC2 1 init BAD!  I QUIT ...
  

What to do when "FSCC problem" error messages from msgctrl don't stop?

• One of the "Hut N TDCs BUSY" LEDs on the ODDAQ hardware monitor box is not in sync with the other 3 LEDs.
• The anti-sorter complaints with frequent warning messages via msgctrl on sukonh repeated once every minute or two minutes, e.g.

  Tue Feb 6 11:36:06 2001/sukant::collector/hut 2 TDC/FSCC problems; starting automatic 50-sec FSCC reset now!

• On the anticollector@sukant monitor window on sukonh you see frequent error messages popping up such as

  ===> TDCs busy for >.1 sec in hut 2 !! ===> No data in OD TDCs, automatic fast FSCC reset executed.

  In this example, there's no data coming from hut 2. So, the problem TDC is in hut 2!

1. The Fastbus power supply is off?
2. The Fastbus power supply has a problem, e.g. a failing voltage (+5V, -5.2V, -2V, +15V, -15V)?
3. The FSCC module (Fastbus Smart Crate Controller) is hung and needs hardware reset?
4. One or more of the TDC modules do not respond to FSCC bus requests? (see also TDC Test Instruction)

• flip "CNTRL PWR ON" button to "off" position,
• wait approx. 30 seconds,
• flip "CNTRL PWR ON" button back to "on" position,
• Wait another 30 seconds or so (for letting the FB controller reboot) before restarting the run. Done!

What if the orange LED (Fifo VME Busy) on the OD-DAQ LED box stays on during the entire run?

This is a sure indicator that the Anti-Collector has lost control over the OD-DAQ VME crate electronics. Possible reasons:

1. The NOVA daemon on sukant might be hung up.

   • In this case, all three online processes on sukant (Anti-Collector, Anti-Sorter, Anti-Sender) cannot communicate with each other and eventually crash sooner or later.

   • What to do then?

     Stop and abort the run and cleanup the network environment!
     See the three steps described above.

2. The VME crate controller (Bit3 module) might be in an undefined state.

   • This can be caused by improper run stops and restarts.

   • What to do?

     A. Stop and abort the run completely.
     B. Go to the central hut and recycle the VME crate power:

     • locate the crate's power switch at the lower part of the crate on the front, and turn off the power,
     • wait approx. 30 seconds and turn the power back on.
     C. Now the run can be restarted.

OD-DAQ Documentation Hyperlinks

(to be expanded)

If you find any errors in this document or if you have any suggestions regarding ODDAQ trouble shooting, please email to H.G.Berns <berns@phys.washington.edu> and/or R.J.Wilkes <wilkes@phys.washington.edu>