Outer Detector High Voltage Paddle Card Upgrade
H.G.Berns, updated
10/14/2001
See also the summary "OD HV Upgrade"
for the 10/2001 SK meeting
[PowerPoint (490kB) or
PDF (724kB) format].
The OD HV Paddle Card serves as a high voltage (1.5...2.5kV) supply
distributer and signal/voltage splitter for a subgroup of 12 OD PMTs.
The original version of this card (see
description) has jumpers for manually disabling individual PMT
channels causing (or suspected to causing) HV current trips and thereby
causing data gaps of the other PMTs hooked up to this card.
The circuitry of one channel is sketched below:
circuit of one channel in the old paddle card design
The procedure of disabling an individual PMT channel has turned out to
often cause headaches, unfortunately. Once a HV voltage has tripped, one
has to find the PMT suspect by:
- turn off HV voltage for the paddle card and unplug the HV supply cable
(red cable with SHV connector on front side),
- unscrew the two fastener bolts in the rear side of the paddle card subrack
holding the frame of the 12 PMT coax cables to the suspected paddle card,
- unplug 50-pin ribbon cable connector to the QTC,
- pull out the paddle card so you can reach the jumpers,
- use an ohmmeter to find the PMT with resistance value below operation
level (approx. 26 MOhms),
- if suspect found this way, then remove appropiate jumper,
- if all PMTs show normal operation resistance, then, well, it's trial
and error, so pick one channel and hope it's the right one...
- push paddle card back into its subrack slot,
- reconnect 50-pin ribbon cable connnector,
- reconnect HV supply cable,
- re-enable HV supply for paddle card and check whether current is at normal
level again,
- if no current trip and all is normal, then procede with last step,
- if current trips again, darn, go back to step 3 and try pulling jumper
of a different PMT channel again,
- reinsert the two fasteners for the card's cable frame in the rear of the subrack;
done.
A couple of weaknesses of this procedure have shown up during the past years of
Super-K operation. E.g. often, the 50-pin ribbon cable has been inserted wrong
(step 7 above), since there are 6 possible ways to mate the header with the jack
on the card, of which only one is correctly lined up. Then a block of OD PMT data
was missing for a while, often undetected for several hours. Another weakness is
the fact that the paddle card often tends to slip out of the subrack's card rails
when pushed back into its slot (step 6) above. This often causes HV arcing to the
neighboring paddle card. Experienced shift people can probably list a couple of
more problems about this procedure...
To minimize the problems with the jumpered paddle card listed above, a new design
is proposed by replacing the jumpers with HV relays. The basic design will be
kept the same as before since there will be no changes with the HV supplies or
QTCs. The proposal uses dip switches for individually disable/enable each
relay manally from the front of the card, without the need for unplugging the
module. See added circuitry sketched for one PMT channel below:
circuit of one channel in upgraded paddle card
This new design allows disabling or enabling of individial OD PMT channels without
the need to remove the paddle card from its position in the subrack. This will save
all physical procedure steps listed above and the headaches caused by them.
Step 5 (finding susepect PMT by resistance measurement) will be replaced by a procedure via
testing each individual channel directly by modifying the HV settings on the LeCroy HV supply,
reading back the voltage and current levels of the card and calculating the PMT resistance from
there (e.g. at 100V, a healthy PMT should draw approx. only 3.8 µA =
approx. 26 MW)
In addition, a header connector for external remote operation of the relays is
implemented, too, allowing to expand the system with a remote control option later. Remote
control option has become an important item since October 1999 when the Super-K operations
were finally started from the Kenkyutou building and on-site shift operations
were reduced to daytime only.
See photo with
captions.
- VME 9U height (14.4") with standard VME C-type front panel and ejector
latches for robust and precise mounting into subrack.
- Two-layer printed circuit board with silk screen labeling on top layer.
- High voltage relay instead of jumper for each PMT channel.
- Front panel DIP switches for manual enable/disable of each channel.
- LEDs for HV status of each PMT channel on front panel (relay on/off display).
- Header jack on front panel for relay remote control input.
[Remote control system is under development]
- LEMO jack on front panel for each PMT signal, allowing easy oscilloscope
signal monitoring without disconnecting the coax ribbon cable to the QTC.
27 Jan. 2000: First 2 prototypes assembled and lab tested for shipment
to Japan.
Installation into OD HV system is scheduled for 30/31 Jan.
2000.
assembled board (top view)
|
assembled board (front panel)
|
Test setup in Seattle lab
|
After burn-in testing of the 2 prototype cards in Seattle, I asked Andy Stachyra to
carry both of them and the 24V powers supply (needed for the relay coil actuation) to Mozumi
since he was scheduled for shift duty from Jan.30 to Feb.12, 2000, anyway. By taking
advantage of a scheduled K2K beam stop on Jan.30, he replaced the first 2 OD paddle cards
in crate 1 in hut 1 with the new prototypes. For more details see the mini report
"OD HV Relay Card Test Run 02/2000".
- The first batch of 42 boards (1 quadrant worth + 2 spares) assembled
by October 31, 2000. A sturdy rack-mount chassis for the 24V DC
poser supply for the relay coils was assembled, too.
- Burn-in tests showed an unexpected problem of magnetic interference
between the relays of neighboring cards, and there seemed to be a
thermal effect, too. After consulting the relay manufacturer and
Physics colleagues I tried inserting mumetal sheets between the
boards for shielding, and the interference problem disappeared fully.
I found that taping a 1.5" strip of mumetal on top of the relays was
sufficient to prevent any magnetic interference (see
photo). To prevent arcing
between the HV traces of a neigboring board with the mumetal I was
recommended to use kapton, a highly insulating material, which also
worked pretty nicely during the tests (see
photo).
- Relay coil power measured with 240 channels activated (1 full crate):
24V, 6.1 Amp.
This is 85% of the capacity of the selected Power-One 24V/7.2A linear
supplies I'm using for each crate. During the lab tests over several
days with 2 crates fully powered, I found the supplies getting fairly
warm. So, I added to heavy-duty rack cooling fans underneath the
supply chassis and now things are nice and cool.
- Photos (click on picture for full resolution):
assembled board
with mumetal |
full crate, front |
front panels,
red LEDs indicate relays
are on (HV
enabled) |
full crate, rear |
- Nov. 18, 2000, 09:00-17:00: Crate 2, hut 1 installed.
- Initially after installation of crate 2, some noise was recorded
in the new run (9521), concentrated in hut 1 crate 2 area.
It turned out that a formerly dead PMT channel (but jumper still in)
was revived by unplugging from the old paddle card into the new relay
cards (cable ID 20269 = paddle no. 1.2.3.2) after it was recorded
dead for more than 2 years. Probably the connector was loose at the
old paddle card.
The noise disappeared fully after approx. 2 hours. We guess it's
a settling-down effect of the gasses inside the PMT once they're
exposed to a HV field after such a long time of inactivity.
- Nov. 20, 2000, 09:00-15:00: Crate 1, hut 1 installed.
- After turning all HV on in crate 1 we found heavy noise in the entire
hut 1 data, affecting not only the OD but the ID as well. It didn't
recover over time, and it even made the LeCroy HV supply crash a few
times, obviously picking up heavy electrical noise pulses nearby.
After several hours of unsuccessful searching for the cause until
late in the night we decided to leave the system running overnight
with HV turned OFF in hut 1 (so, runs 9528 - 9531 don't have full
OD data, probably useless for most analysis groups; sorry!)
- Nov. 21, 2000, 08:00-13:00: further testing of OD in hut 1 to find
noise problem. Finally found, and luckily easy to solve:
The relay DC supply needed to be grounded! The 24V outputs were free
floating and it appeared that the HV field in the reed contacts of
the relays charged up the surrounding coil potential like a series of
small capacitor (maybe a few pF), which then frequently discharged
with little sparcs to the grounded chassis of the DC supply, once the
breakdown voltage was reached.
As soon as grounding was applied to the negative terminals of the 24V
outputs the noise disappeared fully.
Success!
- Photos (click on small picture to see the hi-res photo):
ODDAQ racks in hut1
with new relay cards |
LeCroy HV supply,
HV Relay supply (underneath),
and crate
1 with new boards |
new boards in crate 1,
installed and activated;
red LEDs indicate
which relay
is on (HV enabled) |
HV Relay power supply
underneath LeCroy HV supply.
Note the warning label! |
HV Relay Supply chassis
(top view) before rack installation |
HV Relay Supply chassis
(bottom view) with cooling fans |
Installation of the remaining quadrants at Super-K Apr. 27 - May 2,
2001
- March 2, 2001: production and lab tests of another 128 boards
(3 quadrants worth + 8 spares) and 4 power supplies (3 quadrants
+ 1 spare) finished.
- March 5, 2001: shipment sent to Japan (see
shipping form).
Originally, the installation in the remaining 3 quadrants was
scheduled to occur during the K2K beam run break from 3/29 until
4/4/2001, but the K2K committee decided to continue the beam for
another month and postponed the break to 4/26-5/17 instead.
Therefore, the paddle card installation was also postponed to end of
April 2001:
- April 27+28, 2001: upgrade of quadrant 2 completed.
- April 29+30, 2001: upgrade of quadrant 3 completed.
- May 1+2, 2001: upgrade of quadrant 4 completed.
- May 3, 2001: final tests and checks. All works fine, full
installation is completed now. Success!
NOTE: With the new cards, you don't need to disconnect any cables or
pull any cards for disabling a channel anymore! Instead of pulling a jumper, you
simply turn a relay off via a dip switch located on the front panel.
Procedure for disabling or enabling an OD channel:
- Identify the HV channel which powers the faulty OD PMT using tables
here, or the wire_back program on kingfish
- E.g. tube 408 = paddle no. 1.2.15.1 (hut.crate.card.channel) = HV channel 2.10 in hut 1.
- DISABLE the selected HV channel using the
LeCroy HV controller in the appropriate hut.
- Change the setting of the dip switch for desired channel on its paddle card
- top = channel 1, bottom = channel 12; left position = ON, right
position = OFF
- use a small screwdriver or toothpick to move the dip switch - if you
can't move it with one of your fingers.
- For confirmation, the red LED next to the LEMO monitor jack of that
channel shows whether the relay is on or off.
Note: The LEDs monitor the relay ON/OFF status
only, not whether there is HV or not!
- Finally, ENABLE the selected HV channel. Done!
- (Don't forget to send Bill Kropp an email about the disabled
channel, including base resistance measurement - see procedure
below.)
How to find a problem channel after a HV trip?
- Identify paddle/relay card corresponding to tripped HV channel.
- E.g. use wire_back program on kingfish.
- Verify that tripped HV channel is still off, using the
LeCroy HV supply's display in the
quadrant hut (see
manual).
- Make a note of the DIP switch settings on the appropriate paddle card.
- Turn all DIP switch positions on the selected paddle card to OFF position.
- Test the current draw of each individual channel (only the ones that
were enabled before, of course). Best if you start from the top
channel, i.e. move its DIP switch to ON position, and leave all other
DIP switches to OFF, repeat for each PMT in use.
- Re-ENABLE the tripped HV channel again. Observe the current values
(Meas_uA) on the display of the LeCroy supply.
- A healthy PMT has a base resistance of approx. 26 MW, i.e. the
current should be approx. 59µA at 1600V (including the 1.2 MW resistor on the card in series), ~74µA at
2000V, ~88µA at 2400V, etc. (R=V/I, in case you forgot!).
- If the current is far above the expected value then it must be
the problem channel! E.g. a current of ~1600µA at 2000V means
that the base is shorted.
- DISABLE the HV channel again.
- Repeat steps 5 - 8 until you find the problem channel.
- After locating the problem channel, set its DIP switch to OFF,
then set all others back to ON that were originally ON before.
- Re-ENABLE the HV channel. Double-check that the current is
approximately at the expected value (number_of_enabled_channels x
target_voltage / 26 MW), e.g. 10
enabled channels at 2000V should draw approx. 770µA.
If satisfied, then you're done.
- (Don't forget to send Bill Kropp an email about the finding,
including base resistance.)
Procedure for measuring the PMT base resistance:
- Turn off the HV channel where selected PMT channel is located.
- Set all dip switch positions on the paddle card of the selected PMT location to
OFF position (right), except leave selected channel at ON position.
- Please make a note of the original dip switch settings before
you change anything!
- Set the HV channel to a relatively low voltage setting, e.g. 500V, then
re-enable the channel.
- Again, please make a note of the original HV target voltage before
changing it.
- Wait a few seconds to have the HV voltage settle down to the target
value. Then read the measured voltage and µA levels (Meas_V, Meas_uA)
of that channel.
- The PMT resistance now can be calculated as:
| R (MOhms) = Meas_V / Meas_uA -
1.2 |
- The "1.2" value is from the 1.2 MW
pull-up resistor on the paddle card, which is in series with the
PMT resistance.
- E.g. with Meas_V=500.0 and Meas_uA=18.5, the resulting PMT
resistance is approx. 25.8 MW.
- Disable the HV channel again, then restore the original target voltage
setting.
- Restore the original setting of the 12 dip switch positions.
- Re-enable the HV channel. Done.
I'm planning to install a remote-control unit for the relay operations
later. Once it's installed, above procedures can be automated and
simplified by software, without the need to physically go to the outer
quadrants anymore. But this will probably not be installed before the
Super-K upgrade is done in the fall of 2001.
Procedure for disabling or enabling an OD channel:
