The T-PRO wasn’t warmly received, but engineers within the MAPP region noticed the flexibility of the platform. There was a particularly challenging aspect of stability of the Dorsey-Forbes Transmission Line that could not be readily solved using conventional relays. Minnesota Power (MNP) asked Dr. Swift if a better algorithm could be developed using the APT Relay. It could, so Dr. Zhiying Zhang (who had graduated and now worked for us at Vansco) wrote the algorithm and it was installed at Forbes Substation.
The Original T-PRO
The University of Manitoba’s Department of Engineering has a strong Power Systems group. At the time, Dr. Glenn Swift had a student, Zhiying Zhang, working on his PhD thesis, which embodied the very cool concept that a transformer’s “standard operating life” between oil change and inspection (typically 7 years, by convention) could be extended if the internal temperature was monitored to take into account light loading. Conversely, you could temporarily overload a transformer within reasonable limits, if you took into account the loss of life due to the extra heat generated within the transformer. For instance, here in the north, during the wintertime, you could readily load a transformer for 115% of rating, say, with minimal effect on the life of the transformer, as long as you could model the heat and its affect on life.
As a testbed for their ideas, the university first used a NEC uPD7720 fixed-point DSP in a custom assembly, then moved to a PC-based solution which used a Spectrum Digital PC card containing a TI TMS320C30 floating-point DSP. The university selected Vansco as their partner to build their designs.
The design was a “2 box solution”, with one chassis containing the CTs and PTs, and the other containing an actual PC motherboard and cards, with an umbilical cord between tpoint and he two chassis.
The DSP was programmed bare-metal in ‘C’ and assembler, while the PC ran MS-DOS (eek) and had its user interface on a classic PC keyboard and monitor. The display was cool, using graphics to display operating quantities and trip levels.
It was Dr. Swift who coined the term T-PRO, and APT as short for “Alpha Power Technologies”, although we had to drop the latter after hearing from Alpha Technologies in BC – but “APT Power Technologies” was adopted and stuck. Glenn always was a tireless advocate for a made-in-Manitoba relay.
We did try to market the APT T-PRO, taking it to MIPSYCON and showing it to various utilities in the MAPP region, but it wasn’t quite what they wanted. They wanted a one-box solution… which, later, we decided to develop.
Development Support for TESLA 3000 MPB
The TESLA 3000 MPB was developed for NxtPhase by Fidus in Ottawa. It went fairly well, until the prototype board would just not recognize the Ethernet NIC, at all. Michael Miller asked Elecsys to investigate.
I insisted that we boot LINUX on the board. Fidus advised that it was not feasible to do so. I persisted, and after about a week’s work (customizing the boot parameters and hacking the kernel a bit), I had it booting to LINUX with a small laptop hard drive. Sure enough, the Ethernet NIC would not come up, even though it was fully supported by the installed LINUX driver.
By instrumenting the driver with printk() and observing the result of loading and unloading the driver in the system log, I determined that it was a short between two lines on the PCI bus to the NIC. When Fidus took a close look at the Gerber files, they discovered that the design rules were too tight on the PCI bus on a deep inside layer of the PCB. The PCB was corrected and remade, and the NIC worked.
Later, there was an issue with the operation speed of the MPB core. Fidus was doing the FPGA development in VHDL on Synopsys, but did not have the resources to continue. I took over the development under Synopsys, having to establish VPN to Fidus to obtain the license for it to run. I suspected that the Synopsys tools, as good as they were, could be responsible for my inability to get appropriate performance from the FPGA. I ported the VHDL out from Synopsys back to Xilinx ISE, a byproduct of which was that we could do full development stand-alone in Winnipeg, which was nice.
With a few tweaks, I was able to get the MPB FPGA to simulate out at better-than-required full speed operation, which was proved out in actual operation. I mapped a path to higher speed operation, anticipated as a requirement for future product development.
Production Support and Test of TESLA 3000 AIB using GPIB
After performing the design and full verification of the TESLA 3000 AIB, Elecsys was asked to support the manufacture of the boards at Trilogy-Net in Calgary. There were challenges in getting the boards to work, mostly because of the silly little LM358BP, which is in a tiny 8 microbump package. We had process issues on getting it to solder properly.
In the end, the fault was the layout that I had supervised. Unfortunately, in our desire to pack everything onto one side of the board, I had moved down to the absolute smallest package of the LM358 dual op-amp. Well, in the end the board layout person went to double sided assembly but didn’t tell me until the last minute, meaning that the tiny op-amp stayed. There were larger parts around the op-amp that drew the heat out of the oven, so the op-amp wouldn’t solder consistently. This was a headache until a couple of years ago, when that op-amp was finally replaced by the larger SOIC version. Sigh.
Anyway, we wanted to do a 100% test of the AIBs to ensure proper function before TESLA assembly.
I acquired a number of surplus IEEE-488 / GPIB test equipment and a PC GPIB card, and wrote scripts in PERL to control the test equipment. We were able to drive precise voltages into each channel, to test offset and gain, then certain frequency signals to test filter response. Data was logged, analyzed, then the aggregate data was reviewed continually to calculate statistical process control.
Developed TESLA 3000 Analog Input Board (AIB)
The TESLA 3000 was designed to put the analog-to-digital conversion as close as possible to the rear panel, so the AIB had 6 x TI ADS8364 6 channel 16 bit SAR ADCs on it, with the associated front end and conditioning circuitry. A Xilinx Spartan 3 FPGA directed the acquisition timing, collected the ADC data, and communicated with the TESLA 3000 MPB using a bidirectional high speed LVDS synchronous serial link.
I did the architecture and design of the TESLA 3000 AIB, and oversaw the PCB layout by an external contractor. I specified the operation of the FPGA, how it interfaced to the MPB, and how it was programmed for operation.
The FPGA firmware was developed in Verilog on Xilinx ISE by a contractor at Fidus in Ottawa. When that FPGA developer was unable to complete the job, I took over and finished the project.
Developed the PDR (Power Disturbance Recorder) into the DSR (Dynamic Swing Recorder)
Vansco was an important partner of the IAMC for the development of a new product for Manitoba Hydro – the Power Disturbance Recorder (PDR). This was an innovative system that had a Central Station and multiple Remote Stations, connected by modems. In practice, we had Remote Stations be as far as 1500 km (2400 mi) away, way up north on the Nelson River power system. The Remote Station would detect power disturbances and contact the Central Station to alert it of the occurrence. The Central Station would then connect all other Remote Stations and instruct them to capture a time-synchronized snapshot around the event.
The key concepts here were time synchronization and snapshot retrieval. The Remote Station processors had enough storage to keep very long buffers of data – so even if the disturbance disrupted communications for a time, events would still be captured, network-wide. The communications protocol between the Master and Remote Stations always had an initial multiple-pass time-synchronization exchange to resolve the clock offsets between the stations, without external synchronizing source. Of course, now time sync over wide area is easy, due to ubiquitous satellite navigation & timing systems, but at this time, the PDR implementation was innovative and worked well.
After IAMC closed down, ongoing PDR development was transferred to Vansco. Later, former IAMC developer Michael Miller joined our team and was instrumental in its ongoing development.
The PDR was renamed the Dynamic Swing Recorder (DSR) by our Manitoba Hydro team. That team, especially Dave Fedirchuk, felt that more utilities could use this technology. We tried to market it more widely, but later decided that a newer approach was required – creating a more generalized fault recorder, similar to the leaders at the time, the Rochester Instruments or Mehta Tech recorders. This is where the idea of the TESLA 2000 started to take hold.
Developed the DR2/TESLA Main Processor Board (MPB) and Other DR2 & TESLA Boards
I used to say, “those aren’t electrons flowing on the MPB, those are my blood cells!” In a philosophical sense, it was true – the development of the MPB drained me like no other project effort, before or since.
With multiple processors on board – 2 x PIC 16C MCUs (one for front panel keyboard, one for watchdog processor), a TI TMS320C32 DSP, an Altera Flex 10K FPGA, an Altera MAX 7000 CPLD, 3 x National Semiconductor LM12458 microsequenced ADCs, and a Motorola MC68HC11 MCU for IRIG processing, 3 x NS16550 FIFO UARTs, and a 16 bit PC-104 bus connection, it was quite busy. An X86-based PC-104 card was mounted on top of the MPB – lower powered on the DR2, but much more powerful model on the TESLA.
The concept was that sifting, sorting, compression, and decision making was done on the DSP, where memory was expensive but calculations were easy. The resultant data was streamed through a high-speed DSP-to-X86 FIFO, where the X86 DMA’d it into main memory and stored it to disk. The X86 was used because its memory was plentiful and inexpensive, and disk I/O was fast and easy.
The other boards in the system weren’t as complex, but were just as important. The DR2 AIB, mounted on the bottom of the chassis, had up to 18 miniature PCB-mount instrumentation transformers on it – CTs or PTs, depending on the application – and analog processing to transfer to the ADCs on the MPB. The TESLA AIB was mounted on the rear of the chassis, had more channels but was less sophisticated – non isolated +/- 2.5V (nominal) input. The DR2 and TESLA each had their own interpretation of digital I/O – DR2 with high current relay outputs, and dedicated ultra reliable digital inputs, and the TESLA with fewer and lower level outputs and simpler digital inputs.
The front panel of the DR2 had a 2 line x 24 character FIP display, a small keyboard, and several indicator LEDs, including a large bright red “Target” LED, as is expected on a relay. The TESLA front panel had the smaller LEDs only.
I established the bus structure and pin-outs for all the I/O. The front panel and digital I/O were intentionally designed to be interchangeable, with the intention of future mixed mode configuration, or expansion. Although some of the buses had extra connections (front panel serial port, for instance), this was kept to the ends of the connectors, so combining and splitting ribbon cables would be relatively straightforward. This was very well documented in extensive spreadsheets, which clearly showed connections and levels.
The memory and I/O maps of the processors was also documented in spreadsheets, with well defined bit states and clearly defined defaults.
Later, I wrote an exhaustive 70+ page programming guide to the MPB, which outlined all its features, how each processor interacted, the resources used, assumptions in programming, and how to compose images for execution.
TESLA 2000 Released
The TESLA 2000 was the most sophisticated disturbance recorder of its time. It used two MPBs, connected together by a bidirectional DSP-to-DSP synchronous serial link. This gave the TESLA 36 analog inputs.
However, it was the software which made the TESLA so outstanding. Although the serial port connection could be a pain to set up on MS-Windows, once it was set up, it worked like a dream. The RecordBase / RecordGraph system was amazing and intuitive. It was an instant hit in the Power Utility Community.
L-PRO and T-PRO Released
The delay was stressful, but the results were worth it! Manitoba’s very own 6 input (x 3 phase, always) protective relay became a reality!
The Analog Input Board (AIB) was configured to manage any mix of CTs and PTs, up to 18 discrete inputs.
The L-PRO came out first, then the T-PRO. Their primary hardware difference was the mix of CTs & PTs, and the T-PRO had a separate input for oil temperature transducer.
Challenged More and More: DR2 and TESLA on the Same Platform
We did focus group studies, discussed with customers, went to trade shows and conferences, and finally, settled on the development of two platforms for the power utility market: a TMS320C32-based platform for the relay, which we would call the DR2 (DR1 would have been the units developed using TMS320C30 at the UofM), and a TMS320C44-based platform for the recorder.
The ‘C3x family is fast and inexpensive. The ‘C44 family is well connected and easily networked. No comparable family at the time gave us both.
Management deemed this too expensive and risky, so we were told to go back to the drawing board and do it with only one platform. We gasped… sighed… and started over.
I can’t say that it was wrong. The development was successful… the product met market demand… and, with updates and changes, still in production today.
The core Main Processor Board developed as part of that effort, was in active production and use until being finally retired in 2018. Not bad!