As 2003 drew to a close, it was apparent that NxtPhase was having financial challenges. Its CT & VT developments were costing a lot, and progress was not as quick as hoped. I left NxtPhase to do independent consulting, but then in early 2004, I joined with Jason Fuith in forming Elecsys Solutions. Our goal was to help innovators bring their ideas to production, to assist companies who needed engineering guidance, and to develop our own ideas and products.
Design Time Sync System and Write TimeServ for TESLA
As the developer of the MPB, programmer for the IRIG-B decoder, and designer of the synchronization logic in the FPGA, I had originally documented the way that the time synchronization was to work. It was implemented by our DSP programmer, Bob Jackson, and the X86 programmer, Michael Miller.
Later, during the shift from QNX 2 to QNX 4 (now POSIX-compliant), Michael Miller, now my manager, asked me to revisit the time sync design, update the documentation, and implement a “time tag” system for maintaining accurate representation of time in the buffers and on disk, in the presence of time changes, discontinuities, and disagreements between processors.
I documented the plan, identifying the method of arbitration of time, then wrote TimeServ in its entirety. It’s still in use.
APT Becomes a part of NxtPhase
At the end of 2000, Vansco decided to focus its effort on its core business, and sought to divest the APT Division to a power-utility-related entity. It turned out that a company called NxtPhase, based out of Vancouver BC, had been developing digital instrumentation CTs & PTs for power transmission. These digital devices inherently gave digital information… but nobody in the industry was serious about consuming this digital information, causing them great headaches as they had to transform the information back to analog at the standard 69V and 5A levels! They acquired APT because we were one of the first to do power system relaying using DSP, and were open to digital communications. Our relays had the same inter-DSP connection port that the two MPBs on the TESLA used to communicate between them, so this could be great synergy!
On January 1st, 2001, the APT Division of Vansco became NxtPhase Relays & Recorders.
We remained in the Vansco office at 1301 Clarence Ave. until September, when we moved to the newly renovated building at 74 Scurfield Blvd.
Purchased and Installed Red Hat 5 – and Did Some Hacking
We were promoting the “Manitoba Power Connection” of Manitoba Hydro, Teshmont Consultants, Vansco, Manitoba HVDC Research Centre, and the University of Manitoba. The HVDC Research Centre had a new real-time simulator system called RTDS (Real Time Digital Simulator) and an off-line simulator called EMTDC, both of which used their power-system simulator drafting tool called PSCAD. We wanted to work with these tools, so they counselled us to set up a LINUX machine. I purchased LINUX 5 and installed it on an old X86 machine. It took 3 or 4 tries to get it installed correctly, ha ha.
Then I immediately set out to playing with it… of course.
Because we used QNX, we didn’t have an extensive Ethernet network at Vansco. The APT division had its own local Ethernet, but no connection to the shared printer that we needed in another area of the building.
We did have an extensive ARCnet… which I still believe is far superior to 10Base Ethernet because of its ability to get to near-100% utilization without degredation, but anyways..
We were in the process of decommissioning our QNX desktop network and moving to MS-Windows (hiss, boo, but that’s another story). We had plenty of QNX-proprietary memory-mapped ARCnet cards around. I took a few home and set up computers at home running LINUX, and hacked together the ARCnet and Ethernet drivers to get TCP/IP running over the proprietary ARCnet cards. Then, pulling down the (now unused) ARCnet cables from the ceiling, and dropping a small computer at the other end, I was able to connect to the main network and its print spooler, and give us access to the main printer. A demonstration of the power of LINUX, and what you can do if you have the source code!
Download and Try Pre-1.0 LINUX
Sometime around v0.91 to v0.93, I downloaded LINUX – probably Slackware – and gave it a try. The first time it booted, I said, “yes, yes, that’s the way a real operating system announces itself when it boots!”
However, I didn’t have a practical use for it, so I went back to my QNX hacking.
At the time, we didn’t have a VPN, but we had 33k modems for remote access at Vansco. I was able to log into QNX at work and do my work more-or-less the same as at home. Because everything was terminal-based, it was lightning-fast. Because it was QNX, it was rock-solid stable and could recover from almost any problem.
Developed the PDR (Power Disturbance Recorder) into the DSR (Dynamic Swing Recorder)
Vansco was an important partner of the IAMC for the development of a new product for Manitoba Hydro – the Power Disturbance Recorder (PDR). This was an innovative system that had a Central Station and multiple Remote Stations, connected by modems. In practice, we had Remote Stations be as far as 1500 km (2400 mi) away, way up north on the Nelson River power system. The Remote Station would detect power disturbances and contact the Central Station to alert it of the occurrence. The Central Station would then connect all other Remote Stations and instruct them to capture a time-synchronized snapshot around the event.
The key concepts here were time synchronization and snapshot retrieval. The Remote Station processors had enough storage to keep very long buffers of data – so even if the disturbance disrupted communications for a time, events would still be captured, network-wide. The communications protocol between the Master and Remote Stations always had an initial multiple-pass time-synchronization exchange to resolve the clock offsets between the stations, without external synchronizing source. Of course, now time sync over wide area is easy, due to ubiquitous satellite navigation & timing systems, but at this time, the PDR implementation was innovative and worked well.
After IAMC closed down, ongoing PDR development was transferred to Vansco. Later, former IAMC developer Michael Miller joined our team and was instrumental in its ongoing development.
The PDR was renamed the Dynamic Swing Recorder (DSR) by our Manitoba Hydro team. That team, especially Dave Fedirchuk, felt that more utilities could use this technology. We tried to market it more widely, but later decided that a newer approach was required – creating a more generalized fault recorder, similar to the leaders at the time, the Rochester Instruments or Mehta Tech recorders. This is where the idea of the TESLA 2000 started to take hold.
Developed the DR2/TESLA Main Processor Board (MPB) and Other DR2 & TESLA Boards
I used to say, “those aren’t electrons flowing on the MPB, those are my blood cells!” In a philosophical sense, it was true – the development of the MPB drained me like no other project effort, before or since.
With multiple processors on board – 2 x PIC 16C MCUs (one for front panel keyboard, one for watchdog processor), a TI TMS320C32 DSP, an Altera Flex 10K FPGA, an Altera MAX 7000 CPLD, 3 x National Semiconductor LM12458 microsequenced ADCs, and a Motorola MC68HC11 MCU for IRIG processing, 3 x NS16550 FIFO UARTs, and a 16 bit PC-104 bus connection, it was quite busy. An X86-based PC-104 card was mounted on top of the MPB – lower powered on the DR2, but much more powerful model on the TESLA.
The concept was that sifting, sorting, compression, and decision making was done on the DSP, where memory was expensive but calculations were easy. The resultant data was streamed through a high-speed DSP-to-X86 FIFO, where the X86 DMA’d it into main memory and stored it to disk. The X86 was used because its memory was plentiful and inexpensive, and disk I/O was fast and easy.
The other boards in the system weren’t as complex, but were just as important. The DR2 AIB, mounted on the bottom of the chassis, had up to 18 miniature PCB-mount instrumentation transformers on it – CTs or PTs, depending on the application – and analog processing to transfer to the ADCs on the MPB. The TESLA AIB was mounted on the rear of the chassis, had more channels but was less sophisticated – non isolated +/- 2.5V (nominal) input. The DR2 and TESLA each had their own interpretation of digital I/O – DR2 with high current relay outputs, and dedicated ultra reliable digital inputs, and the TESLA with fewer and lower level outputs and simpler digital inputs.
The front panel of the DR2 had a 2 line x 24 character FIP display, a small keyboard, and several indicator LEDs, including a large bright red “Target” LED, as is expected on a relay. The TESLA front panel had the smaller LEDs only.
I established the bus structure and pin-outs for all the I/O. The front panel and digital I/O were intentionally designed to be interchangeable, with the intention of future mixed mode configuration, or expansion. Although some of the buses had extra connections (front panel serial port, for instance), this was kept to the ends of the connectors, so combining and splitting ribbon cables would be relatively straightforward. This was very well documented in extensive spreadsheets, which clearly showed connections and levels.
The memory and I/O maps of the processors was also documented in spreadsheets, with well defined bit states and clearly defined defaults.
Later, I wrote an exhaustive 70+ page programming guide to the MPB, which outlined all its features, how each processor interacted, the resources used, assumptions in programming, and how to compose images for execution.
TESLA 2000 Released
The TESLA 2000 was the most sophisticated disturbance recorder of its time. It used two MPBs, connected together by a bidirectional DSP-to-DSP synchronous serial link. This gave the TESLA 36 analog inputs.
However, it was the software which made the TESLA so outstanding. Although the serial port connection could be a pain to set up on MS-Windows, once it was set up, it worked like a dream. The RecordBase / RecordGraph system was amazing and intuitive. It was an instant hit in the Power Utility Community.
L-PRO and T-PRO Released
The delay was stressful, but the results were worth it! Manitoba’s very own 6 input (x 3 phase, always) protective relay became a reality!
The Analog Input Board (AIB) was configured to manage any mix of CTs and PTs, up to 18 discrete inputs.
The L-PRO came out first, then the T-PRO. Their primary hardware difference was the mix of CTs & PTs, and the T-PRO had a separate input for oil temperature transducer.
Challenged More and More: DR2 and TESLA on the Same Platform
We did focus group studies, discussed with customers, went to trade shows and conferences, and finally, settled on the development of two platforms for the power utility market: a TMS320C32-based platform for the relay, which we would call the DR2 (DR1 would have been the units developed using TMS320C30 at the UofM), and a TMS320C44-based platform for the recorder.
The ‘C3x family is fast and inexpensive. The ‘C44 family is well connected and easily networked. No comparable family at the time gave us both.
Management deemed this too expensive and risky, so we were told to go back to the drawing board and do it with only one platform. We gasped… sighed… and started over.
I can’t say that it was wrong. The development was successful… the product met market demand… and, with updates and changes, still in production today.
The core Main Processor Board developed as part of that effort, was in active production and use until being finally retired in 2018. Not bad!