When my involvement in the APT Division of Vansco was at its peak, I almost always went to the local Power Engineering Society meetings, which were held during lunch hour at a local hotel. When the chair of the local chapter stepped down, Tom Molinski of Manitoba Hydro, Dave Fedirchuk of APT (seconded from Manitoba Hydro), and I, got together to discuss how to ensure the chapter’s continued success. We decided that I would become chair, and they would assist in finding meeting topics and presenters. This worked very well. Later, other members stepped forward to assist as well. I was chair of PES for 4 years.
Preliminary Proof of Concept Work on ERLPhase Fault Location Algorithm
The very cool thing about sigma-delta conversion is that there are trade-offs that can be made with the data after it is captured – sampling rate versus inherent filtering versus resolution versus noise floor. These are all inter-related.
ERLPhase wanted to investigate travelling wave fault location. Technicians captured mock fault data in the lab, which I modelled in Python, performing analysis on the frequency content, and various processing techniques. I developed a proprietary front-end amplifier in LTspice, then implemented the circuit on a proof-of concept PCB. I wrote Python scripts to import the fault data into an Arbitrary Waveform Generator (often called “an ARB”), and played them into the new board, captured the data on the sigma-delta modulator evaluation board, post-processing the captured data in Python (including wavelet analysis), and displayed the results graphically in 2-D, 3-D and colour plots.
I wrote a report on the effort, providing details and outlining the path of product development.
Develop ERLPhase 2nd Generation DC Analog Isolation Module
The original DC isolation module, developed under my direction around 1999, used an unconventional custom off-line power supply and four Analog Devices AD204 isolation amplifiers, to provide isolated voltage and current input from DC to about 1 kHz. This opened up a new market for recording sensors and telemetry signals on the TESLA, as well as some unique DC-level-sensitive AC applications, such as internal levels in static VAR compensators (SVCs) and DC converters.
Modern applications are much more stringent, however, needing better performance than the AD204 can provide (accuracy, noise level, and isolation), and the power supply couldn’t meet today’s restrictive EMC requirements.
So, I was asked to develop a newer high-performance DC Isolation Module. The basic concepts had been established by Steve Peddle, but he was quite busy on other projects.
The original concept was to perform “intelligent” ADC conversion on isolated “islands”, encoding the data in a protocol, and transferring the multiple streams of this protocol using digital isolation devices back to a central location, where it would be converted back by multiple DAC conversion on a circuit not isolated from the TESLA recoreder (as is required by the TESLA analog front-end).
I came across a potential alternative in the form of isolated sigma-delta modulator converters like the Analog Devices AD7401, which could give a highly accurate, flexible solution. Using an evaluation board, I captured data and performed extensive data analysis in Python. I implemented the sinc3 filter in Python, demonstrated the trade-offs, and explored the possibilities of various configurations.
I evaluated the technologies against each other and wrote a report. After discussions with Steve and ERLPhase management, we decided to proceed with the sigma-delta modulator approach.
We obtained the evaluation kit for the Silicon Labs Universal Bee MCU EFM8UB20, and wrote the initial code as proof-of-concept and to bring up the system, handing it over to the firmware development team to implement the core functions.
I developed and documented the isolation plan, which included double 300V isolation gaps. After a design review, I created the schematic, and had two interconnected PCBs developed. I wrote a verification & validation document with guideline for the firmware implementation.
Work with GE-Reason Brazil on Next Generation COSI-CT Architecture
The original design of what’s now the COSI-CT was developed at NxtPhase in about 1999. When I was with NxtPhase Winnipeg from 2001-2003, I was involved in an informal internal evaluation of the system, and recommended significant changes. It wasn’t that the design was bad, but that it could be improved to make it better suited for the power utility market.
A NxtPhase CT redesign was undertaken around 2003. When I joined Alstom Grid in 2013 to work on the COSI-CT, it was practically identical to the 2003 redesign, but had a few extra components grafted on – an optional internal power supply, and a digital card. The design was tired and had developed serious flaws, which we were trying to fix.
Everyone agreed that a new approach, the next generation of COSI-CT (called “Gen4”), was needed. As part of GE-Alstom after the merger, we undertook the exciting new development of Gen4.
I travelled to Florianopolis, Brazil, and met with the team at GE-Reason. They had developed an amazing array of networking, merging unit, and recording equipment before their acquisition by Alstom in 2015. Together, the engineers at Reason and I, set about to architect COSI-CT Gen4.
After my return to Phoenix, we collaborated constantly, often in communication daily. We worked in Atlassian Confluence, developing the architecture. We consulted with other GE experts in Merging Unit design, and our own marketing team.
Unfortunately, just as the initial architecture was completed in late 2017, GE faced a cash crunch and had to lay off many staff, including me. This was a big disappointment, as we were really excited about the potential for the next generation.
Oh well, hopefully Gen4 of the COSI-CT, whenever it emerges, and in whatever form, is wildly successful.
Develop LINUX Test Tool for GE-Alstom COSI-CT Management Processor
The only way to talk to a COSI-CT Management Processor was the COSI-CT Control Panel program in MS-Windows. It knows the structure of the EEPROM parameter storage in the Management Processor, and interprets it accordingly – for instance, if a single byte value is used, it shows it as an integer 0-255, if it’s a string, it shows as a form fill with fixed length, etc. There is a mode that allows it to write to not-defined areas too… but it is clunky. And, to boot, it’s all in decimal, argh. As I’ve said, if it’s a bit mapped value, give it to me in hexadecimal, don’t make me convert it!
Anyways, COSI-CT Control Panel has access to some control flags that indicate whether the parameters of the COSI-CT are under seal – that is, some of the values cannot be changed, probably because they would alter calibration and therefore current values, which dictate revenue to the power company. When the seal is set, Control Panel will not let you even try to modify these settings. That’s great for end users, but for testing, I can’t tell whether the seal is actually working.
For some projects, the seal is more important, than for others. For instance, if you are just doing general disturbance recording or relaying, there’s no reason for anyone to mess with the calibration, so it’s not as important. However, if it’s on a utility-to-utility intertie, or a generator output, then it might be very important.
We had an instance where it was very important, and needed to know that the seal was working properly. Actually, let my rephrase that – we had a corner case that seemed to indicate that the seal was not working completely correctly. So, I had to correct that issue. But, I couldn’t test it!
The serial line protocol to the Management Processor is well documented, but of course, having the source code of the processor itself was very helpful. There are always ambiguities in protocol specifications, unless you happen to give the specification to someone completely in the dark, and have them develop a device to talk that protocol. Well, this was just such a case… but again, I had the source code.
So, I wrote a program in ‘C’ for LINUX to talk to the Management Processor. It wasn’t intended to replace Control Panel, by any stretch, but it ended up expanding and expanding and expanding (as many of my utilities do), so it can run a “write challenge” against every value, to see if it can be modified under seal, can read all values from the device, write individual or all values into the device, and read every real-time value supported. In fact, I expanded it to support every command on the serial link to the Management Processor.
This was quite useful in determining whether the seal had previously been implemented correctly (hint: there were bugs), and to figure out if it was corrected, but also in talking to devices over TCP/IP – like when units were installed at the SRP Perkins substation, and I was talking over a haltingly slow SSH tunnel link. Control Panel would burp, retry, and give up, but, having complete control over the link, I was able to make the connection, lengthen retries, or whatever it took, to make the connection with my command line program, I was able to arrange to get the data back reliably.
In early development, I cross-compiled the code to run from Cygwin in MS-Windows, but the serial port names in MS-Windows tripped it up. I’m pretty sure it would take little to re-port it to MS-Windows.
I advised management of the program, and chatted with Scott Picker (maintainer of COSI-CT Control Panel) that he might want to use it for a test facility, and I gave them the source code, of course, but as far as I’m aware, nothing ever came of it… sigh.
Develop Wireless Remote Access to GE-Alstom Equipment inside SRP Perkins Substation
Salt River Project Power (SRP) had a mystery in its transmission network – some disturbances were unexplained. By installing a COSI-CT and my newly-developed VT, and a GE-Reason recorder at the Perkins Substation, GE would get some valuable operational data from a local substation, and SRP would find out the cause of the problem.
It would be great to have ongoing telemetry from our equipment in the substation, but of course SRP cannot allow another company like GE to have access to its computer network. I was asked to find a wireless way to communicate with the system.
I asked if SRP would allow us to put up an extra antenna – they were already installing a GPS antenna for us – and they agreed. So I undertook to do a quick survey of potential means of communications, and ended up with a cellular phone router – one of the Multi-Tech MTR-H5 family devices.
I was going to get a SIM through GE Corporate, but the required effort & justification was huge. My manager needed this done quickly. So, I went down to my local T-Mobile store and got a “data & SMS only” SIM (intended for tablets etc.). It was only US$10 a month – which, because I was travelling so much, just put on my expense account thereafter.
Then I tried to contact the router – but couldn’t. Apparently incoming connections to T-Mobile’s wireless devices isn’t generally allowed. This is probably a good thing!
We were also installing a GE DAPserver, which would serve as a data concentrator. I dug around a bit and got root access to the DAPserver (no, it wasn’t easy because we were GE – they don’t give that information out!). I then created a Digital Ocean droplet and coded a continuous SSH connection from the DAPserver to the droplet. There were actually two connections – one by IP address, and one by dynamic DNS hostname – just in case one failed. Each one had reverse traffic tunnels brought up, that would link to the various local devices, like each COSI-CT and the recorder.
Well, it turned out that the COSI-CT only provides a serial interface to its administration functions. The process data comes out on an Ethernet link (generally fibre, but could be twisted pair), but the Management Processor interface is RS-232, ugh. So, I had to buy a Lantronix Xpress-DR+ 2-port serial-to-Ethernet gateway… which was fine.
In any case, it all worked well, in the end. I was actually able to log into the recorder and display phasors – although the link was relatively slow, and performance was poor. For text-based communications, like talking to the COSI-CT, it mostly worked fine. The COSI-CT Control Panel didn’t like the link, because turn-around latency was too slow. Well, Control Panel wasn’t designed for Ethernet either, but Scott Picker added this feature for me, but there was some complication in the negotiation of the link (possibly the original serial auto-line-rate detection) that caused it to drop every time. No matter, my little LINUX Test Tool worked just fine, and enabled me to read and write configuration values as needed.
Although it was clunky and slow, it did work – so I considered it a success!
Develop and Support CCVT Input on GE-Alstom COSI-CT
Update Alstom DIT COSI-CT Management Processor
I was only at Alstom Grid DIT in Phoenix a couple of months, when an question came from management, “Does anyone here know PIC assembly language?”
Well, I do! So I took on the modification of the COSI-CT Management Processor firmware. The code was written circa 2003, and was based on earlier efforts. Good designers re-use working code! Anyway, it had been maintained up until about 2010 by Dylan Stewart, but Dylan now had his hands full as Product Manager. So, I took it on.
It started largely as a minor rework. The processor has explicit support for each stored configuration value inside, knowing which one(s) are bytes, which words, which double words, and which are strings. Things had changed in the configuration, but support for these changes was incomplete. So, I addressed that.
A larger issue was that of supported development environment. The old MPLab 8.x was obsolete, with MPLab X taking its place, but MPLab X would not automatically import MPLab 8.x files, especially complex ones. I installed both development systems, and ensured that the outputs were byte-for-byte equivalent – which you can do with assembly language code! It took a lot of tinkering to get the code to assemble properly under MPLab X, even more difficult to get it to assemble under both, but I managed it.
There was some discussion about the apparent corruption of data values on the backplane bus to and from the other internal cards. There was no error checking on this bus. I wrote a proposal to use CRC on the backplane comms. It was carefully tweaked to fit within the protocol, and minimize overhead. I implemented it on the Management Processor, but due to an error in the implementation on the CT card’s FPGA, there was an amiguity in the link, causing protocol failure. I tried to get this error corrected, but it never was, so the effort was abandoned in place. The code is still there, but it is never used.
Counsel Easun Reyrolle Relay Development Team
ERLPhase Winnipeg had the challenge of bringing a relay development team up to speed at their parent company, Easun Reyrolle of Bangalore, India. ERLPhase had the experience but could not spare the resources. So, I took a term contract to assist in helping to get the Bangalore team started.
Culture was a big challenge, especially over Skype, not having ever met any of the staff in person. I recommended that they get the Indian staff to visit Winnipeg, or (shudder) I would have to visit Bangalore. Well, the Indian team had difficulty in getting to Winnipeg, so I went to Bangalore… and loved it! This was quite the adventure, culture shock, all of the cliches, but overall, a very good experience.
After my return, it was much easier to work with the Bangalore team, now that I knew them well. A great group of people, some of whom I am still in touch with.