Category: All Timeline Stories

All timeline stories.

Develop LINUX Test Tool for GE-Alstom COSI-CT Management Processor

The only way to talk to a COSI-CT Management Processor was the COSI-CT Control Panel program in MS-Windows.  It knows the structure of the EEPROM parameter storage in the Management Processor, and interprets it accordingly – for instance, if a single byte value is used, it shows it as an integer 0-255, if it’s a string, it shows as a form fill with fixed length, etc.  There is a mode that allows it to write to not-defined areas too… but it is clunky.  And, to boot, it’s all in decimal, argh.  As I’ve said, if it’s a bit mapped value, give it to me in hexadecimal, don’t make me convert it!

Anyways, COSI-CT Control Panel has access to some control flags that indicate whether the parameters of the COSI-CT are under seal – that is, some of the values cannot be changed, probably because they would alter calibration and therefore current values, which dictate revenue to the power company.  When the seal is set, Control Panel will not let you even try to modify these settings.  That’s great for end users, but for testing, I can’t tell whether the seal is actually working.

For some projects, the seal is more important, than for others.  For instance, if you are just doing general disturbance recording or relaying, there’s no reason for anyone to mess with the calibration, so it’s not as important.  However, if it’s on a utility-to-utility intertie, or a generator output, then it might be very important.

We had an instance where it was very important, and needed to know that the seal was working properly.  Actually, let my rephrase that – we had a corner case that seemed to indicate that the seal was not working completely correctly.  So, I had to correct that issue.  But, I couldn’t test it!

The serial line protocol to the Management Processor is well documented, but of course, having the source code of the processor itself was very helpful.  There are always ambiguities in protocol specifications, unless you happen to give the specification to someone completely in the dark, and have them develop a device to talk that protocol.  Well, this was just such a case… but again, I had the source code.

So, I wrote a program in ‘C’ for LINUX to talk to the Management Processor.  It wasn’t intended to replace Control Panel, by any stretch, but it ended up expanding and expanding and expanding (as many of my utilities do), so it can run a “write challenge” against every value, to see if it can be modified under seal, can read all values from the device, write individual or all values into the device, and read every real-time value supported.  In fact, I expanded it to support every command on the serial link to the Management Processor.

This was quite useful in determining whether the seal had previously been implemented correctly (hint: there were bugs), and to figure out if it was corrected, but also in talking to devices over TCP/IP – like when units were installed at the SRP Perkins substation, and I was talking over a haltingly slow SSH tunnel link.  Control Panel would burp, retry, and give up, but, having complete control over the link, I was able to make the connection, lengthen retries, or whatever it took, to make the connection with my command line program, I was able to arrange to get the data back reliably.

In early development, I cross-compiled the code to run from Cygwin in MS-Windows, but the serial port names in MS-Windows tripped it up.  I’m pretty sure it would take little to re-port it to MS-Windows.

I advised management of the program, and chatted with Scott Picker (maintainer of COSI-CT Control Panel) that he might want to use it for a test facility, and I gave them the source code, of course, but as far as I’m aware, nothing ever came of it…  sigh.

Develop Wireless Remote Access to GE-Alstom Equipment inside SRP Perkins Substation

Salt River Project Power (SRP) had a mystery in its transmission network – some disturbances were unexplained.  By installing a COSI-CT and my newly-developed VT, and a GE-Reason recorder at the Perkins Substation, GE would get some valuable operational data from a local substation, and SRP would find out the cause of the problem.

It would be great to have ongoing telemetry from our equipment in the substation, but of course SRP cannot allow another company like GE to have access to its computer network.  I was asked to find a wireless way to communicate with the system.

I asked if SRP would allow us to put up an extra antenna – they were already installing a GPS antenna for us – and they agreed.  So I undertook to do a quick survey of potential means of communications, and ended up with a cellular phone router – one of the Multi-Tech MTR-H5 family devices.

I was going to get a SIM through GE Corporate, but the required effort & justification was huge.  My manager needed this done quickly.  So, I went down to my local T-Mobile store and got a “data & SMS only” SIM (intended for tablets etc.).  It was only US$10 a month – which, because I was travelling so much,  just put on my expense account thereafter.

Then I tried to contact the router – but couldn’t.  Apparently incoming connections to T-Mobile’s wireless devices isn’t generally allowed.  This is probably a good thing!

We were also installing a GE DAPserver, which would serve as a data concentrator.   I dug around a bit and got root access to the DAPserver (no, it wasn’t easy because we were GE – they don’t give that information out!).  I then created a Digital Ocean droplet and coded a continuous SSH connection from the DAPserver to the droplet.  There were actually two connections – one by IP address, and one by dynamic DNS hostname – just in case one failed.  Each one had reverse traffic tunnels brought up, that would link to the various local devices, like each COSI-CT and the recorder.

Well, it turned out that the COSI-CT only provides a serial interface to its administration functions.  The process data comes out on an Ethernet link (generally fibre, but could be twisted pair), but the Management Processor interface is RS-232, ugh.  So, I had to buy a Lantronix Xpress-DR+ 2-port serial-to-Ethernet gateway… which was fine.

In any case, it all worked well, in the end.  I was actually able to log into the recorder and display phasors – although the link was relatively slow, and performance was poor.  For text-based communications, like talking to the COSI-CT, it mostly worked fine.  The COSI-CT Control Panel didn’t like the link, because turn-around latency was too slow.  Well, Control Panel wasn’t designed for Ethernet either, but Scott Picker added this feature for me, but there was some complication in the negotiation of the link (possibly the original serial auto-line-rate detection) that caused it to drop every time.  No matter, my little LINUX Test Tool worked just fine, and enabled me to read and write configuration values as needed.

Although it was clunky and slow, it did work – so I considered it a success!

Update Alstom DIT COSI-CT Management Processor

I was only at Alstom Grid DIT in Phoenix a couple of months, when an question came from management, “Does anyone here know PIC assembly language?”

Well, I do!  So I took on the modification of the COSI-CT Management Processor firmware.  The code was written circa 2003, and was based on earlier efforts.  Good designers re-use working code!  Anyway, it had been maintained up until about 2010 by Dylan Stewart, but Dylan now had his hands full as Product Manager.  So, I took it on.

It started largely as a minor rework.  The processor has explicit support for each stored configuration value inside, knowing which one(s) are bytes, which words, which double words, and which are strings.  Things had changed in the configuration, but support for these changes was incomplete.  So, I addressed that.

A larger issue was that of supported development environment.  The old MPLab 8.x was obsolete, with MPLab X taking its place, but MPLab X would not automatically import MPLab 8.x files, especially complex ones.  I installed both development systems, and ensured that the outputs were byte-for-byte equivalent – which you can do with assembly language code!  It took a lot of tinkering to get the code to assemble properly under MPLab X, even more difficult to get it to assemble under both, but I managed it.

There was some discussion about the apparent corruption of data values on the backplane bus to and from the other internal cards.  There was no error checking on this bus.  I wrote a proposal to use CRC on the backplane comms.  It was carefully tweaked to fit within the protocol, and minimize overhead.  I implemented it on the Management Processor, but due to an error in the implementation on the CT card’s FPGA, there was an amiguity in the link, causing protocol failure.  I tried to get this error corrected, but it never was, so the effort was abandoned in place.  The code is still there, but it is never used.

Counsel Easun Reyrolle Relay Development Team

ERLPhase Winnipeg had the challenge of bringing a relay development team up to speed at their parent company, Easun Reyrolle of Bangalore, India.  ERLPhase had the experience but could not spare the resources.  So, I took a term contract to assist in helping to get the Bangalore team started.

Culture was a big challenge, especially over Skype, not having ever met any of the staff in person.  I recommended that they get the Indian staff to visit Winnipeg, or (shudder) I would have to visit Bangalore.  Well, the Indian team had difficulty in getting to Winnipeg, so I went to Bangalore… and loved it!  This was quite the adventure, culture shock, all of the cliches, but overall, a very good experience.

After my return, it was much easier to work with the Bangalore team, now that I knew them well.  A great group of people, some of whom I am still in touch with.