Category: LINUX Development

Implement IRIG-B Decoder in LINUX and on EFM8UB2 Universal Bee

I’ve been advised that one of the biggest challenges in the installation of power utility relays & recorders can be the setup and configuration of the IRIG-B time code distribution.

If there was a small, inexpensive hand-held device that could read the IRIG-B signal, and tell us all its characteristics – modulated/unmodulated, true/inverted, IEEE-1344 extensions active, time zone, offset, and other information – then it would be easier to diagnose the problem and tell the customer what they have to do to get the time sync to work.

I did a search and could not find anything… so after playing with decoding IRIG on my LINUX computer, I bought my own EFM8UB2 Development Kit and proceeded to write a decoder for it, per my blog entry.

The challenge now is to do a user interface for it.  I can easily do a LINUX UI (and did!), but what would be really cool is to (say) have an Android app that could connect to the EFM8UB2 using USB OTG – and display the data there.

I actually go the full Google Android SDK and emulator up & running, and was clicking along, learning the development of Android apps, but then along came Eye on the Ice – which took all of my spare non-work hours time.  One hobby at a time!  I’d like to return to it, and finish, some day soon.

Contribute Update to NTP Project’s IRIG & WWV Tone Generator

I’ve always wanted to create a WWV(H) decoder.  Sigh.  In the late 1980s, I played with switched capacitor filters (MF10 springs to mind, but that’s a relatively low order filter, maybe it was the MAX7490 or MAX7413?), and with NE565 PLLs, to try and pull the 100 Hz subcarrier out of the shortwave audio signal WWV(H), then decode it.  Unfortunately, in this application, PLLs and high order bandpass filters are basically the same thing, and I didn’t get all the way to building the decoder.

Imagine my delight when I realized that the NTP project already has built-in, a WWV(H) decoder… and as a bonus, also an IRIG-B decoder too (modulated only, no unmodulated… yet).  Well, by this time, I didn’t have a shortwave radio handy to pull in WWV with… but, what’s this?   A program called tg in the utils directory is a WWV(H) and modulated IRIG-B generator!

I had a look at it, and sure enough, it would generate WWV(H) and modulated IRIG-B signals, but it was pretty limited, didn’t do any of the IEEE-1344 extensions that we used in power utility applications, nor leap seconds etc.  That wasn’t enough for me.  Hmm, I would rewrite tg to test all these things and generate synthetic IRIG-B.  So, I did.

Oops, it was not built for a modern LINUX sound system… so I ported it.

It was good to be able to test an IRIG-B decoder – I could start at an arbitrary time, insert and remove leap seconds, trigger various conditions, etc.  So I added these functions.

I found that the computer sound card sample rate wasn’t precise.  There were precisely 8,000 samples per second, so after a while, the time emitted by tg would drift.  I modified tg to omit or add an extra cycle (in an unused area of the protocol) to correct for this.  The program would track the sent time and the “apparent” real time with respect to the computer’s real-time clock, and would correct when needed.

I re-mastered a Knoppix image to include tg2 for demonstration of my work.  I thought Elecsys might sell it (or its support) to ERLPhase for cash, but it wasn’t of enough utility to make it worthwhile.

I added unmodulated IRIG-B output, although that was trouble because most sound cards won’t pass frequencies low enough to emit unmodulated IRIG-B.  But I unwarped the filter and did something that almost works, sometimes.

I added sophistication to the WWV(H) output as well, with coded pulses lining up more closely to what the actual station does.

Then I submitted it back to Dr. David Mills for inclusion in the NTP sources… he called my program tg2.

Recently, when reviewing my work and making a few tweaks on tg2 and playing with other parts of NTP, I realized that I had violated Dr. Mills’s coding format guidelines pretty badly.  I’m surprised he put up with it!  So I tried to do better.

Management of Wabtec Embedded Software Team

Wabtec’s Winnipeg plant is actually in Oakbank, about 30 km (18 mi) outside of Winnipeg.  Oakbank is a small community of a few thousand people, surrounded by farmland, and, to the northwest, hills with rock quarries.

This was originally a small Winnipeg company called iders (pronounced by most people as I-D-ers), who had been pioneers in the development of PIN pads for Canadian banks and retailers, and later successfully landed the contract to develop the Conviron CMP4000 environmental growth chamber system, among many projects of varying sizes and complexity.   In the 2001 to 2008 time frame, they also manufactured the NxtPhase TESLA DC Isolation Module, which Horst Koelzow and I had developed.

One of the industries that iders broke into in the early 2000s was electronics for the railroad industry.  Their experience in communications in general, and computer networking in particular, led them over several years, to create what’s now known as the GoLINC ACC (Auxiliary Communications Cage), which has the AAR standard for communications on railroad locomotives.

iders was acquired by GE Transportation in 2016.  In 2018, GE Transportation itself was divested from GE, merging with Wabtec.  The Oakbank plant was still referred to as “GETW”  (GE Transportation Winnipeg) until early 2020, when the migration from GE’s IT systems to Wabtec’s IT systems was mostly complete.

I was hired as the Embedded Software Engineering Manager for GETW in August 2018, with 11 direct reports.  A few days after starting, I was advised that I had 3 hires to make – which, for historical reasons, were each assigned to separate hiring managers, and were each managed by separate recruiters.  The hiring took some time to implement, as we worked out the projected seniority of the hires and reviewed dozens of applicants… but the big thing is that I was busy in the interim.  We made the hires in January 2020.

There was a lot of development going on at GETW.  Some was new, some was remedial work, but it was all very intense.

When the COVID-19 pandemic hit, we were quite fortunate to be able to work effectively from our homes, because Wabtec has quite a good internal communications infrastructure.  There were issues, mind you – we had trouble accessing the Checkmarx static analysis scanner – but we worked around that using a Raspberry Pi that I put on my desk at the office, until the proper routing could be arranged through the VPN.

The pandemic was not as serious in Manitoba as in many other parts of the world.  When things settled down in mid May, we carefully moved back to the office, taking appropriate precautions.

Unfortunately, as with many industries around the world, the locomotive industry was hurt by the economic downturn that accompanied COVID-19.  As a result, my employment with Wabtec came to an end in June.  Many others were let go at the same time, and apparently there will be a long term strategy to consolidate Wabtec’s electronics manufacturing into other locations, and downsizing the office footprint, saving considerable money in the process.

Develop LINUX Test Tool for GE-Alstom COSI-CT Management Processor

The only way to talk to a COSI-CT Management Processor was the COSI-CT Control Panel program in MS-Windows.  It knows the structure of the EEPROM parameter storage in the Management Processor, and interprets it accordingly – for instance, if a single byte value is used, it shows it as an integer 0-255, if it’s a string, it shows as a form fill with fixed length, etc.  There is a mode that allows it to write to not-defined areas too… but it is clunky.  And, to boot, it’s all in decimal, argh.  As I’ve said, if it’s a bit mapped value, give it to me in hexadecimal, don’t make me convert it!

Anyways, COSI-CT Control Panel has access to some control flags that indicate whether the parameters of the COSI-CT are under seal – that is, some of the values cannot be changed, probably because they would alter calibration and therefore current values, which dictate revenue to the power company.  When the seal is set, Control Panel will not let you even try to modify these settings.  That’s great for end users, but for testing, I can’t tell whether the seal is actually working.

For some projects, the seal is more important, than for others.  For instance, if you are just doing general disturbance recording or relaying, there’s no reason for anyone to mess with the calibration, so it’s not as important.  However, if it’s on a utility-to-utility intertie, or a generator output, then it might be very important.

We had an instance where it was very important, and needed to know that the seal was working properly.  Actually, let my rephrase that – we had a corner case that seemed to indicate that the seal was not working completely correctly.  So, I had to correct that issue.  But, I couldn’t test it!

The serial line protocol to the Management Processor is well documented, but of course, having the source code of the processor itself was very helpful.  There are always ambiguities in protocol specifications, unless you happen to give the specification to someone completely in the dark, and have them develop a device to talk that protocol.  Well, this was just such a case… but again, I had the source code.

So, I wrote a program in ‘C’ for LINUX to talk to the Management Processor.  It wasn’t intended to replace Control Panel, by any stretch, but it ended up expanding and expanding and expanding (as many of my utilities do), so it can run a “write challenge” against every value, to see if it can be modified under seal, can read all values from the device, write individual or all values into the device, and read every real-time value supported.  In fact, I expanded it to support every command on the serial link to the Management Processor.

This was quite useful in determining whether the seal had previously been implemented correctly (hint: there were bugs), and to figure out if it was corrected, but also in talking to devices over TCP/IP – like when units were installed at the SRP Perkins substation, and I was talking over a haltingly slow SSH tunnel link.  Control Panel would burp, retry, and give up, but, having complete control over the link, I was able to make the connection, lengthen retries, or whatever it took, to make the connection with my command line program, I was able to arrange to get the data back reliably.

In early development, I cross-compiled the code to run from Cygwin in MS-Windows, but the serial port names in MS-Windows tripped it up.  I’m pretty sure it would take little to re-port it to MS-Windows.

I advised management of the program, and chatted with Scott Picker (maintainer of COSI-CT Control Panel) that he might want to use it for a test facility, and I gave them the source code, of course, but as far as I’m aware, nothing ever came of it…  sigh.

Develop Wireless Remote Access to GE-Alstom Equipment inside SRP Perkins Substation

Salt River Project Power (SRP) had a mystery in its transmission network – some disturbances were unexplained.  By installing a COSI-CT and my newly-developed VT, and a GE-Reason recorder at the Perkins Substation, GE would get some valuable operational data from a local substation, and SRP would find out the cause of the problem.

It would be great to have ongoing telemetry from our equipment in the substation, but of course SRP cannot allow another company like GE to have access to its computer network.  I was asked to find a wireless way to communicate with the system.

I asked if SRP would allow us to put up an extra antenna – they were already installing a GPS antenna for us – and they agreed.  So I undertook to do a quick survey of potential means of communications, and ended up with a cellular phone router – one of the Multi-Tech MTR-H5 family devices.

I was going to get a SIM through GE Corporate, but the required effort & justification was huge.  My manager needed this done quickly.  So, I went down to my local T-Mobile store and got a “data & SMS only” SIM (intended for tablets etc.).  It was only US$10 a month – which, because I was travelling so much,  just put on my expense account thereafter.

Then I tried to contact the router – but couldn’t.  Apparently incoming connections to T-Mobile’s wireless devices isn’t generally allowed.  This is probably a good thing!

We were also installing a GE DAPserver, which would serve as a data concentrator.   I dug around a bit and got root access to the DAPserver (no, it wasn’t easy because we were GE – they don’t give that information out!).  I then created a Digital Ocean droplet and coded a continuous SSH connection from the DAPserver to the droplet.  There were actually two connections – one by IP address, and one by dynamic DNS hostname – just in case one failed.  Each one had reverse traffic tunnels brought up, that would link to the various local devices, like each COSI-CT and the recorder.

Well, it turned out that the COSI-CT only provides a serial interface to its administration functions.  The process data comes out on an Ethernet link (generally fibre, but could be twisted pair), but the Management Processor interface is RS-232, ugh.  So, I had to buy a Lantronix Xpress-DR+ 2-port serial-to-Ethernet gateway… which was fine.

In any case, it all worked well, in the end.  I was actually able to log into the recorder and display phasors – although the link was relatively slow, and performance was poor.  For text-based communications, like talking to the COSI-CT, it mostly worked fine.  The COSI-CT Control Panel didn’t like the link, because turn-around latency was too slow.  Well, Control Panel wasn’t designed for Ethernet either, but Scott Picker added this feature for me, but there was some complication in the negotiation of the link (possibly the original serial auto-line-rate detection) that caused it to drop every time.  No matter, my little LINUX Test Tool worked just fine, and enabled me to read and write configuration values as needed.

Although it was clunky and slow, it did work – so I considered it a success!

Development of Ankle-Brachial-Index Measurement Proof of Concept for Koven

Paul Koven, founder of Koven Technology Canada, was dying, and wanted to see his dream of an automatic machine measuring the ankle-brachial-index of patients.  With a very short time frame, Elecsys was asked to create a proof of concept.

Elecsys produced that proof of concept in about 6 weeks, using an SBC running LINUX, four modified off-the-shelf blood pressure monitors (selected for their ability to be controlled and monitored through a serial port), and a 2 line x 24 character alpha display, and a few buttons.  The result was very large, heavy and not all that pretty – but it worked!  We provided a single unit, with rudimentary build documentation (as much as you can expect with a proof of concept!) and a manual, and they took it to St. Louis, MO to show to Mr. Koven.

Paul did see it before he died, and we got the feedback that it was much appreciated.

Koven’s management wanted to build more of the units, but were dismayed when they found out that it wasn’t a finished product.  They hounded us for months, asking for more construction details, drawings and blueprints, and alternate suppliers for the components.  Time and time again, we patiently explained to them that they had commissioned a proof-of-concept, not even a prototype, and that the each subsequent one would cost the same to build as the first.  We offered to put more engineering into the product to make it more polished and manufacturable, for a cost of course, but each time they declined.  Then, a couple of months later, they would contact us again about details on how to mass-produce the proof-of-concept.

Safe Power Supply Testing on TESLA 3000 MPB

One of the substantial changes for the TESLA 3000 was the incorporation of the DR2/TESLA power supply right onto the MPB.  The DR2/TESLA power supply was an amazing accomplishment, able to operate full power at reasonable efficiency from 38Vdc to 300Vdc.  A custom buck switchmode pre-regulator was used to bring the input voltage down into the range that a Lambda 48V half-brick DC-to-DC converter could handle – which was 36V to 72V nominal.  Below about 54V input, the pre-regulator would lock into full-on, passing through with minimal drop, giving optimal low end operation.

When testing the original DR2/TESLA power supply a decade earlier, we had an elaborate procedure that required a sequential voltage bump, checking critical operation parameters as the test proceeded.  For instance, first going to a low voltage just above where the pre-regulator would start switching, and verifying that it was switching at the proper frequency, and that the output voltage was reasonable and within range.  Without ensuring that the pre-regulator was working properly, a high voltage pass-through would destroy not only the DC-to-DC converter, but also the large reservoir capacitors that we had at its input.  In fact, the reservoir capacitors were dangerous, exploding when overstressed!

Somewhere along the way, the cautious approach was forgotten.  The contract manufacturer making the TESLA 3000 MPBs was experiencing explosive failures in testing.  Their staff was afraid of the board.  They would put a box over the board before testing.

Elecsys put its experience with GPIB to good use.  We obtained more GPIB equipment and a Xantrex XHR300-3.5 power supply, which had a serial interface.  I wrote the first draft of a PERL program to direct the fully automated testing of the TESLA 3000 MPB power supply, then had Nishant Dhruve complete the work.