Elecsys starts with a… Last Call?!?
Elecsys started out with what could have been a “bang”, but ended up being a “whimper”. My “whimpering” from lack of sleep!
Two businessmen from the Riding Mountain area, Terry Ledoux and Rene Roncin, contacted Vansco about development of an remote controlled duck decoy animation system. Vansco was too busy, so they referred them to Elecsys.
Terry Ledoux was a taxidermist based in McCreary MB, who sold his taxidermy business and went full time into being an outfitter and hunting guide. He wanted his new product to be called Last Call, a trade name which he owned.
They had started the development, even had nice rotary moulded enclosures made, but they didn’t know how to do the electronics, so Elecsys took them on.
Elecsys was actually operating out of the attic of Jason’s house. We brought in all our contacts on the job – mechanical development, PCB layout, PIC programming – and ran our credit to the limit to cover all the costs of parts, etc.
The last 48 hours were an absolute nightmare. We worked non-stop, had all kinds of issues – not the least of which was destroying multiple radio modules – they were not only quite expensive, but we also only had so many to work with. I so much needed sleep that I went downstairs to lay down on the couch, while one of my colleagues was going to try to commission another unit with our last radio module… when I had a flash of insight! I had to rush back upstairs and stop them from powering it up! The radios had been destroyed by inductive kickback from the motor drive circuit, which operated from the same supply. We hastily reworked the circuit, and with another 14 hours of work, we delivered two functional prototypes to the customer for a business trip to demonstrate them to Cabela’s in the US.
In the meantime, we were all strapped for cash – the customer, Elecsys, Jason and I ourselves… and had to settle for only a portion of the planned amount. We did cover our costs, but not much more. Needless to say, the customer didn’t proceed with the next stage of the development, which would have cost a lot more money. That was unfortunate, because it was a cool product with a cute name, and I think it would have been a winner.
Some time later, we sat around with Ed Van Humbeck and discussed our experience. He shared with us that something similar happened to him in the early days of Vansco, and counselled us to always get as much as possible up front – to at least cover the bare costs of the project, should the final payment not be forthcoming. We still had much to learn!
Development of Flex-Header Controller for Honey Bee
Brian Fletcher of Honey Bee had contacted Vansco to do a small electronic assembly for them, but Vansco was too large and busy, so they referred him to Elecsys. We happily took on the task of creating a controller for their new 42 foot combine flex header.
The challenge is that, with a flexible cutterbar out front, you cannot rely on simple bumpers to keep the cuttterbar from interfering with the reel. It might flex in the middle, coming up and cutting the plastic tines off of the reel. Those little plastic tines are expensive and very time consuming to replace.
Honey Bee had created six flex points along the length of the cutterbar, where we placed rugged precision pots to measure the displacement of the cutterbar in that region. We placed another precision pot on the reel arm. We were able to calculate interference between the reel and cutterbar, and drive the hydraulics to move the reel up and out of the way.
Life isn’t that simple, though. We needed to have controls in the cab, to configure the system and monitor its operation. We created an in-cab controller with small display, rotary dial and pushbuttons, which communicated to the controller on the implement using CAN. At each end, the processor used was a Microchip PIC18F series MCU.
Over the winter, we performed extensive static testing on a flex header using a portable hydraulic pump, at PAMI in Portage la Prairie. In the spring, we travelled to Honey Bee in Frontier SK to visit the plant and perform more tests. Over the summer, Honey Bee did extensive testing all over, and I attended testing in southern Manitoba, North Dakota, Minnesota, and Illinois, riding on the combine, watching the system operation, and tweaking the control programming.
An unexpected challenge arose, that of the hydraulic capacity of different combines. It turns out that some manufacturers have a huge hydraulic operative capacity, and others quite weak. It’s difficult to accommodate different manufacturers, even with different programming.
Another issue was the retrofit of the controls to the combine. The hydraulic controls weren’t meant to be tinkered with. On some systems, we could just drive the solenoids directly. On others, we had to diode isolate our controls from the existing controls. On others, the diode isolation set off alarms indicating open circuit on the solenoids.
It was going to be a long haul to make the system compatible with all the potential combine systems. In the end, Honey Bee could not invest the capital to make a flexible enough system, and Elecsys could not make this investment either. A further challenge was that the market for the aftermarket flex header that Honey Bee was making, would not bear the several-hundred-dollar premium for such a controller.
Sadly, the controller project ended with only a handful of (very nice looking) prototypes made.
Development of Ankle-Brachial-Index Measurement Proof of Concept for Koven
Paul Koven, founder of Koven Technology Canada, was dying, and wanted to see his dream of an automatic machine measuring the ankle-brachial-index of patients. With a very short time frame, Elecsys was asked to create a proof of concept.
Elecsys produced that proof of concept in about 6 weeks, using an SBC running LINUX, four modified off-the-shelf blood pressure monitors (selected for their ability to be controlled and monitored through a serial port), and a 2 line x 24 character alpha display, and a few buttons. The result was very large, heavy and not all that pretty – but it worked! We provided a single unit, with rudimentary build documentation (as much as you can expect with a proof of concept!) and a manual, and they took it to St. Louis, MO to show to Mr. Koven.
Paul did see it before he died, and we got the feedback that it was much appreciated.
Koven’s management wanted to build more of the units, but were dismayed when they found out that it wasn’t a finished product. They hounded us for months, asking for more construction details, drawings and blueprints, and alternate suppliers for the components. Time and time again, we patiently explained to them that they had commissioned a proof-of-concept, not even a prototype, and that the each subsequent one would cost the same to build as the first. We offered to put more engineering into the product to make it more polished and manufacturable, for a cost of course, but each time they declined. Then, a couple of months later, they would contact us again about details on how to mass-produce the proof-of-concept.
Development Support for TESLA 3000 MPB
The TESLA 3000 MPB was developed for NxtPhase by Fidus in Ottawa. It went fairly well, until the prototype board would just not recognize the Ethernet NIC, at all. Michael Miller asked Elecsys to investigate.
I insisted that we boot LINUX on the board. Fidus advised that it was not feasible to do so. I persisted, and after about a week’s work (customizing the boot parameters and hacking the kernel a bit), I had it booting to LINUX with a small laptop hard drive. Sure enough, the Ethernet NIC would not come up, even though it was fully supported by the installed LINUX driver.
By instrumenting the driver with printk() and observing the result of loading and unloading the driver in the system log, I determined that it was a short between two lines on the PCI bus to the NIC. When Fidus took a close look at the Gerber files, they discovered that the design rules were too tight on the PCI bus on a deep inside layer of the PCB. The PCB was corrected and remade, and the NIC worked.
Later, there was an issue with the operation speed of the MPB core. Fidus was doing the FPGA development in VHDL on Synopsys, but did not have the resources to continue. I took over the development under Synopsys, having to establish VPN to Fidus to obtain the license for it to run. I suspected that the Synopsys tools, as good as they were, could be responsible for my inability to get appropriate performance from the FPGA. I ported the VHDL out from Synopsys back to Xilinx ISE, a byproduct of which was that we could do full development stand-alone in Winnipeg, which was nice.
With a few tweaks, I was able to get the MPB FPGA to simulate out at better-than-required full speed operation, which was proved out in actual operation. I mapped a path to higher speed operation, anticipated as a requirement for future product development.
Developed TESLA 3000 Analog Input Board (AIB)
The TESLA 3000 was designed to put the analog-to-digital conversion as close as possible to the rear panel, so the AIB had 6 x TI ADS8364 6 channel 16 bit SAR ADCs on it, with the associated front end and conditioning circuitry. A Xilinx Spartan 3 FPGA directed the acquisition timing, collected the ADC data, and communicated with the TESLA 3000 MPB using a bidirectional high speed LVDS synchronous serial link.
I did the architecture and design of the TESLA 3000 AIB, and oversaw the PCB layout by an external contractor. I specified the operation of the FPGA, how it interfaced to the MPB, and how it was programmed for operation.
The FPGA firmware was developed in Verilog on Xilinx ISE by a contractor at Fidus in Ottawa. When that FPGA developer was unable to complete the job, I took over and finished the project.
Developed the PDR (Power Disturbance Recorder) into the DSR (Dynamic Swing Recorder)
Vansco was an important partner of the IAMC for the development of a new product for Manitoba Hydro – the Power Disturbance Recorder (PDR). This was an innovative system that had a Central Station and multiple Remote Stations, connected by modems. In practice, we had Remote Stations be as far as 1500 km (2400 mi) away, way up north on the Nelson River power system. The Remote Station would detect power disturbances and contact the Central Station to alert it of the occurrence. The Central Station would then connect all other Remote Stations and instruct them to capture a time-synchronized snapshot around the event.
The key concepts here were time synchronization and snapshot retrieval. The Remote Station processors had enough storage to keep very long buffers of data – so even if the disturbance disrupted communications for a time, events would still be captured, network-wide. The communications protocol between the Master and Remote Stations always had an initial multiple-pass time-synchronization exchange to resolve the clock offsets between the stations, without external synchronizing source. Of course, now time sync over wide area is easy, due to ubiquitous satellite navigation & timing systems, but at this time, the PDR implementation was innovative and worked well.
After IAMC closed down, ongoing PDR development was transferred to Vansco. Later, former IAMC developer Michael Miller joined our team and was instrumental in its ongoing development.
The PDR was renamed the Dynamic Swing Recorder (DSR) by our Manitoba Hydro team. That team, especially Dave Fedirchuk, felt that more utilities could use this technology. We tried to market it more widely, but later decided that a newer approach was required – creating a more generalized fault recorder, similar to the leaders at the time, the Rochester Instruments or Mehta Tech recorders. This is where the idea of the TESLA 2000 started to take hold.
Developed the DR2/TESLA Main Processor Board (MPB) and Other DR2 & TESLA Boards
I used to say, “those aren’t electrons flowing on the MPB, those are my blood cells!” In a philosophical sense, it was true – the development of the MPB drained me like no other project effort, before or since.
With multiple processors on board – 2 x PIC 16C MCUs (one for front panel keyboard, one for watchdog processor), a TI TMS320C32 DSP, an Altera Flex 10K FPGA, an Altera MAX 7000 CPLD, 3 x National Semiconductor LM12458 microsequenced ADCs, and a Motorola MC68HC11 MCU for IRIG processing, 3 x NS16550 FIFO UARTs, and a 16 bit PC-104 bus connection, it was quite busy. An X86-based PC-104 card was mounted on top of the MPB – lower powered on the DR2, but much more powerful model on the TESLA.
The concept was that sifting, sorting, compression, and decision making was done on the DSP, where memory was expensive but calculations were easy. The resultant data was streamed through a high-speed DSP-to-X86 FIFO, where the X86 DMA’d it into main memory and stored it to disk. The X86 was used because its memory was plentiful and inexpensive, and disk I/O was fast and easy.
The other boards in the system weren’t as complex, but were just as important. The DR2 AIB, mounted on the bottom of the chassis, had up to 18 miniature PCB-mount instrumentation transformers on it – CTs or PTs, depending on the application – and analog processing to transfer to the ADCs on the MPB. The TESLA AIB was mounted on the rear of the chassis, had more channels but was less sophisticated – non isolated +/- 2.5V (nominal) input. The DR2 and TESLA each had their own interpretation of digital I/O – DR2 with high current relay outputs, and dedicated ultra reliable digital inputs, and the TESLA with fewer and lower level outputs and simpler digital inputs.
The front panel of the DR2 had a 2 line x 24 character FIP display, a small keyboard, and several indicator LEDs, including a large bright red “Target” LED, as is expected on a relay. The TESLA front panel had the smaller LEDs only.
I established the bus structure and pin-outs for all the I/O. The front panel and digital I/O were intentionally designed to be interchangeable, with the intention of future mixed mode configuration, or expansion. Although some of the buses had extra connections (front panel serial port, for instance), this was kept to the ends of the connectors, so combining and splitting ribbon cables would be relatively straightforward. This was very well documented in extensive spreadsheets, which clearly showed connections and levels.
The memory and I/O maps of the processors was also documented in spreadsheets, with well defined bit states and clearly defined defaults.
Later, I wrote an exhaustive 70+ page programming guide to the MPB, which outlined all its features, how each processor interacted, the resources used, assumptions in programming, and how to compose images for execution.
Worked on an IRIG-B Decoder
Manitoba Hydro wanted an IRIG-B decoder unit. It would listen to modulated IRIG-B, decode the time, and provide details of the time on a serial port. Filipe Fernandes and I did the system architecture. Filipe designed the board, based on a Motorola MC68HC11 development kit, and did the programming in assembly language.
The project was a success, but he was moved off to another project just as it was finishing, so I completed the programming. The design was a success.
This was my first exposure to IRIG-B, and I was fascinated.
Developed the Conviron Integrated Logger Controller (ILC) Card
The CMP3000 has a very sophisticated network operating over isolated RS-422 signals in a ring-bus arrangement. Each controller has a bypass relay which would heal the ring-bus in the event of controller or communication board failure. The Zilog Z8530 Serial Communications Controller (SCC) chip was used to talk HDLC on the ring-bus.
The master of the network was the CMP3300 Data Logger Processor (DLP), which was essentially a hot-rodded CMP3000 controller with different programming. It could manage and control up to 96 CMP3000 controllers on its loop.
To enhance and extend its function, the DLP had a serial port which could be connected to the Host Computer, an IBM PC running QNX. The Host computer could be connected to 3 DLPs, allowing it to manage many more controllers.
Being a general purpose PC running QNX, the Host Computer was able to schedule changes, log all kinds of data, display graphs, and export data to other media (think: floppy disks!).
There came a time when the DLP just seemed clunky. So, I was asked to create the Loop Control Card (LCC), which would mount directly into the IBM PC chassis and replace the DLP. It would be able to control 3 loops of controllers, and have embedded on-board alarm contacts.
By then, the PC of choice was the AT, and Vansco actually was making its own labelled computers, based on components purchased overseas. However, the LCC was designed for the XT bus for maximum compatibility.
The heart of the board was a NEC V25 processor, high-speed (well, for the time) SRAM, and a few PLDs, where I implemented a pseudo-dual-port RAM for control and data transfer. Internal bus faults occurred periodically, which proved almost impossible to track down without the right test equipment. Finally, management went and bought a very expensive (at he time) Fluke/Phillips PM3585/90 logic analyzer, which I proceeded to load up to the maximum, using its dual timing & state analysis to great advantage. In the end, the V25 was every once in a while stealing an extra clock cycle for its prefetch queue, causing a bus timeout. The PC bus specification wouldn’t let me stretch it any further. I was able to rework the logic and make it all come together. You gotta have the right tools!
