More CAN filtering and next ideas...

In the last post I mentioned some warnings and errors appearing in the dashboard due to my hacking of message 0x167. Because the CEM now thinks that the engine is running, it validates some other messages coming from the ECM. Today I found some time to tackle the issue - and it took only 2 hours to not only fix the problem but enhance the situation even more :)
I found out that the status of the alternator and probably even some information about the cooling system (not the temperature itself) is sent via message 0x3d3. I replace the content of this message with values from the other S80: 0x24, 0x42, 0x00, 0x04, 0x75, 0x78, 0x76, 0x08 and after clearing the DTC's, the Dashboard is happy - as will be the experts conducting the regular checks for street worthiness. I didn't bother yet to find out exactly which bytes are relevant.
As I was already at it, I thought somewhere the RPM must be hidden too. In my previous hacks, I only focused on the climate controls and error messages in the dash. I never checked the tachometer. So I checked the logs in Excel again to find messages from the ECM which differ between "ignition on" and "engine idle". In "engine idle" they case must fluctuate a bit because the engine speed is never 100% constant. As Murphy's law requires, I started on the wrong side with the high numbered message id's until I hit message 0x12a and saw the needle jump up to 700rpm. Bingo! The engine speed is encoded in the last two bytes - well, actually only the last 13 bits. As byte 7 already had the value 0xe0 with ignition on (in my car), I simply add the two values. So that's what it looks like now (mind that the array starts at [0] - hence [7] is byte 8:

bytes[6] = 0xe0 | (rpm & 0xff00) >> 8;
bytes[7] = rpm & 0xff;

To transport the rpm of the electric motor over to the CAN filter, I use GEVCU's CAN bus which is connected to the car's high-speed bus and send the value in a new message id 0x129. The filter picks up the value when message 0x129 is received. Then, when message 0x12a is received, it injects the value. Works perfectly fine and now I have a working tachometer - well, except that the instrument doesn't display rpm's between 0 and 500 - it stays at 0 and then jumps to 500 - but what the heck, I can live with that.
The code for the filter can be found on github:

Next ideas

As I don't have a clutch anymore, shifting has become a bit tricky. Gears can only be shifted with zero torque and the synchros are used to adjust the motor speed. This shortens the life of the synchros and is also difficult for drivers unfamiliar with the procedure. Even I get into trouble now and then when the GEVCU dash isn't working and I don't find the spot where I'm coasting (zero torque). So the next step is to grab the vehicle's speed from the CAN bus (not via OBD2, too slow) and from the ratio of engine vs. vehicle speed find out in which gear I'm driving. Then I'll add a capacitive switch to the gear shifter and change the code of GEVCU like follows:

1. When the switch is activated (gear knob touched), the torque is reduced to 0 for 1 sec (or less).
2. With 0 torque, the driver can switch the gear to "neutral".
3. Depending on wether the throttle is in "acceleration" or "deceleration" mode, GEVCU will adjust the motor speed within 0.2-0.5sec so it fits the next lower or higher gear.
4. Torque is again reduced to 0 (or almost 0) so the gear can be inserted.
5. When the knob/switch is released, normal operation is resumed.

I just made up the timing as I have no experience yet on how long the steps will take. I'll have to experiment with the timing as well as the torque applied to adjust the speed quickly enough but without unnecessarily stressing the hardware.
And of course I won't be entering the speed ratios for the different gears.. the GEVCU has to be able to find the different ratios automatically.

0x167, byte 5, bit 6

A couple of months ago I found the CAN message on the low-speed CAN bus which would be sent from the CEM (Central Electronic Module) to the CCM (Climate Control Module) if the engine was running. It would enable all the high-power consumers like seat heater, seat ventilator, rear window de-mist. The only caveat was that while I injected the message, the CEM sent the same one about 2-5 per second to switch things off again. I thought about cutting the CAN wires and inserting a filter. But as the SRS (airbag) is also on that bus, I was hesitant - luckily. Because one week ago, I came up with the idea, why not do the same thing between the ECM (Engine Control Module) and the CEM. Instead of faking input signals to make the ECM believe that an engine is running, I could do it at a later stage: "adjust" the CAN signal from the ECM.

So yesterday I dared and started looking for the high-speed CAN wires. I was lucky. I cut the umbilical of the cable tree open - close to the coolant expansion tank. The twisted blue-black (CAN-High) and green-black (CAN-Lo) wires were right on the top - no digging through 30-40 wires. A courageous cut and a Arduino-Due based CAN shield soldered in and I had a filter ready. ECM on bus 1, rest on bus 0. This way it was a piece of cake to find out which messages are coming from the ECM (0x30, 0x125, 0x12a, 0x143, 0x157, 0x167, 0x315, 0x385, 0x392, 0x3d3, 0x4ca) and which are coming from the other devices (at least 23 other messages). This reduced the reverse-engineering efforts dramatically by about 70%. First I suspected it'd be message id 0x30. Usually these low-end numbers are used for basic status information. Wrong! It seems to be the device's ID - which changes when the engine is running - but yet it wasn't the one yet. So, what did I do?

Another friendly S80 MY08 owner, allowed me to grab some logs from his car with ignition on and with engine idling. With these logs, Excel and Collin Kidders SavyCan, I identified which messages contain different data between the two states. Still a lot! So I started replacing the content of messages one message id at a time, first I used the other car's data for 0x30 --> de-mobilizer got activated. Then 0x125 --> DSTC system failure, then 0x12a --> DSTC error and de-mobilizer, and so on and so on. Most of the other messages had no visible effect.

Then once I uploaded a new firmware change to the Arduino, I got partially lucky: I forgot to switch off ignition and during the upload, the CEM obviously lost communication with the ECM and the whole christmas tree lit up: all seat heaters, fans, BLIS (Blind Spot Indicator System) were enabled for a short time. So today, I just pulled the plug of the Arduino - meaning no messages get forwarded anymore. And although many errors appeared on the dashboard, all the desired devices were available to me. Geeee! This looked promising. So assiduously I started filtering out entire messages - again starting with 0x30 and working my way up. With the sixth attempt, I got lucky: 0x167 must be it! Then I added it again but with the contents from the other S80 and the devices still worked. Next step: only replace the contents of the last 4 bytes and leave the first 4 as they come from the ECM: still ok. Then leave the content of byte 5 and 6 unchanged: not working anymore. With the reference data from the other S80 for byte 5 being 0x22 with ignition on and 0x62 with running engine it became clear quickly that it must be bit 6. So now I just do a bytes[4] |= 0x40 to set the bit and Vida now tells me that the ECM's "power state" is no longer "ignition on" but "running". And now instead of just the working seat heater (most important!) and ventilator and de-mist, I also got a running BLIS and the head-lamps swivel again in the direction I steer. Fantastic! :)

There are only 3 issues which need to be resolved now as the engine appears to be running, more signals are verified:

1. The red "alert" and the battery warning sings light up in the dashboard together with the message that the alternator system has a serious malfunction. The DTC tells me there's a missing message. 
2. Because I didn't attach a coolant temperature sensor or fake the signal, I get a yellow "warning" sing and a message about an issue in the engine coolant system
3. The auxiliary brake pressure pump still doesn't come on when braking abruptly. For a real emergency stop, I have to push the brake very hard until ABS kicks in.
4. Still don't get a RPM reading on the dashboard.

In a daily drive 1. is the most annoying thing. But as the DTC says a message is missing, maybe the ECM just sends one more message containing data from the alternator. Once I find it and inject it, it should go away. Point 2 should be easy to remedy - and even if I can't I could live with it. 

Point 3 is not annoying while driving but it's the most critical in an emergency - especially if other people are driving the car. I don't know what needs to be done to make the BCM (Brake Control Module) switch on the auxiliary pump. Maybe I'll just use the brake pressure which is read in by GEVCU anyway and then send the CAN bus command to turn on/off the auxiliary pump (can grab this from the Vida log as it can be switched on/off from the tool).

Point 4 just needs a bit more fumbling around with the remaining CAN messages. I'm convinced the RPM of the engine is in one of the 11 messages sent from the ECM.

If anyone has information or a good idea on how to resolve these 4 points, I'd appreciate it!

Months of Hell

This post is not in chronological order as I delayed writing it for several months now. As the title suggests, it wasn't the most pleasurable time of my conversion and I "evaded" recapturing the events for a while. But now it's about time to write about these events - and with some distance it's even funny to recap what had occurred (see point 3). Maybe it helps you if you're also stuck in apparently unsolvable problems or if you're not able to track down the source.

As in Voyager's Year of Hell the S80 got some serious problems and just wouldn't want to make its way to the certification agency for 2-3 minor checks. Maybe you remember, in May 2016 I've received a provisional allowance to drive the car. I had to move some weight from the front to the back, correct the wheel angles and get rid of some rust on the brake disks. Nothing serious you might think. Not at all.. everything was taken care of and fixed pretty easily. But in the meantime this has occurred:

1. The bearing of the motor shaft gave in. While driving the motor started vibrating pretty strong. It was quickly identified because the vibrations were related to the motor speed and not the car's speed. So back on the lift, disassemble the motor from the gearbox and confirm that the motor was vibrating. As I already pointed out to Brusa a strange heat build-up at the bearing before I mounted the motor the first time, they were 100% co-operative and handed me a replacement motor.

2. Thanks to the replacement of the motor, I also discovered  a couple of other things (see pics on the right):
• The bolts of the gearbox and adapter plate started to loosen - although I applied Loctite to all of them and torqued them according to specs. When reassembling, I switched to Nordlock washers - I strongly suggest you'd do the same. 
• Burnt grease and pitting of the metal on the adapter shaft: According to experts this points to very high temperatures occurring locally due to quick torque shifts (which in itself is normal). We try to prevent this now by using very high temperature resistant graphit grease.
• Wear and tear at the beginning and the end of the part that's inserted into the motor. This points to a slight misalignment - maybe due to the loosened bolts or due to the construction. But two experts told me that it should be fine now and I don't have to expect the situation to worsen.
• Chipping at the edge of the splines. Probably also due to wear and tear but I had to file it off before reassembly.
3. Once I got everything running vibration free again, I arranged a date with the certification agency. Of course I visited the car wash just before the appointment to make a good impression. And because the car was to be weighted again and I needed every kg, I wanted to get rid of some excess washing water (which equals to about 5kg when full). So I applied the windshield washer thoroughly while driving to the agency on the high-way. Then things happened quickly: I got a warning bell, several unrelated alerts on the dash (like "brake, urgent maintenance required", "ABS inactive", "low voltage", "power saving mode") and a couple of seconds later: No response on the throttle and the car started to slow down. WHAAAT !?! Not now! I moved to the emergency line and came to a stop. The lights in the dash and radio display flickered when I tried to re-start the car. Lights were dim and the dash didn't even indicate that the emergency lights were operating. After working 30min under the hood with my MacBook (ever seen this ?) and cars rushing by at 100kmh, I came up with two theories: Either some water from the car wash must have made its way where it caused a short or the DC-DC converter stopped working. Also the police paid me a visit which at first made me even more nervous but they were really forthcoming. Long story made short, I needed a tow. But again, to save weight, I left the tow hook at home (it weighs another 2kg) and the tow truck didn't have one for Volvo's with him. Of course after spending 1h on the emergency line, I had to cancel the appointment and my phone's battery was almost empty too. How much worse could it get ?!? Remembering my theories, I gambled on one and asked the guy from the emergency service if he had a 12V emergency starter pack with him. Luckily he had, we attached it to my tiny motorcycle lead acid battery and tadaaah, the car came back to life. Another quick look at GEVCU's (the EV control unit) log confirmed it: The DC-DC converter reported an high-voltage under-voltage error and was inoperative. As this couldn't be fixed quickly, we agreed to get off the high-way first, closed the hood as much as possible and drove to the next exit. There we located the next DIY store where I wanted to buy a 12V battery or emergency starter pack. As the motorcycle batteries were much more expensive and probably not fully charged, I decided to buy a starter pack. Hooked up, the car was driving again. To save power, I disconnected the electric power steering pump, switched off all possible consumers and dared to drive back on the high-way. Several km's later, the car started reporting low voltage and various system problem again. I skipped one more exit and took the next one to get as close to home as possible. I used the emergency foot brake to brake to prevent the vacuum pump come to live and rob another couple of mAh on the 12V rail. In a city about 10km from home I had to enter a round-about. Calculated the speed and trajectory of all cars to make it in without braking. But one of the "suckers" didn't leave it as I expected and I had to brake sharp. Hearing the vacuum pump come to life weakly churned at my heart. A quick push on the throttle confirmed: I needed to push differently to get out of the round-about. So I got out of the car and pushed it to the next exit. There was a construction site and they were removing the scaffolding. Luckily I wasn't getting in their way and there was also mains power so I was able to re-charge the starter pack. After an hour, I thought the charge should suffice for the remaining 10min drive and got on the way again. But guess what, not even half way through, the bells came on again and I knew, I was about to become an obstacle to traffic a third time in a row. Another round-about in a small town. A pedestrian on the way to the zebra crossing. You have to know that in Switzerland pedestrians have way right at zebra crossings - cars and trucks have to stop if there's the slightest chance that people want to cross or risk a several hundred CHF fine. So I was under quite some pressure and realising the bad timing, I was going "no no no!", frantically waving at the pedestrian to stay put on the sidewalk and in the end passed by her in a still save but let's say "probably not entirely legal" fashion as she returned the waving arms gesture when I looked in the mirror. I felt sorry as normally I'd never do that. But thanks to this manoeuvre, the car was working until the last incline. There it gave out but I had enough momentum to make it to the "down-hill" part where I made it home without motor support. Another visit to the lift and I found out that the 10A HV fuse for the DC-DC converter was blown. I replaced it with a 8A I had lying around but it blew up again immediately. There was a short on the HV side of the converter and it had to be repaired.
4. To be able to drive without DC-DC converter, I used the car's old 100Ah lead acid battery. Fully charged, it gave me about a 2h drive. These drives were not relaxed ones as the voltage sagged to 11.5V quickly too and the alerts popped up again. But I was able to drive much longer with them in place. But while driving I felt another vibration getting worse and worse around 3300rpm and I thought I'd have to replace the motor another time and maybe something with my construction was completely wrong and destroying bearings. As it turned out much later, the vibrations subsided again once I disabled "oscillation dampening". In low speed situations this mode resulted in heavy fluctuations in torque and maybe wasn't good for the drive-train. But this part caused me a whole lot of head-ache and frustration. So much that together with the DC-DC converter problem, I decided to stow the car away for a couple of weeks. Mentally this was the absolute low point of my conversion. Ev1en the thought of re-inserting the gas engine had crossed my mind. In these situations it's good to have a family to rely on and put things into perspective again. My 9-year old son realised that I was feeling really sad and made me the drawing which says "Dear dad, I'm so proud of you. Kind regards, Philipp" and on the right he adapted the Volvo sign with "1.P" for first place. How much sweeter can you be? No need to tell you that it did wet up more than one eye :)
   


5. And once I thought things were running again, one morning the car would move. This time the motor controller gave me an error: "Temperature Sensor" - although the reported motor temperature was perfectly fine. Checking with Brusa it became evident that there are several sensors and if the ones used to report the temp over CAN appear to be fine, there must be something wrong with the others whose purpose is to shut-down the system in case the motor overheats. As they are redundant for safety and the system also shuts-down if the others report high temperatures, we decided to disable the second safety feature. Luckily previous versions of the motor didn't have this set of sensors and the software is flexible enough so we were able to disable the check through parametrisation. But the whole process also stretched over several weeks.
6. As it took much longer than expected, the re-check at the certification agency was more than over-due. This also put a lot of pressure. I can count myself really lucky that they were so lenient and showed a lot of understanding for my situation.

In December we finally made it to the agency and the car passed final inspection. Since then the only troubles I had were related to heating. Otherwise the car is driving well now (as long as I don't screw up the software by adding new features). So I'm happy again in this area.. and also in others as I find much more time and energy to spend with my family again.

GEVCU Dashboard redesign

The old version of the web based GEVCU dashboard is a bit overloaded with gauges and its look&feel is not of the most modern types. So after seeing some nice gauges in a mobile app presented on evtv, I decided ours needs an overhaul. These were the requirements:

• support dual dials on one gauge (or more)
• support arcs from a start point to the tip of the dial
• improved performance

The original gauge library was created by Mykhailo Stadnyk and it served us well. But we had to switch off animations (interpolations from one value to the next) in order to run it on tablets. This made the dials jump around a bit. One of the main reasons for this was that for every refresh the entire gauge was re-drawn from scratch. So a first change was to introduce layers. The gauge background (circle, numbers, ticks) are now only drawn once and the dials are each drawn on their own layer (canvas stacked above each other). Also the LCD like value displays are drawn onto a separate layer.

But still performance was pretty bad.. on a Macbook Pro, Firefox put the CPU load to 80-90% in a test set-up where all values were redrawn every 10ms. Increasing the interval to 35ms still gives decent animations and dropped to CPU load to about 60%. Better but still not good enough for my Samsung Note 10.1" 2014 tablet. The problem was that JavaScript runs single-threaded. So although the tablet uses a multi-core processor, it only used one to do everything. The solution was to introduce web-workers. They allow to perform background tasks in separate threads. By moving the web-socket communication and calculations to a separate worker and let the main thread do the drawing, we got a huge performance jump. The CPU load dropped to about 45% and the animations started to look better on the tablet. But I wasn't happy yet. Since the specs of web-workers say that a worker can spawn sub-workers, I wanted to go further and not just distribute the work between 2 threads (2 cores). So I decided to spawn a "handler" worker which acts as a coordinator. This handler then spawns a web-socket worker which processes the incoming data into a JSON object. And it also spawns a worker for each dial to be displayed in all the gauges. Each key/value pair received from the web-socket is then forwarded via the handler to the corresponding dial-worker. The result of the dial-worker is forwarded through the handler back to the main thread where the canvas is drawn with the pre-calculated values. This approach would take away the maximum amount of load from the main thread which is responsible for the DOM and drawing the canvas.

But it didn't work out that well. Well, it worked - but only in Firefox. As it turns out, Firefox and MS Edge are the only browsers at that time to implement the full specification of workers. The others (like Opera, Chrome, Android Browser) do only support workers to be created by the main thread. So  in order not to loose the work, I decided to put the handler into a PseudoWorker wrapper. It will still run in the main thread but can be accessed almost like a worker. So in case the other browser implementations catch-up, it's easy to switch.

The arcs was another thing.. to find out how to place the start points correctly was not so easy. Especially as I wanted to be able to have the gauges display one, two, three or more dials. And also be able to "turn around" a dial and not display the values clock-wise but counter-clock-wise with a switch. And of course the arcs also needed to support freely definable gradients. 

All in all, it worked out and now the performance on the tables is good. The dashboard even runs sufficiently well on a raspberry Pi3 with all values changing with every update (which is not realistic). But you'll have to use Android as OS and either Chrome or Opera. Firefox is too slow as are all browsers available on Raspbian. 

Here's a demo video (with faked values) of the new dashboard in action. You might notice the instpirations from Tesla and Volvo gauges but still, they're something different. Oh yes and the charging screen also received an overhaul. So although future versions of GEVU (6.x) will no longer support the ConnectOne ichip wifi, there are still a lot of us out there who got older versions.

Also notice the switches at the top. With them you can temporarely switch on/off regen, creep, EHPS and the heater. Especially on snowy roads it's nice to switch off regen without having to change the configuration of the throttle.

And here's a demo which displays all the possibilities of the new gauges (4 dials in one gauge and also display the highlights as in the old version):