Sonntag, 19. März 2017

Months of Hell

This post is not in chronological order as I delayed writing it for several months now. As the title suggests, it wasn't the most pleasurable time of my conversion and I "evaded" recapturing the events for a while. But now it's about time to write about these events - and with some distance it's even funny to recap what had occurred (see point 3). Maybe it helps you if you're also stuck in apparently unsolvable problems or if you're not able to track down the source.

As in Voyager's Year of Hell the S80 got some serious problems and just wouldn't want to make its way to the certification agency for 2-3 minor checks. Maybe you remember, in May 2016 I've received a provisional allowance to drive the car. I had to move some weight from the front to the back, correct the wheel angles and get rid of some rust on the brake disks. Nothing serious you might think. Not at all.. everything was taken care of and fixed pretty easily. But in the meantime this has occurred:
  1. The bearing of the motor shaft gave in. While driving the motor started vibrating pretty strong. It was quickly identified because the vibrations were related to the motor speed and not the car's speed. So back on the lift, disassemble the motor from the gearbox and confirm that the motor was vibrating. As I already pointed out to Brusa a strange heat build-up at the bearing before I mounted the motor the first time, they were 100% co-operative and handed me a replacement motor.
  2. Thanks to the replacement of the motor, I also discovered  a couple of other things (see pics on the right):
    • The bolts of the gearbox and adapter plate started to loosen - although I applied Loctite to all of them and torqued them according to specs. When reassembling, I switched to Nordlock washers - I strongly suggest you'd do the same. 
    • Burnt grease and pitting of the metal on the adapter shaft: According to experts this points to very high temperatures occurring locally due to quick torque shifts (which in itself is normal). We try to prevent this now by using very high temperature resistant graphit grease.
    • Wear and tear at the beginning and the end of the part that's inserted into the motor. This points to a slight misalignment - maybe due to the loosened bolts or due to the construction. But two experts told me that it should be fine now and I don't have to expect the situation to worsen.
    • Chipping at the edge of the splines. Probably also due to wear and tear but I had to file it off before reassembly.
  3. Once I got everything running vibration free again, I arranged a date with the certification agency. Of course I visited the car wash just before the appointment to make a good impression. And because the car was to be weighted again and I needed every kg, I wanted to get rid of some excess washing water (which equals to about 5kg when full). So I applied the windshield washer thoroughly while driving to the agency on the high-way. Then things happened quickly: I got a warning bell, several unrelated alerts on the dash (like "brake, urgent maintenance required", "ABS inactive", "low voltage", "power saving mode") and a couple of seconds later: No response on the throttle and the car started to slow down. WHAAAT !?! Not now! I moved to the emergency line and came to a stop. The lights in the dash and radio display flickered when I tried to re-start the car. Lights were dim and the dash didn't even indicate that the emergency lights were operating. After working 30min under the hood with my MacBook (ever seen this ?) and cars rushing by at 100kmh, I came up with two theories: Either some water from the car wash must have made its way where it caused a short or the DC-DC converter stopped working. Also the police paid me a visit which at first made me even more nervous but they were really forthcoming. Long story made short, I needed a tow. But again, to save weight, I left the tow hook at home (it weighs another 2kg) and the tow truck didn't have one for Volvo's with him. Of course after spending 1h on the emergency line, I had to cancel the appointment and my phone's battery was almost empty too. How much worse could it get ?!? Remembering my theories, I gambled on one and asked the guy from the emergency service if he had a 12V emergency starter pack with him. Luckily he had, we attached it to my tiny motorcycle lead acid battery and tadaaah, the car came back to life. Another quick look at GEVCU's (the EV control unit) log confirmed it: The DC-DC converter reported an high-voltage under-voltage error and was inoperative. As this couldn't be fixed quickly, we agreed to get off the high-way first, closed the hood as much as possible and drove to the next exit. There we located the next DIY store where I wanted to buy a 12V battery or emergency starter pack. As the motorcycle batteries were much more expensive and probably not fully charged, I decided to buy a starter pack. Hooked up, the car was driving again. To save power, I disconnected the electric power steering pump, switched off all possible consumers and dared to drive back on the high-way. Several km's later, the car started reporting low voltage and various system problem again. I skipped one more exit and took the next one to get as close to home as possible. I used the emergency foot brake to brake to prevent the vacuum pump come to live and rob another couple of mAh on the 12V rail. In a city about 10km from home I had to enter a round-about. Calculated the speed and trajectory of all cars to make it in without braking. But one of the "suckers" didn't leave it as I expected and I had to brake sharp. Hearing the vacuum pump come to life weakly churned at my heart. A quick push on the throttle confirmed: I needed to push differently to get out of the round-about. So I got out of the car and pushed it to the next exit. There was a construction site and they were removing the scaffolding. Luckily I wasn't getting in their way and there was also mains power so I was able to re-charge the starter pack. After an hour, I thought the charge should suffice for the remaining 10min drive and got on the way again. But guess what, not even half way through, the bells came on again and I knew, I was about to become an obstacle to traffic a third time in a row. Another round-about in a small town. A pedestrian on the way to the zebra crossing. You have to know that in Switzerland pedestrians have way right at zebra crossings - cars and trucks have to stop if there's the slightest chance that people want to cross or risk a several hundred CHF fine. So I was under quite some pressure and realising the bad timing, I was going "no no no!", frantically waving at the pedestrian to stay put on the sidewalk and in the end passed by her in a still save but let's say "probably not entirely legal" fashion as she returned the waving arms gesture when I looked in the mirror. I felt sorry as normally I'd never do that. But thanks to this manoeuvre, the car was working until the last incline. There it gave out but I had enough momentum to make it to the "down-hill" part where I made it home without motor support. Another visit to the lift and I found out that the 10A HV fuse for the DC-DC converter was blown. I replaced it with a 8A I had lying around but it blew up again immediately. There was a short on the HV side of the converter and it had to be repaired.
  4. To be able to drive without DC-DC converter, I used the car's old 100Ah lead acid battery. Fully charged, it gave me about a 2h drive. These drives were not relaxed ones as the voltage sagged to 11.5V quickly too and the alerts popped up again. But I was able to drive much longer with them in place. But while driving I felt another vibration getting worse and worse around 3300rpm and I thought I'd have to replace the motor another time and maybe something with my construction was completely wrong and destroying bearings. As it turned out much later, the vibrations subsided again once I disabled "oscillation dampening". In low speed situations this mode resulted in heavy fluctuations in torque and maybe wasn't good for the drive-train. But this part caused me a whole lot of head-ache and frustration. So much that together with the DC-DC converter problem, I decided to stow the car away for a couple of weeks. Mentally this was the absolute low point of my conversion. Ev1en the thought of re-inserting the gas engine had crossed my mind. In these situations it's good to have a family to rely on and put things into perspective again. My 9-year old son realised that I was feeling really sad and made me the drawing which says "Dear dad, I'm so proud of you. Kind regards, Philipp" and on the right he adapted the Volvo sign with "1.P" for first place. How much sweeter can you be? No need to tell you that it did wet up more than one eye :)

  5. And once I thought things were running again, one morning the car would move. This time the motor controller gave me an error: "Temperature Sensor" - although the reported motor temperature was perfectly fine. Checking with Brusa it became evident that there are several sensors and if the ones used to report the temp over CAN appear to be fine, there must be something wrong with the others whose purpose is to shut-down the system in case the motor overheats. As they are redundant for safety and the system also shuts-down if the others report high temperatures, we decided to disable the second safety feature. Luckily previous versions of the motor didn't have this set of sensors and the software is flexible enough so we were able to disable the check through parametrisation. But the whole process also stretched over several weeks.
  6. As it took much longer than expected, the re-check at the certification agency was more than over-due. This also put a lot of pressure. I can count myself really lucky that they were so lenient and showed a lot of understanding for my situation.
In December we finally made it to the agency and the car passed final inspection. Since then the only troubles I had were related to heating. Otherwise the car is driving well now (as long as I don't screw up the software by adding new features). So I'm happy again in this area.. and also in others as I find much more time and energy to spend with my family again.

Freitag, 6. Januar 2017

GEVCU Dashboard redesign

The old version of the web based GEVCU dashboard is a bit overloaded with gauges and its look&feel is not of the most modern types. So after seeing some nice gauges in a mobile app presented on evtv, I decided ours needs an overhaul. These were the requirements:

  • support dual dials on one gauge (or more)
  • support arcs from a start point to the tip of the dial
  • improved performance
The original gauge library was created by Mykhailo Stadnyk and it served us well. But we had to switch off animations (interpolations from one value to the next) in order to run it on tablets. This made the dials jump around a bit. One of the main reasons for this was that for every refresh the entire gauge was re-drawn from scratch. So a first change was to introduce layers. The gauge background (circle, numbers, ticks) are now only drawn once and the dials are each drawn on their own layer (canvas stacked above each other). Also the LCD like value displays are drawn onto a separate layer.
But still performance was pretty bad.. on a Macbook Pro, Firefox put the CPU load to 80-90% in a test set-up where all values were redrawn every 10ms. Increasing the interval to 35ms still gives decent animations and dropped to CPU load to about 60%. Better but still not good enough for my Samsung Note 10.1" 2014 tablet. The problem was that JavaScript runs single-threaded. So although the tablet uses a multi-core processor, it only used one to do everything. The solution was to introduce web-workers. They allow to perform background tasks in separate threads. By moving the web-socket communication and calculations to a separate worker and let the main thread do the drawing, we got a huge performance jump. The CPU load dropped to about 45% and the animations started to look better on the tablet. But I wasn't happy yet. Since the specs of web-workers say that a worker can spawn sub-workers, I wanted to go further and not just distribute the work between 2 threads (2 cores). So I decided to spawn a "handler" worker which acts as a coordinator. This handler then spawns a web-socket worker which processes the incoming data into a JSON object. And it also spawns a worker for each dial to be displayed in all the gauges. Each key/value pair received from the web-socket is then forwarded via the handler to the corresponding dial-worker. The result of the dial-worker is forwarded through the handler back to the main thread where the canvas is drawn with the pre-calculated values. This approach would take away the maximum amount of load from the main thread which is responsible for the DOM and drawing the canvas.
But it didn't work out that well. Well, it worked - but only in Firefox. As it turns out, Firefox and MS Edge are the only browsers at that time to implement the full specification of workers. The others (like Opera, Chrome, Android Browser) do only support workers to be created by the main thread. So  in order not to loose the work, I decided to put the handler into a PseudoWorker wrapper. It will still run in the main thread but can be accessed almost like a worker. So in case the other browser implementations catch-up, it's easy to switch.
The arcs was another thing.. to find out how to place the start points correctly was not so easy. Especially as I wanted to be able to have the gauges display one, two, three or more dials. And also be able to "turn around" a dial and not display the values clock-wise but counter-clock-wise with a switch. And of course the arcs also needed to support freely definable gradients. 
All in all, it worked out and now the performance on the tables is good. The dashboard even runs sufficiently well on a raspberry Pi3 with all values changing with every update (which is not realistic). But you'll have to use Android as OS and either Chrome or Opera. Firefox is too slow as are all browsers available on Raspbian. 
Here's a demo video (with faked values) of the new dashboard in action. You might notice the instpirations from Tesla and Volvo gauges but still, they're something different. Oh yes and the charging screen also received an overhaul. So although future versions of GEVU (6.x) will no longer support the ConnectOne ichip wifi, there are still a lot of us out there who got older versions.
Also notice the switches at the top. With them you can temporarely switch on/off regen, creep, EHPS and the heater. Especially on snowy roads it's nice to switch off regen without having to change the configuration of the throttle.


And here's a demo which displays all the possibilities of the new gauges (4 dials in one gauge and also display the highlights as in the old version):