Throughout this series we've talked about how to read and write signals and data and so forth. But we haven't yet talked about formats of saving CAN data to a file. Since CAN data is generally a complicated data structure consisting of multiple data types, saving it to a file isn't always easy, and you may come up with something that works, but is non-standard and requires non-standard tools. In this series we will talk about reading and writing CAN data as TDMS files and BLF files, as well as talk about other common operations like reading HEX, and S record files, which will be important for later blog posts. This code and examples can be installed by installing the Hooovahh CAN File IO package, which will be referenced in this post, and is available on the downloads page for this blog series.
So how can CAN data be stored into a TDMS file? Well luckily NI came up with a standard that flattens raw CAN frames into an array of bytes, and can then read the data back turning it into frames. When a TDMS file uses this standard format for CAN data, some of NI's tools like DIAdem can read the TDMS file and recognize it has CAN data in it. Luckily NI provides some examples on how to go from CAN frames to a TDMS file and it just requires one channel within one group. The basic example can be found in the CAN Input Stream to TDMS Logfile.vi example, which gets installed with the XNet drivers.
Like many examples done by NI they only go so far, and are usually not the most efficient. That's why I've cleaned up their code, and added a few features like logging CAN IDs as a separate channel for filtering purposes. After installing the Hooovahh CAN File IO package, you'll have a subpalette devoted to reading and writing to a TDMS file. For writing first use the Configure TDMS channel for write, providing all the information you can. Then call the Write CAN frames to log to the TDMS file.
The result will be a TDMS file will has CAN frames data flattened, and written to the channel specified. Using my API there will also be a second channel in the TDMS file recording the CAN IDs of the frames logged. Using this we can do more efficient reads, finding only the frames that have a desired CAN ID. Just like the write process, use the Configure TDMS Channel for Read. You can also use the List CAN Channels to find what channels in a TDMS file contain CAN frames. Then perform a Read CAN Frames. The Read CAN Frames is polymorphic and allows for reading N frames from a given offset, reading frames that match a specific set of IDs, or reading a set of frames from a set of signals in a database or DBC file. The result can be a pretty effective method of reading CAN frames in a file given the information you know about what you want to read. Reading only a subset of the data at a time is also useful for when your log file is very large. Included in the package is an example showing how to read CAN data from a TDMS file, allowing for graphing of signals, given a time window, and IDs to read.
With both APIs sharing some of the same functionality, there are a couple of included examples that allow for converting from one format to the other. This is just a read from one API and a write to the next. This means that other software tools like CANalyzer can be used to process CAN data read by LabVIEW. This also means that CAN logs created by CANalyzer can be converted and then processed by software written in LabVIEW.
You'll notice that both TDMS and BLF data only read and write the raw frames of a CAN log. Things like frame to signal conversion still is done by the software after the read operation. Luckily tools like the ones I've already covered in Part 5 can perform the conversion for you.
If you've made it this far you might be asking yourself, "What in the world does this have to do with CAN?" Well my friend, the automotive CAN bus has been used for several years now to be able to reprogram ECUs in your vehicle. Your car already has a CAN bus for communication, and the OEMs just figured they could use that for reprogramming hardware in the car. If software in an ECU has a critical bug found, you can take your car to a dealership, and they will reprogram it. Of course if that ECU is buried behind the engine getting to it physically will be very difficult. So instead a technician can plug into a diagnostic port under the dash, and a tool can reprogram the ECU without having physical. There are two basic methods of performing what is called CAN Flash, which is flashing an ECU's firmware using CAN. We want to go over a few examples in the next section of CAN flashing, but only once we've understood how to parse and read the firmware, which are HEX and S Rec. can we hope to program the ECU. So you see it does relate to file I/O operations and CAN.
CAN TDMS File
The TDMS file itself is a very interesting topic, and one that I've presented on in the past at NI Week. It is best understood as a file format that is similar to Excel. You have a file (like a workbook) with multiple groups (like a worksheet) with channels (columns in a table) and samples in those channels (rows in a table). TDMS also has all kinds of interesting features like properties.So how can CAN data be stored into a TDMS file? Well luckily NI came up with a standard that flattens raw CAN frames into an array of bytes, and can then read the data back turning it into frames. When a TDMS file uses this standard format for CAN data, some of NI's tools like DIAdem can read the TDMS file and recognize it has CAN data in it. Luckily NI provides some examples on how to go from CAN frames to a TDMS file and it just requires one channel within one group. The basic example can be found in the CAN Input Stream to TDMS Logfile.vi example, which gets installed with the XNet drivers.
Like many examples done by NI they only go so far, and are usually not the most efficient. That's why I've cleaned up their code, and added a few features like logging CAN IDs as a separate channel for filtering purposes. After installing the Hooovahh CAN File IO package, you'll have a subpalette devoted to reading and writing to a TDMS file. For writing first use the Configure TDMS channel for write, providing all the information you can. Then call the Write CAN frames to log to the TDMS file.
The result will be a TDMS file will has CAN frames data flattened, and written to the channel specified. Using my API there will also be a second channel in the TDMS file recording the CAN IDs of the frames logged. Using this we can do more efficient reads, finding only the frames that have a desired CAN ID. Just like the write process, use the Configure TDMS Channel for Read. You can also use the List CAN Channels to find what channels in a TDMS file contain CAN frames. Then perform a Read CAN Frames. The Read CAN Frames is polymorphic and allows for reading N frames from a given offset, reading frames that match a specific set of IDs, or reading a set of frames from a set of signals in a database or DBC file. The result can be a pretty effective method of reading CAN frames in a file given the information you know about what you want to read. Reading only a subset of the data at a time is also useful for when your log file is very large. Included in the package is an example showing how to read CAN data from a TDMS file, allowing for graphing of signals, given a time window, and IDs to read.
CAN BLF File
NI has their standard for logging CAN data to a TDMS, but since TDMS files aren't as common outside of NI software, another option is the BLF file. BLF is a file format primarily used by Vector and their CANalyzer software. CANalyzer has the ability to log, and replay BLF files which is a binary file, in a custom Vector format that is partially documented. Luckily Vector also provides a Windows 32 bit DLL for reading and writing to BLF files. Because the format is more closed there aren't as many features of the BLF API. Still with the API you can perform writing operations, as well as reading N frames from a given offset.With both APIs sharing some of the same functionality, there are a couple of included examples that allow for converting from one format to the other. This is just a read from one API and a write to the next. This means that other software tools like CANalyzer can be used to process CAN data read by LabVIEW. This also means that CAN logs created by CANalyzer can be converted and then processed by software written in LabVIEW.
You'll notice that both TDMS and BLF data only read and write the raw frames of a CAN log. Things like frame to signal conversion still is done by the software after the read operation. Luckily tools like the ones I've already covered in Part 5 can perform the conversion for you.
Reading Hex and S Record Files...really?
Yes really. I know this might seem like the kind of thing that doesn't belong, but believe me it does. Included in the CAN File IO package is a set of tools for reading and parsing HEX and S Record files. HEX and S Rec. files are ASCII text files, that are used in defining what data should be put into what memory address on an embedded device. HEX files are common in things like Arduinos that take the C++ code, and compile it down to the functions the hardware need to perform. Since HEX and S Rec. files are used on all kinds of different hardware with various architectures (8-bit, 32-bit, etc.) this means that there are slight variations on the format used in these files. The formats of these files is clearly defined in the Wikipedia articles that describe them. Just like other examples, this all started from an NI example, this time included in the ECU Measurement and Calibration toolkit. The example was very inefficient, with lots of unnecessary operations, and optimizations in reading it differently. After improving it the result is a VI that takes a file path, of any HEX or S Rec. file type, and will return an array of addresses, and the array of byte data that should be written to those memory address on the device.If you've made it this far you might be asking yourself, "What in the world does this have to do with CAN?" Well my friend, the automotive CAN bus has been used for several years now to be able to reprogram ECUs in your vehicle. Your car already has a CAN bus for communication, and the OEMs just figured they could use that for reprogramming hardware in the car. If software in an ECU has a critical bug found, you can take your car to a dealership, and they will reprogram it. Of course if that ECU is buried behind the engine getting to it physically will be very difficult. So instead a technician can plug into a diagnostic port under the dash, and a tool can reprogram the ECU without having physical. There are two basic methods of performing what is called CAN Flash, which is flashing an ECU's firmware using CAN. We want to go over a few examples in the next section of CAN flashing, but only once we've understood how to parse and read the firmware, which are HEX and S Rec. can we hope to program the ECU. So you see it does relate to file I/O operations and CAN.