Ignys software engineer Stephen Lynch, who recently joined the Ignys team shares his experience of reverse engineering for video software and the different areas engineers need to be aware of. This blog is part of our tech article series.
How touchscreen adds a layer of difficulty for reverse engineering
Reverse engineering a video element by itself is of relatively low difficulty for an experienced engineer.
This is due to the narrow set of functionalities that can be implemented with video.
However once touchscreen functionality is introduced this becomes much harder to work with.
Reverse engineering with videos encounters some interesting problems when touch screens are involved. This is because touch screens vary wildly in their hardware and software implementation.
An example of reverse engineering for video touchscreen
In-car entertainment systems typically work with the following when video is involved:
- Video link
- CAN bus for control information including touch data
The protocol for integrating a touchscreen to a system is very much up to the implementer, for example car manufacturers. Manufacturers often don’t provide their protocols as standard practice.
This often happens if complex confidentiality agreements are in place with one of their subcontractors.
Other reasons not to share this information can include not wanting to give away technology–based innovations or insights into their systems which could be used by a competitor.
As a result of this, software engineers need to try a few things to get the information they need, to successfully carry out reverse engineering.
Plugging in an oscilloscope into the communications bus will in some cases let you decode interfaces such as CAN, ethernet or I2C.
In this instance oscilloscope can be used to probe the CAN bus, in situ, so the connection between the screen and the computer is still active and you are not disconnecting in any way. You can also get probes that pierce into the insulation and the metal so you can view the signal.
This results in information flying through the oscilloscope, and if the equipment supports decoding, this will then decode the CAN packets, breaking them down to give you the raw data that you are looking for.
The raw data collected by itself is often unintelligible, so it is up to the software engineer to figure out what is going on. For example, they will take different recordings of a few seconds with the system in different states. Recording types may include:
- Idle state – When the touchscreen is active but not being interacted with.
- Actively being touched – On various known areas of the touchscreen (the engineer will touch the screen).
- Record whilst the vehicle is being turned off and on (using the ignition key) and seeing what happens during that period.
- Buttons – These can sometimes be present on both sides of the screen and these also need testing. Whilst doing this you make a note of which button you touch at what point so you can clearly compartmentalise the data.
Looking for common features
As part of these recordings, what you touch will be documented which then provides you with what is essentially a puzzle.
At this point you then need to figure out, “is it possible to solve this puzzle?” You are looking for common features between all of them. For example, you sometimes have a keep alive packet.
What is a keep alive packet?
This is a packet which is sent regularly at a certain number of milliseconds or seconds for the screen to tell the computer unit “I’m here and I am still alive.” These are usually easy to identify as this happens at very regular intervals and you can usually see these keep alive packets across every recording.
Start-up activity
Once you have identified the keep alive packet you can start looking at booting–up activity which usually settles down. You should then see something that looks like the idol recording which can help you tease out some of the packets that aren’t to do with the touchscreen functionality.
What is the strategy at this point?
The strategy for the engineer is to identify what parts of the signal is part of the touch information because that’s what you are looking to do something useful with.
You then have recordings of all these different packets, and it becomes a game, you are trying to work out which packets are which. Once you have identified which packets are the keep alive ones and which are present for routing up you should have a much smaller number of packets to identify.
As you start to look at the touch recordings you should see bursts of activity at certain points which gives you an indication of which packets are related to touch.
You’ll see the density of information being transported at different points and it starts to peak where you are touching. There is usually a start and an end packet and some intermediate packets in between. With touch you will generally get that pattern, of course it depends a little on what exactly you are trying to reverse engineer.
A game of graphs
You then need to identify which is the x co-ordinate and the y co-ordinate and how its distributed, this can present itself in many weird and wonderful ways.
This is where excel can really be your friend. There is a lot of playing around with the x and y plot and looking for a continuous draw. This identifies which bits contribute to the x and y co-ordinates, often these have been split up, and chop and change around, hence why investigation is needed.
Whilst this is often done in a logical manner, what the creator deems as logical can vary significantly depending on the inventor.
And in the end, you solve that puzzle.
Other things to consider when engineering with video screens
Hover touches – Sometimes hovering near the screen rather than physically touching it can trigger a reaction and this needs to be identified.
What triggers each feature? – If you are looking to reverse engineer something that isn’t a touchscreen there may be other features that need considering for example the method used to trigger the feature, such as a button. In the case of touch–screen you instigate by touching the screen.
Other communication processes – In a more abstract system you may be looking at how you can trigger a communication process to fire off something so you know, when that happens, what sort of information should be transported. You’ll try different combinations of things to identify what bits of these packets mean what.
There is no one size fits all formula but…
Whilst every project is different the general idea behind it is trying to find ways to hook into these communication paths in situ and figuring out ways of probing it to get a useful response. By poking around in various ways and gathering all the recordings you can identify the common features, and what is different, to work out this puzzle.
And what is the end goal…
What is an example of an end goal in reverse engineering?
One example would be if you had a piece of hardware that was going to communicate between a touch screen and a car’s infotainment system, that hardware will be acting as the man in the middle in this situation, in this communications string. You can have CAN going into it and back out of it. It would have to be carefully designed to be invisible as far as the rest of the system is concerned because it’s not intended to be designed in there, by the original manufacturer. We are reverse engineering it into the system. So, you need to have a good understanding of what is going in and out.
A lot of the time if you are just looking for one piece of information like touch and let’s say you just wanted to filter in and out of touch depending on your circumstances, for example to put a different image on the touch screen, you need to understand it all.
Another example where reverse engineering the CAN buses in a car can be useful is for where an aftermarket device to be fitted into a vehicle. That device may need to know whether the car is stationary or not for safety reasons. One way to determine this is for the device to identify the state of the hand break position. Frequently this information is communicated on the car’s CAN buses however this information is rarely in the public domain and may need to be reverse engineered using similar techniques described above.
What is the end game?
If we wanted to have a system be able to send an external video source to the screen and make use of the touch data being sent from the screen we will need this detailed knowledge of the CAN packets.
This device would take in and output CAN packets. With the knowledge of purpose and structure of these CAN packets we can pass through the “house–keeping” packets such as keep–alive packets ensuring the touch screen and the infotainment system can talk to each other and function as they were designed to. This device would then be able to read the touch packets and send this information back to the external video source in a more standard format such as USB. This device could also ensure that these touch packets would not be delivered to the infotainment system when displaying external video. This prevents a user from accidentally activating features on the infotainment system.
So the end game in this example is we have this piece of hardware that is taking in packets and reproducing them on the other side and leaving alone packets, or filtering, the ones it doesn’t need to allow the car touch screen to be used for external devices. This is an example use case.
Up Next….
A Speed round of questions for Stephen Lynch, our new software engineer
What does reverse engineering mean to you?
The art of reverse engineering is essentially a way to fix a puzzle. You take all the data available, find a way to translate that into an understandable pattern. You can then use that data to create a new way for the existing system to allow the integration of extra technology. Without the new element interfering with the system or being blocked by it in some way.
What do you enjoy the most about reverse engineering?
The challenging puzzle it presents, and the satisfaction of solving it. That moment when I figure out a packet structure and the combination of x and y co-ordinates fit perfectly into place is very rewarding. I also enjoy providing customers with the results so they can use the solution I have created for them.
How difficult is reverse engineering?
I find the difficulty of reverse engineering is linked to two things. Firstly, the complexity of the system and what you are trying to integrate into it, for example a one button system will be easier than a full multi-channel video touchscreen.
Secondly it will depend on the domain experience of the engineer. If an experienced engineer has dealt with many similar projects, they will spot patterns far more easily. They will also detect hidden patterns or anomalies which, if not spotted, could make the process far longer or weaken the end solution.
Whereas a less experienced engineer might require guidance from a more senior engineer or if left alone on a project will need more time. If they don’t have the correct domain experience, they may be unable to complete the project.
In some rare cases the project is simply not viable, regardless of engineer experience, but usually the engineer and designer involved in a project will be able to spot fairly quickly if this is the case, for example if the system uses extensive encryption.
What are you most looking forward to working on at Ignys?
I’ve enjoyed the past two weeks getting to know the team and their projects and I’m looking forward to utilising my software development knowledge to help customers bring great ideas to life.
More about Stephen Lynch
Stephen has wide ranging experience in video Verilog and FPGA design. Stephen has a first-class degree in electronics engineering and PhD from Southampton University.
You can find out more about Stephen and the rest of the Ignys team on our about us page.
Can’t get enough of tech?
Love a good tech blog? – Try our SiC and GaN article next
Seeking software?
Need help? Take a look at our software services and book a discovery call.