Reverse Engineering a Protocol

A QUICK ASIDE before the rest of the post: Today marks six years since I met my wife. Six whole years! We’ve only been married just over 10 months so far, but it’s been absolutely incredible and I know I have a lot more to look forward to. I love you, Courtney! (She also wrote about this.)

Ahem.

I’ve never really worked with protocols on a byte-by-byte level before, though I have read some data sheets on TCP/IP and the AIM chat protocol as part of a proxy program I intended to write at one time (but never did, because I found an existing program that did the same thing). Simply explained, a protocol is just a particular way of doing something. In the context of a computer, a protocol usually defines a structure of information that some hardware or software uses to send data back and forth. If both ends of the connection understand the protocol, then the information flows and everyone is happy. If either end doesn’t understand, then the connection is useless.

HTTP, for example, is a very common protocol that you use every time you visit a web page on the internet. FTP is a protocol used for transferring files. There are literally thousands of protocols in existence, some of which are widely used and some of which have only a very narrow area of application. Many are free, open standards and have lots of documentation available to describe how they work and how to use them. But some are proprietary and have no freely available info. Sometimes you can’t even pay a person or company for the info, either because they guard it too closely or they aren’t around anymore to explain it.

For a project that is actually related to something my mom is doing, I’ve had the opportunity over the last weekend to try to reverse engineer one of these proprietary protocols. I’ve made good progress so far, but I’m not done yet. The device is a stenography machine, and it uses a serial port to send keypress information and other data to the computer. Neither my laptop nor desktop even have a built-in serial port—the technology is too old and most people don’t need it anymore for normal computer usage. However, USB-to-serial adapters are easy to come by and fairly inexpensive, so I got one in the middle of last week and set to work.

To learn the details about a serial protocol, you need a few things:

• A serial port or USB-to-serial adapter (duh)
• A terminal program like RealTerm
• A serial port monitoring program like Free Serial Port Monitor or Device Monitoring Studio (which works in Windows 7)
• Some existing program that can communicate with the device in question
• Pattern recognition skills of some kind, depending on the complexity of the protocol

That last item is kind of ambiguous, but if you look at a set of strings of bytes represented in hexadecimal format and have absolutely no clue where to start, then this may not be the project for you. It is important to be able to recognize which bits stay the same and which ones are different, and to know how to look for things like boundary markers, sequence numbers, and checksums.

Also, having a program that already talks to the device is critical. If all you have is a terminal program, you probably won’t be lucky enough to have the device respond to English-like commands, or even technical ones that seem reasonable. If you have a program that communicates to the device, then you have someplace to start. Use that in combination with the monitoring software to gather as much information as possible about which information packets correspond to which actions, buttons, or events in the software or on the device.

The interesting thing I have discovered first-hand with this is that you really don’t need to know what everything means in order to make something useful. It just needs to work. I’m almost there, too…only one more hurdle to overcome before I have a working proof-of-concept application to experiment with. Coincidentally, the hurdle isn’t even in the protocol, as far as I can tell; there’s a bug in my application that is breaking the captured response data under some set of circumstances that I haven’t figured out yet.

I do love a good challenge though.