Have an idea (roughly) of what you want to achieve. Reverse Engineering can be a very fickle business as Uwe and Eric will be able to testify. Yes, try and build your library with good books, but there's the addage that 'you can't read experience from a text-book'.
Bruce very cleverly summarised what was done by Uwe to start Vag-Com, pretty sound advice for any RE project. So start with both ends of a known working system, then you try to effectively 'get in-between' them. You need a way of watching the initiator of the comms; and how the other end reacts. Then try to mimic one of the ends.
However, the biggest pitfall of RE is that you may never get to know every feature of the system you're trying to crack; as by its very nature many features are undocumented.
My own story:
Right now I'm trying to reverse engineer one of our systems at work; it's not really my actual job, just doing it because I'm interested in the system and if I get it to work it will help with safety. Literally closed design, no documentation although the devices were built in the late 80s/early 90s. The manufacturer will not release documentation because if they think we're interested in a new control program; they'll want to sell one to us instead. Uwe was lucky with KW1281 as there's only one link - my system has four. I quicky established it used RS422 at hardware level, so needed to build a system that had 8(!) RS422 ports. Why 8? Because you need to listen to both ends of each link at the same time.
I managed to write a program that would give an accurate time stamp to the incoming data from each port, so I could later pour hours into how each link interracts with each other. The saving grace is that none of the comms is encrypted..!
Now was the time to write some test progs as the control program; sending my own data. The system wouldn't respond to any of my commands. You go over everything, checking how you calculate checksums etc. - everything, and still nothing. My logs matched the logs from my sniffer program
identically, but still the system would not respond. Very frustrating. However, after a fortnight of banging my head, I discovered that even though I was transmitting at the correct baud rate, the inter-byte time spacing interval from my RS422 cards was incorrect for the old system I was trying to control. (Sound familiar, Uwe, Eric?
)
I take my hat off to Uwe, Eric and not to forget Shaun, because the vast majority of people out there don't really (and probably never will) understand what goes into 'programming' or 'coding'. One day I'll sit at the computer and the code will just flow; another I may sit for half an hour just looking at the cursor flashing... Then there's the days when you look back on some of your previous work and marvel at the fact that you actually wrote that brilliant piece of code!!
Good luck, and let us know how you get on.