To standardize something is to lower entropy. Expectations should be more precise as a result. 

People in the IT industry generally like working with standards. Standards represent an agreed-upon way to do things, so we as developers know in advance what kind of data we need to handle and what is expected of our system.

In theory, this all sounds nice, right?

Well, retain on putting all your hard earned money on that horse. 

In the next few paragraphs, unfortunately, I’ll, have to break the illusion of standards being so “standard” by telling you about our path to implementing a card payment processing system.

The beginning – ISO8583 basics

Majority of us use the ISO8583 standard without realizing it.

ISO8583 defines the protocols and messages used to send and receive credit card transactions. If you ever used a credit card to pay for something, chances are your credit card and transaction info took flight using as vehicle some of predefined ISO8583 message envelopes. 

Our company is one of few that had experience in building card processing systems, and it is being used by some countries in our region by Diners Club International franchises. 

These systems are quite complex and contain a communication module, authorization module, fraud control module, pin management module etc., but what’s important for this story is the infamous communication module.

The communication module is like a host of a fine Diner (pun intended), he is the one in charge of communicating with the outer world and catering the needs of clients. 

Communication module needs to handle ISO8583 protocol messages so that we can unpack them and prepare quality info in a form requested by the rest of the card processing system. Also, they must respond to sender (usually referred to as card acquirers) , using the same ISO8583 standard.

ProudSourceIT went on a deep dive into the inner workings of ISO8583. Since our team already contains strong experts in areas such as network protocols and transactions in financial systems we expected an easy ride (oh c’mon, let me at least live in illusion for a little while).  We explored how it works in much more detail than you need to know here, but feel free to contact us if you want to discuss it in detail.

During the research phase we learned what the ISO8583 messages are all about, what they look like down to the bytes (back to CS school)  that are sent and received, how they fly about the network etc.

By the end of this phase, we were pretty sure nothing major could come up and surprise us.

More versions = more problems

Important thing about this standard that needs to be stated is that the standard itself has many versions. The system we were making needed to support two versions of the protocol (ISO8583:1987 and ISO8583:1993). 

The only difference between those two versions was in the message fields content. Depending on the version some of the fields can vary in type. The structure of the messages is the same in both versions.

To make our system future and fool proof we designed the communication module in such a way so that the acquirer and the rest of the system only see one communication module, regardless of the version. 

We achieved this by making multiple “adapters” for the communication module that were “plugged in” based on what version was used. This allowed us to have a unique message format that was going to other parts of the system so it doesn’t have to know anything about the ISO8583 versions (lucky him). In the future, if we need to support another version we could just “plug in” another adapter and that would be all that’s needed. A very high-level diagram is shown in the image below so you could get an overview of what was done to achieve this multi-version support.

Breaking the illusion

The first version of ISO8583 was released in 1987. Because this means that this particular standard has been around for more than 30 years, we safely assumed that all the kinks have been worked out and all of the systems that used the standard were on the same page.

Boy were we wrong. When we started testing our system with the credit card acquirers, the system worked for some, while for some it didn’t. Hey, the ISO8583 message implementation and the message structure can’t be different than what we read in the standard documentation, right? Also the system worked with some clients (a pat on the back well deserved). Natural deductive reasoning directed us into thinking the problem must be somewhere else. We spent countless hours checking the system deployment and network configurations only to see that the problem wasn’t there. 

After a while, after much discussion, few rounds of voting and many calculations about upcoming casualties we decided it was time to bring out the big guns… yes, we turned on the 

Wireshark. For those unfamiliar, meet wireshark, the best friend of anyone who encountered weird network behavior.

We analyzed the network packages for the acquirers that behaved according to expectations and for those that were not. 

And then it struck us, people can really be idiots (hey, we all have to learn sometime). We saw that different acquirers had a different understanding of the word standard. The ISO8583 documentation clearly stated that the message length in bytes is sent at the beginning of each message. One acquirer decided that was obviously going over the top so he decided to just skip that. Some decided to send longer fields than defined, some decided to completely change the type. We also had occurences of sending bytes instead of ASCII. Also bitmaps were all over the place.

We sat out to find a solution for this conundrum. Some suggested sueing all of them, others suggested various forms of violence while there were creative proposals to create a religion that would preach righteous use of ISO 8583 protocol.

Finally the voice of reason directed us into using technology to solve this problem.

Overcoming the obstacles

Luckily for us, we already designed the system that could support multiple versions of the standard so it wasn’t so much trouble adding support for the specifics that some acquirers had. 

The biggest change we had to make is to allow the adapters to not only parse different message file types based on the version but be able to change the way the whole message is parsed from the beginning.

After implementing all these specifics our high-level architecture diagram now looks like this.

Lessons learned

In this article, I tried to show you that in software development, more often than not things are not as they seem at first. The important thing is that in those moments is to remain calm and never assume some part of the system must work because that part is often the problem.

In finding the right solution, there is probably a tech workaround that can usually save the most nerves. Off course that will once again add extra work to already swamped developers that are never praised for what they do. But, hey how else you imagine the work of a superhero.

And to conclude, unfortunately in IT the standards are sometimes not so standard!