Always Check the Boxes!

“Don’t just check the box!” Chances are you have said or heard this phrase at some point in your career. In case you are not familiar with the term, it refers to a mechanical, “bare minimum” way of doing things. Sometimes it means simply not being creative in your approach. As I will explain, checking the box can actually be a very good strategy to uncover problems, especially if you are in a control assurance function. More precisely, I will call my strategy “Check the boxes.”

Let me take you back to the late ’90s when I was a novice IT auditor in a multinational organization. I had just made the switch to the audit profession after having been a systems developer for a few years.

An Army of Programmers to Fix Y2K
The biggest challenge at that time was the so-called “Year 2000” (or “Y2K”) problem. Many applications in those days used a two-digit representation of the year field (“99” was used to represent the year 1999, for instance). If not remediated properly, it was feared that when the year switched from 1999 to 2000, the year field may become “00” (or something meaningless), leading to processing errors in critical systems. There was a real possibility of nuclear reactors going haywire, patients being administered the wrong dosages, and aircraft not being able to fly or land—a perfect doomsday scenario.

It took an army of programmers, testers, project managers and other IT professionals to fix systems so that no erroneous processing occurred when the date switched from 31 December 1999 to 1 January 2000.

As an auditor, I was to review the Y2K compliance of certain key applications for my organization. As part of this remit, I used to receive a package of tests and documentation from the IT owners that annotated the test results showing date-related processing. Also included was an attestation from IT and the users to certify that the system had been thoroughly tested for Y2K. Upon reviewing the documentation, the auditors made a call whether the system had shown enough readiness for Y2K or if additional documentation or clarity was needed for the scenarios tested.

First Week, Two Packages, Two Systems
In my first week on my job I received 2packages belonging to 2 different systems, which the auditors would use to perform their Y2K certification:

• System 1:  The test details came in a large brown envelope containing loose papers. Many of these showed system diagrams, test results and some handwritten comments on several computer printouts. There was no index of what was being provided. It took me several hours to ascertain the name of the system, let alone the nature of the functionality tested. For the purpose of this article, we will call this application “The Loose Leaf System (LLS).” After a preliminary review I concluded that, at a minimum, the documentation quality was poor. I could not establish for sure if the documentation was complete or accurate. This system was a winner if documentation sloppiness was the evaluation criterion.
• System 2: This documentation came in three neatly packed boxes. Inside each of the boxes were 3 to 4 labeled binders. I did not have any difficulty ascertaining the name of the system or the nature of its functionality. The first binder also had an index that clearly explained what the various binders contained. To keep matters simple, we will call this system the “Three Box System (or TBS).”

As I started my analysis of the documentation for LLS and TBS, it seemed quite obvious who was going to get top marks.

As TBS had all the hallmarks of a well-tested system, I wanted to quickly finish my review of this application first. The first binder contained an index that pointed to other binders labeled 2 through 6. As I lifted the binders to make a nice vertical stack, I noticed that there were only 5 binders in all. The fourth binder was missing. TBS had not supplied the documentation for a key piece of functionality, technically resulting in a case of incomplete documentation. When I called the IT audit manager to trace the missing binder, no one could locate it. I had my very first audit observation on my hands as a result of this discovery. The final audit result for TBS…“Failed audit” and my comment was to revise and resubmit the documentation for audit approval.

LLS also required follow-ups with its IT manager. I was, however, able to establish, to my satisfaction, that the system tests were indeed carried out and the loose sheaf of papers did contain the necessary evidence to point to a complete and accurate set of test results. Notwithstanding this, it too was a case of bad documentation quality. The audit result…“Passed audit” with minor comments on documentation.

Which System Worked Best?
Just so you are all not kept in suspense about the fate of TBS, after several weeks, I received a replacement for the missing fourth binder, as the team had to retest the functionality since they could not locate the original test documentation. During the retest, the IT manager told me they found other bugs that were not uncovered the first time around. So, my discovery of the missing binder actually helped the IT team further strengthen the TBS system.

This experience led me to coin my own principle:  “Always check the boxes”—the audit equivalent of never judge a book by its cover.

Over the years, I have found many uses for and identified several deficiencies as a result of this principle. It instilled in me the discipline to open and verify every binder, data file and artifact received from my clients. The act of thoroughly checking the boxes, or the relevant audit artifacts, has often me helped me find problems. A purist may point out that this is, after all, an application of the well-established audit dictum, Trust, but Verify. Hopefully, there is no argument over that.

Editor’s note:  Subramanian Annaswamy, CISA, CAMS, CSQA, has several years of audit experience in leading financial services organizations. He is currently a technology audit director in a leading bank in New York. The opinions expressed above are his personal views and do not represent the view of his current or former employers.

Subramanian Annaswamy, CISA, CAMS, CSQA

[ISACA Now Blog]