Assessing Control Effectiveness — An Essential Part of Every Risk Assessment

Control effectiveness is measured by looking at the maturity of the process. Most people agree that mature processes are documented, but why? Transferring knowledge from the human brain requires conversion from tacit knowledge to explicit knowledge, so that it can be shared, reviewed, updated and tested. Think about it. If we relied on tacit knowledge all the time, there is a good chance that the outcomes would be different every time the process was executed, unless they had a plan to follow, which is where explicit knowledge comes into play. Quality management requires that we integrate feedback loops to push a process even higher in maturity. Continuously monitoring and making adjustments to perfect the process can only be achieved with explicit knowledge.

Building the perfect control to mitigate risk is one thing, but making sure that it gets implemented, monitored and maintained adequately, so that it is functioning 100 percent, is yet another. This requires the assessment of competence for those employees or contractors who have been assigned the responsibility to get the job done! I like to leverage my knowledge as a teacher using Bloom’s Taxonomy. I create at least six basic questions to determine how much the employee knows. For example, I recently created a one-page assessment for CyberSecurity Leader.

We have evaluated the control for maturity and assessed competence of the administrator. Now we need to verify and validate that the control is functioning as planned. There are similar approaches that work, from using quality assurance techniques to penetration testing. This part should be looked at every time changes occur that touch the control in any way. We need a solid baseline for assessment control effectiveness, and to accomplish that, I like to integrate the use of design qualification (DQ), installation qualifications (IQ), operational qualifications (OQ) and performance qualifications (PQ).

Based on my experience, the most secure systems are those that establish and maintain absolute control over the environment. I often joke with senior management about my number-one rule, “No surprises!” Quality management is deep in knowledge about establishing control and assessing process, so it is only logical that we would assimilate this knowledge into information security.

DQ is the architecture, or specifications, used to build a service or product. Any changes must be strictly controlled; so, while DQ sets out the design, IQ defines the specifications, or standard operating procedures, for installing a new piece of software or hardware. Control design is a related topic that would allow you to map where this control applies within the risk universe as it mitigates risk to a specific asset that is used to deliver a service or product. OQ documents the configuration specifications that could be recorded in the configuration management database used by ITIL also ISO 20000. Once everything has been documented and procedures have been followed, the PQs are reviewed. What were the expected response times? How can we optimize them to meet customer expectations?

Whomever gets the job of reviewing control effectiveness should be looking at three key elements—maturity, competence and testing—to verify and validate that what we said we would do we have actually achieved. The importance of assessing control effectiveness during regular audits is obvious. The assessment of control effectiveness during risk assessment as part of the risk management and governance process is absolutely crucial to provide all the facts to management quantified in a meaningful way.

I have seen plenty of external audits that have gone in a direction where hundreds of thousands of dollars and sometimes millions are spent on new controls that may not have been necessary if the current investment in security was better quantified and managed. This is an evidence approach that can easily be shared and reviewed and scrutinized. Too much control could negatively impact the business model, organizational culture, agility or time to market and resilience by creating more complexity that is expensive to maintain and difficult to replicate in emergency situations.

Mark E.S. Bernard, CISA, CISM, CGEIT, CRISC, CISSP, ISO 27001 Lead Auditor

[Source: ISACA]