9-minute listen
A key component to any successful and mature Information Technology (IT) team is a disciplined change control process. Few things are more frustrating to a CIO than having a service affecting outage and not being able to immediately identify recent changes to the environment. This article will discuss recommendations and provide an Excel document template for creating a risk-based change management process.
Highlights
- Change control requires rigor, discipline, auditing, and a commitment from IT leadership to hold people accountable that do not follow the process.
- A written policy that employees understand and follow is critical for success.
- Objective, risk-based change management criteria are much more effective than subjective ones.
- This resource comes with a downloadable template.
RECOMMENDATIONS
- Create a change management policy that involves the necessary stakeholders when drafting the policy.
- Create a tiered model based on level of risk, less risky changes are approved by supervisors, higher level risks are approved by upper management.
- Ensure change control boards (CCBs) are appropriately staffed and taken seriously by employees.
- Using the risk matrix and including it as an artifact of each change request allows for auditing and management reviews.
TECHNICAL DETAILS
We were asked to assist an IT organization with putting some structure around their change control process. After several failed changes occurred within the enterprise which caused service affecting outages, a management assessment was conducted on change control and several opportunities for improvement were identified.
As we began to understand how change control was being executed and after talking to several of the IT employees, a few common themes were found. One issue was that the level of risk associated with a change was completely subjective, with no way to put it into context. One IT systems engineer may rate a change completely different than another, because of their individual training and experience. Another issue was that managers were not reviewing change plans completely and even when change plans were clearly either not done at all or lacking sufficient detail, there were no consequences.
This organization had a CCB and a written policy about change control. Unfortunately, often the CCB did not always hear about the changes because it was very subjective as to what was and what was not presented to the CCB. Even when the CCB did hear of the changes being proposed, many did not understand the context of the changes. The written policy was extremely outdated and did not properly reflect how the CCB, or change management in general, was actually performed.
To eliminate the issue of subjective change plans this Change Control Risk Matrix was created, which measures several key areas of a change plan:
- Assuming the change fails and the back out plan is unsuccessful, how many users could be affected?
- Assuming the change fails and the back out plan is unsuccessful, what are the potential risks to program operations?
- Assuming the change fails and the back out plan is unsuccessful, what are the potential security risks?
- Assuming the change fails and the back out plan is unsuccessful, what are the potential reputation and image risks to IT?
- Based on past changes of a similar nature and back out plan testing, what is the probability that this risk will be realized?
Each of these sections have different levels of severity and they are all to be answered from the perspective of if the change fails, what is the risk. The probability of the failure then adds weight to how the IT employee answered the other questions.
As the employee answers each of the dropdown questions, the form automatically begins rating the risk. After answering all questions, an overall risk level is provided for the change plan. This risk matrix is required to be uploaded to their electronic change management application as an artifact. No change plans are allowed to be approved without an attached matrix form. The overall risk score is also entered into a field in their change management application and are broken down as follows:
Risk level 1: Supervisor approval only
Risk level 2: Supervisor and Manager approval
Risk level 3: Change control board approval
Risk level 4: Change control board approval (higher risk)
Risk level 5: Approval from the CIO and CISO required
As each level becomes progressively riskier, it requires additional approval. For example, a risk level 5 would require the approval of the supervisor, manager, CCB, the CISO, and the CIO.
The risk matrix allows apples-to-apples comparisons and takes away the subjectivity. It also allows change plans to be reviewed and audited much easier and provides managers with risk-based decisions when approving or disapproving change plans. We also implemented random auditing, having the process management team within IT randomly select several change plans each month and ensuring the risk matrix forms are completed correctly.
The risk matrix forms are designed so that any reasonable IT employee who completed the matrix for the same change would arrive at the same risk level.
Simultaneous to the creation of the risk matrix form was updating the change control policy to reflect the new risk-based change process. The new policy required that all changes must have a completed risk matrix associated with them prior to approval and defined the different risk levels. The policy also discusses random audits, consequences for not following the policy, roles and responsibilities of the change board, and other common items that would be expected in a change control policy (veto power, maintenance windows, emergency changes, etc.).
After the risk matrix was designed and vetted with key IT stakeholders, mandatory training was provided to IT employees on how to properly complete the risk matrix forms and how to attach them to the change plans. This training also covered the updated policy and provided a baseline of what was expected for employees related to change control.
In the spirit of information sharing, the policy and training also advised employees that they have the ability to treat a change higher than it may be rated on the matrix, but never lower. For example, a change may be rated as a level 2 without needing CCB approval (for example, a very small change to a field in an application), however an employee can always request CCB review for situational awareness.
This process has been in place now for several years with excellent results. Unplanned outages have drastically fallen and when outages do occur change plans and risk forms are easily obtainable. Executive IT management has awareness of the highest risks and cybersecurity is not only a requirement to be considered for each change, but also a mandatory voter in CCB meetings.
A copy of the IT Change Control Risk Matrix can be downloaded below. The form is currently protected so only the dropdown fields and information at the bottom of the form can be completed. There is no password required to unprotect the form though. A tab on the bottom of the form called “DATA” contains the dropdown questions and weights.
You can customize this for your IT environment if you wish. Once customizations are done, the DATA tab can be hidden and the form password protected. Please note that this form works best in Excel for Windows and will need some minor formatting changes if used in Excel for Mac.
Download the resource here:
