Incident Response Planning in a Nutshell: Leveraging Lessons Learned
“Some of the best lessons we ever learn are learned from past mistakes. The error of the past is the wisdom and success of the future.” – Dale Turner, American singer-songwriter and record producer
As final part of our 5-part series about Incident Response Planning (IRP), this article dives deeper into how the often-overlooked lessons learned exercise can help organizations improve their incident response planning capabilities going forward.
Find more detailed information about IRP in the articles below:
Why IRP lessons learned sessions are important
While often overlooked, lessons learned sessions are crucial to improving an organization’s security posture and readiness to face security incidents in the future. They help evaluate incident response performance, identify challenges and improve incident response capabilities going forward. For lessons learned sessions to be effective, they need to contain at minimum 4 different functions, follow a strict process, provide answers to specific questions about the incident and establish clearly-defined action items for follow up and continuous improvement.
According to NIST Special Publication 800-61, Rev. 2, “one of the most important parts of incident response is also the most often omitted: learning and improving”. In fact, only 58% of organizations regularly review and update their incident response processes – a number far too low to guarantee effective safeguards against security incidents (The SANS Institute, 2017).
This article covers:
- which key elements should be covered,
- when they should take place,
- who should participate, and
- what actions should be taken to move your organization forward after suffering from a security incident.
What are lessons learned sessions?
The Project Management Institute (PMI) defined as “the learning gained from the process of performing the project”. In the context of security incidents, they usually take place after a security incident has occurred and has been mitigated.
The SANS Institute identified Lessons Learned as one of the 6 critical stages of the Incident Response Process (after Preparation, Identification, Containment, Eradication and Recovery).
Lessons learned sessions are recommended to evaluate your mistakes, take stock of what happened during the incident and assess how your team has dealt with mitigating the impact of the incident. The purpose of a lessons learned session to share and use the knowledge gathered from a security incident to:
- Facilitate the recurrence of positive outcomes (“let’s repeat what went well”)
- Prevent the recurrence of negative outcomes (“let’s avoid making the same mistakes”)
Lessons Learned Benefits
Lessons learned sessions offer a variety of benefits for the incident response team, the organization as well as existing and future incidents. A properly executed lessons learned exercise can help organizations:
- Learn from mistakes. Especially during a security incident, organizations tend to operate in crisis mode and are too overwhelmed to execute all steps flawlessly. Identifying where mistakes have been made will help self-evaluate performance and set the tone for continuous improvement.
- Understand where problems occurred. Despite its negative impact, a security incident can represent a valuable opportunity to dig deeper into where exactly errors were made, which vulnerabilities led to the incident, whether security controls were effective and whether there are any security gaps in the organization that could be fixed. Getting all relevant parties together during this exercise will help understand the problems from a 360-degree view.
- Recognize success. Too often, organizations tend to forget about what actually went well during a security incident. In addition to analyzing what didn’t go well, it is equally important to recognize what went well – whether it relates to employee performance, effective processes in place, crisis communications etc. Identifying success stories will encourage positive performance and leverage exemplary behavior for future incidents.
- Retain organizational knowledge. During a crisis, documentation is often the last thing on the mind of those who are dealing with the crisis. Keep in mind that it is crucial to document the good, the bad and the ugly to retain this knowledge and make it available for future use.
- Reduce future risk. If similar security incidents occur in the future, a lessons learned session can help prevent similar mistakes and thereby reduce the risk to the organization as a whole.
- Improve future performance. By analyzing how an organization dealt with a security incident, it can optimize its incident response performance for future similar scenarios. In addition to having an Incident Response Plan in place, lessons learned sessions contribute to adapting and improving this plan so that the organization can be better prepared.
►►►Tip: Make sure to briefly go over these benefits at the beginning of each lessons learned session. This will help remind session attendees why their feedback and active participation is important.
What is the process to follow?
To make sure that lessons learned sessions are as effective as possible, they should follow a predefined, standardized 5-step process. Once all findings are collected, they should be documented and shared with all relevant parties. Then, findings should be analyzed and stored in a repository that is accessible to all relevant parties. This way, findings can be retrieved to be used for future potential incidents.
When should they take place?
The lessons learned session should take place as soon as possible following a security incident, preferably within a couple of days. The sooner the session is organized, the fresher the incident will be in the memory of those who have dealt with the incident.
In certain cases, it may also make sense to begin the lessons learned exercise prior to the end of the incident, depending on the complexity and criticality of the incidents. For example, organizations experiencing a security incident that takes a couple of months to mitigate should not wait until the incident is fully mitigated. Scheduling a lessons learned session while the incident is still ongoing may reveal relevant insights that could turn out to be useful for mitigation activities.
►►►Tip: If your organization is experiencing a security incident, ask yourself when it would make most sense to have a lessons learned session, not whether you have the time (chances are, you’ll never have the time…).
Who should participate?
The participants of a lessons learned session will vary depending on the size and structure of the organization, but should at least include the following functions:
- An organizer. Make sure to designate somebody with excellent organizational skills to schedule these sessions. Ideally, this person should have basic knowledge about the incident as well as the incident response handling function, and be comfortable adhering to a strict schedule, following the process and conducting all required follow-up activities. Project managers are typically well suited to take on this role.
- A notetaker. To make sure that all content of the lessons learned session is properly documented, make sure to designate somebody whose sole responsibility it is to take notes. Ideally, this person should not be actively involved in the security incident and document the session from an objective and neutral perspective. A member of the administrative team is usually a safe bet. Note that the session organizer cannot be the notetaker – wearing two hats is dangerous!
- A member from the IT/IT security team. For lessons learned sessions to be effective, they need to include at least one member of the IT or security division. Representing the voice of those who actually mitigated the incident will help gather relevant technical feedback necessary to evaluate how the incident went and what measures may be necessary to strengthen the organization’s security posture.
- A member from executive management. Depending on how your C-Suite is organized, you should have at least one member of the executive team join the lessons learned session, e.g. your CIO, CSO or CEO. The executive is equipped with the decision-making power necessary to authorize appropriate actions that should be implemented and can leverage his or her high-level view of the organization, its shareholders and stakeholders to guide the discussion.
►►►Tip: Depending on how your organization is structured, you may want to include additional functions such as Legal, HR, Communications/PR etc. Before organizing a lessons learned session, make sure to look at the key roles involved in dealing with the incident and choose your session members accordingly.
Which questions should be answered?
Although lessons learned sessions vary depending on the nature, scope and impact of the security incident, they should follow a similar structure and provide answers to at least these questions:
- What happened exactly during the incident, and at what times?
- Are documented procedures in place for incident handling? If yes, were they followed when the incident occurred? If not, what are the plans for establishing such procedures?
- How well did we perform in dealing with the incident? Did management act appropriately? Did employees act appropriately? If not, what can be improved?
- Is there anything we should do differently next time?
- Have we communicated the right amount of information to the right channels? If not, how can we ensure that post-incident communication is improved?
- Do we need any additional tools, resources or strategies that will help us better detect, evaluate and mitigate security incidents in the future?
What are the next steps?
Don’t be surprised if your lessons learned session unveils a myriad of issues and gaps about your current incident response handling practice. That’s a good thing! At the end of every lessons learned session, the person in charge of documenting the session should summarize which identified problem areas were revealed, what the proposed action items/solutions are to fix said problem areas, who will be responsible to take care of these action items, and when these solutions are expected to be implemented.
Action items examples:
- Your lessons learned session may reveal that your employees aren’t trained enough to deal with a security incident, thus legitimizing the need for a company-wide employee security awareness program.
- Your environment is not secure enough and may have too many exploitable vulnerabilities that could be patched after a thorough penetration testing
- You’re lacking the necessary time and resources to monitor your environment for potential threats and intrusions, and may need to look into options for 24/7 monitoring.
- You realize that your information security program is incomplete and needs to be fine-tuned to become more robust and help your team deal with security incidents.
►►►Tip: For lessons learned exercises to be effective, make sure that concrete action items are established, roles and responsibilities are assigned and specific dates are defined.
Related Blog: 5 Mistakes You Need to Avoid After a Data Breach
This concludes our 5-part blog series about Incident Response Planning. Be sure to subscribe to our blog to stay tuned for more content about cybersecurity, incident response and securing your organization.
