Cybersecurity Metrics Reporting Model (CMRM)

The ComplianceForge Cybersecurity Metrics Reporting Model (CMRM) takes a practical view towards implementing a sustainable metrics reporting capability. At the end of the day, executive management (e.g., CIO, CEO, Board of Directors (BoD), etc.) want an answer to a relatively-straightforward question: “Are we secure?” In order for a CISO to honestly provide an answer, it requires a way for the CISO to measure and quantify an “apples and oranges” landscape where processes and technologies lack both uniform risk weighting and abilities to capture metrics. The SMRM helps solve this aspect of dissimilarity by utilizing a weighted approach to metrics that generate Key Performance Indexes (KPXs) as a way to logically-organize and report individual metrics. Using KPX enables the SMRM to provide a reasonable and defendable answer.

Key Takeaways - Cybersecurity Metrics Reporting Model

Executives want a simple answer to Are we secure? The CMRM provides a structured way to quantify and visualize that answer.
The model uses a weighted approach to metrics that generates Key Performance Indexes (KPXs) to logically organize reporting.
The answer is best expressed as a numerical score on a spectrum from not secure to secure, based on the organization's risk profile.
The CMRM solves the Garbage In, Garbage Out (GIGO) problem by starting with the answer and working backwards to define what data is needed.
Individual metrics feed into KPXs, KPXs feed into the overall score. Enabling long-term trending to evaluate the impact of security initiatives.
The model can be automated in GRC / IRM tools for continuous reporting.

Metrics & Analytics

Garbage In, Garbage Out (GIGO) Problem

Metrics / analytics reporting is plagued by the Garbage In, Garbage Out (GIGO) problem. Often GIGO issue is rooted in executives trying to explain their perceived needs for metrics to cybersecurity practitioners in a way that describes the design of a "football bat" (e.g., nonsensical solution). How ComplianceForge addressed this GIGO problem is start with answering the most important question (e.g., are we secure") and working backwards in a manner to defend that answer.

The “Are we secure?” question is best answered as a numerical score. This quantifiable score is used to visualize the score against a numerical spectrum to provides context, based on the risk profile of the organization. The numerical score would land between “not secure” and “secure” on the spectrum, according to a baseline score definition that would be specific to the organization. This can provide long-term trending to evaluate the direct impact of certain security initiatives. The SMRM can be automated in a Governance, Risk & Compliance (GRC) or Integrated Risk Management (IRM) platform, but it comes as a Microsoft Excel spreadsheet as part of ComplianceForge’s Digital Security Program (DSP). The “Are we secure?” question can be both tracked to display trending and drilled down into KPXs, or individual metrics, to identify why the score changed.

Cybersecurity Metrics Reporting Model example

Key Performance Index (KPX) is essentially a term that we use to normalize the various metrics in each category. One area of contention with metrics is defining what a KPI or KRI is since people tend to butcher the terminology. Our approach to defining those terms are shown below:

Key Performance Indexes (KPXs)

KPXs are logical groupings of KPIs that allow an organization to monitor an index of metrics about a specific capability or team.

KPXs are used to answer the question, “Is the XYZ capability operating effectively?” where that capability is an aggregation of multiple individual metrics.
KPXs may be weighted to highlight risk-heavy topics of concern.
KPXs may be nested underneath other KPXs to report the hierarchical nature of metrics that help answer the question of “Are we secure?”

KPIs and KRIs are not hierarchical metrics, but are individual metrics that are deemed important to monitor, based on the specific risk or value associated with that metric:

Key Performance Indicators (KPIs)

KPIs are “rearward facing” and focus on historical trending to evaluate performance.
KPIs should not be weighted.
KPIs are indicators that enable an organization to monitor its progress towards achieving its defined performance targets.
KPIs are used to answer the question, “Are we achieving our desired levels of performance?” for a specific control.

Key Risk Indicators (KRIs)

KRIs are “forward facing” and focus on identifying a future-looking trend that impacts risk.
KRIs should not be weighted.
KRIs are indicators that enable an organization to define its risk profile and monitor changes to that profile.
KRIs are used to answer the question, “Are we within our desired risk tolerance level?” for a specific control.

The metrics shown in this model are included in the ComplianceForge Security, Compliance & Resilience Program (SCRP) product. Being transparent on the subject, the entire point of a "canned solution" for metrics is to provide a starting point where someone else does the heavy lifting for you to get to a 70-80% solution that someone within your organization can then run with to customize for your specific needs. This is where ComplianceForge is a business accelerator - we enable you to hit the ground running with your cybersecurity documentation that can takes months or years to create on your own. The "heavy lifting" of the equation is what we provide, not the finalized metrics product. That is really where the demarcation is between what ComplianceForge offers for metrics and how an organization would customize the remaining since you have the organization-specific knowledge side of the metrics equation that cannot be templatized.

ComplianceForge does not sell the SMRM, KPIs, KRIs on their own, since the metrics are part of the DSP solution. With the 1-1 mapping relationship between the DSP and the Secure Controls Framework (SCF), the DSP can help operationalize the SCF controls in a meaningful and efficient manner, so that is something to consider for organizations that want to fully adopt the SCF as its control structure and maximize its effectiveness.

We are happy to discuss our products. Please feel free to contact us with your questions.

Common Questions

Frequently-Asked Questions

Here are answers to common questions about cybersecurity metrics and reporting:

What is the Cybersecurity Metrics Reporting Model (CMRM)?

The Cybersecurity Metrics Reporting Model (CMRM) is a free ComplianceForge guide for answering, with a defensible number, the executive question of whether your organization is secure. It groups metrics into weighted Key Performance Indexes (KPXs) that roll up into a score placing your organization between not secure and secure, customized to your risk profile. Tracking that score over time shows your leadership the effect of security investments.

What is the difference between a KPI and a KRI in cybersecurity?

A Key Performance Indicator (KPI) looks backward and measures progress toward defined performance targets, while a Key Risk Indicator (KRI) looks forward and tracks changes to the organization's risk profile. A KPI answers whether the team is achieving the desired level of performance. A KRI answers whether the organization is staying within acceptable risk tolerance and helps spot emerging risk trends. In the CMRM, both are unweighted individual measurements.

What is a Key Performance Index (KPX)?

A Key Performance Index (KPX) is a logical grouping of KPIs that lets an organization monitor an index of metrics about a specific capability or team. Each KPX answers whether a given capability is operating effectively. Unlike individual KPIs, KPXs can be weighted to emphasize areas that carry more risk, and they can be stacked hierarchically so detailed team metrics roll up into executive reporting.

What cybersecurity metrics should be reported to the board?

Board reporting should focus on a few metrics that show whether the organization is secure and trending in the right direction, rather than raw operational counts. The CMRM works backward from a single weighted score and then identifies the data needed to defend it, which avoids the "garbage in, garbage out" problem. NIST SP 800-55 Vol. 1 similarly frames security measures around judging the adequacy of in-place policies, procedures and controls.

Is there a cybersecurity metrics template?

Yes, the metrics shown in the CMRM are included in ComplianceForge's Security, Compliance & Resilience Program (SCRP) as a Microsoft Excel spreadsheet. We provide it as a starting point that gets you to a 70 to 80 percent solution, which you then refine with your local operational knowledge. Because our documentation maps to the Secure Controls Framework (SCF), the metrics tie to controls, and you can automate the model in a GRC or IRM platform.