COTS Threat Detection Evaluation for National Security Decision Support

Challenge

A government organisation operating within the national security domain recognised that the threat landscape was rapidly evolving in both sophistication and scale. As malicious activity became more complex -spanning physical, digital, and socio-economic areas - teams responsible for early detection and mitigation faced increasing pressure to act quickly and accurately.

To strengthen their operational capabilities, the organisation sought to explore a data-driven decision-support system capable of semi-automated threat identification. The intention was to augment human expertise with technology that could surface early indicators, prioritise risks, and recommend potential mitigation routes.
‍
However, selecting the right tool was a significant challenge. With a wide and fragmented market of Commercial Off-The-Shelf (COTS) threat-detection platforms, the key question became: Which solution could reliably detect diverse threats, integrate with existing workflows, and meaningfully support human-in-the-loop decision making?

Solution

Butterfly Data undertook a comprehensive landscape review and comparative evaluation of leading COTS products, using structured analysis and evidence-based scoring.

The project began by mapping the national security domains most vulnerable to emerging threats, such as public health, energy, economic stability, and cyber operations. Using this framework, we identified a mix of established market leaders and specialist niche providers offering automated threat detection or probabilistic modelling capabilities.

Each solution was evaluated against the following criteria:

• Threat Coverage: Types of threats the tool could detect or infer
• Mitigation Support: Availability of built-in mitigation pathways or recommended actions
• Automation & Flexibility: Ability to scale, adapt to new data sources, and support semi-automated workflows
• Simplicity & Integrability: Ease of deployment within existing systems and operational environments
• Data Diversity: Capacity to handle text, geospatial data, time-series signals, and multi-modal inputs
• Real-World Validation: Evidence from implementations, case studies, and user feedback

The final outputs included a detailed analytical report supported by:

• A structured assessment matrix
• Multi-tool comparison tables
• A Sunburst diagram visualising how threats map across available tools
• Graphical analyses using synthetic data to illustrate potential cross-domain threat mapping

Results

The evaluation provided a clear, evidence-based understanding of the threat-detection tools available on the market. Multiple COTS platforms and Bayesian Belief Network (BBN) tools were assessed and scored against customer-defined criteria, each receiving a rating out of five to support transparent comparison.

In addition to the written report and annex, we produced a series of visual artefacts, including Figma diagrams and a sunburst visualisation, to help stakeholders interpret key differences quickly and confidently.

The work concluded with a vendor demonstration event, where shortlisted suppliers were invited to present their products’ functionality. This enabled the organisation to directly compare usability, performance, and feature sets, supporting an informed decision on the most suitable technology to enhance national security threat detection and decision-making processes.