Ian Costa is a co-author of this article.
There has been great speculation of how AI will transform business, but how does this technology apply in practice? StoneTurn’s cybersecurity, forensic data science and risk and controls experts discuss how they developed an LLM-based review to cross-reference and assess large volumes of information stored in documents and files in different formats against the National Institute of Standards and Technology’s Cybersecurity Framework (“NIST CSF”).
The Client’s Issue
Cybersecurity assessments most often begin with cross-referencing the organization’s basic cybersecurity policies and procedures against the minimum requirements of an industry framework such as the NIST CSF. The assessment continues with identifying enhancements to develop, streamline, and refine cybersecurity processes, policies, and protocols to safeguard the organization better and satisfy regulatory requirements.
However, cross-referencing an organization’s policies against the NIST CSF takes considerable time and experience because the information appears to spread across various documents. StoneTurn sought a technological solution to streamline the process.
How StoneTurn Leveraged Technology
StoneTurn’s researchers ingested a sample client’s information security policies and procedures as well as the industry framework into our securely held, locally hosted Large Language Model (LLM). The client’s documents included many dozens of documents and files in different formats.
We then crafted a series of prompts to address each requirement in the framework, resulting in LLM-generated responses to each requirement. The responses gave our analysts a foundation that catalyzed the cross-referencing process by magnitudes.
We took the results of the LLM and matched them with the human review of the information conducted during the engagement. This human review involved experienced professionals who carefully reviewed every document and assessed the organization’s adherence to each control. Many of the LLM’s outputs mirrored the results of the human-led assessment. The results that needed enhancements coming out of the LLM tool would have helped kickstart a discussion among the experienced professionals on the control and how it is implemented in the organization.
Image: As the image illustrates, in the same amount of time that our team reviewed the documents manually, our LLM based model was able to review exponentially more documents, with human oversight for accuracy.
So What?
Our LLM-generated process showed how technology can eliminate cumbersome and repetitive tasks in a review involving assessing client information against established criteria. Compliance officers, for example, can apply this approach to assess the design effectiveness of an organization’s compliance policies, processes, and controls. Outside of the compliance function, this approach can be applied to other departments that typically find themselves stretched, including tasks such as targeted contract review (General Counsel’s office), validation of documentation accuracy and completeness (Chief Technology Officer), and helping ensure ongoing messaging and content is aligned with company strategy (Chief Marketing Officer).
The process also demonstrated the superiority of using technology to analyze and funnel massive amounts of information across numerous documents and files in different formats into a single cohesive set.
Utilizing LLMs to create a “first draft” of observations enables human analysts to kickstart their analysis, although organizations must be cautious not to over-rely on LLM. The results are not 100% accurate because due to the nature of this new technology, and thus all output requires human review and oversight by subject matter experts to validate the outcomes and use the information to structure time-saving enhancements.
In the not-too-distant future, StoneTurn expects this technology to be able to complete a compliance document review from start to finish. This will free up more time for higher-level tasks, such as assisting with remediation, performing tests, and validating controls. It also provides experienced recommendations for enhancements to make compliance assessments increasingly robust and secure, offering a promising outlook for the future of compliance reviews. As technology continues to transform business, our professionals are staying one step ahead to provide the most timely, efficient, and effective use cases to help solve clients’ most pressing needs. Combining human experience with technological advancement can help drive results more quickly and more robustly as we move into the future.
If you have any questions or would like to find out more about this topic please reach out to Michael Costa, Kashif Sheikh or Jonny Frank.
To receive StoneTurn Insights, sign up for our newsletter.
