AI and LLM Logging Opportunities
We are in an era where artificial intelligence (“AI”) and large language model (“LLM”) initiatives are well in progress and part of a larger push to “capture fierce advantages possible with AI.”[1] Whether or not those advantages exist or translate to a measurable return[2] remains to be seen.
Current indications are that companies are marching forward with AI/LLM initiatives, including the experimentation, use, and deployment of such technologies internally and externally. While AI has the potential to help streamline processes and afford more time for high-value endeavors, there are also risks: it seems on a near weekly basis, another organization has to come clean about errors in legal filings or reporting because of AI hallucinations. Cybersecurity is a high-stakes area, similar to legal processes, where organizations can seldom afford to get something wrong. The fictional character “Dr. Ian Malcom” would likely classify this as an era where firms are preoccupied with whether or not they could integrate AI/LLM technologies, and not asking if they should. Cautionary tales are a reminder to exercise ethics, empathy, and maintain core literacy aptitudes (such as digital literacy, media literacy, and information literacy, to name a few) as entities explore any new technology.
Driven by an evolving cyber threat landscape, high-profile security journalism, and successful cyberattacks, cybersecurity initiatives have become standard practice across industries—often mandated by regulatory bodies. The basic level of “cybersecurity hygiene” that firms have today was previously uncommon or simply not considered years ago. Thanks to these evolved practices, organizations are better positioned to approach AI/LLMs from a cybersecurity angle today than the early 2010s when modern ransomware campaigns began.
For organizations of all sizes seeking to explore how AI/LLM technologies can be deployed or integrated into their workflow, several considerations should be undertaken in order to understand how this new technology can best be applied to maximize results and minimize security risks.
How Cybersecurity Teams Can Harness AI
In this article, we explore how two tools in this ecosystem, Microsoft Copilot and OpenWebUI, can be beneficial to cybersecurity teams as they seek better security for the organizations they support.
Microsoft Copilot
Within a Microsoft 365 (“M365”) environment, Copilot interactions are logged in the Unified Audit Log (“UAL”), a centralized record of user and administrator activities across various M365 services. Although the UAL retention period depends on the customer’s licensing tier, it is widely accepted as a core data source for security investigations, forensic analysis, and compliance auditing. The Audit logs for Copilot and AI applications documentation[3] is provided by Microsoft and is incredibly useful for analyzing records relevant to Copilot. Below are a few notable attributes for security teams beginning an investigation involving Copilot.
- AccessedResources includes references to all resources that Copilot accessed in response to the user’s request.
- Messages include prompt and response details, including a “JailbreakDetected” flag to indicate if the message contained an AI jailbreak attempt[4].
- Contexts include details about where the origin of the prompt, such as the file, app, or service the prompt was made from.
- RecordType includes the category of Copilot or AI application the account interacted with.
Microsoft’s own documentation provides detail and visibility into the possible records of interest. As Copilot evolves over time, corresponding records in the UAL are expected to be adjusted. It would behoove teams to bookmark and review the official documentation for review at scheduled intervals. For a more practical and technical dive on what is available today, check out Using Microsoft 365 Security Audit Logs to Track Copilot Usage by Martina Grom[5].
Open WebUI
Open WebUI[6] is a self-hosted platform designed to work with offline AI/LLM platforms such as Ollama[7] and LM Studio[8]. Open WebUI provides an Understanding Open WebUI document,[9] which users should review to better understand what information is logged and where. There are several application server/backend logging levels detailed, each which may provide a firm with useful targeted debugging and security operations. To get more detailed logs, including prompts, the global logging level must be changed to DEBUG or NOTSET by setting the GLOBAL_LOG_LEVEL environment variable. Test different logging levels to ensure that logs of value are captured versus verbose log which may provide little to no value.
A healthy amount of caution should be taken when adjusting logging levels to capture prompts. Ultimately, it’s not just logging a record of user input, it may also create a residual security and privacy concern. It’s not unreasonable to work under the assumption that those logs may contain sensitive, confidential, or proprietary information that users may have submitted either with or without authorization. It’s crucial for cybersecurity teams to treat log generation, availability, and retention with the same level of care as other sensitive data. By doing so, they will avoid creating a new data source that an adversary could leverage.
Conclusion
Third-party and local AI/LLM technologies may provide data useful for cybersecurity teams. Work with development and operations teams to understand the technology stack behind AI/LLM initiatives to capitalize on relevant logging opportunities. Ensure that logging, access, and retention better inform security operations and maintain alignment with operational security.
AI/LLM systems generate unique telemetry across multiple layers—from API calls and token usage to model interactions and data flows—that traditional security tools may not capture by default, if at all. Collaborate early with engineering teams to identify what data is available from AI platforms (inference logs, prompt/response metadata, authentication events, rate limiting triggers) and where it resides (cloud provider logs, application logs, model serving infrastructure). Establish retention policies that balance security investigation needs with privacy requirements and storage costs, particularly for sensitive data that may appear in prompts or outputs.
Reach out to Marc Padilla with any questions and find out more about how StoneTurn can help.
To receive StoneTurn Insights, sign up for our newsletter.
[1] https://corpgov.law.harvard.edu/2025/04/02/ai-in-focus-in-2025-boards-and-shareholders-set-their-sights-on-ai/
[2] https://www.forbes.com/sites/kolawolesamueladebayo/2025/08/18/why-enterprise-ai-still-cant-deliver-on-its-promise/
[3] https://learn.microsoft.com/en-us/purview/audit-copilot
[4] https://www.microsoft.com/en-us/security/blog/2024/06/04/ai-jailbreaks-what-they-are-and-how-they-can-be-mitigated/
[5] https://blog.atwork.at/post/Track-Copilot-Usage-with-Security-Audit-Logs
[6] https://openwebui.com/
[7] https://ollama.com/
[8] https://lmstudio.ai/
[9] https://docs.openwebui.com/getting-started/advanced-topics/logging
