If you say the term "third party risk" within earshot of a CISO, you might see them start sweating and looking uncomfortable. Most companies these days have dozens or hundreds of integrations with third-party software vendors, and each one of those connections is a risk.

Ideally, you could enforce least privilege access with these third parties, monitor their activities, and respond when the third-party integrations misbehave. Realistically, however, this almost never happens. Security teams have too much to do, building a monitoring strategy for third parties is incredibly time intensive, and time-consuming to maintain.

Well...that was before LLMs. With the right integration between your logs and your engineers you can build detections that monitor your third parties in only a 3-5 minutes, seriously. Here's a concrete example of how we started monitoring the Vanta auditor role in our AWS account.

Baselining a third party's activity 

It didn't take much to get started in understanding what the Vanta Auditor is doing with the RunReveal MCP Server. We input a simple prompt into Cursor and 60 seconds later we had answers:

Prompt: Using runreveal, can you summarize the actions taken by the vanta integration in AWS, in the past 30 days?

Even though we gave it pointers about where to look first, it first started looking for available tables to then find out the structure of the logs it needed to query. The LLM discovered aws_cloudtrail_logs (a normalized version of CloudTrail's logs we have in RunReveal) and then started querying this table to answer the question we asked.

As a security engineer, this short but detailed overview is incredibly helpful. Accurately profiling a tool's operations over 30 days can take quite a bit of manual work and doing it via an LLM saves a bunch of time.

Based on that summary, it looks like the Vanta auditor was doing what I would expect it to be doing. It also covered errors that the Vanta auditor AWS role has run into and made a few basic recommendations. Neat!

This is great, but we aren't detecting anything quite yet...

Continuous monitoring of a third party

RunReveal provides an interface to write streaming detections using the standard Sigma format or query detections that search logs on a scheduled timer. What's really nice is that the MCP server RunReveal provides can test query detections against your live data, and most LLMs are very well-versed in generating YAML, so both types of detections are easy to manage.

Let's put this to the test and generate some detections that can do continuous monitoring of our vanta auditor role.

Prompt: Can you write sigma detections to alert when: There is any other action taken in the IAM? If the user attempts to use any other role than auditor?

We wanted to know roughly that the Vanta auditor role wouldn't try to do anything we deemed sensitive (in this case we specifically asked for IAM), and we also wanted to know if the role ever attempted to move laterally.

It's a simple and unscientific prompt, but that's all it took for the MCP server to generate three helpful detections. The first sigma rule looks for any sensitive IAM activity that the Vanta auditor did.

title: Vanta Unauthorized IAM Actions
id: 5f8c9d2a-3b1e-4c7f-9a6d-8e2b5c4f7a9b
status: experimental
description: Detects when Vanta performs unauthorized IAM actions beyond read-only audit activities
author: RunReveal
date: 2025/05/27
references:
  - https://docs.aws.amazon.com/IAM/latest/APIReference/
tags:
  - attack.privilege_escalation
  - attack.persistence
  - attack.t1078
logsource:
  product: aws
  service: cloudtrail
detection:
  selection_vanta:
    userIdentity.arn|contains: 'vanta'
    eventSource: 'iam.amazonaws.com'
  selection_unauthorized_actions:
    eventName:
      # User Management - Write Operations
      - 'CreateUser'
      - 'DeleteUser'
      - 'UpdateUser'
      - 'CreateLoginProfile'
      - 'UpdateLoginProfile' 
      - 'DeleteLoginProfile'
      - 'ChangePassword'
      - 'UpdateAccessKey'
      - 'CreateAccessKey'
      - 'DeleteAccessKey'
      - 'EnableMFADevice'
      - 'DeactivateMFADevice'
      - 'DeleteVirtualMFADevice'
      - 'CreateVirtualMFADevice'
      # Role Management - Write Operations
      - 'CreateRole'
      - 'DeleteRole'
      - 'UpdateRole'
      - 'UpdateAssumeRolePolicy'
      - 'AttachRolePolicy'
      - 'DetachRolePolicy'
      - 'PutRolePolicy'
      - 'DeleteRolePolicy'
      # Policy Management - Write Operations
      - 'CreatePolicy'
      - 'DeletePolicy'
      - 'CreatePolicyVersion'
      - 'DeletePolicyVersion'
      - 'SetDefaultPolicyVersion'
      - 'AttachUserPolicy'
      - 'DetachUserPolicy'
      - 'PutUserPolicy'
      - 'DeleteUserPolicy'
      - 'AttachGroupPolicy'
      - 'DetachGroupPolicy'
      - 'PutGroupPolicy'
      - 'DeleteGroupPolicy'
      # Group Management - Write Operations
      - 'CreateGroup'
      - 'DeleteGroup'
      - 'UpdateGroup'
      - 'AddUserToGroup'
      - 'RemoveUserFromGroup'
      # Instance Profile Management
      - 'CreateInstanceProfile'
      - 'DeleteInstanceProfile'
      - 'AddRoleToInstanceProfile'
      - 'RemoveRoleFromInstanceProfile'
      # Account Settings
      - 'UpdateAccountPasswordPolicy'
      - 'DeleteAccountPasswordPolicy'
  condition: selection_vanta and selection_unauthorized_actions
fields:
  - eventTime
  - eventName
  - eventSource
  - userIdentity.arn
  - userIdentity.principalId
  - sourceIPAddress
  - userAgent
  - errorCode
  - errorMessage
falsepositives:
  - Legitimate Vanta role expansion (should be coordinated)
  - Manual testing by security team using Vanta credentials
level: high

The next detection is watching for any unauthorized role assumption as prompted. Without too much trouble, the detection starts looking for the vanta-auditor role performing any AssumeRole calls that it shouldn't be, and any other unexpected role assumptions given the way the Vanta auditor role is currently used.

title: Vanta Unauthorized Role Assumption
id: 2d5f8a9c-6b4e-4c9f-8a7d-5e9b2c4f8a9d
status: experimental
description: Detects unauthorized role assumptions related to Vanta - either Vanta principals moving laterally to non-auditor roles, or external principals attempting to assume Vanta roles
author: RunReveal
date: 2025/06/04
references:
  - https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html
tags:
  - attack.privilege_escalation
  - attack.t1078.004
  - attack.lateral_movement
  - attack.t1550.001
logsource:
  product: aws
  service: cloudtrail
detection:
  selection_sts_assume_role:
    eventSource: 'sts.amazonaws.com'
    eventName: 'AssumeRole'
  
  # Scenario 1: Vanta-assumed identity tries to assume non-Vanta roles (lateral movement)
  vanta_lateral_movement:
    userIdentity.arn|contains: 'vanta'
    requestParameters|contains: 'roleArn'
  vanta_lateral_movement_filter:
    requestParameters|contains: 'vanta-auditor'
  
  # Scenario 2: External principals trying to assume any Vanta role
  external_to_vanta:
    requestParameters|contains: 'vanta'
  external_to_vanta_filter:
    userIdentity.arn|contains: 'vanta'
  
  # Scenario 3: Suspicious session patterns (non-standard Vanta session names)
  suspicious_vanta_session:
    requestParameters|contains: 'vanta'
    requestParameters|contains: 'roleSessionName'
  legitimate_vanta_session:
    requestParameters|contains: 'Vanta-'
  
  condition: 
    selection_sts_assume_role and (
      (vanta_lateral_movement and not vanta_lateral_movement_filter) or
      (external_to_vanta and not external_to_vanta_filter) or
      (suspicious_vanta_session and not legitimate_vanta_session)
    )

fields:
  - eventTime
  - eventName
  - userIdentity.arn
  - userIdentity.principalId
  - userIdentity.type
  - requestParameters
  - sourceIPAddress
  - userAgent
  - errorCode
  - errorMessage
  - awsRegion

falsepositives:
  - New authorized Vanta roles being deployed
  - Role name changes in Vanta infrastructure
  - Legitimate administrative access during Vanta setup
  - Vanta service updates that introduce new role patterns

level: high

It's worth noting that unprompted, the LLM recognized that the role only uses a single IP when connecting to our infrastructure and it also created a detection that would get triggered if there was any call in our logs that was using the role from a different IP. We checked the accuracy of this and were able to find docs indicating that this detection was indeed a good one to keep.

title: Vanta Suspicious Network Activity
id: 6a9c2d5f-4b8e-4e9f-9a7d-2e5b8c4f9a2d
status: experimental
description: Detects Vanta activity from unexpected IP addresses or user agents
author: RunReveal
date: 2025/05/27
references:
  - Internal Vanta integration documentation
tags:
  - attack.initial_access
  - attack.t1078
logsource:
  product: aws
  service: cloudtrail
detection:
  selection_vanta:
    userIdentity.arn|contains: 'vanta'
  filter_expected_ip:
    sourceIPAddress: '34.227.127.165'
  filter_expected_useragent:
    userAgent|startswith: 'aws-sdk-js/3.'
    userAgent|contains: 'exec-env/AWS_ECS_FARGATE'
  condition: selection_vanta and not (filter_expected_ip and filter_expected_useragent)
fields:
  - eventTime
  - eventName
  - eventSource
  - userIdentity.arn
  - sourceIPAddress
  - userAgent
  - awsRegion
  - errorCode
falsepositives:
  - Vanta infrastructure changes or updates
  - New Vanta deployment regions
  - SDK version updates
level: medium

But what's the other stuff in the detection about aws-sdk-js and AWS_ECS_FARGATE? Well it turns out that the credentials are passed around internally in AWS and reused. One AWS API call might result in numerous CloudTrail logs, and we verified that this detection was accurately filtering out internal AWS API calls where the source IP addresses come from the private class B ranges.

What's next for MCP

This investigation demonstrates how LLMs, when properly focused, can transform third-party security monitoring from a time-consuming manual process into an efficient, automated workflow. By combining RunReveal's MCP server with Claude's analytical capabilities, we were able to quickly fingerprint a third-party, understand which actions would be unusual, and come up with a basic monitoring strategy to look for new behavior in just a handful of minutes.

A key insight we've had is that LLMs excel when given specific goals and tools, but struggle with open-ended tasks. By combining LLMs with raw access to our security logs in a structured way, it can easily baseline the way things should operate, cover the threat scenarios you're worried about, and generate effective detections.

There's a lot of hype around LLMs and for good reason. This kind of task would have been extremely tedious and time-consuming. At RunReveal, we think of the LLMs as some of our hardest working teammates that are always willing to do the grunt work—and there's a lot more we'll be announcing soon about how LLMs can help security teams with their monitoring!