Key takeaways
- Empirical research from Unit 42 analyzed over 40,000 identities across 125 cloud environments, proving that runtime API calls form distinct functional behavioral clusters.
- Static Identity and Access Management (IAM) permissions only audit theoretical capability, leaving environments vulnerable to over-privileged roles and credential masquerading.
- An unsupervised machine learning pipeline using UMAP and HDBSCAN successfully mapped major functional roles, including console administrators, CI/CD runners, and backup services.
- High-dimensional cluster models can be distilled using L1-regularized logistic regression into concise, weighted rules that execute natively within standard SQL queries.
- Operationalizing behavioral baselines allows detection systems to catch anomalous privilege abuse without the performance overhead of running continuous machine learning pipelines.
Analyzing cloud audit logs through unsupervised behavioral clustering enables security teams to classify identities by what they actually execute rather than relying on static, over-privileged permissions. Recent research from Unit 42 demonstrates that clustering over 40,000 Amazon Web Services identities across 125 enterprise environments reveals distinct functional roles—such as administrators, backup services, and DevOps pipelines. By applying Uniform Manifold Approximation and Projection (UMAP) and Hierarchical Density-Based Spatial Clustering of Applications with Noise (HDBSCAN), teams can map empirical API usage and distill cluster definitions into lightweight SQL heuristics. This operational approach detects masquerading and privilege escalation without requiring continuous, resource-heavy machine learning pipelines.
Auditing Actual API Behavior Versus Static Permissions
Cloud security programs traditionally evaluate identity risk through Cloud Security Posture Management (CSPM), which inspects assigned policies and permissions within Identity and Access Management (IAM) configurations. While enforcing the principle of least privilege remains a best practice, enterprise environments frequently diverge from this ideal. Engineering teams accumulate excessive entitlements to avoid deployment bottlenecks, leaving long-lived credentials with broad administrative or multi-service capabilities that are never exercised during normal operations.
This operational gap creates a significant visibility dilemma for security operations centers. In AWS alone, more than 240 services expose over 15,000 discrete API operations. When an identity performs discovery operations—such as listing storage buckets or enumerating account roles—the intent depends entirely on context. A centralized inventory scanner executing these calls represents expected operational behavior; a backup service role invoking those same APIs suggests that an attacker has compromised credentials and begun reconnaissance.
Attackers routinely exploit this posture blind spot through masquerading techniques. Adversaries leverage existing over-privileged roles, blend into benign resource naming conventions, and execute unauthorized tasks that technically fall within the identity’s assigned IAM boundaries. Relying solely on configuration audits cannot distinguish legitimate entitlement usage from active exploitation. Cloud Detection and Response (CDR) must therefore shift focus from theoretical capability to observed runtime activity.
| Dimension | Static Permission Auditing (CSPM) | Behavioral Activity Analysis (CDR) |
|---|---|---|
| Primary Telemetry | IAM policies, role trust documents, SCPs | Cloud audit logs (e.g., AWS CloudTrail events) |
| Core Evaluation | What an identity is permitted to invoke | What an identity actively executes in production |
| Blind Spots | Over-privileged dormant access, benign masquerading | Cold-start identities, infrequent seasonal tasks |
| Operational Overhead | Periodic configuration and API scraping | Event stream parsing and feature vectorization |
| Detection Mechanism | Policy syntax matching and drift reporting | Statistical deviation from functional cluster baselines |
The Unsupervised Clustering Pipeline
To establish whether cloud identities follow consistent, predictable behavioral patterns across disparate organizations, Unit 42 constructed an unsupervised data processing pipeline. The architecture ingests raw event data and translates heterogeneous operational calls into structured mathematical coordinates through four distinct phases:
- Cloud Audit Log Ingestion: Raw management and data plane event records are captured from AWS CloudTrail across multiple accounts and aggregated for processing.
- Feature Vectorization: Every evaluated identity is mapped to a high-dimensional boolean vector across a global vocabulary of approximately 15,000 potential API operations. A vector index receives a value of true if the identity invoked the corresponding operation within the two-month observation window, and false otherwise. This yields an extremely sparse matrix dominated by zero values.
- Dimensionality Reduction via UMAP: High-dimensional sparse vectors present severe computational challenges for density calculations. Uniform Manifold Approximation and Projection compresses the feature space while preserving both local and global neighborhood structures. The pipeline employs cosine similarity as the distance metric, evaluating the directional alignment of active API sets rather than raw call volume. Dimensionality reduction executes in two parallel passes: an initial compression down to 32 continuous dimensions for clustering algorithms, and a secondary compression to two dimensions for spatial visualization.
- Density-Based Clustering via HDBSCAN: The 32-dimensional dense embeddings are processed by Hierarchical Density-Based Spatial Clustering of Applications with Noise. HDBSCAN identifies clusters across varying density gradients without requiring an arbitrary pre-selected cluster count, while systematically segregating anomalous identities as unclustered noise.
This multi-stage architecture demonstrated that cloud identities naturally coalesce into distinct behavioral clusters. In a dataset encompassing more than 40,000 identities across 125 environments, the 30 largest clusters accounted for approximately 20,000 identities, demonstrating that machine and human actors converge on specific functional patterns regardless of organizational boundaries.
Decoding Functional Roles Across 40,000 Identities
Once spatial clusters are generated, security analysts must decode the actual business function represented by each group. Relying strictly on identity names or resource tags is unreliable, as naming conventions differ widely and can be spoofed. The research applied four complementary analytical techniques to validate cluster identity:
- Operation Frequency Analysis: Measuring the percentage of identities within a cluster that invoke specific APIs. In the largest identified cluster, comprising roughly 5,000 identities across more than 100 cloud projects, 94% of identities executed
ConsoleLogin. In contrast, fewer than 1% of identities across all remaining clusters ever invoked this operation. Approximately 60% of this cluster also invoked default management console operations such asGetCostAndUsageandGetCostForecast, establishing a clear administrative user signature. - Class-Based TF-IDF (c-TF-IDF) Scoring: Adapting text-mining metrics to cloud operations uncovers distinguishing behaviors that may not dominate raw volume counts. Calculating c-TF-IDF scores identifies APIs that are frequent within a target cluster but exceptionally rare across the global population. For the console administrator group, operations such as
ListNotificationHubsachieved high uniqueness scores, serving as distinct behavioral markers. - Attribute-Based Spatial Mapping: Filtering the global map by specific parameters, such as API calls tied to Amazon S3, EC2, or AWS Lambda, provides structural validation of workload clusters. Overlaying metadata filters demonstrated that identities containing the substring
admincongregated tightly within the console login cluster. - Algorithmic Substring Mining: Rather than searching for subjective keywords, the pipeline utilized Generalized Suffix Trees to extract the most common naming substrings across clustered identities. Within the administrative cluster, one of the most frequent patterns was
AWSReservedSSO_AdministratorAccess_, the default role prefix created by AWS IAM Identity Center.
Extending this methodology across the remaining dataset revealed distinct, repeatable clusters for both interactive users and automated services. The clustering model successfully isolated profiles for continuous integration and delivery (CI/CD) systems, Infrastructure as Code (IaC) runners, monitoring and security tooling, backup agents, networking controllers, and FinOps cost-governance platforms.
Distilling Complex ML Into Production SQL Heuristics
While UMAP and HDBSCAN provide powerful analytical frameworks for offline research and baseline discovery, deploying them directly into production runtime detection introduces severe operational friction. Continuously running dense embedding transformations and spatial clustering pipelines over gigabytes of streaming audit logs demands substantial computing infrastructure, increases processing latency, and introduces black-box complexity that complicates incident triage.
To overcome this limitation, the research team demonstrated a model distillation methodology that bridges the gap between machine learning discovery and operational execution. Once target behavioral clusters are established, a binary logistic regression classifier is trained specifically on the original sparse boolean vectors to identify membership for that role.
Because the feature space encompasses 15,000 operations, standard regression models would assign marginal coefficients to thousands of irrelevant APIs. To eliminate noise, the training process applies L1 regularization (Lasso). The L1 penalty forces the mathematical weights of non-essential operations to absolute zero, retaining only a small, highly predictive subset of defining API calls.
This regularized classifier transforms the behavioral cluster into a transparent, explainable decision formula consisting of a few dozen weighted operations. Because inference simply calculates the weighted sum of observed API calls against a fixed threshold, the mathematical logic can be implemented directly within standard SQL queries in cloud data warehouses, security data lakes, or SIEM platforms.
Security engineering teams can run these SQL-based heuristics on scheduled query engines to continuously assign behavioral roles to new and evolving identities. When paired with Cloud Detection and Response telemetry, these dynamic role baselines supply critical context for rule evaluation, allowing platforms to fire high-confidence alerts aligned with MITRE ATT&CK techniques:
- Account Manipulation (T1098.003): Flagging unexpected IAM role modifications, cross-account trust adjustments, or sensitive S3 bucket ACL exposures when invoked by identities outside established administrative or IaC clusters.
- Impair Defenses (T1562): Identifying attempts to delete AWS Web Application Firewall (WAF) ACLs or disable logging when executed by accounts not associated with security management tooling.
- Inhibit System Recovery (T1490): Detecting AWS Backup vault deletion calls originating from non-backup service principals.
- Cloud Administration Command (T1651): Alerting on AWS Systems Manager (SSM) command execution when initiated by identities that typically operate exclusively within restricted data-plane boundaries.
Bottom line
Static IAM audits verify permission guardrails, but they cannot assess runtime intent or prevent credential misuse across over-privileged cloud accounts. The Unit 42 behavioral clustering research proves that cloud identities naturally fall into distinct, measurable operational roles when evaluated by active API invocations. By using unsupervised clustering to map the threat surface and distilling those findings into regularized SQL heuristics, security teams can implement scalable, explainable role classification. Organizations should integrate behavioral telemetry into their detection workflows to catch privilege abuse and defense evasion that bypass traditional posture checks.
