Enforce continuation diagnostics on smoke tests

.agents/skills/apm-integrations/SKILL.md

@@ -150,8 +150,11 @@ After `updateAgentJarIntegrationsGoldenFile` runs, commit the updated `metadata/
 
 **If `checkDecoratorAnalyticsConfigurations` fails:** a name returned by the decorator's `instrumentationNames()` is missing `DD_TRACE_<NAME>_ANALYTICS_ENABLED` / `DD_TRACE_<NAME>_ANALYTICS_SAMPLE_RATE` entries in `metadata/supported-configurations.json` — add them per [Supported Configurations](references/supported-configurations.md).
 
-**If tests fail:** verify span lifecycle order (start → activate → error → close → finish), helper registration,
-and `contextStore()` map entries match actual usage.
+**If tests fail:** verify span lifecycle order (start → activate → error → close → finish), helper
+registration, and `contextStore()` map entries match actual usage. If the output contains
+`Scope/continuation timeline`, read and follow
+`.agents/skills/fix-continuation-leakage/SKILL.md`; fix the broken ownership edge rather than
+adding `strictTraceWrites(false)` or disabling the diagnostic.
 
 ## Step 11 – Checklist before finishing
 

.agents/skills/fix-continuation-leakage/SKILL.md

@@ -0,0 +1,109 @@
+---
+name: fix-continuation-leakage
+description: >
+  Diagnose and fix scope or continuation lifecycle failures in dd-trace-java instrumentation
+  tests. Use when a test reports a continuation leak, double resolution, activation after resolve,
+  or an unclosed scope, or when strictTraceWrites(false) appears to hide one. Reads the automatic
+  diagnostic timeline, finds the broken lifecycle edge, fixes it, and explains it with a compact
+  Mermaid diagram.
+user-invocable: true
+context: fork
+allowed-tools:
+  - Bash
+  - Read
+  - Edit
+  - Glob
+  - Grep
+  - AskUserQuestion
+---
+
+# Fix continuation leakage
+
+Instrumentation tests run the diagnostic automatically. A failure includes the capture, resume,
+resolution, scope, thread, timing, and callsite data needed to find the missing lifecycle edge.
+
+## Work the failure
+
+1. Resolve the actual Gradle module path and failing suite from CI or the module's tasks. Nesting
+   varies (for example, `netty:netty-4.1` versus `kotlin-coroutines-1.3`). Use `forkedTest` for
+   `*ForkedTest*` classes, or the suite-specific forked task such as `latestDepForkedTest`.
+   Run the smallest failing test with full output:
+
+```bash
+set -o pipefail
+./gradlew :dd-java-agent:instrumentation:<module-path>:<test-task> --tests '<FQCN-or-pattern>' --info 2>&1 | tee /tmp/scopediag-run.txt
+```
+
+2. Verify the selected test ran in that task's XML results; a successful build with no discovered
+   tests is not validation. Find `Scope/continuation timeline` in the output. If Gradle hides it, inspect the test XML's
+   `<system-out>` under the module's `build/test-results` directory.
+3. Follow the failing record from its first event:
+   - `LEAKED` / `NEVER_CLOSED`: find the success, error, cancellation, and rejection exits that
+     skipped `release()` or `close()`.
+   - `DOUBLE_FINISH`: find two owners of the same cleanup.
+   - `ACTIVATE_AFTER_RESOLVE`: find work scheduled after ownership ended.
+   - `LATE_FINISH` / `CLOSE_WRONG_THREAD`: advisory evidence; verify whether ordering is valid.
+   - `[deferred-cleanup]`: a root iteration scope transferred cleanup to the bounded iteration
+     cleaner. It may remain open at the test boundary and is not a leak. Do not generalize this to
+     other `ITERATION` scopes; an unregistered iteration scope must still close normally.
+4. Classify the captured work before changing code:
+   - For a real asynchronous operation, repair success, failure, cancellation, and rejection
+     cleanup.
+   - If the test started the work, wait for its terminal event and dispose or close it before the
+     test ends.
+   - If a framework initializer creates a permanent sentinel with no context consumer, disable
+     propagation only around that creation boundary. Match the exact type and method, and update
+     `knownMatchingTypes()` when shortcut matching is used.
+     Check static initialization (`<clinit>`): first use under an active request can capture its
+     context in singleton tasks, including shaded Netty `GlobalEventExecutor` sentinels. Reproduce
+     first use in a fresh JVM; prewarming can hide the bug. Do not suppress all class initializers.
+   - If an executor replaces a task before delegating, avoid capturing the discarded task while
+     preserving capture for the task actually submitted.
+   - For intentionally delayed work, wait for its documented terminal event rather than
+     suppressing propagation.
+   - For a context swap, verify both restoration and resource cleanup. Restore or close the
+     returned ownership object in `finally`; do not ignore every swap.
+5. Prefer a test-lifecycle fix when production behavior is correct. Otherwise fix ownership where
+   it breaks, with one owner and `try/finally` cleanup across every exit.
+6. Rerun the failing test, then its module. Validate the leaked record and root-trace publication
+   separately from trace-count or arrival-order assertions; fixing a leak may expose an unrelated
+   flaky assertion.
+
+## Fixture setup failures
+
+Automatic recording may start after `setupSpec()` or equivalent fixture initialization. For an
+initialization error or a trace wait inside setup, temporarily record around that setup block and
+remove the diagnostic scaffolding after finding the owner.
+
+Apply process-wide configuration before starting servers, actor systems, executors, or other
+long-lived fixtures. Use a forked test or recreate the fixture when its static state cannot be
+reset safely.
+
+## Do not hide evidence
+
+Do not make the test green with `strictTraceWrites(false)` or
+`@TrackScopeContinuations(enabled=false, reason="...")`. Those hide evidence. The opt-out requires
+a reason and is only for a proven diagnostic incompatibility. If the failure is genuinely
+intermittent, treat that as a flaky-test finding, keep diagnostics enabled, and link the `@Flaky`
+annotation to a tracked issue.
+
+## Assess production impact
+
+An unresolved continuation blocks normal reference-count completion, not necessarily publication.
+The production `PendingTrace` buffer can still write finished spans; strict tests remove that
+delayed-write fallback, but partial flush remains possible. Do not infer lost traces or a fixed
+UI delay from a diagnostic failure. Check the collector and configuration; see
+[continuation effects](../../../docs/how_instrumentations_work.md#continuation-effects).
+
+Memory retention requires a reachable owner of the continuation/context; it does not prove the
+whole trace remains retained or memory grows without bound. Wrong parentage requires activation
+of unrelated context or a leaked active scope. Separate the observed lifecycle defect from its
+possible production effects and from test-only cleanup failures.
+
+## Explain it to a human
+
+Lead with one sentence: what was captured, which cleanup edge was missing, and where. Cite the
+timeline callsites. Then include a small Mermaid `flowchart LR`; use green for healthy edges, red
+for the broken edge, and label thread handoffs. Use a Gantt only when timing itself caused the bug.
+
+End with the code fix and the exact tests that passed.

.claude/skills/fix-continuation-leakage/SKILL.md

@@ -0,0 +1,23 @@
+---
+name: fix-continuation-leakage
+description: >
+  Diagnose and fix scope or continuation lifecycle failures in dd-trace-java instrumentation
+  tests. Use when a test reports a continuation leak, double resolution, activation after resolve,
+  or an unclosed scope, or when strictTraceWrites(false) appears to hide one. Reads the automatic
+  diagnostic timeline, finds the broken lifecycle edge, fixes it, and explains it with a compact
+  Mermaid diagram.
+user-invocable: true
+context: fork
+allowed-tools:
+  - Bash
+  - Read
+  - Edit
+  - Glob
+  - Grep
+  - AskUserQuestion
+---
+
+# Fix continuation leakage
+
+Read `.agents/skills/fix-continuation-leakage/SKILL.md` in full and follow it. That file is the
+shared playbook for repository agents.

dd-java-agent/agent-aiguard/build.gradle

@@ -22,6 +22,7 @@ dependencies {
   implementation project(':communication')
 
   testImplementation project(':utils:test-utils')
+  testImplementation project(':dd-trace-core')
   testImplementation libs.bundles.junit5
   testImplementation libs.bundles.mockito
   testImplementation('org.skyscreamer:jsonassert:1.5.3')

dd-java-agent/agent-aiguard/src/main/java/com/datadog/aiguard/AIGuardInternal.java

@@ -364,7 +364,7 @@ public Evaluation evaluate(final List<Message> messages, final Options options)
               .url(url)
               .method("POST", new MoshiJsonRequestBody(moshi, messages, meta));
       headers.forEach(request::header);
-      try (final Response response = client.newCall(request.build()).execute()) {
+      try (final Response response = executeRequest(tracer, request.build())) {
         final Map<String, Object> result;
         try {
           result = parseResponseBody(response);
@@ -455,6 +455,18 @@ public Evaluation evaluate(final List<Message> messages, final Options options)
     }
   }
 
+  private Response executeRequest(final AgentTracer.TracerAPI tracer, final Request request)
+      throws IOException {
+    final boolean asyncPropagation = tracer.isAsyncPropagationEnabled();
+    try {
+      // The synchronous HTTP call must not pass its context to pool maintenance threads.
+      tracer.setAsyncPropagationEnabled(false);
+      return client.newCall(request).execute();
+    } finally {
+      tracer.setAsyncPropagationEnabled(asyncPropagation);
+    }
+  }
+
   @SuppressWarnings("unchecked")
   private Map<String, Object> parseResponseBody(final Response response) throws IOException {
     final ResponseBody body = response.body();

dd-java-agent/agent-aiguard/src/test/java/com/datadog/aiguard/AIGuardContextPropagationTest.java

@@ -0,0 +1,143 @@
+package com.datadog.aiguard;
+
+import static java.util.Collections.emptyMap;
+import static java.util.Collections.singletonList;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertSame;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+import static org.mockito.ArgumentMatchers.any;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.verify;
+import static org.mockito.Mockito.when;
+
+import datadog.context.ContextScope;
+import datadog.trace.api.aiguard.AIGuard;
+import datadog.trace.bootstrap.instrumentation.api.AgentSpan;
+import datadog.trace.bootstrap.instrumentation.api.AgentTracer;
+import datadog.trace.common.writer.ListWriter;
+import datadog.trace.core.CoreTracer;
+import datadog.trace.core.DDSpan;
+import java.io.IOException;
+import java.util.List;
+import java.util.Properties;
+import okhttp3.Call;
+import okhttp3.HttpUrl;
+import okhttp3.MediaType;
+import okhttp3.OkHttpClient;
+import okhttp3.Protocol;
+import okhttp3.Request;
+import okhttp3.Response;
+import okhttp3.ResponseBody;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.tabletest.junit.TableTest;
+
+class AIGuardContextPropagationTest {
+  private AgentTracer.TracerAPI originalTracer;
+  private CoreTracer tracer;
+  private ListWriter writer;
+
+  @BeforeEach
+  void setUp() {
+    originalTracer = AgentTracer.get();
+    writer = new ListWriter();
+  }
+
+  @AfterEach
+  void tearDown() {
+    AgentTracer.forceRegister(originalTracer);
+    tracer.close();
+  }
+
+  @TableTest({
+    "scenario                    | callerPropagation | outcome | scopeDepthLimit",
+    "allow with evaluation scope | {true, false}     | ALLOW   | 100            ",
+    "block with evaluation scope | {true, false}     | BLOCK   | 100            ",
+    "error with evaluation scope | {true, false}     | ERROR   | 100            ",
+    "allow at scope depth limit  | {true, false}     | ALLOW   | 1              ",
+    "block at scope depth limit  | {true, false}     | BLOCK   | 1              ",
+    "error at scope depth limit  | {true, false}     | ERROR   | 1              "
+  })
+  void isolatesHttpWorkAndRestoresCaller(
+      boolean callerPropagation, Outcome outcome, int scopeDepthLimit) throws Exception {
+    Properties properties = new Properties();
+    properties.setProperty("trace.scope.depth.limit", Integer.toString(scopeDepthLimit));
+    tracer =
+        CoreTracer.builder()
+            .withProperties(properties)
+            .writer(writer)
+            .strictTraceWrites(true)
+            .build();
+    AgentTracer.forceRegister(tracer);
+    HttpUrl url = HttpUrl.get("http://localhost/evaluate");
+    OkHttpClient client = mock(OkHttpClient.class);
+    Call call = mock(Call.class);
+    when(client.newCall(any(Request.class))).thenReturn(call);
+    when(call.execute())
+        .thenAnswer(
+            invocation -> {
+              DDSpan span = (DDSpan) tracer.activeSpan();
+              assertEquals(
+                  scopeDepthLimit == 1 ? "parent" : "ai_guard", span.getOperationName().toString());
+              assertFalse(tracer.isAsyncPropagationEnabled());
+              if (outcome == Outcome.ERROR) {
+                throw new IOException("Transport failed");
+              }
+              String action = outcome == Outcome.BLOCK ? "DENY" : "ALLOW";
+              return new Response.Builder()
+                  .request(new Request.Builder().url(url).build())
+                  .protocol(Protocol.HTTP_1_1)
+                  .code(200)
+                  .message("OK")
+                  .body(
+                      ResponseBody.create(
+                          MediaType.get("application/json"),
+                          "{\"data\":{\"attributes\":{\"action\":\""
+                              + action
+                              + "\",\"is_blocking_enabled\":true}}}"))
+                  .build();
+            });
+    AIGuardInternal evaluator = new AIGuardInternal(url, emptyMap(), client);
+    List<AIGuard.Message> messages = singletonList(AIGuard.Message.message("user", "Hello"));
+    AgentSpan parent = tracer.startSpan("test", "parent");
+    try (ContextScope ignored = tracer.activateSpan(parent)) {
+      tracer.setAsyncPropagationEnabled(callerPropagation);
+      if (outcome == Outcome.ERROR) {
+        assertThrows(
+            AIGuard.AIGuardClientError.class,
+            () -> evaluator.evaluate(messages, AIGuard.Options.DEFAULT));
+      } else if (outcome == Outcome.BLOCK) {
+        assertThrows(
+            AIGuard.AIGuardAbortError.class,
+            () -> evaluator.evaluate(messages, AIGuard.Options.DEFAULT));
+      } else {
+        assertEquals(
+            AIGuard.Action.ALLOW,
+            evaluator.evaluate(messages, AIGuard.Options.DEFAULT).getAction());
+      }
+      assertSame(parent, tracer.activeSpan());
+      assertEquals(callerPropagation, tracer.isAsyncPropagationEnabled());
+      verify(call).execute();
+    } finally {
+      parent.finish();
+    }
+    writer.waitForTraces(1);
+    assertEquals(1, writer.size());
+    assertEquals(2, writer.get(0).size());
+    DDSpan evaluation =
+        writer.get(0).stream()
+            .filter(span -> "ai_guard".contentEquals(span.getOperationName()))
+            .findFirst()
+            .get();
+    assertEquals(parent.getSpanId(), evaluation.getParentId());
+    assertTrue(evaluation.isFinished());
+  }
+
+  enum Outcome {
+    ALLOW,
+    BLOCK,
+    ERROR
+  }
+}

dd-java-agent/agent-bootstrap/src/main/java/datadog/trace/bootstrap/instrumentation/java/lang/VirtualThreadState.java

@@ -2,6 +2,7 @@
 
 import datadog.context.Context;
 import datadog.context.ContextContinuation;
+import datadog.trace.api.GenericClassValue;
 
 /**
  * This class holds the saved context and scope continuation for a virtual thread.
@@ -10,6 +11,15 @@
  * mount/unmount.
  */
 public final class VirtualThreadState {
+  private static final ClassValue<Boolean> PROPAGATE_CONTEXT =
+      GenericClassValue.of(
+          type -> !type.getName().equals("jdk.internal.net.http.HttpClientImpl$SelectorManager"));
+
+  /** The HTTP selector belongs to its client, not to the request creating the client. */
+  public static boolean shouldPropagateContext(Runnable task) {
+    return PROPAGATE_CONTEXT.get(task.getClass());
+  }
+
   /** The virtual thread's saved context (scope stack snapshot). */
   private Context context;
 

dd-java-agent/agent-tooling/src/main/resources/datadog/trace/agent/tooling/bytebuddy/matcher/ignored_class_name.trie

@@ -107,6 +107,7 @@
 0 sun.rmi.server.*
 0 sun.rmi.transport.*
 0 sun.nio.ch.FileChannelImpl
+0 sun.nio.ch.Poller$Pollers
 # USM needs it
 0 sun.security.ssl.*
 0 javax.net.ssl.SSLSocket
@@ -133,6 +134,7 @@
 0 akka.http.impl.engine.client.PoolMasterActor
 0 akka.http.impl.engine.client.pool.NewHostConnectionPool$*
 0 akka.http.impl.engine.http2.Http2Ext
+0 akka.http.impl.engine.server.HttpServerBluePrint$TimeoutAccessImpl
 0 akka.http.impl.engine.server.HttpServerBluePrint$TimeoutAccessImpl$*
 0 akka.http.impl.util.StreamUtils$*
 # saves ~0.1s skipping ~233 classes