Contents
- Tracing Concepts — Trace, Span, Context
- Dependencies — Micrometer + OTEL Bridge
- Exporting to Zipkin
- Exporting via OTLP (Grafana Tempo)
- Automatic Context Propagation
- Custom Spans with Tracer API
- Baggage — Carrying Data Across Services
- Log Correlation — traceId in Every Log Line
- Trace — the complete journey of one request across all services. Identified by a traceId (shared across all spans).
- Span — a single unit of work within a trace (e.g., an HTTP call, a DB query). Has a spanId, start time, duration, status, and tags.
- Parent span — a span that triggered another. Forms a tree of causality within a trace.
- Context propagation — the mechanism that passes traceId + spanId from one service to the next, typically via HTTP headers (traceparent for W3C TraceContext, or X-B3-TraceId for Zipkin B3).
- Baggage — arbitrary key-value data that travels with the trace context (e.g., tenant-id, user-id).
Spring Boot 3 uses Micrometer Tracing as the tracing API (replacing Spring Cloud Sleuth). It bridges to either OpenTelemetry SDK or Brave (Zipkin's library), keeping your application code independent of the backend.
<!-- Core tracing — Micrometer Tracing with OpenTelemetry bridge -->
<dependency>
<groupId>io.micrometer</groupId>
<artifactId>micrometer-tracing-bridge-otel</artifactId>
</dependency>
<!-- OTLP exporter (for Grafana Tempo, Jaeger, any OTEL collector) -->
<dependency>
<groupId>io.opentelemetry</groupId>
<artifactId>opentelemetry-exporter-otlp</artifactId>
</dependency>
<!-- OR: Zipkin exporter (Brave bridge) -->
<!--
<dependency>
<groupId>io.micrometer</groupId>
<artifactId>micrometer-tracing-bridge-brave</artifactId>
</dependency>
<dependency>
<groupId>io.zipkin.reporter2</groupId>
<artifactId>zipkin-reporter-brave</artifactId>
</dependency>
-->
<!-- Actuator needed for /actuator/health and trace sampling config -->
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-actuator</artifactId>
</dependency>
# application.yml
management:
tracing:
sampling:
probability: 1.0 # 1.0 = 100% of requests traced (use 0.1 in high-traffic prod)
zipkin:
tracing:
endpoint: http://zipkin:9411/api/v2/spans # for Brave/Zipkin bridge
<!-- Brave bridge + Zipkin reporter -->
<dependency>
<groupId>io.micrometer</groupId>
<artifactId>micrometer-tracing-bridge-brave</artifactId>
</dependency>
<dependency>
<groupId>io.zipkin.reporter2</groupId>
<artifactId>zipkin-reporter-brave</artifactId>
</dependency>
management:
tracing:
sampling:
probability: 1.0
zipkin:
tracing:
endpoint: http://localhost:9411/api/v2/spans
# Run Zipkin locally for development
docker run -d -p 9411:9411 openzipkin/zipkin
# Open http://localhost:9411 to browse traces
# application.yml — OTLP exporter to Grafana Tempo or any OTEL collector
management:
tracing:
sampling:
probability: 1.0
otlp:
tracing:
endpoint: http://tempo:4318/v1/traces # Grafana Tempo OTLP HTTP endpoint
# Or gRPC:
# endpoint: http://tempo:4317
# docker-compose.yml — Grafana stack for local tracing
services:
tempo:
image: grafana/tempo:latest
ports:
- "3200:3200" # query
- "4318:4318" # OTLP HTTP
command: ["-config.file=/etc/tempo.yaml"]
grafana:
image: grafana/grafana:latest
ports:
- "3000:3000"
environment:
- GF_AUTH_ANONYMOUS_ENABLED=true
OTLP (OpenTelemetry Protocol) is the vendor-neutral standard. Exporting via OTLP means you can switch between Jaeger, Tempo, Honeycomb, Datadog, or any OTEL-compatible backend by changing a single endpoint URL — no code changes.
With Micrometer Tracing on the classpath, Spring Boot auto-instruments all common I/O operations — no code changes needed:
- WebClient / RestClient — injects traceparent header on outbound requests
- RestTemplate — same with TracingClientHttpRequestInterceptor
- Feign — same with tracing interceptor auto-configured
- Spring MVC / WebFlux — extracts incoming traceparent and creates server spans
- Kafka — propagates via message headers (traceparent)
- JDBC / R2DBC — creates db spans for SQL queries
- @Scheduled / @Async — creates spans for scheduled and async methods
// Downstream service — trace context is extracted automatically from request headers
@RestController
public class OrderController {
@GetMapping("/orders/{id}")
public Order getOrder(@PathVariable Long id) {
// Micrometer creates a "GET /orders/{id}" server span automatically
// traceId from the incoming "traceparent" header is used
return orderService.findById(id);
}
}
// Upstream service calling downstream — traceparent injected automatically
@Service
@RequiredArgsConstructor
public class CheckoutService {
private final WebClient webClient;
public Mono<Inventory> checkInventory(Long productId) {
return webClient.get()
.uri("http://inventory-service/products/{id}/stock", productId)
// "traceparent: 00-{traceId}-{spanId}-01" added automatically
.retrieve()
.bodyToMono(Inventory.class);
}
}
import io.micrometer.tracing.Tracer;
import io.micrometer.tracing.Span;
@Service
@RequiredArgsConstructor
public class OrderService {
private final Tracer tracer;
public Order processOrder(CreateOrderRequest req) {
// Create a custom child span for a meaningful business operation
Span span = tracer.nextSpan()
.name("order.process")
.tag("order.customer-id", req.customerId().toString())
.tag("order.item-count", String.valueOf(req.items().size()))
.start();
try (Tracer.SpanInScope ws = tracer.withSpan(span)) {
Order order = doProcessOrder(req);
span.tag("order.id", order.id().toString());
span.event("order.confirmed");
return order;
} catch (Exception ex) {
span.error(ex);
throw ex;
} finally {
span.end();
}
}
}
// Simpler — @NewSpan and @ContinueSpan AOP annotations
@Service
public class InventoryService {
@NewSpan("inventory.reserve") // creates a new child span automatically
public boolean reserveStock(@SpanTag("product.id") Long productId, int qty) {
return doReserve(productId, qty);
}
}
import io.micrometer.tracing.BaggageField;
// Writing baggage at the entry point (e.g., API gateway or first service)
@Component
public class TenantBaggagePropagator implements Filter {
private static final BaggageField TENANT_ID = BaggageField.create("tenant-id");
@Override
public void doFilter(ServletRequest req, ServletResponse res, FilterChain chain)
throws IOException, ServletException {
String tenantId = ((HttpServletRequest) req).getHeader("X-Tenant-Id");
if (tenantId != null) {
TENANT_ID.updateValue(tenantId); // stored in trace context, propagated downstream
}
chain.doFilter(req, res);
}
}
// Reading baggage in any downstream service
@Service
public class TenantAwareService {
private static final BaggageField TENANT_ID = BaggageField.create("tenant-id");
public String getCurrentTenant() {
return TENANT_ID.getValue(); // extracted from incoming trace context headers
}
public List<Product> getProducts() {
String tenant = getCurrentTenant();
return productRepo.findByTenant(tenant);
}
}
Baggage is propagated in every outbound request header — keep values small (IDs, flags) and never put sensitive data (tokens, passwords) in baggage. Baggage adds overhead proportional to the number of services in the call chain.
# application.yml — enable MDC tracing fields in logs
logging:
pattern:
level: "%5p [${spring.application.name:},%X{traceId:-},%X{spanId:-}]"
# Logback automatically picks up traceId/spanId from MDC when Micrometer Tracing is present
# Sample log output — every line includes traceId and spanId
INFO [order-service,65f2c3a1e4b7d8f9,3a1e4b7d] Started OrderServiceApplication
INFO [order-service,65f2c3a1e4b7d8f9,3a1e4b7d] Processing order for customer 42
INFO [order-service,65f2c3a1e4b7d8f9,8c4f2b1e] Reserving stock for product 101
INFO [order-service,65f2c3a1e4b7d8f9,8c4f2b1e] Stock reserved successfully
# Grep all logs for a single request across all services using the traceId:
# grep "65f2c3a1e4b7d8f9" /var/log/order-service.log /var/log/inventory-service.log
// Add custom fields to MDC alongside the automatic traceId/spanId
@Component
public class UserContextLoggingFilter extends OncePerRequestFilter {
@Override
protected void doFilterInternal(HttpServletRequest req, HttpServletResponse res,
FilterChain chain) throws ServletException, IOException {
String userId = req.getHeader("X-User-Id");
try {
if (userId != null) MDC.put("userId", userId);
chain.doFilter(req, res);
} finally {
MDC.remove("userId");
}
}
}
In Grafana Loki, log lines are automatically linked to traces when they share the same traceId. Clicking a trace in Grafana Tempo will show correlated log lines — giving you the full picture of what happened without switching between tools.