Gumdrop Telemetry

← Back to Main Page

Gumdrop includes a native OpenTelemetry implementation for distributed tracing, metrics, and observability. The telemetry system integrates with Gumdrop's event-driven architecture and exports data via OTLP/HTTP or OTLP/gRPC to any OpenTelemetry Collector, or to local JSONL files for offline analysis and debugging. Built-in instrumentation is provided for HTTP, SMTP, IMAP, POP3, and FTP servers, as well as the session replication cluster. The API enables custom instrumentation for any Gumdrop-based service.

Contents

• Overview
• Configuration
• Built-in Instrumentation
• Custom Instrumentation
• Metrics
• Distributed Tracing
• OTLP Exporter
• File Exporter (JSONL)
• JMX Bridge

Overview

The telemetry system provides:

• Distributed tracing - trace requests across service boundaries using W3C Trace Context (traceparent header)
• Metrics collection - counters, histograms, and gauges with configurable aggregation temporality (delta or cumulative)
• Hierarchical spans - represent operations as a tree of spans with timing, attributes, and events
• OpenTelemetry compatibility - exports via OTLP/HTTP or OTLP/gRPC to any OpenTelemetry Collector, or to JSONL files for local analysis
• Native integration - uses Gumdrop's HTTP client for non-blocking export with SelectorLoop affinity and endpoint pooling
• Zero-dependency - self-contained protobuf serialization, no external OpenTelemetry SDK required

Core Concepts

• Trace - a collection of spans representing a single request flow; identified by a 128-bit trace ID
• Span - a unit of work with name, timing, attributes, events, and status; forms parent-child relationships
• SpanKind - categorises the span's role: SERVER, CLIENT, PRODUCER, CONSUMER, or INTERNAL
• Attribute - key-value metadata attached to spans (string, boolean, integer, or double)
• Event - timestamped occurrence within a span (e.g., "message sent", "exception thrown")

Configuration

Telemetry is configured via the TelemetryConfig component in gumdroprc. HTTPS is strongly recommended for OTLP export to protect telemetry data in transit:

<component id="telemetry" class="org.bluezoo.gumdrop.telemetry.TelemetryConfig">
    <!-- Service identification -->
    <property name="service-name">my-mail-server</property>
    <property name="service-version">1.0.0</property>
    <property name="service-namespace">production</property>
    <property name="deployment-environment">prod-east</property>
    
    <!-- OTLP endpoint (OpenTelemetry Collector) - use HTTPS in production -->
    <!-- For HTTP/protobuf use port 4318; for gRPC use port 4317 -->
    <property name="endpoint">https://otel-collector:4318</property>
    <property name="protocol">http/protobuf</property>
    
    <!-- TLS configuration for OTLP export -->
    <property name="truststore-file">/etc/gumdrop/otlp-truststore.p12</property>
    <property name="truststore-pass">changeit</property>
    
    <!-- Metrics configuration -->
    <property name="metrics-temporality-name">cumulative</property>
    <property name="metrics-interval-ms">60000</property>
</component>

The presence of a TelemetryConfig on a server enables telemetry; there is no separate "enabled" flag. Individual signal types (traces, logs, metrics) can be disabled if needed.

Configuration Properties

Associating Telemetry with Servers

To enable telemetry for a server, reference the configuration:

<service id="smtp" class="org.bluezoo.gumdrop.smtp.SimpleRelayService">
    <property name="telemetry-config" ref="#telemetry"/>
    <listener class="org.bluezoo.gumdrop.smtp.SMTPListener" port="25"/>
</service>

<service id="http" class="org.bluezoo.gumdrop.servlet.ServletService">
    <property name="telemetry-config" ref="#telemetry"/>
    <listener class="org.bluezoo.gumdrop.http.HTTPListener">
        <property name="port">8080</property>
    </listener>
</service>

Built-in Instrumentation

HTTP Server

When telemetry is enabled, the HTTP server automatically creates spans for each request:

• Span name: "HTTP {method}" (e.g., "HTTP GET")
• SpanKind: SERVER
• Attributes:
  • http.method - request method
  • http.url - request URL
  • http.status_code - response status
  • http.client.address - client IP
• Distributed tracing: respects incoming traceparent header
• Response: includes traceparent header in response

Metrics:

SMTP Server

The SMTP server creates hierarchical traces for connections and sessions:

• Connection trace: root span covering the TCP connection
  • net.peer.ip - client IP address
  • net.transport - TCP
• Session spans: child spans for each SMTP transaction (MAIL FROM to RSET/QUIT)
  • smtp.session_number - session index
  • smtp.client_hostname - HELO/EHLO hostname
  • smtp.mail_from - sender address
  • smtp.auth.mechanism - authentication method
  • smtp.auth.user - authenticated user
• Events: RCPT TO recipients, STARTTLS, message acceptance

Metrics:

IMAP Server

The IMAP server creates hierarchical traces for connections and sessions:

• Connection trace: root span covering the TCP connection
• Session span: child span for the IMAP session
  • net.peer.ip - client IP address
  • net.transport - TCP
• Authenticated span: child span from authentication to logout
  • enduser.id - authenticated username
  • imap.auth.mechanism - authentication method
• Events: STARTTLS, SELECT mailbox, APPEND operations

Metrics:

POP3 Server

The POP3 server creates traces similar to IMAP:

• Connection trace: root span covering the TCP connection
• Session span: child span for the POP3 session
• Authenticated span: child span from authentication to QUIT
  • enduser.id - authenticated username
  • pop3.mailbox.count - number of messages
  • pop3.mailbox.size - total mailbox size
• Events: STARTTLS, message retrieval, message deletion

Metrics:

FTP Server

The FTP server creates traces for connections and file operations:

• Connection trace: root span covering the control connection
• Session span: child span for the FTP session
• Authenticated span: child span from login to QUIT
  • enduser.id - authenticated username
  • ftp.cwd - current working directory
• Events: AUTH TLS, file transfers (RETR, STOR), deletions

Metrics:

DNS Server

The DNS server creates spans for incoming queries and upstream proxy operations. Metrics are tagged with dns.transport to distinguish UDP, DoT, and DoQ listeners.

Metrics:

WebSocket Server

When telemetry is configured on the parent HTTP listener, WebSocketServerMetrics records connection and frame-level metrics.

Metrics:

Session Cluster

The session cluster provides metrics for distributed session replication when clustering is enabled for web applications:

Metrics:

Attribute values:

• context: The servlet context name (e.g., "/myapp")
• type (messages/bytes): "ping", "replicate", "passivate", "delta", or "fragment"
• type (replication duration): "full" or "delta"

Custom Instrumentation

Custom servers and handlers can add their own instrumentation using the telemetry API.

Creating Traces

// Get telemetry config (from server or connection)
TelemetryConfig config = server.getTelemetryConfig();

// Create a new trace with root span
Trace trace = config.createTrace("Process Order", SpanKind.SERVER);

// Or continue from incoming traceparent
String traceparent = request.getHeader("traceparent");
Trace trace = config.createTraceFromTraceparent(traceparent, "Process Order", SpanKind.SERVER);

Working with Spans

// Add attributes to current span
trace.addAttribute("order.id", orderId);
trace.addAttribute("customer.id", customerId);
trace.addAttribute("order.total", 149.99);

// Add events to mark significant points
trace.addEvent("Payment validated");
trace.addEvent("Inventory reserved");

// Start a child span for a sub-operation
Span childSpan = trace.startSpan("Query Database", SpanKind.CLIENT);
try {
    // ... database operation ...
    childSpan.setStatusOk();
} catch (Exception e) {
    childSpan.recordException(e);
} finally {
    childSpan.end();
}

// End the trace (exports automatically)
trace.end();

Span Status

// Set status explicitly
span.setStatusOk();
span.setStatusError("Validation failed");
span.setStatus(SpanStatus.UNSET);

// Record exception (sets ERROR status and adds event)
span.recordException(exception);

Example: Custom Endpoint Handler

public class MyProtocolHandler implements ProtocolHandler {
    
    private Trace endpointTrace;
    
    @Override
    public void connected(Endpoint endpoint) {
        // Create endpoint-level trace
        TelemetryConfig config = endpoint.getServerEndpoint().getTelemetryConfig();
        if (config != null && config.isTracesEnabled()) {
            endpointTrace = config.createTrace("MyProtocol endpoint", SpanKind.SERVER);
            endpointTrace.addAttribute("net.peer.ip", endpoint.getRemoteAddress());
        }
    }
    
    protected void processRequest(MyRequest request) {
        // Start a span for this request
        Span requestSpan = null;
        if (endpointTrace != null) {
            requestSpan = endpointTrace.startSpan("Process request");
            requestSpan.addAttribute("request.type", request.getType());
        }
        
        try {
            // Process request...
            if (requestSpan != null) {
                requestSpan.addEvent("Request processed");
                requestSpan.setStatusOk();
            }
        } catch (Exception e) {
            if (requestSpan != null) {
                requestSpan.recordException(e);
            }
            throw e;
        } finally {
            if (requestSpan != null) {
                requestSpan.end();
            }
        }
    }
    
    @Override
    public void disconnected() {
        if (endpointTrace != null) {
            endpointTrace.end();
        }
    }
}

Example: Servlet Instrumentation

public class InstrumentedServlet extends HttpServlet {
    
    @Override
    protected void doPost(HttpServletRequest req, HttpServletResponse resp)
            throws ServletException, IOException {
        
        // Get the stream's span (created automatically by HTTP server)
        Span httpSpan = ((GumdropHttpServletRequest) req).getSpan();
        
        if (httpSpan != null) {
            // Add application-specific attributes
            httpSpan.addAttribute("app.action", req.getParameter("action"));
            httpSpan.addAttribute("app.user", req.getRemoteUser());
            
            // Start child span for business logic
            Span bizSpan = httpSpan.startChild("Business Logic", SpanKind.INTERNAL);
            try {
                processBusinessLogic(req, resp);
                bizSpan.setStatusOk();
            } catch (Exception e) {
                bizSpan.recordException(e);
                throw e;
            } finally {
                bizSpan.end();
            }
        } else {
            // Telemetry disabled, just process normally
            processBusinessLogic(req, resp);
        }
    }
}

Metrics

Gumdrop provides OpenTelemetry-compatible metrics collection with synchronous and asynchronous instruments.

Instrument Types

Creating Instruments

// Get a meter from TelemetryConfig
Meter meter = config.getMeter("org.bluezoo.gumdrop.http");

// Synchronous counter
LongCounter requestCounter = meter.counterBuilder("http.server.requests")
    .setDescription("Total HTTP requests received")
    .setUnit("requests")
    .build();

// Synchronous histogram with explicit buckets
DoubleHistogram latencyHistogram = meter.histogramBuilder("http.server.duration")
    .setDescription("HTTP request latency")
    .setUnit("ms")
    .setExplicitBuckets(5, 10, 25, 50, 100, 250, 500, 1000)
    .build();

// Up-down counter for current state
LongUpDownCounter activeConnections = meter.upDownCounterBuilder("http.server.active_connections")
    .setDescription("Currently active connections")
    .build();

Recording Measurements

// Record with attributes
requestCounter.add(1, Attributes.of(
    "http.method", "GET",
    "http.status_code", 200
));

latencyHistogram.record(45.2, Attributes.of("http.method", "GET"));

// Track active connections
activeConnections.add(1);   // Connection opened
activeConnections.add(-1);  // Connection closed

Observable Instruments

Observable instruments use callbacks invoked at collection time, ideal for values maintained externally:

// Observable gauge for memory usage
meter.gaugeBuilder("process.memory.used")
    .setDescription("Used memory in bytes")
    .setUnit("bytes")
    .buildWithCallback(new ObservableCallback() {
        public void observe(ObservableMeasurement measurement) {
            Runtime rt = Runtime.getRuntime();
            measurement.record(rt.totalMemory() - rt.freeMemory());
        }
    });

// Observable counter for OS statistics
meter.observableCounterBuilder("process.cpu.time")
    .setDescription("CPU time used by process")
    .setUnit("ns")
    .buildWithCallback(new ObservableCallback() {
        public void observe(ObservableMeasurement measurement) {
            measurement.record(getProcessCpuTime());
        }
    });

Aggregation Temporality

Metrics can be exported with different temporalities:

• DELTA - Values represent change since last export. Preferred for push-based systems and stateless collectors.
• CUMULATIVE - Values represent total since process start. Preferred for Prometheus-style scrapers.

<property name="metrics-temporality-name">delta</property>

Distributed Tracing

Gumdrop supports W3C Trace Context for distributed tracing across services.

Incoming Traces

When a request includes a traceparent header, the server continues that trace rather than starting a new one. The format is:

traceparent: 00-{trace-id}-{span-id}-{flags}

Example: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01

Outgoing Requests

When making outbound requests with the HTTP client, call client.setTrace(state.getTrace()) before connect(). The traceparent header is then added automatically to every request, so the distributed trace remains connected without manual header handling:

HTTPClient client = new HTTPClient(loop, downstreamHost, downstreamPort);
client.setTrace(state.getTrace());  // Propagates traceparent automatically
client.connect(new HTTPClientHandler() {
    public void onConnected(Endpoint endpoint) {
        HTTPRequest request = client.post("/api");
        request.header("Content-Type", "application/json");
        request.send(responseHandler);
    }
    // ...
});

If you omit setTrace(), no traceparent header is sent and the downstream service will start a new trace. You can still add the header manually via request.header("traceparent", trace.getTraceparent()) if needed, but automatic propagation is preferred.

This creates a connected chain of spans across services, visible in trace visualization tools.

SpanKind for Distributed Calls

OTLP Exporter

The OTLP exporter sends telemetry data to an OpenTelemetry Collector. Two protocols are supported:

• OTLP/HTTP (default) - protocol="http/protobuf", uses port 4318, streams protobuf directly to HTTP endpoints. Maximally efficient for HTTP-based collectors.
• OTLP/gRPC - protocol="grpc", uses port 4317, sends gRPC-framed protobuf to collector gRPC services. Use when the collector expects gRPC.

HTTPS should be used in production to protect sensitive telemetry data including trace context, service identifiers, and custom attributes.

Architecture

• Uses Gumdrop's native HTTP client with SelectorLoop affinity
• Supports HTTP/2 and HTTP/1.1 with TLS (ALPN negotiation)
• Asynchronous batching reduces network overhead
• Background thread handles serialization and export
• Large payloads are chunked for efficient transfer
• gRPC mode uses the same protobuf encoding with gRPC framing

TLS Configuration

For HTTPS endpoints, configure a truststore containing the CA certificate(s) that signed the collector's certificate:

<component id="telemetry" class="org.bluezoo.gumdrop.telemetry.TelemetryConfig">
    <property name="service-name">my-service</property>
    <property name="endpoint">https://otel-collector:4318</property>
    
    <!-- Truststore containing the CA certificate for the collector -->
    <property name="truststore-file">/etc/gumdrop/otlp-truststore.p12</property>
    <property name="truststore-pass">changeit</property>
</component>

Create a PKCS12 truststore containing your CA certificate:

# Import a CA certificate into a new PKCS12 truststore
keytool -importcert -alias otel-ca \
    -file /path/to/ca-cert.pem \
    -keystore otlp-truststore.p12 \
    -storetype PKCS12 \
    -storepass changeit

If no truststore is configured and the endpoint uses HTTPS, the default JVM truststore will be used. This is suitable when the collector uses a certificate signed by a well-known CA.

Collector Configuration

Example OpenTelemetry Collector configuration with TLS to receive Gumdrop traces:

# otel-collector-config.yaml (HTTP receiver on 4318)
receivers:
  otlp:
    protocols:
      http:
        endpoint: 0.0.0.0:4318
        tls:
          cert_file: /etc/otel/server.crt
          key_file: /etc/otel/server.key

# For gRPC (port 4317), add:
#      grpc:
#        endpoint: 0.0.0.0:4317

exporters:
  jaeger:
    endpoint: jaeger:14250
    tls:
      insecure: true
  
  # Or export to other backends
  # zipkin:
  #   endpoint: http://zipkin:9411/api/v2/spans
  # otlp:
  #   endpoint: tempo:4317

service:
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [jaeger]

Authentication

For authenticated endpoints, use the headers property:

<property name="headers">Authorization=Bearer token123,X-Org-Id=my-org</property>

Complete Example Configuration

<!-- Telemetry configuration with HTTPS -->
<component id="telemetry" class="org.bluezoo.gumdrop.telemetry.TelemetryConfig">
    <property name="service-name">mail-gateway</property>
    <property name="service-version">2.1.0</property>
    <property name="deployment-environment">production</property>
    <property name="service-instance-id">mail-gw-01</property>
    
    <!-- HTTPS endpoint (recommended for production) -->
    <property name="endpoint">https://otel-collector.monitoring:4318</property>
    <property name="truststore-file">/etc/gumdrop/otlp-truststore.p12</property>
    <property name="truststore-pass">changeit</property>
    
    <property name="batch-size">256</property>
    <property name="flush-interval-ms">3000</property>
    <property name="metrics-interval-ms">30000</property>
</component>

<!-- SMTP service with telemetry -->
<service id="smtp" class="org.bluezoo.gumdrop.smtp.SimpleRelayService">
    <property name="telemetry-config" ref="#telemetry"/>
    <property name="handler-factory" ref="#smtpHandler"/>
    <listener class="org.bluezoo.gumdrop.smtp.SMTPListener" port="25"/>
</service>

<!-- HTTP servlet service with telemetry -->
<service id="http" class="org.bluezoo.gumdrop.servlet.ServletService">
    <property name="telemetry-config" ref="#telemetry"/>
    <property name="web-app" ref="#webapp"/>
    <listener class="org.bluezoo.gumdrop.http.HTTPListener">
        <property name="port">8080</property>
    </listener>
</service>

Shutdown Guarantees

Gumdrop ensures telemetry data is not lost during connection errors or JVM shutdown:

• Connection error handling - Read errors, write errors, and unexpected disconnects are recorded in telemetry with exception details before the trace ends
• Try-finally patterns - All protocol connections use try-finally blocks to guarantee telemetry spans are ended even when exceptions occur
• JVM shutdown hook - When an exporter is configured, a shutdown hook is automatically registered to flush pending telemetry data before the JVM exits
• Force flush - The OTLPExporter.forceFlush() method can be called programmatically to immediately export all pending data

The shutdown hook ensures that telemetry data buffered for batching is exported when the server is stopped gracefully. For immediate data visibility during development, reduce flush-interval-ms to export more frequently.

File Exporter (JSONL)

The OTLPFileExporter writes telemetry data to local files in OpenTelemetry OTLP JSON Lines format. Each line is a complete OTLP JSON object (ExportTraceServiceRequest, ExportLogsServiceRequest, or ExportMetricsServiceRequest) followed by a newline.

The file exporter is useful for:

• Local development - inspect telemetry output without running a collector
• Offline analysis - process JSONL files with standard tools (jq, grep, custom scripts)
• Debugging - capture telemetry data for later review
• Batch ingestion - feed JSONL files into pipelines or data lakes
• Testing - verify instrumentation produces correct spans and metrics

Architecture

• Separate .jsonl files for each signal type (traces, logs, metrics) as required by the specification
• When no file paths are configured, all signals are written to stdout
• Background thread handles serialization and flushing, avoiding blocking the caller
• Buffered I/O with configurable buffer size (default 8KB)
• Files are opened in append mode; parent directories are created automatically
• Graceful shutdown flushes all pending data before closing files

Configuration

Set exporter-type to file and provide paths for each signal type. Any signal without a configured path writes to stdout.

<component id="telemetry" class="org.bluezoo.gumdrop.telemetry.TelemetryConfig">
    <property name="service-name">my-service</property>
    <property name="exporter-type">file</property>

    <!-- Separate files per signal type -->
    <property name="file-traces-path">/var/log/otel/traces.jsonl</property>
    <property name="file-logs-path">/var/log/otel/logs.jsonl</property>
    <property name="file-metrics-path">/var/log/otel/metrics.jsonl</property>

    <!-- Optional: tune I/O buffer size -->
    <property name="file-buffer-size">16384</property>

    <property name="metrics-interval-ms">60000</property>
</component>

Stdout Mode

When no file paths are set, the exporter writes all signals to stdout. This is the simplest way to see telemetry output during development:

<component id="telemetry" class="org.bluezoo.gumdrop.telemetry.TelemetryConfig">
    <property name="service-name">my-service</property>
    <property name="exporter-type">file</property>
    <!-- No file paths = all output goes to stdout -->
</component>

Working with JSONL Output

The output files use standard JSON Lines format, one JSON object per line. You can process them with common tools:

# View traces with jq
cat /var/log/otel/traces.jsonl | jq .

# Find slow spans (duration > 1s)
cat /var/log/otel/traces.jsonl | jq '
  .resourceSpans[].scopeSpans[].spans[]
  | select(.endTimeUnixNano - .startTimeUnixNano > 1000000000)'

# Count metrics by name
cat /var/log/otel/metrics.jsonl | jq -r '
  .resourceMetrics[].scopeMetrics[].metrics[].name' | sort | uniq -c

# Tail traces in real time
tail -f /var/log/otel/traces.jsonl | jq .

JMX Bridge

When metrics are enabled, Gumdrop exposes OpenTelemetry metrics via JMX MBeans. This allows JMX-based monitoring tools (jconsole, VisualVM, Prometheus JMX exporter) to access the same metrics without requiring OTLP export. OpenTelemetry remains the single source of truth; the JMX bridge reads from it on each attribute access.

Object Name and Attributes

Metrics are exposed under the domain org.bluezoo.gumdrop with type Telemetry:

org.bluezoo.gumdrop:type=Telemetry

Metric names use underscores instead of dots (e.g., http.server.requests becomes http_server_requests). Counters and gauges appear as single numeric attributes. Histograms expose four attributes per metric:

Example attributes for HTTP server metrics:

• http_server_requests - Total requests (counter)
• http_server_active_requests - Requests in progress (gauge)
• http_server_active_connections - Active connections (gauge)
• http_server_request_duration_count, _sum, _min, _max - Request duration histogram

Configuration

The JMX bridge is enabled by default when metrics are enabled. Disable it with:

<component id="telemetry" class="org.bluezoo.gumdrop.telemetry.TelemetryConfig">
    <property name="metrics-enabled">true</property>
    <property name="jmx-bridge-enabled">false</property>
</component>

The bridge registers automatically during TelemetryConfig.init() and unregisters on shutdown. No OTLP or file exporter is required; the JMX bridge works with metrics alone.

Using with Monitoring Tools

jconsole / VisualVM: Connect to the Gumdrop process and navigate to org.bluezoo.gumdrop → Telemetry to view metric values.

Prometheus JMX Exporter: Configure the Prometheus JMX Exporter to scrape the Gumdrop JVM. The exporter will discover the Telemetry MBean and expose its attributes as Prometheus metrics.

← Back to Main Page | HTTP Server & Client | SMTP Server & Client | IMAP Server | POP3 Server

Gumdrop Telemetry