We're excited to announce the release of @platformatic/kafka-opentelemetry, a comprehensive OpenTelemetry instrumentation package that brings enterprise-grade observability to your Kafka applications built with @platformatic/kafka.
Why We Built This
Modern distributed systems rely heavily on Apache Kafka for reliable message streaming, but gaining visibility into message flows, performance bottlenecks, and error patterns across your Kafka infrastructure has traditionally been challenging. With the growing adoption of OpenTelemetry as the standard for observability, we recognized the need for seamless integration between Platformatic's Kafka library and the OpenTelemetry ecosystem.
Key Features
🚀 Zero-Configuration Setup
Get started with comprehensive Kafka tracing in just a few lines of code. The instrumentation automatically detects and traces all producer and consumer operations without requiring changes to your existing Kafka code.
📊 Complete Observability Stack
Distributed Tracing: Full producer and consumer trace spans with proper parent-child relationships
Metrics Collection: Counters for sent/consumed messages and duration histograms for operations
Error Tracking: Automatic capture and correlation of Kafka errors with traces
🎯 OpenTelemetry Standards Compliance
Follows official OpenTelemetry semantic conventions for messaging systems, ensuring compatibility with all major observability platforms, including Jaeger, Zipkin, Honeycomb, DataDog, and more.
⚡ No Monkey Patching
Built using Node.js's native diagnostic_channel rather than monkey patching, ensuring reliable and performant instrumentation that won't interfere with your application's behavior.
Quick Start
npm install @platformatic/kafka-opentelemetry
import { NodeTracerProvider } from '@opentelemetry/sdk-trace-node'
import { ConsoleSpanExporter } from '@opentelemetry/sdk-trace-node'
import { SimpleSpanProcessor } from '@opentelemetry/sdk-trace-node'
import { registerInstrumentations } from '@opentelemetry/instrumentation'
import { KafkaInstrumentation } from '@platformatic/kafka-opentelemetry'
const provider = new NodeTracerProvider()
provider.addSpanProcessor(new SimpleSpanProcessor(new ConsoleSpanExporter()))
provider.register()
registerInstrumentations({ instrumentations: [new KafkaInstrumentation()]})
import { Producer, Consumer } from '@platformatic/kafka'
What Gets Traced
The instrumentation provides comprehensive visibility into:
Producer Operations: Message publishing with topic, partition, and key attributes
Consumer Operations: Message processing with full context propagation
API Operations: Internal Kafka protocol operations (Metadata, Produce, Fetch, etc.)
Error Conditions: Failed sends, processing errors, and connection issues
Metrics Out of the Box
Beyond tracing, you'll get essential Kafka metrics automatically:
messaging.client.sent.messages - Track producer throughput
messaging.client.consumed.messages - Monitor consumer processing
messaging.client.operation.duration - Kafka API operation latency
messaging.process.duration - Message processing time
Perfect for Modern Architectures
Whether you're building microservices, event-driven architectures, or real-time data pipelines, this instrumentation gives you the observability foundation needed for:
Performance Monitoring: Identify slow consumers and producers
Error Tracking: Quick root cause analysis of message failures
Capacity Planning: Understand traffic patterns and scaling needs
SLA Monitoring: Track message processing latency and success rates
TypeScript First
Built with full TypeScript support, providing an excellent developer experience with strong typing and IntelliSense support throughout.
Get Started Today
The package is available on npm and ready for production use. The GitHub repository has detailed documentation, examples, and contribution guidelines.
We're excited to see how this helps teams gain better visibility into their Kafka-based systems. Try it out and let us know what you think!
Built with ❤️ by the Platformatic team. Licensed under Apache-2.0.
