@platformatic/kafka Now Supports Confluent Schema Registry

If you run Kafka in production, you can’t skip schema evolution. Teams need clear data types, compatibility checks, and a safe way to update contracts without breaking consumers or downstream services.

Before now, using @platformatic/kafka with Confluent Schema Registry meant writing extra code to connect the pieces. With @platformatic/kafka v1.27.0, that’s no longer needed.

@platformatic/kafka now has built-in support for Confluent Schema Registry, including:

• AVRO

• Protocol Buffers

• JSON Schema

• Basic and Bearer authentication

• Automatic schema fetch and caching

• Integrated Producer and Consumer hooks

You get schema-aware messaging, and the project still focuses on being fast and predictable for Node.js Kafka clients.

Why This Matters

Most schema registry integrations add complexity where you don’t want it: in the message serialization and deserialization paths. Fetching remote schemas is asynchronous, but encoding and decoding should stay synchronous for speed and consistency.

Put simply, network I/O and cache coordination should happen before the main data processing, not during it. Keeping these steps separate helps maintain stable throughput and latency as traffic increases.

This release introduces a two-layer architecture to keep that separation clear:

1. Low-level hooks for async pre-processing:

   • beforeSerialization

   • beforeDeserialization

2. High-level registry API via ConfluentSchemaRegistry

In practice, this means schemas are fetched and cached before encode/decode happens, so your serializers and deserializers stay synchronous when messages are processed.

This gives application teams a simpler way to think about things: do the asynchronous prep first, then keep codec behavior predictable during main processing.

At a high level, the flow is:

• Extract schema ID from message metadata (producer) or wire payload (consumer).

• Resolve schema from local cache when available.

• On cache miss, fetch asynchronously via beforeSerialization/beforeDeserialization hooks and cache the schema.

• Run synchronous serialization/deserialization with the resolved schema.

In multi-instance deployments, that cache layer can be backed by Redis or Valkey, so workers share schema state across nodes while keeping encode/decode synchronous in the hot path.

What You Can Do Now

You can connect a registry directly to both the Producer and Consumer, letting @platformatic/kafka handle schema-aware serialization from start to finish.

This is especially helpful when several services publish and consume the same topics on different deployment cycles, since consistent schema handling is a must.

import { Consumer, Producer } from '@platformatic/kafka'
import { ConfluentSchemaRegistry } from '@platformatic/kafka/registries'

const registry = new ConfluentSchemaRegistry({
  url: 'http://localhost:8081'
})

const producer = new Producer({
  clientId: 'orders-producer',
  bootstrapBrokers: ['localhost:9092'],
  registry
})

const consumer = new Consumer({
  groupId: 'orders-consumers',
  clientId: 'orders-consumer',
  bootstrapBrokers: ['localhost:9092'],
  registry
})

When producing, pass schema IDs in message metadata:

await producer.send({
  messages: [
    {
      topic: 'orders',
      key: { orderId: 101 },
      value: { customerId: 'cust-44', total: 129.99 },
      metadata: {
        schemas: {
          key: 10,
          value: 11
        }
      }
    }
  ]
})

When consuming, payloads are automatically decoded with the cached schema. If a schema isn’t found, the registry fetches it before deserialization continues.

This makes it easy to move from custom codec code to a single registry integration in your client setup.

Authentication and Enterprise Scenarios

Schema Registry deployments are often protected. The new integration includes:

• Basic auth (username + password)

• Bearer token auth (token)

• Dynamic credentials via providers

This makes it easier to connect to managed or secured registry instances without writing custom transport code. It also makes credential rotation simpler when you use providers.

If your setup uses short-lived credentials, provider functions let you refresh tokens and secrets without having to rebuild your producer or consumer logic.

Performance and Reliability Considerations

One main design goal was to avoid unnecessary overhead to message processing.

The implementation focuses on cache locality and step-by-step pre-processing:

• Schema IDs are extracted from the wire format (or message metadata).

• Unknown schemas are fetched once and cached.

• Repeated schema IDs in a batch are resolved from the cache.

• Encode/decode continues in synchronous paths.

This setup cuts down on unnecessary async work while still supporting remote schema registries safely. It also helps keep throughput and performance steady, as you’d expect from a Node.js client.

Operationally, this also makes failures easier to understand. Schema resolution errors happen during fetch or preparation, while codec errors are still linked to payload and schema compatibility.

Also Included in This Release

The v1.27.0 release also shipped quality improvements around consumer behaviour and protocol handling, with broad test coverage and new playground clients for:

• AVRO

• Protobuf

• JSON Schema

• Authenticated Schema Registry setups

The end result is a production-ready integration you can try out quickly, starting in local development and moving to secure production registries.

Experimental API Notice

ConfluentSchemaRegistry and its related hooks are currently experimental. They may change in minor or patch releases as we keep improving them based on real-world use and feedback.

If you plan to use this in production, make sure to pin your versions and check the release notes. We’ll keep refining the API based on feedback from real deployments.

If your team is rolling this out, here’s a practical way to start:

1. Start with one topic and one schema format (typically AVRO or JSON Schema)

2. Validate serialization/deserialization behaviour in staging with real payloads.

3. Expand topic coverage and introduce auth/credential providers as needed.

Getting Started

Install the package:

npm install @platformatic/kafka

For Protobuf support, also install:

npm install protobufjs

Next, follow the full integration guide in the documentation:

• Confluent Schema Registry docs

• v1.27.0 release notes

If you give it a try, we’d love to hear your feedback at hello@platformatic.dev. Real-world schema workflows will help shape the next version of this API and guide our priorities for future improvements.

Thanks for building with us! 🚀