How to build a write-intensive system with Kafka and tRPC

Introduction

In recent times, data has been more than just bytes and bits. It's the backbone of countless industries and the heart of many modern applications. Systems today are not just required to read large volumes of data. They must also handle massive write operations. This is where write-intensive systems are introduced. These systems prioritize the ability to record data rapidly and efficiently, often needing to cater to high-velocity data sources like IoT sensors, financial transactions, or social media activity feeds.

In between scalability and reliability lies Apache Kafka. Kafka is renowned for its durability, fault tolerance, and high throughput capabilities, especially when it comes to write-heavy workloads. As a distributed event streaming platform, it facilitates real-time data pipelines and stream processing. On the other hand, tRPC is a modern framework for building typesafe APIs, that amplifies the power of RPC (Remote Procedure Call) with TypeScript's static typing. It promises a robust way to create and manage APIs, ensuring efficient communication between services.

Queue as an alternative for building a write-intensive system

Before we dive deep into Kafka, it's worth noting that many systems leverage queuing mechanisms to handle write-intensive workloads.

In traditional systems architecture, queues have been the cornerstone of handling bursts of data and ensuring that backend systems aren't overwhelmed. They act as buffers, storing messages or data temporarily until the consuming system or service is ready to process them. This asynchronous nature of queues helps in decoupling the producing and consuming systems, thereby ensuring that the speed mismatches between producers and consumers don't lead to system overloads.

For instance, think of an e-commerce platform during a Black Friday sale. The influx of user activity and orders is huge. A simple database might not be able to handle the writes directly, but with a queue, the orders can be processed in a controlled manner without crashing the system.

An example of Queue software is RabbitMQ. It is an open-source message broker that supports multiple messaging protocols. RabbitMQ is known for its robustness, ease of use, and support for a wide range of plugins. Queues can absorb large bursts of write operations, serving as a buffer.

Where to use Kafka

Kafka, in many ways, is an evolution of this concept, bringing in additional benefits like durability, scalability, and the ability to stream data in real time.

While traditional queuing systems are effective, they have their limitations, especially when data persistence, ordering, and real-time processing are paramount. This is where Kafka shines.

Real-world use cases

Given these strengths, Kafka is ideal for scenarios that involve real-time analytics, log aggregation, stream processing, and, of course, write-intensive systems where data durability and scalability are crucial.

The purpose and benefits of using Kafka and tRPC in building a write-intensive systems

Illustration of building a write-intensive system:

This diagram illustrates building a write-intensive system, emphasizing Kafka's data streaming and tRPC's type-safe APIs. It showcases foundational setups, optimization techniques, and the importance of resilience and peak performance.

Kafka provides a platform for handling vast amounts of data, but when it comes to interfacing or presenting that data to other systems, APIs become indispensable. tRPC's seamless integration with TypeScript, combined with its efficient and type-safe approach, complements Kafka's data-handling prowess. Together, they create a powerful ecosystem for building write-intensive systems.

Kafka's Strengths

tRPC's power

By combining Kafka's event streaming capabilities with tRPC's efficient API management, developers can build systems that not only handle massive write operations but also present this data to downstream services in a streamlined, typesafe, and efficient manner. The synergy between these tools provides a foundation that can support the demanding requirements of modern, data-heavy applications.

Write-intensive systems

In the evolving digital landscape, the demand for systems that primarily write data as opposed to reading it is surging. Such systems are the backbone for scenarios where real-time capture and processing of data are paramount.

Characteristics

Challenges

Use cases

Financial transaction systems, large-scale logging systems, real-time analytics, telemetry data capture from thousands of IoT devices, and event-driven architectures.

System design considerations for write-intensive workloads

Apache Kafka overview

Kafka, born at LinkedIn and later donated to the Apache Software Foundation, has rapidly become the go-to platform for event streaming. With its distributed architecture, it's uniquely positioned to handle vast amounts of data with low latency. Its design, centered around the immutable commit log, ensures that data is stored sequentially, allowing multiple consumers to read data in real-time or retrospectively. It makes it ideal for applications like write-intensive systems and more. Learn more here.

Key Features

In a write-intensive system, the flow and volume of data are paramount. The producer-broker-consumer model of Kafka aligns perfectly with this:

tRPC overview

By leveraging the strengths of TypeScript, tRPC has revolutionized the RPC paradigm, ensuring type safety in client-server communications. In systems where the integrity and shape of data are critical, tRPC's enforcement of type consistency is invaluable. It simplifies API development, streamlines client-server interactions, and integrates seamlessly with modern frameworks. For write-intensive systems, where efficient and error-free data exchange is paramount, tRPC offers a robust solution. Learn more here.

Key features and benefits

Use cases

Role in the system

The efficient transmission and processing of data are critical in a write-intensive system. tRPC stands as the bridge ensuring that data flows seamlessly between services, databases, and clients. Its typesafe nature further guarantees that this data remains consistent and reliable, making it an invaluable tool in the realm of such demanding systems.

Hosted Kafka solutions

While setting up Kafka manually provides a great deal of control over your infrastructure, it's worth considering hosted Kafka solutions like upstash to simplify the process and reduce operational overhead, especially when building write-intensive systems. Hosted Kafka services offer several advantages that can enhance the efficiency and scalability of your data streaming setup.

Upstash is a Serverless Data Platform with Redis and Kafka support. It offers a comprehensive Kafka hosting service that simplifies the implementation of Kafka for various use cases, including write-intensive systems.

Benefits of hosted Kafka solutions

While hosted solutions offer compelling benefits, it's essential to note that the choice between a self-hosted Kafka setup and a hosted solution depends on your project's specific requirements. Factors like control, budget, and customization needs should guide your decision-making process.

Implementing the system with Kafka

Before you start, it's important to set things up correctly. For a self-hosted manual setup, you'll need to follow these steps:

Downloading Kafka

You can download Kafka from the official Apache Kafka website. Choose the appropriate version based on your requirements.

Installation

After downloading, you extract the Kafka binaries and navigate into the Kafka directory:

tar -xzf kafka_2.13-2.8.0.tgz
cd kafka_2.13-2.8.0

This command extracts the downloaded Kafka tarball and then changes the directory to the Kafka installation.

Configuration

Kafka configurations are mostly held in the config directory. For starters, you might need to adjust the broker settings in server.properties.

For example, to change the default port:

listeners=PLAINTEXT://:9093

This configuration updates the Kafka broker to listen on port 9093.

Starting Zookeeper and Kafka

Kafka relies on Zookeeper for distributed cluster management. First, we start Zookeeper:

bin/zookeeper-server-start.sh config/zookeeper.properties

These commands first start the ZooKeeper, which Kafka uses for maintaining configurations and leadership elections among broker nodes. Following that, the Kafka broker itself is started.

Once Zookeeper is up, we initiate the Kafka server:

bin/kafka-server-start.sh config/server.properties

Creating Kafka producers in JavaScript

A Kafka producer sends messages (events) to Kafka topics.

Here's a flowchart illustrating the process of setting up a Kafka producer using JavaScript:

A: Initialize a Node.js project.

B: Install the kafka-node library.

C: Set up the Kafka producer.

D: Connect to the Kafka broker.

E: Wait for the producer to be ready.

F: Send a message to the Kafka topic.

G: Handle any errors and log the outputs.

Setting up the Node.js project

Initialize a new Node.js project and install the required library:

npm init -y
npm install kafka-node

Writing the producer code:

const kafka = require('kafka-node');

// Connect to a Kafka broker
const client = new kafka.KafkaClient({kafkaHost: 'localhost:9092'});
const producer = new kafka.Producer(client);

producer.on('ready', () => {
  // Message to send to the Kafka topic
  const messages = [{ topic: 'test-topic', messages: 'Hello Kafka' }];
  
  producer.send(messages, (err, data) => {
    if (err) console.log('Error:', err);
    else console.log('Message sent:', data);
  });
});

producer.on('error', (err) => {
  console.error('There was an error:', err);
});

Here:

Kafka brokers and topic partitions

At the core of Kafka are topics, brokers, and partitions:

Creating a topic

bin/kafka-topics.sh --create --topic test-topic --partitions 3 --replication-factor 1 --bootstrap-server localhost:9092

The above command creates a topic named test-topic with 3 partitions. The replication factor of 1 means data is not duplicated across brokers. This setup is okay for local development but in production, you might want a higher replication factor for fault tolerance.

Creating Kafka consumers in JavaScript

While producers send messages to topics, consumers read these messages.

Writing the Consumer Code:

const kafka = require('kafka-node');

const client = new kafka.KafkaClient({kafkaHost: 'localhost:9092'});
const consumer = new kafka.Consumer(client, [{ topic: 'test-topic', partition: 0 }]);

consumer.on('message', (message) => {
  console.log('Received Message:', message.value);
});

consumer.on('error', (err) => {
  console.error('There was an error:', err);
});

Here:

It's important to note that Kafka, with its extensive ecosystem, offers a wide range of settings and optimizations that are specifically designed for certain use cases. You'll occasionally need to adjust settings as you build and implement your write-intensive system to fit your particular requirements.

Implementing a system with tRPC

tRPC offers an easy way of creating typed end-to-end remote procedure calls. In the context of write-intensive systems, tRPC serves as a powerful tool to handle vast amounts of incoming data, transforming and ingesting it into the system.

Installation

npm install @trpc/client @trpc/server zod

Setting up a basic tRPC server

// server.js
const { createRouter, createServer } = require('@trpc/server');
const { z } = require('zod');

const router = createRouter()
  .query('hello', {
    resolve: () => 'Hello tRPC!',
  });

const server = createServer({
  router,
  createContext: () => ({}),
});

server.listen(4000);

Here:

Creating tRPC Procedures

Once you've set up your tRPC server, the next step involves defining procedures for various operations. Let's consider a use-case where the system should handle user registrations via Kafka events.

Defining a user registration procedure:

const { z } = require('zod');

const userSchema = z.object({
  id: z.string(),
  name: z.string(),
  email: z.string().email(),
});

const router = createRouter()
  .mutation('registerUser', {
    input: userSchema,
    resolve: async ({ input }) => {
      // Mock: Save to a database here
      console.log('User registered:', input);
      return { status: 'success' };
    },
});

Here:

Integrating tRPC with Kafka

With the tRPC procedure in place, the next step is to trigger it when a Kafka consumer reads a user registration event. The following diagram provides a step-by-step flow of how the Kafka consumer integrates with tRPC to handle user registration events.

Kafka Consumer with tRPC integration

const { createRouter } = require('@trpc/server');
const kafka = require('kafka-node');

const client = new kafka.KafkaClient({ kafkaHost: 'localhost:9092' });
const consumer = new kafka.Consumer(client, [{ topic: 'user-registrations', partition: 0 }]);

consumer.on('message', async (message) => {
  // Assume the message contains a user object for registration
  const user = JSON.parse(message.value);

  // Calling the tRPC procedure
  const response = await router.mutation.registerUser.resolve({ input: user });
  console.log(response);
});

consumer.on('error', (err) => {
  console.error('There was an error:', err);
});

Here:

With this, we've taken a dive into setting up a system using tRPC, integrating it with Kafka, and setting the stage for a write-intensive workload. The powerful combination of tRPC's type-safety and Kafka's robustness can help ensure data accuracy, resilience, and scalability.

Optimizing the system for high-volume writes

Building a write-intensive system with Kafka and tRPC is just the start. To cater to high-volume writes, we need to optimize the system for performance, resilience, and scalability.

Batch processing with Kafka

Batch processing is a method where a set of data is processed and stored in a group, or batch, rather than individually. Kafka supports batch processing out of the box.

const producer = new kafka.Producer(client, { requireAcks: 1, batchSize: 1000 });

By setting batchSize, you're indicating the number of messages that should be batched together.

Asynchronous writes

To further improve throughput, consider producing messages asynchronously:

const messagesToProduce = Array(1000).fill().map((_, i) => ({ topic: 'test-topic', messages: `Message ${i}` }));

messagesToProduce.forEach(message => {
  producer.send([message], (err, data) => {
    if (err) console.log('Error:', err);
  });
});

We're generating 1000 messages and sending them asynchronously to Kafka. This maximizes the utilization of resources and accelerates write operations.

Load balancing and partitioning in Kafka

To distribute the load, increase the number of topic partitions and consumers. Multiple consumers can read from different partitions simultaneously, parallelizing processing.

Error handling and retry mechanisms

Ensure your system is resilient by handling errors and retries:

producer.on('error', (err) => {
  // Retry logic
  setTimeout(() => {
    producer.send(message, callback);
  }, 5000);
});

In this code, if the producer encounters an error, the code retries sending the message after a 5-second delay. Incorporating exponential back-off and setting a max retry limit can further enhance this logic.

Optimizing a write-intensive system requires a blend of architectural decisions and configuration tweaks. Each optimization technique comes with its trade-offs, so understanding the requirements and constraints of the specific system is crucial. Whether it's increasing throughput with batch processing, ensuring data integrity with synchronous writes, or guaranteeing resilience with error handling, each technique has a role in ensuring the system performs optimally under high load.

Conclusion

Building a write-intensive system can be a complex task, requiring an understanding of various technologies, system design principles, and optimization techniques.

In this comprehensive guide, we dissected the art of constructing a write-intensive system using Kafka and tRPC. Kafka's prowess in high-volume data streaming paired with tRPC's type-safe and efficient APIs ensures a system that's both robust and performant. From foundational setups to advanced optimization techniques like batch processing and asynchronous writes, we've laid out a roadmap to navigate the complexities of such systems, helping you achieve resilience and peak performance.