How to Deploy Apache Pulsar as a Google Pub/Sub Alternative

Apache Pulsar is a distributed messaging and streaming platform that combines high-throughput message delivery with durable storage and built-in serverless computing. It supports multi-tenancy, geo-replication, and tiered storage. Google Pub/Sub is a fully managed messaging service that provides asynchronous messaging between applications with automatic scaling and global availability.

This article explains how to deploy Apache Pulsar as a self-hosted alternative to Google Pub/Sub. It covers Docker-based installation, Pulsar Manager UI setup, multi-tenancy configuration, Pulsar Functions for serverless processing, IO connectors for data integration, authentication and authorization, and migration strategies from Google Pub/Sub.

Key Differences Between Apache Pulsar and Google Pub/Sub

Google Pub/Sub is a fully managed messaging service tightly integrated with the Google Cloud ecosystem. Apache Pulsar offers comparable functionality with infrastructure independence and advanced features like built-in Functions and tiered storage.

Deployment Model: Google Pub/Sub runs as a managed service with automatic scaling. Pulsar deploys on any infrastructure you control, including on-premises servers, virtual machines (VMs), or Kubernetes clusters.
Multi-Tenancy: Google Pub/Sub uses projects for isolation. Pulsar provides native multi-tenancy with tenants, namespaces, and topic-level policies for granular resource management.
Storage Architecture: Google Pub/Sub abstracts storage entirely. Pulsar uses Apache BookKeeper for durable message storage with optional tiered storage to offload older data to object storage.
Serverless Computing: Google Pub/Sub integrates with Cloud Functions for event processing. Pulsar includes built-in Functions that run alongside the broker without external dependencies.
Pricing: Google Pub/Sub charges based on data volume and operations. Pulsar is open-source with no licensing costs beyond infrastructure.

Understanding Apache Pulsar Architecture

Apache Pulsar maps to Google Pub/Sub concepts while providing additional flexibility through its layered architecture.

Google Pub/Sub	Apache Pulsar	Description
Topics	Topics	Named channels for publishing messages.
Subscriptions	Subscriptions	Named consumers that receive messages from topics.
Push Subscriptions	Pulsar Functions	Push subscriptions deliver messages to HTTP endpoints. Use Pulsar Functions to forward messages to external HTTP endpoints when migrating this pattern.
Ordering Keys	Key-Shared Subscriptions	Ordered delivery for messages with the same ordering key.
Dead-Letter Topics	Dead-Letter Topics	Stores messages that cannot be processed.
IAM Policies	Authorization Policies	Role-based access control for topics and namespaces.
Projects (Pub/Sub scope)	Tenants/Namespaces	Pub/Sub topics and subscriptions are scoped to a GCP project. Pulsar tenants provide equivalent Pub/Sub-level isolation, with namespaces as subdivisions within a tenant.

The Pulsar architecture consists of:

Brokers: Stateless servers that handle message routing, load balancing, and client connections. Brokers do not store messages directly.
BookKeeper (Bookies): Distributed storage layer that provides durable, replicated message storage with low-latency writes.
Metadata Store: Coordination and configuration layer for cluster metadata, policies, and leader election. In modern standalone Docker deployments, Pulsar uses RocksDB by default instead of an external ZooKeeper service.
Pulsar Manager: Web-based user interface (UI) for managing tenants, namespaces, topics, and monitoring cluster health.
Pulsar Functions: Lightweight serverless computing framework for stream processing without external systems.

Prerequisites

Before you begin, you need to:

Have access to a Linux-based server as a non-root user with sudo privileges.
Install Docker and Docker Compose.
Create a DNS A record pointing to your server's IP address (for example, pulsar.example.com).

Deploy Apache Pulsar with Docker

Docker provides the fastest path to a running Pulsar cluster. This setup uses the official Apache Pulsar images in standalone mode, running the broker, BookKeeper, and metadata store in one container. The broker exposes port 6650 for binary connections and port 8080 for the Admin REST API. Traefik handles HTTPS for the Pulsar Manager UI. The broker advertises localhost, so pulsar-admin and pulsar-client commands in this article run via docker exec rather than from outside the container.

Create the project directory with subdirectories for data, configuration, functions, and IO connectors.
console
```
$ mkdir -p ~/pulsar/{data,conf,connectors,functions}
```
Navigate to the project directory.
console
```
$ cd ~/pulsar
```
Create a .env file to store configuration values.
console
```
$ nano .env
```
Add the following environment variables. Replace pulsar.example.com with your domain, admin@example.com with your email address, and PULSAR_MANAGER_PASSWORD with a strong password (minimum 6 characters) for the Pulsar Manager admin account.
ini
```
PULSAR_DOMAIN=pulsar.example.com
ACME_EMAIL=admin@example.com
PM_USERNAME=admin
PM_PASSWORD=PULSAR_MANAGER_PASSWORD
PM_EMAIL=admin@example.com
```
Save and close the file.
Set permissions on the data directory so the container can write to it.
console
```
$ chmod 777 data
```
Create the admin initialization script. This script runs once on startup to create the Pulsar Manager admin account using the credentials from your .env file.
console
```
$ nano init-admin.sh
```

Add the following content.

                            bash
                            
                        
#!/bin/sh
set -e
CSRF=$(curl -s http://pulsar-manager:7750/pulsar-manager/csrf-token)
curl -s \
  -H "X-XSRF-TOKEN: $CSRF" \
  -H "Cookie: XSRF-TOKEN=$CSRF;" \
  -H "Content-Type: application/json" \
  -X PUT http://pulsar-manager:7750/pulsar-manager/users/superuser \
  -d "{\"name\":\"$PM_USERNAME\",\"password\":\"$PM_PASSWORD\",\"description\":\"admin\",\"email\":\"$PM_EMAIL\"}"

Save and close the file.

Make the script executable.
console
```
$ chmod +x init-admin.sh
```
Add your user to the docker group to run Docker commands without sudo.
console
```
$ sudo usermod -aG docker $USER
```
Apply the group membership to the current session.
console
```
$ newgrp docker
```
Create the Docker Compose configuration file.
console
```
$ nano docker-compose.yml
```

Add the following configuration:

                            yaml
                            
                        
services:
  traefik:
    image: traefik:v3.7.0
    container_name: traefik
    command:
      - "--providers.docker=true"
      - "--providers.docker.exposedbydefault=false"
      - "--entrypoints.web.address=:80"
      - "--entrypoints.websecure.address=:443"
      - "--certificatesresolvers.le.acme.email=${ACME_EMAIL}"
      - "--certificatesresolvers.le.acme.storage=/letsencrypt/acme.json"
      - "--certificatesresolvers.le.acme.httpchallenge=true"
      - "--certificatesresolvers.le.acme.httpchallenge.entrypoint=web"
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - "/var/run/docker.sock:/var/run/docker.sock:ro"
      - "./letsencrypt:/letsencrypt"
    restart: unless-stopped

  pulsar:
    image: apachepulsar/pulsar:4.2.2
    container_name: pulsar
    command: bin/pulsar standalone --advertised-address localhost
    ports:
      - "6650:6650"
      - "8080:8080"
    volumes:
      - ./data:/pulsar/data
      - ./connectors:/pulsar/connectors
      - ./functions:/pulsar/functions
    environment:
      - PULSAR_MEM=-Xms2g -Xmx2g -XX:MaxDirectMemorySize=1g
    restart: unless-stopped

  pulsar-manager:
    image: apachepulsar/pulsar-manager:v0.4.0
    container_name: pulsar-manager
    environment:
      - SPRING_CONFIGURATION_FILE=/pulsar-manager/pulsar-manager/application.properties
    depends_on:
      - pulsar
    healthcheck:
      test: ["CMD-SHELL", "wget -qO- http://localhost:7750/pulsar-manager/csrf-token >/dev/null 2>&1"]
      interval: 10s
      timeout: 5s
      retries: 12
      start_period: 45s
    labels:
      - "traefik.enable=true"
      - "traefik.http.services.pulsar-manager.loadbalancer.server.port=9527"
      - "traefik.http.routers.pulsar-manager.rule=Host(`${PULSAR_DOMAIN}`)"
      - "traefik.http.routers.pulsar-manager.entrypoints=websecure"
      - "traefik.http.routers.pulsar-manager.tls=true"
      - "traefik.http.routers.pulsar-manager.tls.certresolver=le"
    restart: unless-stopped

  pulsar-manager-init:
    image: curlimages/curl:8.19.0
    container_name: pulsar-manager-init
    depends_on:
      pulsar-manager:
        condition: service_healthy
    restart: "no"
    volumes:
      - ./init-admin.sh:/init-admin.sh:ro
    entrypoint: ["sh", "/init-admin.sh"]
    environment:
      - PM_USERNAME
      - PM_PASSWORD
      - PM_EMAIL

Save and close the file.

Start the stack.
console
```
$ docker compose up -d
```
Verify that the traefik, pulsar, and pulsar-manager containers show a running status.
console
```
$ docker compose ps
```
The output shows all three containers with a status of Up.
Check the Pulsar startup logs for any errors.
console
```
$ docker compose logs pulsar
```
The output shows the broker and BookKeeper starting. Wait until the broker is fully ready before proceeding.
Verify that the admin user was created by the init service.
console
```
$ docker logs pulsar-manager-init
```
The output shows {"message":"Add super user success, please login"}. The pulsar-manager-init container exits after this one-time setup and does not restart.

Access Pulsar Manager UI

Pulsar Manager provides a web-based interface for managing clusters, tenants, namespaces, topics, subscriptions, and monitoring broker metrics. Pulsar creates two default tenants on startup: public, used for the default namespace, and pulsar, used for internal system topics.

Open a web browser and navigate to https://pulsar.example.com, replacing pulsar.example.com with your configured domain.
Log in with the administrator credentials created during setup and click Log in.
Click New Environment to register your Pulsar cluster.
Fill in the dialog fields:
- Enter a name in the Environment Name field (for example, local).
- Enter http://pulsar:8080 in the Service URL field.
- Enter http://pulsar:6650 in the Bookie URL field.
- Click Confirm.
Click the environment name in the list to navigate to the management interface. Verify that the left sidebar shows Tenants, Namespaces, Topics, and Tokens under the Management section.

Configure Multi-Tenancy

Pulsar's multi-tenancy model provides logical isolation between different teams, applications, or customers. Each tenant can have multiple namespaces, and each namespace can contain multiple topics with independent policies.

Create a new tenant using the Pulsar admin CLI.

                            console
                            
                        
$ docker exec -it pulsar bin/pulsar-admin tenants create my-tenant \
    --admin-roles admin \
    --allowed-clusters standalone

Create a namespace within the tenant.

                            console
                            
$ docker exec -it pulsar bin/pulsar-admin namespaces create my-tenant/production

Configure retention policies for the namespace. This example retains messages for 7 days or up to 10 GB.

                            console
                            
                        
$ docker exec -it pulsar bin/pulsar-admin namespaces set-retention my-tenant/production \
    --size 10G \
    --time 7d

Configure message Time-to-Live (TTL) to automatically expire unacknowledged messages after 1 hour.

                            console
                            
                        
$ docker exec -it pulsar bin/pulsar-admin namespaces set-message-ttl my-tenant/production \
    --messageTTL 3600

Create a topic within the namespace.

                            console
                            
$ docker exec -it pulsar bin/pulsar-admin topics create persistent://my-tenant/production/orders

Verify that the topic was created.
console
```
$ docker exec -it pulsar bin/pulsar-admin topics list my-tenant/production
```
The output displays the full topic name in the format persistent://my-tenant/production/orders.
Open the NAMESPACES tab for the my-tenant tenant in Pulsar Manager to view its namespaces.

Deploy Pulsar Functions

Pulsar Functions provide lightweight serverless computing for stream processing directly within the Pulsar cluster. Functions can transform, filter, route, or aggregate messages without external systems like Apache Flink or Spark.

Create a sample Python function that converts messages to uppercase in the functions directory you created earlier.
console
```
$ nano ~/pulsar/functions/uppercase_function.py
```

Add the following code:

                            python
                            
                        
from pulsar import Function

class UppercaseFunction(Function):
    def process(self, input, context):
        return input.upper()

Save and close the file.

Deploy the function to process messages from an input topic and publish results to an output topic.

                            console
                            
                        
$ docker exec -it pulsar bin/pulsar-admin functions create \
    --name uppercase \
    --tenant public \
    --namespace default \
    --inputs persistent://public/default/input-topic \
    --output persistent://public/default/output-topic \
    --py /pulsar/functions/uppercase_function.py \
    --classname uppercase_function.UppercaseFunction

Verify that the function is running.

                            console
                            
                        
$ docker exec -it pulsar bin/pulsar-admin functions status \
    --tenant public \
    --namespace default \
    --name uppercase

The output shows the function instances and their running status.

Test the function by producing a message to the input topic.
console
```
$ docker exec -it pulsar bin/pulsar-client produce persistent://public/default/input-topic \
    --messages "hello world"
```
The Pulsar client prints several INFO log lines during startup. That is expected. The command is successful when it ends with 1 messages successfully produced.

Consume messages from the output topic.

                            console
                            
                        
$ docker exec -it pulsar bin/pulsar-client consume persistent://public/default/output-topic \
    --subscription-name test-sub \
    --num-messages 1 \
    --subscription-position Earliest

The output displays content:HELLO WORLD, confirming the function processed the message.

Set Up Pulsar IO Connectors

Pulsar IO provides a framework for moving data between Pulsar and external systems. Source connectors ingest data from external systems into Pulsar topics, while sink connectors export data from Pulsar topics to external databases, storage systems, or services.

The standard Pulsar Docker image does not include built-in connector NAR files. Download the connector you need before deploying it. This section demonstrates the file source connector, which reads files from a local directory and publishes their content as messages to a Pulsar topic.

Download the file source connector NAR file into the connectors directory you created earlier.

                            console
                            
                        
$ curl -L -o ~/pulsar/connectors/pulsar-io-file-4.2.2.nar \
    https://downloads.apache.org/pulsar/pulsar-4.2.2/connectors/pulsar-io-file-4.2.2.nar

Restart the Pulsar container to load the connector. Running admin commands before the broker is fully ready can return transient BookKeeper availability errors.
console
```
$ docker compose restart pulsar
```
Wait for the broker to be fully ready before proceeding. Run the following command repeatedly until it returns ["standalone"].
console
```
$ curl -s http://localhost:8080/admin/v2/clusters
```
List available source connectors to verify the file connector loaded.
console
```
$ docker exec -it pulsar bin/pulsar-admin sources available-sources
```
The output shows file with its description.
Create the directory that the connector will monitor for new files.
console
```
$ mkdir -p ~/pulsar/data/input-files
```
Grant the Pulsar container write access to the input directory. The container runs as a non-root user, so open permissions are required for it to read files placed here.
console
```
$ chmod 777 ~/pulsar/data/input-files
```
Create the file source connector configuration file.
console
```
$ nano file-source-config.yaml
```

Add the following configuration:

                            yaml
                            
                        
configs:
  inputDirectory: /pulsar/data/input-files
  recurse: false
  keepFile: true
  fileFilter: '[^\.].*'
  minimumFileAge: 0

Save and close the file.

Copy the configuration into the container.
console
```
$ docker cp file-source-config.yaml pulsar:/pulsar/
```

Deploy the file source connector to publish file contents to a topic.

                            console
                            
                        
$ docker exec -it pulsar bin/pulsar-admin sources create \
    --name file-source \
    --tenant public \
    --namespace default \
    --destination-topic-name persistent://public/default/file-data \
    --source-type file \
    --source-config-file /pulsar/file-source-config.yaml

Verify the source connector is running.

                            console
                            
                        
$ docker exec -it pulsar bin/pulsar-admin sources status \
    --tenant public \
    --namespace default \
    --name file-source

The output shows "running": true when the connector is active.

Create a file in the input directory with two lines.

                            console
                            
$ printf "log entry 1\nlog entry 2\n" > ~/pulsar/data/input-files/test.txt

Consume the messages ingested by the connector.
console
```
$ docker exec -it pulsar bin/pulsar-client consume persistent://public/default/file-data \
    --subscription-name file-reader \
    --num-messages 2 \
    --subscription-position Earliest
```
The output displays two messages. Each message contains one line from the file in the content field and includes file.name, file.path, and file.modified.time in the properties field. The messages may arrive in any order. The final line confirms success. If you repeat this step, use a different value for --subscription-name each time, as an existing subscription resumes from where it last stopped and will not replay already-consumed messages.
```
----- got message -----
key:[test.txt_1], properties:[...], content:log entry 1
----- got message -----
key:[test.txt_2], properties:[...], content:log entry 2
2 messages successfully consumed
```

Configure Authentication and Authorization

Pulsar supports multiple authentication mechanisms including Transport Layer Security (TLS) certificates, JSON Web Tokens (JWT), and OAuth 2.0. This section configures JWT-based authentication for client connections.

Generate a secret key for signing JWT tokens.

                            console
                            
                        
$ docker exec -it pulsar bin/pulsar tokens create-secret-key \
    --output /pulsar/data/my-secret.key

Generate an admin token using the secret key and save it to a file.

                            console
                            
                        
$ docker exec pulsar bin/pulsar tokens create \
    --secret-key file:///pulsar/data/my-secret.key \
    --subject admin > ~/pulsar/data/admin-token.txt

Generate a client token for application access and save it to a file.

                            console
                            
                        
$ docker exec pulsar bin/pulsar tokens create \
    --secret-key file:///pulsar/data/my-secret.key \
    --subject app-client > ~/pulsar/data/app-client-token.txt

Stop the Pulsar container to apply configuration changes.
console
```
$ docker compose stop pulsar
```

Export the default Pulsar configuration as a starting point.

                            console
                            
$ docker run --rm apachepulsar/pulsar:4.2.2 cat /pulsar/conf/standalone.conf > conf/standalone.conf

Open the configuration file to append the authentication settings.
console
```
$ nano conf/standalone.conf
```

Scroll to the end of the file and append the following authentication settings.

                            ini
                            
                        
authenticationEnabled=true
authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderToken
tokenSecretKey=file:///pulsar/data/my-secret.key

authorizationEnabled=true
authorizationProvider=org.apache.pulsar.broker.authorization.PulsarAuthorizationProvider
superUserRoles=admin

brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.AuthenticationToken
brokerClientAuthenticationParameters=file:///pulsar/data/admin-token.txt

The brokerClientAuthenticationPlugin and brokerClientAuthenticationParameters entries allow internal broker components such as the Functions worker to authenticate against the broker when JWT is enabled. Save and close the file.

Open the Docker Compose configuration file.
console
```
$ nano docker-compose.yml
```
Add the following line to the volumes block under the pulsar service.
yaml
```
- ./conf/standalone.conf:/pulsar/conf/standalone.conf
```
Save and close the file.
Start the Pulsar container with the new configuration. Wait for the broker to be fully ready before running authenticated commands.
console
```
$ docker compose up -d pulsar
```

Grant permissions to the client role on a namespace.

                            console
                            
                        
$ docker exec -it pulsar bin/pulsar-admin \
    --auth-plugin org.apache.pulsar.client.impl.auth.AuthenticationToken \
    --auth-params "token:$(cat ~/pulsar/data/admin-token.txt)" \
    namespaces grant-permission public/default \
    --role app-client \
    --actions produce,consume

Test authenticated access using the client token saved earlier.

                            console
                            
                        
$ docker exec -it pulsar bin/pulsar-client \
    --auth-plugin org.apache.pulsar.client.impl.auth.AuthenticationToken \
    --auth-params "token:$(cat ~/pulsar/data/app-client-token.txt)" \
    produce persistent://public/default/test-topic \
    --messages "authenticated message"

Authenticate Pulsar Manager

Pulsar Manager makes its own admin API calls to the broker, so it also needs a token now that authentication is enabled. Without this, the UI returns 401 Unauthorized on every tenant and topic request.

Export the default Pulsar Manager configuration as a starting point. This reads the file from the image directly rather than the running container, so it stays safe to rerun even after the file is bind-mounted in a later step.
console
```
$ docker run --rm --entrypoint cat apachepulsar/pulsar-manager:v0.4.0 \
    /pulsar-manager/pulsar-manager/application.properties > pulsar-manager-application.properties
```
Open the file to set the broker JWT properties.
console
```
$ nano pulsar-manager-application.properties
```
Locate the following properties and set their values:
ini
```
backend.jwt.token=YOUR_ADMIN_TOKEN
jwt.broker.token.mode=SECRET
jwt.broker.secret.key=file:///pulsar-manager/broker-secret.key
```
Replace YOUR_ADMIN_TOKEN with the contents of ~/pulsar/data/admin-token.txt. Save and close the file.
Open the Docker Compose configuration file.
console
```
$ nano docker-compose.yml
```
Add a volumes block under the pulsar-manager service with the following lines.
yaml
```
volumes:
  - ./pulsar-manager-application.properties:/pulsar-manager/pulsar-manager/application.properties:ro
  - ./data/my-secret.key:/pulsar-manager/broker-secret.key:ro
  - ./pulsar-manager-data:/pulsar-manager/pulsar-manager/dbdata
```
A bind mount keeps this configuration in place if the container is ever recreated. A plain docker cp into the container only lasts until that happens. The dbdata mount covers Pulsar Manager's own database, which stores the admin login and any registered environments. Without it, both are lost on every recreation. Save and close the file.
Copy Pulsar Manager's existing database into the new mount point so the current admin login and environment carry over, then recreate the container to apply the configuration.
console
```
$ mkdir -p pulsar-manager-data
$ docker cp pulsar-manager:/pulsar-manager/pulsar-manager/dbdata/. pulsar-manager-data/
$ docker compose up -d pulsar-manager
```

Verify the Deployment

Each of the following checks covers an independent failure domain. These steps assume you completed Configure Authentication and Authorization earlier. All pulsar-admin and pulsar-client invocations require --auth-plugin org.apache.pulsar.client.impl.auth.AuthenticationToken and --auth-params "token:$(cat ~/pulsar/data/admin-token.txt)". Omitting them causes HTTP 401 Unauthorized.

Navigate to https://pulsar.example.com, log in with your configured credentials, and verify that the cluster dashboard loads correctly.
Test that the Pulsar binary protocol port accepts TCP connections. Replace SERVER-IP with your server's IP address. This confirms the port is open. Use docker exec for pulsar-admin and pulsar-client operations, since the broker advertises localhost for lookups.
console
```
$ nc -zv SERVER-IP 6650
```
Verify that the Admin REST API responds correctly using the admin token saved earlier.
console
```
$ curl -s -H "Authorization: Bearer $(cat ~/pulsar/data/admin-token.txt)" \
    http://localhost:8080/admin/v2/clusters
```
The output displays ["standalone"] for a single-node deployment.

Create the verification topic if it does not already exist.

                            console
                            
                        
$ docker exec -it pulsar bin/pulsar-admin \
    --auth-plugin org.apache.pulsar.client.impl.auth.AuthenticationToken \
    --auth-params "token:$(cat ~/pulsar/data/admin-token.txt)" \
    topics create persistent://public/default/verify-test

Verify that the topic appears in the topic list. JWT tokens generated with bin/pulsar tokens create are valid for broker authentication but do not appear in Pulsar Manager's Tokens page.

                            console
                            
                        
$ docker exec -it pulsar bin/pulsar-admin \
    --auth-plugin org.apache.pulsar.client.impl.auth.AuthenticationToken \
    --auth-params "token:$(cat ~/pulsar/data/admin-token.txt)" \
    topics list public/default

Produce a test message to the verification topic.

                            console
                            
                        
$ docker exec -it pulsar bin/pulsar-client \
    --auth-plugin org.apache.pulsar.client.impl.auth.AuthenticationToken \
    --auth-params "token:$(cat ~/pulsar/data/admin-token.txt)" \
    produce persistent://public/default/verify-test \
    --messages "test message"

Consume the test message with a new subscription name. If you repeat this check, use a different subscription name such as verify-sub-2. --subscription-position Earliest only applies when a subscription is first created.

                            console
                            
                        
$ docker exec -it pulsar bin/pulsar-client \
    --auth-plugin org.apache.pulsar.client.impl.auth.AuthenticationToken \
    --auth-params "token:$(cat ~/pulsar/data/admin-token.txt)" \
    consume persistent://public/default/verify-test \
    --subscription-name verify-sub-1 \
    --num-messages 1 \
    --subscription-position Earliest

Check the status of the uppercase function.

                            console
                            
                        
$ docker exec -it pulsar bin/pulsar-admin \
    --auth-plugin org.apache.pulsar.client.impl.auth.AuthenticationToken \
    --auth-params "token:$(cat ~/pulsar/data/admin-token.txt)" \
    functions status \
    --tenant public \
    --namespace default \
    --name uppercase

A healthy function shows "running" : true.

Migrate from Google Pub/Sub to Apache Pulsar

Migrating from Google Pub/Sub to Pulsar involves mapping Google concepts to Pulsar equivalents, updating client configurations, and transferring message schemas and data. The examples below assume you completed Configure Authentication and Authorization earlier in this article.

Topic and Subscription Migration

Google Pub/Sub topics map directly to Pulsar persistent topics, and subscriptions map to Pulsar subscriptions with different types.

Topics: Create a Pulsar topic for each Pub/Sub topic using the format persistent://tenant/namespace/topic-name.
Subscriptions: Map Pub/Sub subscriptions to Pulsar subscription types:
- Pull subscriptions map to Shared or Key_Shared subscriptions for competing consumers.
- Push subscriptions can be replaced with Pulsar Functions for serverless processing.

Application Code Updates

Replace the Google Cloud Pub/Sub client library with the Pulsar client library for your language. Pulsar provides official clients for Java, Python, Go, Node.js, C++, and C#.

Message Schema Migration

Google Pub/Sub supports Avro and Protocol Buffer schemas per topic. Pulsar includes a built-in schema registry. Refer to the Pulsar schema documentation for the schema format Pulsar expects and how to register schemas using pulsar-admin.

Push Subscriptions to Pulsar Functions

Google Pub/Sub push subscriptions deliver messages to HTTP endpoints. Replace these with Pulsar Functions that forward messages to external services. Refer to the Pulsar Functions documentation for implementation details.

Dead-Letter Topic Configuration

Google Pub/Sub routes failed messages to a dead-letter topic after the maximum delivery attempts are exceeded. Pulsar supports equivalent behavior through dead-letter policies configured on subscriptions. Refer to the Pulsar dead-letter topic documentation for configuration details.

Data Migration Strategy

For topics with existing messages that need to be preserved:

Dual-Write Period: Configure producers to write to both Pub/Sub and Pulsar during the migration window.
Backfill Historical Data: Export messages from Pub/Sub to Cloud Storage using an export subscription, then replay the exported files into Pulsar using a custom producer script.
Cutover: Once consumers are migrated and caught up, disable the Pub/Sub producers.

Migration Considerations

Message Ordering: Pub/Sub delivers messages in order per ordering key when the publisher sets an ordering key and the subscription has message ordering enabled. Use Pulsar Key_Shared subscriptions for equivalent behavior.
Message Retention: Pub/Sub retains messages for 7 days by default. Configure Pulsar namespace retention policies to match.
Authentication: Replace Google Cloud IAM with Pulsar JWT tokens or integrate with your identity provider using OAuth 2.0.
Monitoring: Replace Cloud Monitoring with Pulsar's built-in Prometheus metrics endpoint at http://localhost:8080/metrics/.
Client Libraries: Pulsar provides official clients for Java, Python, Go, Node.js, C++, and C#. Verify client library availability for your technology stack.
Throughput Limits: Pub/Sub has per-project quotas. Pulsar throughput depends on your infrastructure sizing.
Exactly-Once Semantics: Pub/Sub provides at-least-once delivery by default. Pulsar supports exactly-once semantics with transactional producers.

Conclusion

You have deployed Apache Pulsar as a self-hosted alternative to Google Pub/Sub using Docker, configured multi-tenancy with namespaces and retention policies, deployed Pulsar Functions for serverless stream processing, set up IO connectors for data integration, enabled JWT-based authentication and authorization, and applied migration strategies for topics, schemas, and dead-letter queues. For more information, refer to the official Apache Pulsar documentation.

Tags:

Open Source

GCP

How to Deploy Apache Pulsar as a Google Pub/Sub Alternative

How to Deploy Apache Pulsar as a Google Pub/Sub Alternative

Key Differences Between Apache Pulsar and Google Pub/Sub

Understanding Apache Pulsar Architecture

Prerequisites

Deploy Apache Pulsar with Docker

Access Pulsar Manager UI

Configure Multi-Tenancy

Deploy Pulsar Functions

Set Up Pulsar IO Connectors

Configure Authentication and Authorization

Authenticate Pulsar Manager

Verify the Deployment

Migrate from Google Pub/Sub to Apache Pulsar

Topic and Subscription Migration

Application Code Updates

Message Schema Migration

Push Subscriptions to Pulsar Functions

Dead-Letter Topic Configuration

Data Migration Strategy

Migration Considerations

Conclusion

Comments

Products

Features

Solutions

Marketplace

Resources

Company

Tech Talks

Vultr Blogs