How to Deploy Apache Pulsar as a Google Pub/Sub Alternative

Apache Pulsar is a distributed messaging and streaming platform that combines high-throughput message delivery with durable storage and built-in serverless computing. It supports multi-tenancy, geo-replication, and tiered storage. Google Pub/Sub is a fully managed messaging service that provides asynchronous messaging between applications with automatic scaling and global availability.

This article explains how to deploy Apache Pulsar as a self-hosted alternative to Google Pub/Sub. It covers Docker-based installation, Pulsar Manager UI setup, multi-tenancy configuration, Pulsar Functions for serverless processing, IO connectors for data integration, authentication and authorization, and migration strategies from Google Pub/Sub.

Key Differences Between Apache Pulsar and Google Pub/Sub

Google Pub/Sub is a fully managed messaging service tightly integrated with the Google Cloud ecosystem. Apache Pulsar offers comparable functionality with infrastructure independence and advanced features like built-in Functions and tiered storage.

• Deployment Model: Google Pub/Sub runs as a managed service with automatic scaling. Pulsar deploys on any infrastructure you control, including on-premises servers, virtual machines (VMs), or Kubernetes clusters.
• Multi-Tenancy: Google Pub/Sub uses projects for isolation. Pulsar provides native multi-tenancy with tenants, namespaces, and topic-level policies for granular resource management.
• Storage Architecture: Google Pub/Sub abstracts storage entirely. Pulsar uses Apache BookKeeper for durable message storage with optional tiered storage to offload older data to object storage.
• Serverless Computing: Google Pub/Sub integrates with Cloud Functions for event processing. Pulsar includes built-in Functions that run alongside the broker without external dependencies.
• Pricing: Google Pub/Sub charges based on data volume and operations. Pulsar is open-source with no licensing costs beyond infrastructure.

Understanding Apache Pulsar Architecture

Apache Pulsar maps to Google Pub/Sub concepts while providing additional flexibility through its layered architecture.

The Pulsar architecture consists of:

• Brokers: Stateless servers that handle message routing, load balancing, and client connections. Brokers do not store messages directly.
• BookKeeper (Bookies): Distributed storage layer that provides durable, replicated message storage with low-latency writes.
• Metadata Store: Coordination and configuration layer for cluster metadata, policies, and leader election. In modern standalone Docker deployments, Pulsar uses RocksDB by default instead of an external ZooKeeper service.
• Pulsar Manager: Web-based user interface (UI) for managing tenants, namespaces, topics, and monitoring cluster health.
• Pulsar Functions: Lightweight serverless computing framework for stream processing without external systems.

Prerequisites

Before you begin, you need to:

• Have access to a Linux-based server as a non-root user with sudo privileges.
• Install Docker and Docker Compose.
• Create a DNS A record pointing to your server's IP address (for example, pulsar.example.com).

Deploy Apache Pulsar with Docker

Docker provides the fastest path to a running Pulsar cluster. This setup uses the official Apache Pulsar images in standalone mode, running the broker, BookKeeper, and metadata store in one container. The broker exposes port 6650 for binary connections and port 8080 for the Admin REST API. Traefik handles HTTPS for the Pulsar Manager UI. The broker advertises localhost, so pulsar-admin and pulsar-client commands in this article run via docker exec rather than from outside the container.

1. Create the project directory with subdirectories for data, configuration, functions, and IO connectors.

   CONSOLE

   Copy

   $ mkdir -p ~/pulsar/{data,conf,connectors,functions}

   Explain Code

   Powered by Vultr AgentBeta

2. Navigate to the project directory.

   CONSOLE

   Copy

   $ cd ~/pulsar

   Explain Code

   Powered by Vultr AgentBeta

3. Create a .env file to store configuration values.

   CONSOLE

   Copy

   $ nano .env

   Explain Code

   Powered by Vultr AgentBeta

4. Add the following environment variables. Replace pulsar.example.com with your domain, admin@example.com with your email address, and PULSAR_MANAGER_PASSWORD with a strong password (minimum 6 characters) for the Pulsar Manager admin account.

   INI

   Copy

   PULSAR_DOMAIN=pulsar.example.com
   ACME_EMAIL=admin@example.com
   PM_USERNAME=admin
   PM_PASSWORD=PULSAR_MANAGER_PASSWORD
   PM_EMAIL=admin@example.com

   Explain Code

   Powered by Vultr AgentBeta

   Save and close the file.

5. Set permissions on the data directory so the container can write to it.

   CONSOLE

   Copy

   $ chmod 777 data

   Explain Code

   Powered by Vultr AgentBeta

6. Create the admin initialization script. This script runs once on startup to create the Pulsar Manager admin account using the credentials from your .env file.

   CONSOLE

   Copy

   $ nano init-admin.sh

   Explain Code

   Powered by Vultr AgentBeta

7. Add the following content.

   BASH

   Copy

   #!/bin/sh
   set -e
   CSRF=$(curl -s http://pulsar-manager:7750/pulsar-manager/csrf-token)
   curl -s \
     -H "X-XSRF-TOKEN: $CSRF" \
     -H "Cookie: XSRF-TOKEN=$CSRF;" \
     -H "Content-Type: application/json" \
     -X PUT http://pulsar-manager:7750/pulsar-manager/users/superuser \
     -d "{\"name\":\"$PM_USERNAME\",\"password\":\"$PM_PASSWORD\",\"description\":\"admin\",\"email\":\"$PM_EMAIL\"}"

   Explain Code

   Powered by Vultr AgentBeta

   Save and close the file.

8. Make the script executable.

   CONSOLE

   Copy

   $ chmod +x init-admin.sh

   Explain Code

   Powered by Vultr AgentBeta

9. Add your user to the docker group to run Docker commands without sudo.

   CONSOLE

   Copy

   $ sudo usermod -aG docker $USER

   Explain Code

   Powered by Vultr AgentBeta

10. Apply the group membership to the current session.

    CONSOLE

    Copy

    $ newgrp docker

    Explain Code

    Powered by Vultr AgentBeta

11. Create the Docker Compose configuration file.

    CONSOLE

    Copy

    $ nano docker-compose.yml

    Explain Code

    Powered by Vultr AgentBeta

12. Add the following configuration:

    YAML

    Copy

    services:
      traefik:
        image: traefik:v3.7.0
        container_name: traefik
        command:
          - "--providers.docker=true"
          - "--providers.docker.exposedbydefault=false"
          - "--entrypoints.web.address=:80"
          - "--entrypoints.websecure.address=:443"
          - "--certificatesresolvers.le.acme.email=${ACME_EMAIL}"
          - "--certificatesresolvers.le.acme.storage=/letsencrypt/acme.json"
          - "--certificatesresolvers.le.acme.httpchallenge=true"
          - "--certificatesresolvers.le.acme.httpchallenge.entrypoint=web"
        ports:
          - "80:80"
          - "443:443"
        volumes:
          - "/var/run/docker.sock:/var/run/docker.sock:ro"
          - "./letsencrypt:/letsencrypt"
        restart: unless-stopped
    
      pulsar:
        image: apachepulsar/pulsar:4.2.2
        container_name: pulsar
        command: bin/pulsar standalone --advertised-address localhost
        ports:
          - "6650:6650"
          - "8080:8080"
        volumes:
          - ./data:/pulsar/data
          - ./connectors:/pulsar/connectors
          - ./functions:/pulsar/functions
        environment:
          - PULSAR_MEM=-Xms2g -Xmx2g -XX:MaxDirectMemorySize=1g
        restart: unless-stopped
    
      pulsar-manager:
        image: apachepulsar/pulsar-manager:v0.4.0
        container_name: pulsar-manager
        environment:
          - SPRING_CONFIGURATION_FILE=/pulsar-manager/pulsar-manager/application.properties
        depends_on:
          - pulsar
        healthcheck:
          test: ["CMD-SHELL", "wget -qO- http://localhost:7750/pulsar-manager/csrf-token >/dev/null 2>&1"]
          interval: 10s
          timeout: 5s
          retries: 12
          start_period: 45s
        labels:
          - "traefik.enable=true"
          - "traefik.http.services.pulsar-manager.loadbalancer.server.port=9527"
          - "traefik.http.routers.pulsar-manager.rule=Host(`${PULSAR_DOMAIN}`)"
          - "traefik.http.routers.pulsar-manager.entrypoints=websecure"
          - "traefik.http.routers.pulsar-manager.tls=true"
          - "traefik.http.routers.pulsar-manager.tls.certresolver=le"
        restart: unless-stopped
    
      pulsar-manager-init:
        image: curlimages/curl:8.19.0
        container_name: pulsar-manager-init
        depends_on:
          pulsar-manager:
            condition: service_healthy
        restart: "no"
        volumes:
          - ./init-admin.sh:/init-admin.sh:ro
        entrypoint: ["sh", "/init-admin.sh"]
        environment:
          - PM_USERNAME
          - PM_PASSWORD
          - PM_EMAIL

    Explain Code

    Powered by Vultr AgentBeta

    Save and close the file.

13. Start the stack.

    CONSOLE

    Copy

    $ docker compose up -d

    Explain Code

    Powered by Vultr AgentBeta

14. Verify that the traefik, pulsar, and pulsar-manager containers show a running status.

    CONSOLE

    Copy

    $ docker compose ps

    Explain Code

    Powered by Vultr AgentBeta

    The output shows all three containers with a status of Up.

15. Check the Pulsar startup logs for any errors.

    CONSOLE

    Copy

    $ docker compose logs pulsar

    Explain Code

    Powered by Vultr AgentBeta

    The output shows the broker and BookKeeper starting. Wait until the broker is fully ready before proceeding.

16. Verify that the admin user was created by the init service.

    CONSOLE

    Copy

    $ docker logs pulsar-manager-init

    Explain Code

    Powered by Vultr AgentBeta

    The output shows {"message":"Add super user success, please login"}. The pulsar-manager-init container exits after this one-time setup and does not restart.

Access Pulsar Manager UI

Pulsar Manager provides a web-based interface for managing clusters, tenants, namespaces, topics, subscriptions, and monitoring broker metrics. Pulsar creates two default tenants on startup: public, used for the default namespace, and pulsar, used for internal system topics.

1. Open a web browser and navigate to https://pulsar.example.com, replacing pulsar.example.com with your configured domain.

2. Log in with the administrator credentials created during setup and click Log in.

   

3. Click New Environment to register your Pulsar cluster.

4. Fill in the dialog fields:

   • Enter a name in the Environment Name field (for example, local).
   • Enter http://pulsar:8080 in the Service URL field.
   • Enter http://pulsar:6650 in the Bookie URL field.
   • Click Confirm.

   

5. Click the environment name in the list to navigate to the management interface. Verify that the left sidebar shows Tenants, Namespaces, Topics, and Tokens under the Management section.

Configure Multi-Tenancy

Pulsar's multi-tenancy model provides logical isolation between different teams, applications, or customers. Each tenant can have multiple namespaces, and each namespace can contain multiple topics with independent policies.

1. Create a new tenant using the Pulsar admin CLI.

   CONSOLE

   Copy

   $ docker exec -it pulsar bin/pulsar-admin tenants create my-tenant \
       --admin-roles admin \
       --allowed-clusters standalone

   Explain Code

   Powered by Vultr AgentBeta

2. Create a namespace within the tenant.

   CONSOLE

   Copy

   $ docker exec -it pulsar bin/pulsar-admin namespaces create my-tenant/production

   Explain Code

   Powered by Vultr AgentBeta

3. Configure retention policies for the namespace. This example retains messages for 7 days or up to 10 GB.

   CONSOLE

   Copy

   $ docker exec -it pulsar bin/pulsar-admin namespaces set-retention my-tenant/production \
       --size 10G \
       --time 7d

   Explain Code

   Powered by Vultr AgentBeta

4. Configure message Time-to-Live (TTL) to automatically expire unacknowledged messages after 1 hour.

   CONSOLE

   Copy

   $ docker exec -it pulsar bin/pulsar-admin namespaces set-message-ttl my-tenant/production \
       --messageTTL 3600

   Explain Code

   Powered by Vultr AgentBeta

5. Create a topic within the namespace.

   CONSOLE

   Copy

   $ docker exec -it pulsar bin/pulsar-admin topics create persistent://my-tenant/production/orders

   Explain Code

   Powered by Vultr AgentBeta

6. Verify that the topic was created.

   CONSOLE

   Copy

   $ docker exec -it pulsar bin/pulsar-admin topics list my-tenant/production

   Explain Code

   Powered by Vultr AgentBeta

   The output displays the full topic name in the format persistent://my-tenant/production/orders.

7. Open the NAMESPACES tab for the my-tenant tenant in Pulsar Manager to view its namespaces.

Deploy Pulsar Functions

Pulsar Functions provide lightweight serverless computing for stream processing directly within the Pulsar cluster. Functions can transform, filter, route, or aggregate messages without external systems like Apache Flink or Spark.

1. Create a sample Python function that converts messages to uppercase in the functions directory you created earlier.

   CONSOLE

   Copy

   $ nano ~/pulsar/functions/uppercase_function.py

   Explain Code

   Powered by Vultr AgentBeta

2. Add the following code:

   PYTHON

   Copy

   from pulsar import Function
   
   class UppercaseFunction(Function):
       def process(self, input, context):
           return input.upper()

   Explain Code

   Powered by Vultr AgentBeta

   Save and close the file.

3. Deploy the function to process messages from an input topic and publish results to an output topic.

   CONSOLE

   Copy

   $ docker exec -it pulsar bin/pulsar-admin functions create \
       --name uppercase \
       --tenant public \
       --namespace default \
       --inputs persistent://public/default/input-topic \
       --output persistent://public/default/output-topic \
       --py /pulsar/functions/uppercase_function.py \
       --classname uppercase_function.UppercaseFunction

   Explain Code

   Powered by Vultr AgentBeta

4. Verify that the function is running.

   CONSOLE

   Copy

   $ docker exec -it pulsar bin/pulsar-admin functions status \
       --tenant public \
       --namespace default \
       --name uppercase

   Explain Code

   Powered by Vultr AgentBeta

   The output shows the function instances and their running status.

5. Test the function by producing a message to the input topic.

   CONSOLE

   Copy

   $ docker exec -it pulsar bin/pulsar-client produce persistent://public/default/input-topic \
       --messages "hello world"

   Explain Code

   Powered by Vultr AgentBeta

   The Pulsar client prints several INFO log lines during startup. That is expected. The command is successful when it ends with 1 messages successfully produced.

6. Consume messages from the output topic.

   CONSOLE

   Copy

   $ docker exec -it pulsar bin/pulsar-client consume persistent://public/default/output-topic \
       --subscription-name test-sub \
       --num-messages 1 \
       --subscription-position Earliest

   Explain Code

   Powered by Vultr AgentBeta

   The output displays content:HELLO WORLD, confirming the function processed the message.

Set Up Pulsar IO Connectors

Pulsar IO provides a framework for moving data between Pulsar and external systems. Source connectors ingest data from external systems into Pulsar topics, while sink connectors export data from Pulsar topics to external databases, storage systems, or services.

The standard Pulsar Docker image does not include built-in connector NAR files. Download the connector you need before deploying it. This section demonstrates the file source connector, which reads files from a local directory and publishes their content as messages to a Pulsar topic.

1. Download the file source connector NAR file into the connectors directory you created earlier.

   CONSOLE

   Copy

   $ curl -L -o ~/pulsar/connectors/pulsar-io-file-4.2.2.nar \
       https://downloads.apache.org/pulsar/pulsar-4.2.2/connectors/pulsar-io-file-4.2.2.nar

   Explain Code

   Powered by Vultr AgentBeta

2. Restart the Pulsar container to load the connector. Running admin commands before the broker is fully ready can return transient BookKeeper availability errors.

   CONSOLE

   Copy

   $ docker compose restart pulsar

   Explain Code

   Powered by Vultr AgentBeta

3. Wait for the broker to be fully ready before proceeding. Run the following command repeatedly until it returns ["standalone"].

   CONSOLE

   Copy

   $ curl -s http://localhost:8080/admin/v2/clusters

   Explain Code

   Powered by Vultr AgentBeta

4. List available source connectors to verify the file connector loaded.

   CONSOLE

   Copy

   $ docker exec -it pulsar bin/pulsar-admin sources available-sources

   Explain Code

   Powered by Vultr AgentBeta

   The output shows file with its description.

5. Create the directory that the connector will monitor for new files.

   CONSOLE

   Copy

   $ mkdir -p ~/pulsar/data/input-files

   Explain Code

   Powered by Vultr AgentBeta

6. Grant the Pulsar container write access to the input directory. The container runs as a non-root user, so open permissions are required for it to read files placed here.

   CONSOLE

   Copy

   $ chmod 777 ~/pulsar/data/input-files

   Explain Code

   Powered by Vultr AgentBeta

7. Create the file source connector configuration file.

   CONSOLE

   Copy

   $ nano file-source-config.yaml

   Explain Code

   Powered by Vultr AgentBeta

8. Add the following configuration:

   YAML

   Copy

   configs:
     inputDirectory: /pulsar/data/input-files
     recurse: false
     keepFile: true
     fileFilter: '[^\.].*'
     minimumFileAge: 0

   Explain Code

   Powered by Vultr AgentBeta

   Save and close the file.

9. Copy the configuration into the container.

   CONSOLE

   Copy

   $ docker cp file-source-config.yaml pulsar:/pulsar/

   Explain Code

   Powered by Vultr AgentBeta

10. Deploy the file source connector to publish file contents to a topic.

    CONSOLE

    Copy

    $ docker exec -it pulsar bin/pulsar-admin sources create \
        --name file-source \
        --tenant public \
        --namespace default \
        --destination-topic-name persistent://public/default/file-data \
        --source-type file \
        --source-config-file /pulsar/file-source-config.yaml

    Explain Code

    Powered by Vultr AgentBeta

11. Verify the source connector is running.

    CONSOLE

    Copy

    $ docker exec -it pulsar bin/pulsar-admin sources status \
        --tenant public \
        --namespace default \
        --name file-source

    Explain Code

    Powered by Vultr AgentBeta

    The output shows "running": true when the connector is active.

12. Create a file in the input directory with two lines.

    CONSOLE

    Copy

    $ printf "log entry 1\nlog entry 2\n" > ~/pulsar/data/input-files/test.txt

    Explain Code

    Powered by Vultr AgentBeta

13. Consume the messages ingested by the connector.

    CONSOLE

    Copy

    $ docker exec -it pulsar bin/pulsar-client consume persistent://public/default/file-data \
        --subscription-name file-reader \
        --num-messages 2 \
        --subscription-position Earliest

    Explain Code

    Powered by Vultr AgentBeta

    The output displays two messages. Each message contains one line from the file in the content field and includes file.name, file.path, and file.modified.time in the properties field. The messages may arrive in any order. The final line confirms success. If you repeat this step, use a different value for --subscription-name each time, as an existing subscription resumes from where it last stopped and will not replay already-consumed messages.

    ----- got message -----
    key:[test.txt_1], properties:[...], content:log entry 1
    ----- got message -----
    key:[test.txt_2], properties:[...], content:log entry 2
    2 messages successfully consumed

Configure Authentication and Authorization

Pulsar supports multiple authentication mechanisms including Transport Layer Security (TLS) certificates, JSON Web Tokens (JWT), and OAuth 2.0. This section configures JWT-based authentication for client connections.

1. Generate a secret key for signing JWT tokens.

   CONSOLE

   Copy

   $ docker exec -it pulsar bin/pulsar tokens create-secret-key \
       --output /pulsar/data/my-secret.key

   Explain Code

   Powered by Vultr AgentBeta

2. Generate an admin token using the secret key and save it to a file.

   CONSOLE

   Copy

   $ docker exec pulsar bin/pulsar tokens create \
       --secret-key file:///pulsar/data/my-secret.key \
       --subject admin > ~/pulsar/data/admin-token.txt

   Explain Code

   Powered by Vultr AgentBeta

3. Generate a client token for application access and save it to a file.

   CONSOLE

   Copy

   $ docker exec pulsar bin/pulsar tokens create \
       --secret-key file:///pulsar/data/my-secret.key \
       --subject app-client > ~/pulsar/data/app-client-token.txt

   Explain Code

   Powered by Vultr AgentBeta

4. Stop the Pulsar container to apply configuration changes.

   CONSOLE

   Copy

   $ docker compose stop pulsar

   Explain Code

   Powered by Vultr AgentBeta

5. Export the default Pulsar configuration as a starting point.

   CONSOLE

   Copy

   $ docker run --rm apachepulsar/pulsar:4.2.2 cat /pulsar/conf/standalone.conf > conf/standalone.conf

   Explain Code

   Powered by Vultr AgentBeta

6. Open the configuration file to append the authentication settings.

   CONSOLE

   Copy

   $ nano conf/standalone.conf

   Explain Code

   Powered by Vultr AgentBeta

7. Scroll to the end of the file and append the following authentication settings.

   INI

   Copy

   authenticationEnabled=true
   authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderToken
   tokenSecretKey=file:///pulsar/data/my-secret.key
   
   authorizationEnabled=true
   authorizationProvider=org.apache.pulsar.broker.authorization.PulsarAuthorizationProvider
   superUserRoles=admin
   
   brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.AuthenticationToken
   brokerClientAuthenticationParameters=file:///pulsar/data/admin-token.txt

   Explain Code

   Powered by Vultr AgentBeta

   The brokerClientAuthenticationPlugin and brokerClientAuthenticationParameters entries allow internal broker components such as the Functions worker to authenticate against the broker when JWT is enabled. Save and close the file.

8. Open the Docker Compose configuration file.

   CONSOLE

   Copy

   $ nano docker-compose.yml

   Explain Code

   Powered by Vultr AgentBeta

9. Add the following line to the volumes block under the pulsar service.

   YAML

   Copy

         - ./conf/standalone.conf:/pulsar/conf/standalone.conf

   Explain Code

   Powered by Vultr AgentBeta

   Save and close the file.

10. Start the Pulsar container with the new configuration. Wait for the broker to be fully ready before running authenticated commands.

    CONSOLE

    Copy

    $ docker compose up -d pulsar

    Explain Code

    Powered by Vultr AgentBeta

11. Grant permissions to the client role on a namespace.

    CONSOLE

    Copy

    $ docker exec -it pulsar bin/pulsar-admin \
        --auth-plugin org.apache.pulsar.client.impl.auth.AuthenticationToken \
        --auth-params "token:$(cat ~/pulsar/data/admin-token.txt)" \
        namespaces grant-permission public/default \
        --role app-client \
        --actions produce,consume

    Explain Code

    Powered by Vultr AgentBeta

12. Test authenticated access using the client token saved earlier.

    CONSOLE

    Copy

    $ docker exec -it pulsar bin/pulsar-client \
        --auth-plugin org.apache.pulsar.client.impl.auth.AuthenticationToken \
        --auth-params "token:$(cat ~/pulsar/data/app-client-token.txt)" \
        produce persistent://public/default/test-topic \
        --messages "authenticated message"

    Explain Code

    Powered by Vultr AgentBeta

Authenticate Pulsar Manager

Pulsar Manager makes its own admin API calls to the broker, so it also needs a token now that authentication is enabled. Without this, the UI returns 401 Unauthorized on every tenant and topic request.

1. Export the default Pulsar Manager configuration as a starting point. This reads the file from the image directly rather than the running container, so it stays safe to rerun even after the file is bind-mounted in a later step.

   CONSOLE

   Copy

   $ docker run --rm --entrypoint cat apachepulsar/pulsar-manager:v0.4.0 \
       /pulsar-manager/pulsar-manager/application.properties > pulsar-manager-application.properties

   Explain Code

   Powered by Vultr AgentBeta

2. Open the file to set the broker JWT properties.

   CONSOLE

   Copy

   $ nano pulsar-manager-application.properties

   Explain Code

   Powered by Vultr AgentBeta

3. Locate the following properties and set their values:

   INI

   Copy

   backend.jwt.token=YOUR_ADMIN_TOKEN
   jwt.broker.token.mode=SECRET
   jwt.broker.secret.key=file:///pulsar-manager/broker-secret.key

   Explain Code

   Powered by Vultr AgentBeta

   Replace YOUR_ADMIN_TOKEN with the contents of ~/pulsar/data/admin-token.txt. Save and close the file.

4. Open the Docker Compose configuration file.

   CONSOLE

   Copy

   $ nano docker-compose.yml

   Explain Code

   Powered by Vultr AgentBeta

5. Add a volumes block under the pulsar-manager service with the following lines.

   YAML

   Copy

   volumes:
     - ./pulsar-manager-application.properties:/pulsar-manager/pulsar-manager/application.properties:ro
     - ./data/my-secret.key:/pulsar-manager/broker-secret.key:ro
     - ./pulsar-manager-data:/pulsar-manager/pulsar-manager/dbdata

   Explain Code

   Powered by Vultr AgentBeta

   A bind mount keeps this configuration in place if the container is ever recreated. A plain docker cp into the container only lasts until that happens. The dbdata mount covers Pulsar Manager's own database, which stores the admin login and any registered environments. Without it, both are lost on every recreation. Save and close the file.

6. Copy Pulsar Manager's existing database into the new mount point so the current admin login and environment carry over, then recreate the container to apply the configuration.

   CONSOLE

   Copy

   $ mkdir -p pulsar-manager-data
   $ docker cp pulsar-manager:/pulsar-manager/pulsar-manager/dbdata/. pulsar-manager-data/
   $ docker compose up -d pulsar-manager

   Explain Code

   Powered by Vultr AgentBeta

Verify the Deployment

Each of the following checks covers an independent failure domain. These steps assume you completed Configure Authentication and Authorization earlier. All pulsar-admin and pulsar-client invocations require --auth-plugin org.apache.pulsar.client.impl.auth.AuthenticationToken and --auth-params "token:$(cat ~/pulsar/data/admin-token.txt)". Omitting them causes HTTP 401 Unauthorized.

1. Navigate to https://pulsar.example.com, log in with your configured credentials, and verify that the cluster dashboard loads correctly.

2. Test that the Pulsar binary protocol port accepts TCP connections. Replace SERVER-IP with your server's IP address. This confirms the port is open. Use docker exec for pulsar-admin and pulsar-client operations, since the broker advertises localhost for lookups.

   CONSOLE

   Copy

   $ nc -zv SERVER-IP 6650

   Explain Code

   Powered by Vultr AgentBeta

3. Verify that the Admin REST API responds correctly using the admin token saved earlier.

   CONSOLE

   Copy

   $ curl -s -H "Authorization: Bearer $(cat ~/pulsar/data/admin-token.txt)" \
       http://localhost:8080/admin/v2/clusters

   Explain Code

   Powered by Vultr AgentBeta

   The output displays ["standalone"] for a single-node deployment.

4. Create the verification topic if it does not already exist.

   CONSOLE

   Copy

   $ docker exec -it pulsar bin/pulsar-admin \
       --auth-plugin org.apache.pulsar.client.impl.auth.AuthenticationToken \
       --auth-params "token:$(cat ~/pulsar/data/admin-token.txt)" \
       topics create persistent://public/default/verify-test

   Explain Code

   Powered by Vultr AgentBeta

5. Verify that the topic appears in the topic list. JWT tokens generated with bin/pulsar tokens create are valid for broker authentication but do not appear in Pulsar Manager's Tokens page.

   CONSOLE

   Copy

   $ docker exec -it pulsar bin/pulsar-admin \
       --auth-plugin org.apache.pulsar.client.impl.auth.AuthenticationToken \
       --auth-params "token:$(cat ~/pulsar/data/admin-token.txt)" \
       topics list public/default

   Explain Code

   Powered by Vultr AgentBeta

6. Produce a test message to the verification topic.

   CONSOLE

   Copy

   $ docker exec -it pulsar bin/pulsar-client \
       --auth-plugin org.apache.pulsar.client.impl.auth.AuthenticationToken \
       --auth-params "token:$(cat ~/pulsar/data/admin-token.txt)" \
       produce persistent://public/default/verify-test \
       --messages "test message"

   Explain Code

   Powered by Vultr AgentBeta

7. Consume the test message with a new subscription name. If you repeat this check, use a different subscription name such as verify-sub-2. --subscription-position Earliest only applies when a subscription is first created.

   CONSOLE

   Copy

   $ docker exec -it pulsar bin/pulsar-client \
       --auth-plugin org.apache.pulsar.client.impl.auth.AuthenticationToken \
       --auth-params "token:$(cat ~/pulsar/data/admin-token.txt)" \
       consume persistent://public/default/verify-test \
       --subscription-name verify-sub-1 \
       --num-messages 1 \
       --subscription-position Earliest

   Explain Code

   Powered by Vultr AgentBeta

8. Check the status of the uppercase function.

   CONSOLE

   Copy

   $ docker exec -it pulsar bin/pulsar-admin \
       --auth-plugin org.apache.pulsar.client.impl.auth.AuthenticationToken \
       --auth-params "token:$(cat ~/pulsar/data/admin-token.txt)" \
       functions status \
       --tenant public \
       --namespace default \
       --name uppercase

   Explain Code

   Powered by Vultr AgentBeta

   A healthy function shows "running" : true.

Migrate from Google Pub/Sub to Apache Pulsar

Migrating from Google Pub/Sub to Pulsar involves mapping Google concepts to Pulsar equivalents, updating client configurations, and transferring message schemas and data. The examples below assume you completed Configure Authentication and Authorization earlier in this article.

Topic and Subscription Migration

Google Pub/Sub topics map directly to Pulsar persistent topics, and subscriptions map to Pulsar subscriptions with different types.

• Topics: Create a Pulsar topic for each Pub/Sub topic using the format persistent://tenant/namespace/topic-name.
• Subscriptions: Map Pub/Sub subscriptions to Pulsar subscription types:
  • Pull subscriptions map to Shared or Key_Shared subscriptions for competing consumers.
  • Push subscriptions can be replaced with Pulsar Functions for serverless processing.

Application Code Updates

Replace the Google Cloud Pub/Sub client library with the Pulsar client library for your language. Pulsar provides official clients for Java, Python, Go, Node.js, C++, and C#.

Message Schema Migration

Google Pub/Sub supports Avro and Protocol Buffer schemas per topic. Pulsar includes a built-in schema registry. Refer to the Pulsar schema documentation for the schema format Pulsar expects and how to register schemas using pulsar-admin.

Push Subscriptions to Pulsar Functions

Google Pub/Sub push subscriptions deliver messages to HTTP endpoints. Replace these with Pulsar Functions that forward messages to external services. Refer to the Pulsar Functions documentation for implementation details.

Dead-Letter Topic Configuration

Google Pub/Sub routes failed messages to a dead-letter topic after the maximum delivery attempts are exceeded. Pulsar supports equivalent behavior through dead-letter policies configured on subscriptions. Refer to the Pulsar dead-letter topic documentation for configuration details.

Data Migration Strategy

For topics with existing messages that need to be preserved:

• Dual-Write Period: Configure producers to write to both Pub/Sub and Pulsar during the migration window.
• Backfill Historical Data: Export messages from Pub/Sub to Cloud Storage using an export subscription, then replay the exported files into Pulsar using a custom producer script.
• Cutover: Once consumers are migrated and caught up, disable the Pub/Sub producers.

Migration Considerations

• Message Ordering: Pub/Sub delivers messages in order per ordering key when the publisher sets an ordering key and the subscription has message ordering enabled. Use Pulsar Key_Shared subscriptions for equivalent behavior.
• Message Retention: Pub/Sub retains messages for 7 days by default. Configure Pulsar namespace retention policies to match.
• Authentication: Replace Google Cloud IAM with Pulsar JWT tokens or integrate with your identity provider using OAuth 2.0.
• Monitoring: Replace Cloud Monitoring with Pulsar's built-in Prometheus metrics endpoint at http://localhost:8080/metrics/.
• Client Libraries: Pulsar provides official clients for Java, Python, Go, Node.js, C++, and C#. Verify client library availability for your technology stack.
• Throughput Limits: Pub/Sub has per-project quotas. Pulsar throughput depends on your infrastructure sizing.
• Exactly-Once Semantics: Pub/Sub provides at-least-once delivery by default. Pulsar supports exactly-once semantics with transactional producers.

Conclusion

You have deployed Apache Pulsar as a self-hosted alternative to Google Pub/Sub using Docker, configured multi-tenancy with namespaces and retention policies, deployed Pulsar Functions for serverless stream processing, set up IO connectors for data integration, enabled JWT-based authentication and authorization, and applied migration strategies for topics, schemas, and dead-letter queues. For more information, refer to the official Apache Pulsar documentation.