Monitoring Laravel Queues with Horizon and Vultr Managed Databases for Caching

Published on June 28, 2023•Updated on June 20, 2024

Introduction

Horizon is a queue manager for Laravel that gives you a dashboard and code-driven configuration to monitor and manage your queues. This guide explains how to use the Vultr Managed Database for Caching with Laravel Horizon, including the steps to make Laravel Horizon production-ready.

Prerequisites

Before you begin:

Deploy a Ubuntu 22.04 server on Vultr.
Create a non-root user with sudo privileges.
Install LEMP on the server.
Install NodeJS on the server.
Deploy a Vultr Managed Database for Caching.

Create the MySQL Database

Log in to MySQL.
```
 $ sudo mysql
```
Create a new database.
```
 CREATE DATABASE example_database;
```

Create a new MySQL user.

 CREATE USER 'example_user'@'%' IDENTIFIED WITH caching_sha2_password BY 'password';

Grant the user permission to the database.

 GRANT ALL ON example_database.* TO 'example_user'@'%';

Exit MySQL.
```
 EXIT
```
Log in to MySQL with the new user.
```
 mysql -u example_user -p
```
Check if the user has access to the database.
```
 SHOW DATABASES;
```

Set Up the Web Server

This guide uses the LEMP stack, and Laravel 10 requires at least PHP version 8.1. Verify the installed PHP version on your server using the following command.

$ php --version

Output:

PHP 8.1.2-1ubuntu2.11 (cli) (built: Feb 22 2023 22:56:18(NTS)
Copyright (c) The PHP Group
Zend Engine v4.1.2, Copyright (c) Zend Technologies with Zend OPcache v8.1.2-1ubuntu2.11, Copyright (c)by Zend Technologies

Install the necessary PHP extensions.

 $ sudo apt install php-{cli,fpm,mysql,gd,soap,mbstring,bcmath,common,xml,curl,imagick,zip,redis}

Install Composer using the following commands.

 $ php -r "copy('https://getcomposer.    installer'composer-setup.php');"
 $ php composer-setup.php
 $ php -r "unlink('composer-setup.php');"
 $ sudo mv composer.phar /usr/local/bin/composer

Create a new user for the PHP FastCGI Process Manager (PHP-FPM) pool. This user runs the PHP-FPM process and has full privileges to the website files. Setting up a new user adds improves the security and isolation for your website.
```
 $ sudo adduser website
```
Add the user to the Nginx web server group.
```
 $ sudo usermod -a -G website www-data
```
Create the website files directory.
```
 $ sudo mkdir /var/www/website
```
Grant the user ownership privileges to the directory.
```
 $ sudo chown website:website /var/www/website
```

Copy the default PHP-FPM pool configuration as a template for the new pool.

 $ sudo cp /etc/php/8.1/fpm/pool.d/www.conf /etc/php/8.1/fpm/pool.d/website.conf

Open the pool configuration file.
```
 $ sudo nano /etc/php/8.1/fpm/pool.d/website.conf
```
Make changes to the file as follows:
- [www] to [website].
- user = www-data to user = website.
- group = www-data to group = website.
- listen = /run/php/php8.1-fpm.sock to listen = /run/php/php8.1-fpm-website.sock.
Save and close the file.
Delete the default PHP-FPM pool configuration.
```
 $ sudo rm /etc/php/8.1/fpm/pool.d/www.conf
```
Restart PHP-FPM:
```
 $ sudo systemctl restart php8.1-fpm
```

Configure Nginx

Disable the default Nginx configuration.

 $ sudo unlink /etc/nginx/sites-enabled/default

Create a new Nginx configuration file.

 $ sudo nano /etc/nginx/sites-available/website

Add the following configurations to the file. Replace example.com with your domain name.

 server {
     listen 80;
     listen [::]:80;

     server_name example.com;

     root /var/www/website/public;

     index index.html index.htm index.php;

     charset utf-8;

     client_max_body_size 10m;
     client_body_timeout 60s;

     location / {
         try_files $uri $uri/ /index.php?$query_string;
     }

     location = /favicon.ico { access_log off; log_not_found off; }
     location = /robots.txt  { access_log off; log_not_found off; }

     access_log /var/log/example.com-access.log;
     error_log  /var/log/example.com-error.log error;

     error_page 404 /index.php;

     location ~ \.php$ {
         include snippets/fastcgi-php.conf;
         fastcgi_pass unix:/run/php/php8.1-fpm-website.sock;
         fastcgi_buffers 32 32k;
         fastcgi_buffer_size 32k;
     }
 }

Save and close the file.

Enable Nginx configuration.

 $ sudo ln -s /etc/nginx/sites-available/website /etc/nginx/sites-enabled/

Check the Nginx configuration for errors.
```
 $ sudo nginx -t
```
Restart Nginx.
```
 $ sudo systemctl restart nginx
```

Create a Laravel Project

The example project in this guide is a newsletter app that sends emails to subscribers. It utilizes the Laravel queues to process email delivery in the background. The app also uses the Laravel Breeze starter kits to scaffold the user authentication system.

You need to run the Laravel configuration tasks as the website user. Switch to the account by running the following command.
```
 $ sudo su - website
```
Change to the website directory.
```
 $ cd /var/www/website
```

Using Composer, create a new Laravel project.

 $ composer create-project laravel/laravel .

Open the .env file.
```
 $ nano .env
```
Set the Laravel mail driver to log. This blocks Laravel from sending the actual email and writes all emails to your log files for inspection.
```
 MAIL_MAILER=log
```
Add your MySQL database configurations.
```
 DB_DATABASE=example_database
 DB_USERNAME=example_user
 DB_PASSWORD=password
```
Replace the above values with your actual database settings as follows:
- DB_DATABASE: MySQL database name.
- DB_USERNAME: MySQL username that has permission to the database.
- DB_PASSWORD: MySQL user password.
Save and close the .env file.

Install Laravel Breeze.

 $ composer require laravel/breeze --dev

Scaffold your application using the Breeze Blade stacks.
```
 $ php artisan breeze:install blade
```
Create a Subscriber Eloquent model and migration.
```
 $ php artisan make:model Subscriber -m
```

Open the migration file.

 $ nano database/migrations/2023_05_14_165944_create_subscribers_table.php

Add the following columns for the subscribers schema.

 Schema::create('subscribers', function (Blueprint $table) {
     $table->id();
     $table->string('name');
     $table->string('email');
     $table->timestamps();
 });

Create a factory for the Subscriber model.

 $ php artisan make:factory SubscriberFactory -m Subscriber

Open the factory class.

 $ nano database/factories/SubscriberFactory.php

Update the definition method with the following code.

 public function definition()
 {
     return [
         'name' => fake()->name(),
         'email' => fake()->unique()->safeEmail(),
     ];
 }

Open the database seeder file.

 $ nano database/seeders/DatabaseSeeder.php

Add the following code to the run method.

 public function run()
 {
     \App\Models\User::factory()->create([
         'name' => 'Test User',
         'email' => 'test@example.com',
     ]);

     \App\Models\Subscriber::factory()->count(100)->create();
 }

Run the database migrations.
```
 $ php artisan migrate
```
Run the database seeder.
```
 $ php artisan db:seed
```

Create Mailable Class

Generate a Mailable class using the Artisan command.
```
 $ php artisan make:mail SendNewsletter -m
```
Open the Mailable class.
```
 $ nano app/Mail/SendNewsletter.php
```

Add subject and content property to the class.

 /**
  * The email subject.
  *
  * @var string
  */
 public $subject;

 /**
  * The email content.
  *
  * @var string
  */
 public $content;

Update the constructor method with the following code.

 public function __construct($subject, $content)
 {
     $this->subject = $subject;
     $this->content = $content;
 }

Update the envelope method with the following code.

 public function envelope()
 {
     return new Envelope(
         subject: $this->subject,
     );
 }

Save and close the file.

The full Mailable class function should lok like the one below.

  <?php

  namespace App\Mail;

  use Illuminate\Bus\Queueable;
  use Illuminate\Contracts\Queue\ShouldQueue;
  use Illuminate\Mail\Mailable;
  use Illuminate\Mail\Mailables\Content;
  use Illuminate\Mail\Mailables\Envelope;
  use Illuminate\Queue\SerializesModels;

  class SendNewsletter extends Mailable
  {
    use Queueable, SerializesModels;


    public $subject;

      /**
      * The email content.
      *
      * @var string
      */
  public $content;

  /**
  * Create a new message instance.
  *
  * @param string $subject
  * @param string $content
  * @return void
  */
 public function __construct($subject, $content)
 {
     $this->subject = $subject;
     $this->content = $content;
 }

 /**
  * Get the message envelope.
  *
  * @return \Illuminate\Mail\Mailables\Envelope
  */
 public function envelope()
 {
     return new Envelope(
         subject: $this->subject,
     );
 }

 /**
  * Get the message content definition.
  *
  * @return \Illuminate\Mail\Mailables\Content
  */
 public function content()
 {
     return new Content(
         markdown: 'mail.send-newsletter',
     );
 }

 /**
  * Get the attachments for the message.
  *
  * @return array
  */
 public function attachments()
 {
     return [];
    }
  }

Open the mail template.

 $ nano resources/views/mail/send-newsletter.blade.php

Update it with the following code.

 <x-mail::message>
 {{ $content }}
 </x-mail::message>

Create the Controller Class

Generate the DashboardController class.

 $ php artisan make:controller DashboardController

Open the DashboardController class.

 $ nano app/Http/Controllers/DashboardController.php

Load necessary classes at the top of the file.

 use Illuminate\Support\Facades\Mail;
 use App\Models\Subscriber;
 use App\Mail\SendNewsletter;

Create the index action with the following code.

 public function index()
 {
    $subscribers = Subscriber::paginate(20);
    return view('dashboard', compact('subscribers'));
 }

Create the newsletter action that shows a form to broadcast the email.
```
 public function newsletter()
 {
    return view('newsletter');
 }
```

Create the sendNewsletter action that processes the email delivery to all subscribers.

 public function sendNewsletter(Request $request)
 {
    $request->validate([
        'subject' => 'required',
        'content' => 'required',
    ]);

    $subscribers = Subscriber::get();

    foreach ($subscribers as $subscriber) {
        Mail::to($subscriber->email)->queue(new SendNewsletter($request->subject, $request->content));
    }

    return redirect()->route('newsletter.form')->with('message', 'Success');
 }

Below is the full content of the DashboardController class:

 <?php

 namespace App\Http\Controllers;

 use Illuminate\Http\Request;
 use Illuminate\Support\Facades\Mail;
 use App\Models\Subscriber;
 use App\Mail\SendNewsletter;

 class DashboardController extends Controller
 {
    public function index()
    {
        $subscribers = Subscriber::paginate(20);
        return view('dashboard', compact('subscribers'));
    }

    public function newsletter()
    {
        return view('newsletter');
    }

    public function sendNewsletter(Request $request)
    {
        $request->validate([
            'subject' => 'required',
            'content' => 'required',
        ]);


        $subscribers = Subscriber::get();

        foreach ($subscribers as $subscriber) {
            Mail::to($subscriber->email)->queue(new SendNewsletter($request->subject, $request->content));
        }

        return redirect()->route('newsletter.form')->with('message', 'Success');
    }
 }

Save and close the file.

Defining Routes

Open the routes/web.php file.
```
 $ nano routes/web.php
```
Add the DashboardController at the top of the file.
```
 use App\Http\Controllers\DashboardController;
```

Update the dashboard route with the following code.

 R oute::get('/dashboard', [DashboardController::class, 'index'])->middleware(['auth', 'verified'])->name('dashboard');

Add the newsletter routes:

 Route::get('/newsletter', [DashboardController::class, 'newsletter'])->middleware(['auth', 'verified'])->name('newsletter.form');
 Route::post('/newsletter', [DashboardController::class, 'sendNewsletter'])->middleware(['auth', 'verified'])->name('newsletter.send');

Save and close the file.

Create Views

Open the resources/views/dashboard.blade.php file.
```
 $ nano resources/views/dashboard.blade.php
```

Update it with the following code.

 <x-app-layout>
    <x-slot name="header">
        <h2 class="font-semibold text-xl text-gray-800 leading-tight">
            {{ __('Dashboard') }}
        </h2>
    </x-slot>

    <div class="py-12">
        <div class="max-w-7xl mx-auto sm:px-6 lg:px-8">
            <div class="sm:flex sm:items-center">
                <div class="sm:flex-auto">
                  <h3 class="text-xl font-semibold text-gray-900">Subscribers ({{ $subscribers->total() }})</h3>
                </div>
                <div class="mt-4 sm:mt-0 sm:ml-16 sm:flex-none">
                    <a href="{{ route('newsletter.form') }}" class="inline-flex items-center justify-center rounded-md border border-transparent bg-indigo-600 px-4 py-2 text-sm font-medium text-white shadow-sm hover:bg-indigo-700 focus:outline-none focus:ring-2 focus:ring-indigo-500 focus:ring-offset-2 sm:w-auto">
                        Send Newsletter
                    </a>
                </div>
            </div>
            <div class="overflow-hidden shadow ring-1 ring-black ring-opacity-5 md:rounded-lg mt-8">
                <table class="min-w-full divide-y divide-gray-300">
                    <thead class="bg-gray-50">
                        <tr>
                            <th scope="col" class="py-3.5 pl-4 pr-3 text-left text-sm font-semibold text-gray-900 sm:pl-6">Name</th>
                            <th scope="col" class="px-3 py-3.5 text-left text-sm font-semibold text-gray-900">Email</th>
                        </tr>
                    </thead>
                    <tbody class="divide-y divide-gray-200 bg-white">
                        @foreach ($subscribers as $subscriber)
                        <tr>
                            <td class="whitespace-nowrap py-4 pl-4 pr-3 text-sm font-medium text-gray-900 sm:pl-6">{{ $subscriber->name }}</td>
                            <td class="whitespace-nowrap px-3 py-4 text-sm text-gray-500">{{ $subscriber->email }}</td>
                        </tr>
                        @endforeach
                    </tbody>
                </table>
            </div>

            <div class="mt-8">{{ $subscribers->links() }}</div>
        </div>
    </div>
 </x-app-layout>

Create a resources/views/newsletter.blade.php file with the following code.

 <x-app-layout>
    <x-slot name="header">
        <h2 class="font-semibold text-xl text-gray-800 leading-tight">
            {{ __('Send Newsletter') }}
        </h2>
    </x-slot>

    <div class="py-12">
        <div class="max-w-7xl mx-auto sm:px-6 lg:px-8">
            <div class="p-4 sm:p-8 bg-white shadow sm:rounded-lg">
                <div class="max-w-xl mx-auto">
                    @if (session()->has('message'))
                        <div class="rounded-md bg-green-50 p-4 text-sm font-medium text-green-800">{{ session('message') }}</div>
                    @endif
                    <form method="post" action="{{ route('newsletter.send') }}" class="mt-6 space-y-6">
                        @csrf

                        <div>
                            <x-input-label for="subject" :value="__('Subject')" />
                            <x-text-input id="subject" name="subject" type="text" class="mt-1 block w-full" required autofocus autocomplete="subject" />
                            <x-input-error class="mt-2" :messages="$errors->get('subject')" />
                        </div>

                        <div>
                            <x-input-label for="content" :value="__('Content')" />
                            <textarea id="content" rows="10" name="content" class="mt-1 block w-full border-gray-300 focus:border-indigo-500 focus:ring-indigo-500 rounded-md shadow-sm" required autocomplete="content"></textarea>
                            <x-input-error class="mt-2" :messages="$errors->get('content')" />
                        </div>

                        <div class="flex items-center gap-4">
                            <x-primary-button>{{ __('Send') }}</x-primary-button>
                        </div>
                    </form>
                </div>
            </div>
        </div>
    </div>
 </x-app-layout>

Compile the front-end assets.
```
 $ npm run build
```

Install and Configure Laravel Horizon

Install Horizon using Composer.
```
 $ composer require laravel/horizon
```
Publish Horizon assets.
```
 $ php artisan horizon:install
```

Configure the Horizon Queue Worker

Open the Horizon configuration file.
```
 $ nano config/horizon.php
```

Navigate to the environments configuration option.

 'environments' => [
    'production' => [
        'supervisor-1' => [
            'maxProcesses' => 10,
            'balanceMaxShift' => 1,
            'balanceCooldown' => 3,
        ],
    ],

    'local' => [
        'supervisor-1' => [
            'maxProcesses' => 3,
        ],
    ],
 ],

In this option, configure the queue worker behavior for each environment. For example, set the value of the maxProcesses option in the production environment to 20. This allows the worker to scale up to 20 processes in production.

 'production' => [
    'supervisor-1' => [
        'maxProcesses' => 20,
        'balanceMaxShift' => 1,
        'balanceCooldown' => 3,
    ],
 ],

Configure the Redis® Connection

Open the .env file.
```
 $ nano .env
```
Update the queue connection to Redis®.
```
 QUEUE_CONNECTION=redis
```
Add Vultr Managed Database for Caching configurations.
```
 REDIS_HOST=tls://[YOUR_VULTR_REDIS_HOST]
 REDIS_PASSWORD=password
 REDIS_PORT=6379
```
Replace the default values with your actual database details as below.
- REDIS_HOST – Vultr Managed Database for Caching host.
- REDIS_PASSWORD – Enter your Vultr Managed Database for Caching password.
- REDIS_PORT – The Vultr Managed Database for Caching port.
Save and close the .env file.

Prepare Horizon for Production

Horizon exposes the dashboard at the /horizon path. In production, you should protect this path using the authorization gate that controls access to the dashboard.

Open the HorizonServiceProvider.php file.

 $ nano app/Providers/HorizonServiceProvider.php

In the gate method, add the user's email that you want to allow to the Horizon dashboard.

 protected function gate(): void
 {
    Gate::define('viewHorizon', function ($user) {
        return in_array($user->email, [
            'test@example.com'
        ]);
    });
 }

Save and close the file.

Open the .env file.
```
 $ nano .env
```
Disable debug messages.
```
 APP_DEBUG=false
```
Change the app environment to production.
```
 APP_ENV=production
```

Run Horizon on Production using Supervisor

Install Supervisor.
```
 $ sudo apt-get install supervisor
```
Create a Supervisor configuration file for Horizon.
```
 $ sudo nano /etc/supervisor/conf.d/horizon.conf
```

Add the following configurations to the file.

 [program:horizon]
 process_name=%(program_name)s
 command=php /var/www/website/artisan horizon
 autostart=true
 autorestart=true
 user=website
 redirect_stderr=true
 stdout_logfile=/var/log/horizon.log
 stopwaitsecs=3600

Start the Supervisor process as below.

 $ sudo supervisorctl reread
 $ sudo supervisorctl update
 $ sudo supervisorctl start horizon

Secure the Website with Trusted Let's Encrypt SSL Certificates

Install the Certbot Let's Encrypt tool.
```
 $ sudo snap install --classic certbot
```

Activate the Certbot command.

 $ sudo ln -s /snap/bin/certbot /usr/bin/certbot

Generate the SSL certificates. Replace example.com with your actual domain name, and user@example.com with your email address.
```
 $ sudo certbot --nginx -d example.com -d www.example.com -m user@example.com --agree-tos
```
Test that Certbot auto-renews the SSL certificate.
```
 $ certbot renew --dry-run
```
In a web browser, visit your domain and verify that it serves HTTPS requests.
```
 https://example.com
```

Configure Firewall

By default, Uncomplicated Firewall (UFW) is active on Vultr servers, configure the firewall to allow HTTP and HTTPS traffic as below.

Allow the HTTP port 80.
```
 $ sudo ufw allow 80/tcp
```
Allow HTTPS port 443.
```
 $ sudo ufw allow 443/tcp
```
Restart the firewall to save changes.
```
 $ sudo ufw reload
```
Verify that the rules are added to the firewall table.
```
 $ sudo ufw status
```

Test the Application

In a web browser, open the application.
```
 https://example.com
```
The default Laravel landing page should display.
Click the login link and log in with a user from the database seeder.
Click the Send Newsletter button on the dashboard page.
Fill in the Subject and Content field.
Click the Send button.
The application then creates a job queue to send the email newsletter to each subscriber.

Monitor the Jobs in Horizon Dashboard

Log in as a user with access to the Horizon dashboard.
Visit the Horizon dashboard at the https://[YOUR_DOMAIN_NAME]/horizon path.
Check the Horizon status. If it's active, it means that Horizon is running and can process all your job queues.
See your job queues at the Pending Jobs, Completed Jobs, or Failed Jobs menus depending on their status.
If the job fails, see the job exception logs.

Check the Email Delivery

For testing purposes, this guide uses the log driver for the Laravel mail. It means that the sample app doesn't send the actual emails. It writes all emails in the log file. To check if the app successfully sends an email, open the laravel.log file.

$ nano storage/logs/laravel.log

All email content should display in the log file as below:

[2023-05-14 12:11:52] local.DEBUG: From: Laravel <hello@example.com>
To: jschmitt@example.org
Subject: tes
MIME-Version: 1.0
Date: Sun, 14 May 2023 12:11:52 +0000
Message-ID: <8dc916eb3d9248ee1b10a99f5576b424@example.com>
Content-Type: multipart/alternative; boundary=mxw5N1XT
...

Conclusion

In this guide you have used the Vultr Managed Database for Caching with Laravel Horizon and used it to set up a production-ready application. Laravel Horizon allows you to monitor job queues in a single dashboard. You can see the exception logs if a job fails and can retry it. It also has a queue worker that you can configure differently in each environment using a single configuration file.

Monitoring Laravel Queues with Horizon and Vultr Managed Databases for Caching

Introduction

Prerequisites

Create the MySQL Database

Set Up the Web Server

Configure Nginx

Create a Laravel Project

Create Mailable Class

Create the Controller Class

Defining Routes

Create Views

Install and Configure Laravel Horizon

Configure the Horizon Queue Worker

Configure the Redis® Connection

Prepare Horizon for Production

Run Horizon on Production using Supervisor

Secure the Website with Trusted Let's Encrypt SSL Certificates

Configure Firewall

Test the Application

Monitor the Jobs in Horizon Dashboard

Check the Email Delivery

Conclusion

More Information

Products

Features

Marketplace

Resources

Company