Monitoring Laravel Queues with Horizon and Vultr Managed Databases for Caching

Updated on June 20, 2024
Monitoring Laravel Queues with Horizon and Vultr Managed Databases for Caching header image

Introduction

Horizon is a queue manager for Laravel that gives you a dashboard and code-driven configuration to monitor and manage your queues. This guide explains how to use the Vultr Managed Database for Caching with Laravel Horizon, including the steps to make Laravel Horizon production-ready.

Prerequisites

Before you begin:

Create the MySQL Database

  1. Log in to MySQL.

     $ sudo mysql
  2. Create a new database.

     CREATE DATABASE example_database;
  3. Create a new MySQL user.

     CREATE USER 'example_user'@'%' IDENTIFIED WITH caching_sha2_password BY 'password';
  4. Grant the user permission to the database.

     GRANT ALL ON example_database.* TO 'example_user'@'%';
  5. Exit MySQL.

     EXIT
  6. Log in to MySQL with the new user.

     mysql -u example_user -p
  7. Check if the user has access to the database.

     SHOW DATABASES;

Set Up the Web Server

This guide uses the LEMP stack, and Laravel 10 requires at least PHP version 8.1. Verify the installed PHP version on your server using the following command.

$ php --version

Output:

PHP 8.1.2-1ubuntu2.11 (cli) (built: Feb 22 2023 22:56:18(NTS)
Copyright (c) The PHP Group
Zend Engine v4.1.2, Copyright (c) Zend Technologies with Zend OPcache v8.1.2-1ubuntu2.11, Copyright (c)by Zend Technologies
  1. Install the necessary PHP extensions.

     $ sudo apt install php-{cli,fpm,mysql,gd,soap,mbstring,bcmath,common,xml,curl,imagick,zip,redis}
  2. Install Composer using the following commands.

     $ php -r "copy('https://getcomposer.    installer'composer-setup.php');"
     $ php composer-setup.php
     $ php -r "unlink('composer-setup.php');"
     $ sudo mv composer.phar /usr/local/bin/composer
  3. Create a new user for the PHP FastCGI Process Manager (PHP-FPM) pool. This user runs the PHP-FPM process and has full privileges to the website files. Setting up a new user adds improves the security and isolation for your website.

     $ sudo adduser website
  4. Add the user to the Nginx web server group.

     $ sudo usermod -a -G website www-data
  5. Create the website files directory.

     $ sudo mkdir /var/www/website
  6. Grant the user ownership privileges to the directory.

     $ sudo chown website:website /var/www/website
  7. Copy the default PHP-FPM pool configuration as a template for the new pool.

     $ sudo cp /etc/php/8.1/fpm/pool.d/www.conf /etc/php/8.1/fpm/pool.d/website.conf
  8. Open the pool configuration file.

     $ sudo nano /etc/php/8.1/fpm/pool.d/website.conf

    Make changes to the file as follows:

    • [www] to [website].
    • user = www-data to user = website.
    • group = www-data to group = website.
    • listen = /run/php/php8.1-fpm.sock to listen = /run/php/php8.1-fpm-website.sock.

    Save and close the file.

  9. Delete the default PHP-FPM pool configuration.

     $ sudo rm /etc/php/8.1/fpm/pool.d/www.conf
  10. Restart PHP-FPM:

     $ sudo systemctl restart php8.1-fpm

Configure Nginx

  1. Disable the default Nginx configuration.

     $ sudo unlink /etc/nginx/sites-enabled/default
  2. Create a new Nginx configuration file.

     $ sudo nano /etc/nginx/sites-available/website
  3. Add the following configurations to the file. Replace example.com with your domain name.

     server {
         listen 80;
         listen [::]:80;
    
         server_name example.com;
    
         root /var/www/website/public;
    
         index index.html index.htm index.php;
    
         charset utf-8;
    
         client_max_body_size 10m;
         client_body_timeout 60s;
    
         location / {
             try_files $uri $uri/ /index.php?$query_string;
         }
    
         location = /favicon.ico { access_log off; log_not_found off; }
         location = /robots.txt  { access_log off; log_not_found off; }
    
         access_log /var/log/example.com-access.log;
         error_log  /var/log/example.com-error.log error;
    
         error_page 404 /index.php;
    
         location ~ \.php$ {
             include snippets/fastcgi-php.conf;
             fastcgi_pass unix:/run/php/php8.1-fpm-website.sock;
             fastcgi_buffers 32 32k;
             fastcgi_buffer_size 32k;
         }
     }

    Save and close the file.

  4. Enable Nginx configuration.

     $ sudo ln -s /etc/nginx/sites-available/website /etc/nginx/sites-enabled/
  5. Check the Nginx configuration for errors.

     $ sudo nginx -t
  6. Restart Nginx.

     $ sudo systemctl restart nginx

Create a Laravel Project

The example project in this guide is a newsletter app that sends emails to subscribers. It utilizes the Laravel queues to process email delivery in the background. The app also uses the Laravel Breeze starter kits to scaffold the user authentication system.

  1. You need to run the Laravel configuration tasks as the website user. Switch to the account by running the following command.

     $ sudo su - website
  2. Change to the website directory.

     $ cd /var/www/website
  3. Using Composer, create a new Laravel project.

     $ composer create-project laravel/laravel .
  4. Open the .env file.

     $ nano .env
  5. Set the Laravel mail driver to log. This blocks Laravel from sending the actual email and writes all emails to your log files for inspection.

     MAIL_MAILER=log
  6. Add your MySQL database configurations.

     DB_DATABASE=example_database
     DB_USERNAME=example_user
     DB_PASSWORD=password

    Replace the above values with your actual database settings as follows:

    • DB_DATABASE: MySQL database name.
    • DB_USERNAME: MySQL username that has permission to the database.
    • DB_PASSWORD: MySQL user password.

    Save and close the .env file.

  7. Install Laravel Breeze.

     $ composer require laravel/breeze --dev
  8. Scaffold your application using the Breeze Blade stacks.

     $ php artisan breeze:install blade
  9. Create a Subscriber Eloquent model and migration.

     $ php artisan make:model Subscriber -m
  10. Open the migration file.

     $ nano database/migrations/2023_05_14_165944_create_subscribers_table.php
  11. Add the following columns for the subscribers schema.

     Schema::create('subscribers', function (Blueprint $table) {
         $table->id();
         $table->string('name');
         $table->string('email');
         $table->timestamps();
     });
  12. Create a factory for the Subscriber model.

     $ php artisan make:factory SubscriberFactory -m Subscriber
  13. Open the factory class.

     $ nano database/factories/SubscriberFactory.php
  14. Update the definition method with the following code.

     public function definition()
     {
         return [
             'name' => fake()->name(),
             'email' => fake()->unique()->safeEmail(),
         ];
     }
  15. Open the database seeder file.

     $ nano database/seeders/DatabaseSeeder.php
  16. Add the following code to the run method.

     public function run()
     {
         \App\Models\User::factory()->create([
             'name' => 'Test User',
             'email' => 'test@example.com',
         ]);
    
         \App\Models\Subscriber::factory()->count(100)->create();
     }
  17. Run the database migrations.

     $ php artisan migrate
  18. Run the database seeder.

     $ php artisan db:seed

Create Mailable Class

  1. Generate a Mailable class using the Artisan command.

     $ php artisan make:mail SendNewsletter -m
  2. Open the Mailable class.

     $ nano app/Mail/SendNewsletter.php
  3. Add subject and content property to the class.

     /**
      * The email subject.
      *
      * @var string
      */
     public $subject;
    
     /**
      * The email content.
      *
      * @var string
      */
     public $content;
  4. Update the constructor method with the following code.

     public function __construct($subject, $content)
     {
         $this->subject = $subject;
         $this->content = $content;
     }
  5. Update the envelope method with the following code.

     public function envelope()
     {
         return new Envelope(
             subject: $this->subject,
         );
     }

    Save and close the file.

    The full Mailable class function should lok like the one below.

      <?php
    
      namespace App\Mail;
    
      use Illuminate\Bus\Queueable;
      use Illuminate\Contracts\Queue\ShouldQueue;
      use Illuminate\Mail\Mailable;
      use Illuminate\Mail\Mailables\Content;
      use Illuminate\Mail\Mailables\Envelope;
      use Illuminate\Queue\SerializesModels;
    
      class SendNewsletter extends Mailable
      {
        use Queueable, SerializesModels;
    
    
        public $subject;
    
          /**
          * The email content.
          *
          * @var string
          */
      public $content;
    
      /**
      * Create a new message instance.
      *
      * @param string $subject
      * @param string $content
      * @return void
      */
     public function __construct($subject, $content)
     {
         $this->subject = $subject;
         $this->content = $content;
     }
    
     /**
      * Get the message envelope.
      *
      * @return \Illuminate\Mail\Mailables\Envelope
      */
     public function envelope()
     {
         return new Envelope(
             subject: $this->subject,
         );
     }
    
     /**
      * Get the message content definition.
      *
      * @return \Illuminate\Mail\Mailables\Content
      */
     public function content()
     {
         return new Content(
             markdown: 'mail.send-newsletter',
         );
     }
    
     /**
      * Get the attachments for the message.
      *
      * @return array
      */
     public function attachments()
     {
         return [];
        }
      }
  6. Open the mail template.

     $ nano resources/views/mail/send-newsletter.blade.php
  7. Update it with the following code.

     <x-mail::message>
     {{ $content }}
     </x-mail::message>

Create the Controller Class

  1. Generate the DashboardController class.

     $ php artisan make:controller DashboardController
  2. Open the DashboardController class.

     $ nano app/Http/Controllers/DashboardController.php
  3. Load necessary classes at the top of the file.

     use Illuminate\Support\Facades\Mail;
     use App\Models\Subscriber;
     use App\Mail\SendNewsletter;
  4. Create the index action with the following code.

     public function index()
     {
        $subscribers = Subscriber::paginate(20);
        return view('dashboard', compact('subscribers'));
     }
  5. Create the newsletter action that shows a form to broadcast the email.

     public function newsletter()
     {
        return view('newsletter');
     }
  6. Create the sendNewsletter action that processes the email delivery to all subscribers.

     public function sendNewsletter(Request $request)
     {
        $request->validate([
            'subject' => 'required',
            'content' => 'required',
        ]);
    
        $subscribers = Subscriber::get();
    
        foreach ($subscribers as $subscriber) {
            Mail::to($subscriber->email)->queue(new SendNewsletter($request->subject, $request->content));
        }
    
        return redirect()->route('newsletter.form')->with('message', 'Success');
     }
  7. Below is the full content of the DashboardController class:

     <?php
    
     namespace App\Http\Controllers;
    
     use Illuminate\Http\Request;
     use Illuminate\Support\Facades\Mail;
     use App\Models\Subscriber;
     use App\Mail\SendNewsletter;
    
     class DashboardController extends Controller
     {
        public function index()
        {
            $subscribers = Subscriber::paginate(20);
            return view('dashboard', compact('subscribers'));
        }
    
        public function newsletter()
        {
            return view('newsletter');
        }
    
        public function sendNewsletter(Request $request)
        {
            $request->validate([
                'subject' => 'required',
                'content' => 'required',
            ]);
    
    
            $subscribers = Subscriber::get();
    
            foreach ($subscribers as $subscriber) {
                Mail::to($subscriber->email)->queue(new SendNewsletter($request->subject, $request->content));
            }
    
            return redirect()->route('newsletter.form')->with('message', 'Success');
        }
     }

    Save and close the file.

Defining Routes

  1. Open the routes/web.php file.

     $ nano routes/web.php
  2. Add the DashboardController at the top of the file.

     use App\Http\Controllers\DashboardController;
  3. Update the dashboard route with the following code.

     R oute::get('/dashboard', [DashboardController::class, 'index'])->middleware(['auth', 'verified'])->name('dashboard');
  4. Add the newsletter routes:

     Route::get('/newsletter', [DashboardController::class, 'newsletter'])->middleware(['auth', 'verified'])->name('newsletter.form');
     Route::post('/newsletter', [DashboardController::class, 'sendNewsletter'])->middleware(['auth', 'verified'])->name('newsletter.send');

    Save and close the file.

Create Views

  1. Open the resources/views/dashboard.blade.php file.

     $ nano resources/views/dashboard.blade.php
  2. Update it with the following code.

     <x-app-layout>
        <x-slot name="header">
            <h2 class="font-semibold text-xl text-gray-800 leading-tight">
                {{ __('Dashboard') }}
            </h2>
        </x-slot>
    
        <div class="py-12">
            <div class="max-w-7xl mx-auto sm:px-6 lg:px-8">
                <div class="sm:flex sm:items-center">
                    <div class="sm:flex-auto">
                      <h3 class="text-xl font-semibold text-gray-900">Subscribers ({{ $subscribers->total() }})</h3>
                    </div>
                    <div class="mt-4 sm:mt-0 sm:ml-16 sm:flex-none">
                        <a href="{{ route('newsletter.form') }}" class="inline-flex items-center justify-center rounded-md border border-transparent bg-indigo-600 px-4 py-2 text-sm font-medium text-white shadow-sm hover:bg-indigo-700 focus:outline-none focus:ring-2 focus:ring-indigo-500 focus:ring-offset-2 sm:w-auto">
                            Send Newsletter
                        </a>
                    </div>
                </div>
                <div class="overflow-hidden shadow ring-1 ring-black ring-opacity-5 md:rounded-lg mt-8">
                    <table class="min-w-full divide-y divide-gray-300">
                        <thead class="bg-gray-50">
                            <tr>
                                <th scope="col" class="py-3.5 pl-4 pr-3 text-left text-sm font-semibold text-gray-900 sm:pl-6">Name</th>
                                <th scope="col" class="px-3 py-3.5 text-left text-sm font-semibold text-gray-900">Email</th>
                            </tr>
                        </thead>
                        <tbody class="divide-y divide-gray-200 bg-white">
                            @foreach ($subscribers as $subscriber)
                            <tr>
                                <td class="whitespace-nowrap py-4 pl-4 pr-3 text-sm font-medium text-gray-900 sm:pl-6">{{ $subscriber->name }}</td>
                                <td class="whitespace-nowrap px-3 py-4 text-sm text-gray-500">{{ $subscriber->email }}</td>
                            </tr>
                            @endforeach
                        </tbody>
                    </table>
                </div>
    
                <div class="mt-8">{{ $subscribers->links() }}</div>
            </div>
        </div>
     </x-app-layout>
  3. Create a resources/views/newsletter.blade.php file with the following code.

     <x-app-layout>
        <x-slot name="header">
            <h2 class="font-semibold text-xl text-gray-800 leading-tight">
                {{ __('Send Newsletter') }}
            </h2>
        </x-slot>
    
        <div class="py-12">
            <div class="max-w-7xl mx-auto sm:px-6 lg:px-8">
                <div class="p-4 sm:p-8 bg-white shadow sm:rounded-lg">
                    <div class="max-w-xl mx-auto">
                        @if (session()->has('message'))
                            <div class="rounded-md bg-green-50 p-4 text-sm font-medium text-green-800">{{ session('message') }}</div>
                        @endif
                        <form method="post" action="{{ route('newsletter.send') }}" class="mt-6 space-y-6">
                            @csrf
    
                            <div>
                                <x-input-label for="subject" :value="__('Subject')" />
                                <x-text-input id="subject" name="subject" type="text" class="mt-1 block w-full" required autofocus autocomplete="subject" />
                                <x-input-error class="mt-2" :messages="$errors->get('subject')" />
                            </div>
    
                            <div>
                                <x-input-label for="content" :value="__('Content')" />
                                <textarea id="content" rows="10" name="content" class="mt-1 block w-full border-gray-300 focus:border-indigo-500 focus:ring-indigo-500 rounded-md shadow-sm" required autocomplete="content"></textarea>
                                <x-input-error class="mt-2" :messages="$errors->get('content')" />
                            </div>
    
                            <div class="flex items-center gap-4">
                                <x-primary-button>{{ __('Send') }}</x-primary-button>
                            </div>
                        </form>
                    </div>
                </div>
            </div>
        </div>
     </x-app-layout>
  4. Compile the front-end assets.

     $ npm run build

Install and Configure Laravel Horizon

  1. Install Horizon using Composer.

     $ composer require laravel/horizon
  2. Publish Horizon assets.

     $ php artisan horizon:install

Configure the Horizon Queue Worker

  1. Open the Horizon configuration file.

     $ nano config/horizon.php
  2. Navigate to the environments configuration option.

     'environments' => [
        'production' => [
            'supervisor-1' => [
                'maxProcesses' => 10,
                'balanceMaxShift' => 1,
                'balanceCooldown' => 3,
            ],
        ],
    
        'local' => [
            'supervisor-1' => [
                'maxProcesses' => 3,
            ],
        ],
     ],

    In this option, configure the queue worker behavior for each environment. For example, set the value of the maxProcesses option in the production environment to 20. This allows the worker to scale up to 20 processes in production.

     'production' => [
        'supervisor-1' => [
            'maxProcesses' => 20,
            'balanceMaxShift' => 1,
            'balanceCooldown' => 3,
        ],
     ],

Configure the Redis® Connection

  1. Open the .env file.

     $ nano .env
  2. Update the queue connection to Redis®.

     QUEUE_CONNECTION=redis
  3. Add Vultr Managed Database for Caching configurations.

     REDIS_HOST=tls://[YOUR_VULTR_REDIS_HOST]
     REDIS_PASSWORD=password
     REDIS_PORT=6379

    Replace the default values with your actual database details as below.

    • REDIS_HOST – Vultr Managed Database for Caching host.
    • REDIS_PASSWORD – Enter your Vultr Managed Database for Caching password.
    • REDIS_PORT – The Vultr Managed Database for Caching port.

    Save and close the .env file.

Prepare Horizon for Production

Horizon exposes the dashboard at the /horizon path. In production, you should protect this path using the authorization gate that controls access to the dashboard.

  1. Open the HorizonServiceProvider.php file.

     $ nano app/Providers/HorizonServiceProvider.php
  2. In the gate method, add the user's email that you want to allow to the Horizon dashboard.

     protected function gate(): void
     {
        Gate::define('viewHorizon', function ($user) {
            return in_array($user->email, [
                'test@example.com'
            ]);
        });
     }

    Save and close the file.

  3. Open the .env file.

     $ nano .env
  4. Disable debug messages.

     APP_DEBUG=false
  5. Change the app environment to production.

     APP_ENV=production

Run Horizon on Production using Supervisor

  1. Install Supervisor.

     $ sudo apt-get install supervisor
  2. Create a Supervisor configuration file for Horizon.

     $ sudo nano /etc/supervisor/conf.d/horizon.conf
  3. Add the following configurations to the file.

     [program:horizon]
     process_name=%(program_name)s
     command=php /var/www/website/artisan horizon
     autostart=true
     autorestart=true
     user=website
     redirect_stderr=true
     stdout_logfile=/var/log/horizon.log
     stopwaitsecs=3600
  4. Start the Supervisor process as below.

     $ sudo supervisorctl reread
     $ sudo supervisorctl update
     $ sudo supervisorctl start horizon

Secure the Website with Trusted Let's Encrypt SSL Certificates

  1. Install the Certbot Let's Encrypt tool.

     $ sudo snap install --classic certbot
  2. Activate the Certbot command.

     $ sudo ln -s /snap/bin/certbot /usr/bin/certbot
  3. Generate the SSL certificates. Replace example.com with your actual domain name, and user@example.com with your email address.

     $ sudo certbot --nginx -d example.com -d www.example.com -m user@example.com --agree-tos
  4. Test that Certbot auto-renews the SSL certificate.

     $ certbot renew --dry-run
  5. In a web browser, visit your domain and verify that it serves HTTPS requests.

     https://example.com

Configure Firewall

By default, Uncomplicated Firewall (UFW) is active on Vultr servers, configure the firewall to allow HTTP and HTTPS traffic as below.

  1. Allow the HTTP port 80.

     $ sudo ufw allow 80/tcp
  2. Allow HTTPS port 443.

     $ sudo ufw allow 443/tcp
  3. Restart the firewall to save changes.

     $ sudo ufw reload
  4. Verify that the rules are added to the firewall table.

     $ sudo ufw status

Test the Application

  1. In a web browser, open the application.

     https://example.com
  2. The default Laravel landing page should display.

  3. Click the login link and log in with a user from the database seeder.

  4. Click the Send Newsletter button on the dashboard page.

  5. Fill in the Subject and Content field.

  6. Click the Send button.

  7. The application then creates a job queue to send the email newsletter to each subscriber.

Monitor the Jobs in Horizon Dashboard

  1. Log in as a user with access to the Horizon dashboard.
  2. Visit the Horizon dashboard at the https://[YOUR_DOMAIN_NAME]/horizon path.
  3. Check the Horizon status. If it's active, it means that Horizon is running and can process all your job queues.
  4. See your job queues at the Pending Jobs, Completed Jobs, or Failed Jobs menus depending on their status.
  5. If the job fails, see the job exception logs.

Check the Email Delivery

For testing purposes, this guide uses the log driver for the Laravel mail. It means that the sample app doesn't send the actual emails. It writes all emails in the log file. To check if the app successfully sends an email, open the laravel.log file.

$ nano storage/logs/laravel.log

All email content should display in the log file as below:

[2023-05-14 12:11:52] local.DEBUG: From: Laravel <hello@example.com>
To: jschmitt@example.org
Subject: tes
MIME-Version: 1.0
Date: Sun, 14 May 2023 12:11:52 +0000
Message-ID: <8dc916eb3d9248ee1b10a99f5576b424@example.com>
Content-Type: multipart/alternative; boundary=mxw5N1XT
...

Conclusion

In this guide you have used the Vultr Managed Database for Caching with Laravel Horizon and used it to set up a production-ready application. Laravel Horizon allows you to monitor job queues in a single dashboard. You can see the exception logs if a job fails and can retry it. It also has a queue worker that you can configure differently in each environment using a single configuration file.

More Information