How to Install GPT4ALL - An Opensource Chatbot Application for Running Large Language Models (LLMs)

GPT4All is an open-source graphical desktop application running large language models (LLMs) locally. GPT4All supports most desktop operating systems including macOS, Windows, and Linux to run LLMs without an active internet connection. It supports GGUF format models from sources including Hugging Face and integrates with API-based providers including Groq, OpenAI, and Mistral to run models privately.

This guide covers how to install GPT4All, download, run, and chat with LLMs locally. You will install the latest GPT4All version, set up API-based access, and chat with LLMs using local documents like spreadsheets, PDFs, text documents, notes, and configuration files.

Prerequisites

Before you begin:

• Access a Vultr Cloud GPU or CPU-powered computer running Windows, or a macOS system, or Ubuntu (with GUI) based Linux server as a non-root user with sudo privileges.
• Create a domain A record for your domain name pointing to the workstation's IP address. This guide uses gpt4all.example.com domain for demonstration purposes.

Install GPT4All

You can install GPT4All on Windows, macOS, or Linux. In the following sections, install GPT4All depending on your operating system.

Configure GPT4All to Run LLMs

GPT4All does not include any built-in models by default, allowing you to download your models from the built-in repository or Hugging Face. GPT4All supports distilled, instruct, reasoning, uncensored, and censored models you can download and add to GPT4All. Follow the steps below to add new models, manage all downloaded models, and create a new chat to interact with the installed models.

1. Open GPT4All from your applications menu.

2. Click Models on the main navigation menu.

   

3. Click Add Model to explore the available models. Choose from any of the following options on the Explore Models page:

   

   • GPT4All: Provides models hosted in the built-in GPT4All repository depending on your installed version. Models in the GPT4All category are tested and compatible with GPT4All.
   • Remote Providers: Links to remote model providers, including Groq, OpenAI, and Mistral, which require an Internet connection for model inference.
   • Hugging Face: Includes GGUF format models hosted on Hugging Face. You can search and download new models from Hugging Face using keywords to run with GPT4All.

4. Scroll or search for a specific model, such as Llama3.2 1B Instruct, and verify the model requirements, including:

   

   • File Size: The total size of the model files should be available as free space on your workstation.
   • RAM Required: The total required memory to run the model within the GPT4All application interface.
   • Parameters: The model's total parameters on which it's trained.
   • Quant: The GPT4All Qt dependency package version that's required to run the model.
   • Type: The model's architecture or the base model family.

5. Click Download to download the model.

6. Click Models on the navigation menu.

7. Verify that the model is downloaded and available in GPT4All.

   

   You have downloaded a model from the existing sources in GPT4All. You can browse all existing models and invoke each model independently on the Chats page.

Use GPT4All to Chat with LLMs Locally

Downloaded models are available in GPT4All on the Chats tab. All chats are private and local on your workstation. Follow the steps below to chat with the downloaded models locally available on your system without using the Internet.

1. Click Chats on the main navigation menu.

2. Click Select a Model and choose the model.

   

3. Enter a prompt in the Send a message field and press Enter to start the chat.

4. Monitor the number of tokens and the model response rate.

5. Verify that the model responds with a result that matches your input text prompt.

   

6. Enter another prompt in the Send a message field and press Enter to continue chatting with the model referencing the original prompt to fine-tune the generated results.

7. Click Delete next to a chat in the history pane to delete it.

8. Click Edit to modify the chat title, or click New Chat to start a new chat.

RAG: Reference Local Documents Using GPT4All

GPT4All supports Retrieval-Augmented Generation (RAG) by default with the LocalDocs menu option. You can upload multiple documents and chat with LLM using your uploaded documents to enhance the quality of the generated text. Follow the steps below to create a new collection and chat with local documents in GPT4All.

1. Click Settings and select LocalDocs.

   

2. Verify the list of supported file extensions.

3. Click the Embeddings Device option and select CUDA if available to use your GPU device for embedding the local documents.

4. Click LocalDocs on the main navigation menu.

5. Click Add Doc Collection to create a new collection of local documents.

   

6. Enter the collection name and click Browse to select the path containing your documents.

7. Click Create Collection to scan all documents in your specified source directory.

8. Verify the embeddings count and the model in use, nomic-embed-text-v1.5 by default, and monitor progress for each document in the path.

   

9. Verify that the Collection includes the documents you uploaded to use with GPT4All.

10. Click Chats.

11. Start a new chat and click LocalDocs in the top-right to select the documents collection in your chat to perform RAG operations referencing your uploaded documents.

    

Download and Use GGUF-Format Models with GPT4All

GGUF (GPT-Generated Unified Format) is a model format designed to store, load, and infer large language models efficiently. It's optimized for memory usage and fast loading times, making it suitable for applications that require high performance and low latency. You can download .gguf format models from platforms like Hugging Face and manually add the model to GPT4All for use within the application interface. Follow the steps below to download and use .gguf format models from Hugging Face with GPT4All.

1. Open a web browser, such as Firefox.

2. Visit Hugging Face and search for a model to download.

3. For example, enter Mistral GGUF in the search bar to browse the available models.

   

4. Select your desired model from the list, such as mistralai/Devstral-Small-2507_gguf.

5. Click Files and Versions to browse the available version files.

6. Select the quantization version to browse the available .gguf files. For instance, you can select the Devstral-Small-2507-Q4_K_M.gguf

7. Verify the file size of the specific quantised model.

8. Open a new terminal window.

9. Check the disk usage on your workstation and verify that it has enough space to store the model files.

   console Copy

   $ df -h
   

10. In your browser, click on the download icon of your model version.

11. Once the download is complete, navigate to the Downloads folder in your terminal session.

    console Copy

    $ cd ~/Downloads
    

12. Move the downloaded .gguf file to the .local/share/nomic.ai/GPT4All/ directory in your user's home directory.

    console Copy

    $ mv Devstral-Small-2507-Q4_K_M.gguf ~/.local/share/nomic.ai/GPT4All/
    

13. List the .local/share/nomic.ai/GPT4All/ directory and verify the available .gguf model files.

    console Copy

    $ ls ~/.local/share/nomic.ai/GPT4All/
    

    Output:

    Devstral-Small-2507-Q4_K_M.gguf  Llama-3.2-1B-Instruct-Q4_0.gguf  localdocs_v3.db  test_write.txt

14. Close GPT4All if it's open on your workstation.

15. Open GPT4All again to load the newly added models.

16. Navigate to the Models tab and verify that the newly added model is available.

    

17. Navigate to Chats, click New Chat, and select the new model.

18. Enter a new text prompt like write a short story about a robot learning to code, with a focus on arrays in the message field, and press Enter to generate a response using the model.

    

Enable the GPT4All API Server

GPT4All includes a built-in API server that lets you interact with models using REST API endpoints. The API allows you to list and run available models, allowing you to integrate GPT4All into your existing applications. Follow the steps below to configure the local API server and enable connections to the GPT4All port.

1. Click Settings on the main navigation menu.

2. Navigate to the Advanced section within the application options.

   

3. Click Enable Local API Server to enable API access on your server.

4. Verify the default API Server Port for accessing the GPT4All API.

5. Open a new terminal window.

6. Send a GET request to the /v1/models endpoint using the GPT4All localhost port to list all available models on the server.

   console Copy

   $ curl http://localhost:4891/v1/models
   

   The API returns a JSON output containing the list of available AI models.

7. Send a GET request to the /v1/models/<model-name> endpoint to get details of a specific model. Replace Devstral-Small-2507-Q4_K_M.gguf with a model available on your GPT4All server.

   console Copy

   $ curl http://localhost:4891/v1/models/Devstral-Small-2507-Q4_K_M.gguf
   

8. Send a POST request to the /v1/completions endpoint to generate text completions.

   console Copy

   $ curl -X POST http://localhost:4891/v1/completions \
     -H "Content-Type: application/json" \
     -d '{
       "model": "Devstral-Small-2507-Q4_K_M.gguf",
       "prompt": "What is the sum of 8, 5, and negative 200?",
       "max_tokens": 500
     }'
   

   The API returns a response to the prompt in JSON format and a few other details.

Configure Nginx as a Reverse Proxy to Forward API Requests on Linux

GPT4All listens for API requests using the 127.0.0.1 localhost address, which limits all connection requests to your local system. Installing and configuring Nginx as a reverse proxy enables secure network connections to the GPT4All API server by forwarding all requests to the localhost network address.

1. Update the APT package information index.

   console Copy

   $ sudo apt update
   

2. Install Nginx web server.

   console Copy

   $ sudo apt install nginx -y
   

3. Create a new gpt4all.conf virtual host configuration in the /etc/nginx/sites-available directory.

   console Copy

   $ sudo nano /etc/nginx/sites-available/gpt4all.conf
   

4. Add the following configurations to the gpt4all.conf file. Replace gpt4all.example.com with your domain name.

   ini Copy

   server {
       listen 80;
       server_name gpt4all.example.com;
   
       location / {
           proxy_pass http://127.0.0.1:4891;
           proxy_http_version 1.1;
           proxy_set_header Upgrade $http_upgrade;
           proxy_set_header Connection 'upgrade';
           proxy_set_header Host $host;
           proxy_cache_bypass $http_upgrade;
       }
   }
   

   Save the file and exit the text editor.

   In the above configuration, Nginx forwards all connection requests from the gpt4all.example.com domain to the GPT4All API server port 4891, allowing you to send API requests to the server remotely.

5. To enable it, link the gpt4all.conf configuration to the /etc/nginx/sites-enabled directory.

   console Copy

   $ sudo ln -s /etc/nginx/sites-available/gpt4all.conf /etc/nginx/sites-enabled/gpt4all.conf
   

6. Remove the default Nginx virtual host configuration.

   console Copy

   $ sudo rm /etc/nginx/sites-enabled/default
   

7. Test the Nginx configuration for syntax errors.

   console Copy

   $ sudo nginx -t
   

   Output:

   nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
   nginx: configuration file /etc/nginx/nginx.conf test is successful

8. Restart Nginx to apply the configuration changes.

   console Copy

   $ sudo systemctl restart nginx
   

9. Allow network connections to Nginx through the default firewall.

   console Copy

   $ sudo ufw allow "Nginx Full"
   

10. Reload UFW to apply the firewall configuration changes.

    console Copy

    $ sudo ufw reload
    

11. Install Certbot to generate Let's Encrypt certificates for all connections to the gpt4all.example.com domain.

    console Copy

    $ sudo apt install -y certbot python3-certbot-nginx
    

12. Generate a new SSL certificate for the gpt4all.example.com domain. Replace admin@example.com with your email address.

    console Copy

    $ sudo certbot --nginx -d gpt4all.example.com -m admin@example.com --agree-tos
    

13. Restart Nginx to apply the Let's Encrypt certificate changes.

    console Copy

    $ sudo systemctl restart nginx
    

Send API Requests to the GPT4All API Server

GPT4All now accepts remote API requests from a reverse proxy that forwards connections to the localhost API server port. Follow the steps below to send remote API requests to GPT4All using your gpt4all.example.com domain and run models on the server.

1. Send a GET request to the /v1/models endpoint using your gpt4all.example.com domain to list all available models.

   console Copy

   $ curl https://gpt4all.example.com/v1/models
   

2. Send a GET request to the /v1/models/<model-name> endpoint to get the details of a specific model.

   console Copy

   $ curl https://gpt4all.example.com/v1/models/Qwen2-1.5B-Instruct
   

Troubleshooting

Running GPT4All may return multiple errors after installation due to resource limitations or improper configurations. Depending on your base operating system, follow the recommendations below to troubleshoot common GPT4All errors on your workstation.

$ sudo apt install ubuntu-desktop -y

Conclusion

In this guide, you have installed GPT4All and run large language models (LLMs) on your workstation. GPT4All supports multiple types of models including open-source models and API-provided models from providers like OpenAI. All chats and model interactions are private, allowing you to perform inference and run LLMs locally. For more information and application configurations, visit the GPT4All documentation.