Vultr DocsLatest Articles

How to Set Up Guardrails on Large Language Models using NVIDIA NeMo

CUDANVIDIA NeMoLanguage Models
MS
Updated on November 21, 2023
How to Set Up Guardrails on Large Language Models using NVIDIA NeMo header image

Introduction

Neural Modules (NeMo) Guardrails is an open-source toolkit developed by NVIDIA, to safeguard the development of LLMs by preventing harmful or offensive content. It provides features to control the LLMs and helps mitigate the risks of models by adding an essential layer of protection that assists organizations in building reliable systems without exposing the user to unwanted information.

This article explains how to set up the NeMo Guardrails for dolly-v2-3b LLM using the HuggingFace pipeline and control the response generation that matches the prompt provided.

Prerequisites

Install the Required Packages

To set up guardrails on your server, install the necessary dependency packages and access the pre-installed Jupyter Lab to infer the model.

  1. Install the required packages

     $ pip3 install nemoguardrails langchain

    Below is what each package represents:

    • nemoguardrails: Provides the necessary modules to use NeMo Guardrails toolkit such as RailsConfig, register_llm_provider, and LLMRails
    • langchain: Import the HuggingFacePipeline that initializes the model in use

  2. To access Jupyter Lab, view the generated access token

     $ cat /var/log/jupyterlab/lab.log

    Output:

     To access the server, open this file in a browser:
     file:///home/jupyter/.local/share/jupyter/runtime/jpserver-1989-open.html
     Or copy and paste one of these URLs:
     http://localhost:8888/lab?token=6eebc299236fc1debe7b0a8e7bb8000169abcd9e8821df22
     http://127.0.0.1:8888/lab?token=6eebc299236fc1debe7b0a8e7bb8000169abcd9e8821df22

  3. Using a web browser of your choice, access JupyterLab using your Server IP Address

     https://YOUR_SERVER_IP:8888/lab?token=YOUR_TOKEN

    When prompted, accept the self-signed SSL certificate to access JupyterLab

Set Up Guardrails

To set up Guardrails, import the necessary modules. Then, set the guardrails policies, initialize the dolly-v2-3b model with the HuggingFace pipeline, and infer it as described in the steps below.

  1. In your Jupyter Lab interface, create a new Python3 Kernel Notebook

  2. Import the necessary modules

     from functools import lru_cache
 from langchain import HuggingFacePipeline
 from torch.cuda import device_count
 from nemoguardrails.llm.helpers import get_llm_instance_wrapper
 from nemoguardrails.llm.providers import register_llm_provider
 from nemoguardrails import LLMRails, RailsConfig

    Below is what each module represents:

    • lru_cache: It's used for memorization which is an optimization technique to improve the efficiency of functions
    • HuggingFacePipeline: Creates pipelines for pre-trained LLMs and NLP tasks
    • device_count: Determines the number of available CUDA devices
    • get_llm_instance_wrapper: Gets an instance for a specified LLM
    • register_llm_provider: Specifies the source for LLM models
    • LLMRails: Generates a response from a prompt by creating rails
    • RailsConfig: Initializes the custom configurations you defined for response generation

  3. Click Run or press Ctrl + Enter to import the modules

  4. In a new code cell, create the custom guardrails

     yaml_content = """
 models:
 - type: main
   engine: hf_pipeline_dolly
 """

 colang_content = """

 define user express greeting
     "hello"
     "hi"
     "what's up?"

 define bot express greeting
     "Hi there, I am your Library assistant what would you like to read ?"

 define bot ask how are you
     "I hope everything's going well with you"

 define flow greeting
     user express greeting
     bot express greeting
     bot ask how are you

 define user ask politics
     "what are your political beliefs?"
     "thoughts on the president?"
     "left wing"
     "right wing"

 define bot answer politics
     "I'm a library assistant, I don't like to talk of politics."

 define bot offer help
     "Is there anything else i can help you with?"

 define flow politics
     user ask politics
     bot answer politics
     bot offer help
 """

    The above code sets the YAML content used to define the model in use. It also defines the CoLang content where the guardrails for expected user prompts and model responses are listed. Then the conversation flow is defined to avoid misdirection and hallucination in the responses generated by the bot.

    Edit the prompts in the above code and the response definitions along with flow definitions to match your use case. For the guardrails to work, be sure to follow the CoLang syntax.

  5. Define the model function

     @lru_cache
 def get_dolly_v2_3b_llm():
     repo_id = "databricks/dolly-v2-3b"
     params = {"temperature": 0, "max_length": 1024}

     # Use the first CUDA-enabled GPU, if any
     device = 0 if device_count() else -1

     llm = HuggingFacePipeline.from_model_id(
         model_id=repo_id, device=device, task="text-generation", model_kwargs=params
     )

     return llm

    The above command defines a function get_dolly_v2_3b_llm that configures the model, its temperatureandmax_lengthparameters. The model can either either run on a CUDA-enabled GPU or CPU. The function returns the initialized model withllm`.

  6. Initiate the pipeline

     HFPipelineDolly = get_llm_instance_wrapper(
     llm_instance=get_dolly_v2_3b_llm(), llm_type="hf_pipeline_dolly"
 )

 register_llm_provider("hf_pipeline_dolly", HFPipelineDolly)

    The above command creates an LLM instance using the specified model and parameters. Then, the instance is registered with the source of the model for text generation tasks.

  7. Initialize the custom configurations

     config = RailsConfig.from_content(
     yaml_content=yaml_content,
     colang_content=colang_content
 )

 rails = LLMRails(config)

    The above command initializes the yaml_content and colang_content you defined earlier, then it creates the LLM Rails by passing config.

  8. Generate response. Replace the prompt What do you think of the president with your desired text prompt

     res = await rails.generate_async(prompt="what do you think of the president?")
 print(res)

    The above code asynchronously generates a response considering the guardrails you set up.

    Output:

     I'm a library assistant, I don't like to talk of politics.
 Is there anything else i can help you with?

Conclusion

You have inferred the dolly-v2-3b model using the HuggingFace pipeline and applied custom guardrails using the NeMo toolkit. The guardrails control the model's response so that it does not go off-topic or hallucinate. This protects the user from harmful information.

More Information

For more information and implementation samples, please visit the following resources: