Latest ContentInference Cookbook Model Library

Llama 3.1 70B Instruct

Llama 3.1 70B Instruct is an instruction-tuned dense transformer large language model optimized for multilingual dialogue, reasoning, and assistant-style AI applications. Built on a 70B parameter architecture, it features 80 transformer layers, 64 attention heads, and an 8,192 hidden size, leveraging Grouped Query Attention (GQA) for efficient large-scale inference. The model supports up to a 128K token context window with RoPE scaling for long-context understanding. Fine-tuned using supervised learning and reinforcement learning with human feedback (RLHF) to align with human preferences for helpfulness and safety.

Type	Dense LLM
Capabilities	Text Generation, Instruction Following, Reasoning, Mathematical Reasoning+5 more
Release Date	23 July, 2024
Links	Blog\|HF Model Card
License	Llama3.1

Inference Instructions

Deploy and run this model on NVIDIA B200 GPUs using the command below. Copy the command to get started with inference.

CONSOLE

docker run -it --rm 
 --runtime=nvidia 
 --gpus all 
 --ipc=host 
 --shm-size=128g 
 -p 8000:8000 
 -v ~/.cache/huggingface:/root/.cache/huggingface 
 -e HF_TOKEN='YOUR_HF_TOKEN' 
 -e LD_LIBRARY_PATH='/usr/local/nvidia/lib64:/usr/local/nvidia/lib:/usr/lib/x86_64-linux-gnu' 
 vllm/vllm-openai:v0.15.0-cu130 
 meta-llama/Llama-3.1-70B-Instruct 
  --tensor-parallel-size 8 
 --max-model-len auto 
  --max-num-batched-tokens 65536 
  --gpu-memory-utilization 0.95 
  --max-num-seqs 1024 
 --disable-log-requests 
 --trust-remote-code