Latest ContentInference Cookbook Model Library

Gemma 3 270M IT

Gemma-3-270m-it is a lightweight instruction-tuned dense transformer language model derived from the same research used for Gemini models. Built on a 270M parameter architecture, it features 18 transformer layers, 4 attention heads, and a 640 hidden size, using a combination of sliding-window and full attention mechanisms for efficient context processing. The model supports a 32K token context window and multilingual capabilities across 140+ languages, and is optimized for lightweight deployment on resource-constrained environments such as laptops and edge infrastructure.

Type	Dense LLM
Capabilities	Text Generation, Instruction Following, Reasoning, Mathematical Reasoning+4 more
Links	Paper\|Blog\|HF Model Card
License	Gemma

Inference Instructions

Deploy and run this model on NVIDIA B200 GPUs using the command below. Copy the command to get started with inference.

CONSOLE

docker run -it --rm 
 --runtime=nvidia 
 --gpus all 
 --ipc=host 
 --shm-size=64g 
 -p 8000:8000 
 -v ~/.cache/huggingface:/root/.cache/huggingface 
 -e HF_TOKEN='YOUR_HF_TOKEN' 
 -e LD_LIBRARY_PATH='/usr/local/nvidia/lib64:/usr/local/nvidia/lib:/usr/lib/x86_64-linux-gnu' 
 vllm/vllm-openai:v0.15.0-cu130 
 google/gemma-3-270m-it 
  --tensor-parallel-size 4 
 --max-model-len auto 
  --max-num-batched-tokens 65536 
  --gpu-memory-utilization 0.95 
 --block-size 32 
 --max-num-seqs 1024 
 --disable-log-requests 
 --trust-remote-code

Note

Ensure tp is set to 1, 2, or 4 to match attention head divisibility and force --block-size 32 to bypass the known FlashInfer bug associated with the model's 256 head dimension.