how to use OCI Ampere A1 instance for LLM

AI

Deploying LLMs on OCI Ampere A1 Instances

OCI Ampere A1 instances utilize Ampere Altra ARM-based CPUs, which are highly cost-effective for inferencing small- to mid-sized Large Language Models (LLMs) like Llama 3 (8B) or Mistral (7B). These instances can be used through automated "AI Quick Actions" or manual deployment with optimized libraries like llama.cpp.

Price-Performance and Always Free Eligibility

A1 instances can offer up to a 2.9x price/performance advantage over similar x86-based cloud instances for small batch sizes (1–4). You can often run smaller models on the OCI Always Free tier, which provides up to 4 OCPUs and 24 GB of RAM on Ampere A1 at no cost.

Optimization

Use Ampere Optimized AI Frameworks and quantized models (GGUF/AWQ) to significantly improve tokens-per-second (TPS) and time-to-first-token (TTFT).

Llmfit and Hardware Detection

Llmfit is inherently aware of arm64 architectures because it is built in Rust and evaluates systems strictly by scanning their available raw system resources (CPU architecture, RAM, GPU/VRAM) to match them with appropriate LLM quantization levels. However, llmfit is designed primarily as a hardware profiling and model-matching tool; it is not explicitly aware of the specific compiler and kernel intricacies required to maximize performance on OCI Ampere A1 shapes.

Hardware Detection and Model Profiling

When run on a VM.Standard.A1.Flex instance, the llmfit GitHub repository tool correctly identifies the machine as an arm64 Linux environment. It captures the allocated OCPU core count and the system RAM (up to 512 GB). It evaluates how open-source models (like Llama 3 or Mistral) will fit within that RAM. It calculates memory allocation overhead and recommends the highest possible quantization format (like 4-bit or 8-bit GGUF) that can run without out-of-memory errors.

Runtime Support and Limitations

Llmfit integrates with arm64-compatible engines like llama.cpp and Ollama. However, llmfit calculates metrics based on generalized hardware formulas, which misses OCI Ampere-specific performance nuances, such as Custom Vector Extensions and Speed Overestimation/Underestimation.

Recommended Workflow

Use llmfit to quickly determine which quantized models will physically fit into your A1 instance's allocated memory. Once you know your ideal model size, bypass standard scripts and deploy using Ampere-optimized container images or OCI Data Science AI Quick Actions to ensure the hardware's math engines are fully utilized.

Back to all articles