banana.cpp

A modular, pure C++ implementation of a small language model inference engine supporting multiple model architectures.

01Overview

banana.cpp is a pure C++ LLM inference engine designed for modularity and extensibility. It supports multiple small language model architectures including SmolLM2, Llama 3.2, and Qwen variants, with built-in support for FP16/BF16 precision and modern architectural features like GQA, RoPE, and SwiGLU.

02Supported Models

SmolLM2135M, 360M, 1.7B parameter variants
Llama 3.21B, 3B parameter variants
Qwen2.5-0.5B, 3-0.6B parameter variants

03Key Features

Modular ArchitectureLayer-based design makes it easy to add new architectures
Mixed PrecisionNative FP16 and BF16 support for efficient inference
Modern AttentionGQA, MHA, and MQA attention mechanisms
Position EncodingsRoPE (Rotary Position Embedding) implementation
Activation FunctionsSwiGLU and standard MLP layers
NormalizationRMSNorm and LayerNorm implementations
Auto-detectionModel registry automatically detects model type from config
HuggingFace IntegrationBuilt-in model downloader for HuggingFace models

04Architecture

The engine uses a clean, modular layer-based architecture:

LayersReusable components (Attention, MLP, Normalization, RoPE)
ModelsCompose layers based on configuration
TokenizersSeparate BPE logic from chat template handling
RegistryAuto-detect model type from config.json

This design philosophy makes it straightforward to add new model architectures by composing existing layers, introduce new layer types without modifying models, and support multiple chat templates and tokenizer formats.

05Usage

# List supported models
./banana-cpp --list-models

# Download and run a model
./banana-cpp --model smollm2-360m --download
./banana-cpp --model smollm2-360m --prompt "What is the capital of France?"

# Interactive mode
./banana-cpp --model smollm2-360m --interactive

# Custom settings
./banana-cpp --model llama-3.2-1b \
    --temperature 0.8 \
    --top-k 40 \
    --top-p 0.95 \
    --max-tokens 512

06Building

mkdir build && cd build
cmake ..
make -j$(nproc)

07Project Structure

include/config/Model and tokenizer configuration structures
include/core/Core tensor data structures and operations
include/layers/Neural network layer implementations
include/models/Model architecture definitions
include/tokenizers/BPE tokenization and chat templates
include/registry/Model auto-detection system
include/utils/Model loading and HuggingFace downloading