Thinking Machines Releases Inkling, a 1-Trillion-Parameter Multimodal Open Model

According to Hugging Face, Thinking Machines has released Inkling, a 1-trillion-parameter open-weights multimodal model that natively processes image, audio, and text inputs with a 1M-token context window. The model was trained on 45 trillion tokens spanning text, images, audio, and video. Inkling is available in full BF16 precision and a quantized NVFP4 variant, with day-one framework support across transformers, SGLang, and llama.cpp.

Inkling’s Sparse Architecture and Parameter Efficiency

The model employs a decoder-only Mixture-of-Experts design with 975 billion total parameters but only 41 billion active at inference time, achieved through 256 experts. This parameter sparsity enables faster inference without sacrificing model capacity, a critical trade-off for deployed multimodal systems. According to the Hugging Face announcement, the active parameter count of 41B positions Inkling competitively against single-modality models while handling three input modalities simultaneously.

Architectural Innovations for Cross-Modal Reasoning

Inkling incorporates three distinctive architectural components optimized for multimodal understanding. First, the model uses relative attention rather than rotary positional embeddings (RoPE), with each attention layer learning position information directly in attention logits through a per-token, per-head relative feature projection. Second, it employs hybrid attention alternating between global attention over the full 1M-token context and sliding-window attention at a 1:5 ratio, with the final layer using global attention to build feature-rich representations. Third, the model includes short 1D convolution (SConv) layers operating over hidden states, which Hugging Face describes as helping local attention while freeing the attention and MoE modules from redundant local computation.

Deployment and Quantization Options

Beyond the full-precision variant, Inkling’s NVFP4 quantization is described as well-calibrated for production inference. The model also includes speculative MTP (Multi-Token Prediction) layers designed to accelerate generation throughput. Framework-level integration across transformers, SGLang, and llama.cpp reduces implementation friction for teams evaluating the model for domain adaptation via fine-tuning.

Why This Matters

Inkling’s release widens the open-weights multimodal landscape at trillion-parameter scale. For teams building cross-modal reasoning applications—where image-text understanding alone proves insufficient and audio context adds value—the 1M-token context combined with sparse parameter activation reduces the cost-per-inference barrier that typically constrains large multimodal deployments. The explicit emphasis on fine-tuning readiness suggests Thinking Machines is positioning Inkling as an adaptation foundation rather than a finished product, appealing to enterprise organizations seeking to customize multimodal pipelines without proprietary API lock-in.