shoehorn

shoehorn is a tool that fits large language models into the exact memory budget of your hardware. Instead of relying on preset quantizations that either waste headroom or fail at load time, it starts from the amount of VRAM you actually have, subtracts what inference needs, and solves a per-tensor mixed-precision assignment that lands within a rounding error of the remainder—routinely using 99.99% of the budget, sometimes to the byte. This means you can run a model that would otherwise be too large, or run a given model with higher precision than standard quantizations would allow.

It works with GGUF models and uses llama.cpp as the inference backend. The local web app measures your machine, streams the fit, renders the budget as a tape measure, and shows a perplexity number indicating what the fit cost. It then provides a Chat button for immediate interaction. shoehorn is designed for anyone who wants to run open-source language models on their own hardware—whether a Mac with 8–128 GB of unified memory, a GPU with 8–32 GB of VRAM, or a custom workstation—and wants to maximize quality without manual trial and error.

The software is MIT-licensed, implemented from scratch in Rust (no llama.cpp code linked), and outputs standard GGUF v3 that any llama.cpp-compatible loader can use. It's available for macOS (Apple Silicon), Linux (x86-64, NVIDIA/AMD), and Windows (x86-64, NVIDIA).