qwen-ane-llm
Qwen LLM inference on the Apple Neural Engine: Core ML model conversion and ANE optimization for on-device, energy-efficient, privacy-preserving local inference.
Why I built it
The Apple Neural Engine is powerful, underdocumented, and mostly ignored by the LLM ecosystem. That combination is exactly what makes me curious. This project is me finding out, hands on the hardware, what it takes to get a modern open-source LLM running on it.
What it does
Converts Qwen language models to Core ML and runs them on the Apple Neural Engine via the ANE-LM framework, for on-device, energy-efficient inference without any cloud dependency. A companion project, qwen-speech-mlx, extends the same exploration to speech models using Apple’s MLX framework.
What it taught me
On-device inference is a different discipline from API integration: model formats, quantization, and hardware constraints dominate. It connects directly to the local-first thread that runs through this blog.