AI Summary
A developer has managed to run the GLM 5.2 language model on a computer with limited resources, achieving communication speeds of 0.1 tokens per second. The project, named Colibrì, utilizes a mixture-of-experts approach to optimize performance on a machine with 32GB of RAM.
- The developer tested GLM 5.2 on a standard computer to evaluate its performance without running out of memory (OOM).
- The model was converted to int4 format, and various techniques were implemented to manage long context usage.
- The Colibrì project activates approximately 40 billion parameters per token from a 744 billion parameter model, with only 11 GB changing per token.
- The dense components of the model remain in RAM, while the routed experts are stored on disk and streamed as needed.
- The project was developed on a 12-core laptop with 25 GB of RAM, and the developer is open to feedback and collaboration.
glmllmperformancetestingcomputing