Choose local language models without falling for marketing
The best choice does not begin with a model name. It begins with an honest description of the task, the data and the environment you actually have.
"Which local model is the best?" sounds like a sensible question, yet it quickly points in the wrong direction. A model is not simply good or bad. For a particular task, on particular hardware, with particular expectations of speed, quality and confidentiality, it may be more or less suitable. Clarify those conditions first and you need less marketing to make a better decision.
Rankings and announcements change weekly; your task stays. That is exactly the leverage: knowing your own case precisely lets you place any new release within minutes — does it fit my constraints or not? — instead of chasing every name.
Start with the work, then the model
A local language model is not a trophy to win. It is a tool inside a workflow. Selection therefore begins with a scene: should it sort notes, revise text, answer questions from confidential documents, explain code, or prepare a repeatable draft? A concrete scene makes clear what the outcome has to accomplish.
Describe not only the desired answer, but also the standard. Is a slower answer acceptable when it is more careful? Must it work without internet access? Is a draft for human review enough, or must it retrieve information accurately from supplied material? Only these questions give words such as quality, speed and privacy a practical meaning.
Four constraints shape the choice
The first constraint is the task itself. A model that writes freely is assessed differently from one that summarises supplied material. The second is the environment: available compute, memory, storage, energy and willingness to wait. The third is the handling of data. The fourth is operability: who sets up the system, updates it, observes failures and helps when it stops working?
These are not technical footnotes. They determine whether a project survives everyday work after an impressive demonstration. A small, stable workflow can be more valuable than a large model that is pleasant to use only under ideal conditions.
It helps to write the four constraints down before looking at any candidate. They then act as a filter: much drops out without a test because it exceeds the environment or violates the data frame. What remains is a short list — and only that list is worth comparing.
Size is not a quality verdict
Larger models can offer more room on some tasks. At the same time, they generally require more resources and may respond more slowly. Smaller models can be quicker, easier to operate and entirely sufficient for narrowly framed work. This does not create a fixed ranking; it creates a test question: which level of quality actually changes a decision in this piece of work?
Technical settings do not turn a model label into a guarantee either. They affect how demanding a model is to operate and how it behaves in an environment. Looking only at figures misses the more useful observation: on real examples, does the system produce an outcome that people can review and use?
So it pays to always think „better" with a suffix: better for what, under which conditions, at what cost? A model that phrases a nuance more elegantly but computes twice as long and pushes the machine to its limit is the worse choice for many everyday tasks — and for some, the only right one.
Comparison means repeating the same case
A fair comparison does not need ten colourful demos. It needs a few real cases run under the same conditions. Set the material, task, desired format and evaluation in advance. Then inspect more than the first impression: does the answer stay with the material? Does it say what it does not know? Does it keep the agreed format? Is the waiting time acceptable for this work?
Document the result briefly. A small test sheet with the case, outcome, error, correction effort and operating experience protects against memory bias. It also reveals whether a model only shines in one fortunate example or supports a task reliably enough.
A law firm wants meeting notes summarised locally. The test case: the same three anonymised transcripts, the same instruction, the same target format — decision, open points, deadlines as a list. Candidate A writes elegantly but invents a deadline in one transcript that appears nowhere. Candidate B sounds wooden but stays with the material all three times and marks one unclear passage as unclear. On the test sheet, B wins clearly: less correction effort, more reliable handling of uncertainty. Without the fixed case, first impressions would have crowned A.
Local use is a responsibility, not a property
Working locally can change data paths and dependencies. It does not automatically make data safe or outcomes correct. Access, stored files, backups, updates and approvals remain part of the system. Professional review remains with the people who use the outcome.
The most mature decision is therefore rarely 'local only' or 'online only'. It is: for which task, which data and which risk is which working location appropriate? When you can answer that clearly, you are not choosing a label. You are building a workflow that fits your standards.
The local model choice card
A single card holds the decision together. It is filled in before the first test — and the test case in it stays the same for every candidate.
# MY CHOICE CARD
**Work case**
I want it to …
**Material and data**
It works with … These contents may / may not …
**Outcome standard**
A useful outcome is …
**Environment**
Available: … Acceptable waiting time: …
**Human review**
Before use, someone checks …
**Test case**
The same case for every comparison: …A good local model is not the one with the loudest promise. It is the one whose strengths, limits and operation fit a real task. When you describe that case clearly and test it repeatedly, selection becomes an explainable decision instead of a collection of names and impressions.
Worksheet: Make a reasoned model choice
Choose a task you would genuinely try with a local system. The goal is not to crown a "best" model; it is to prepare a reasoned decision.
Define one work case. Describe the task, material and the person who will use the outcome.
Make constraints visible. Record requirements for data, available environment, acceptable waiting time and the kind of human review.
Build a comparison case. Write one task and its desired output format. Keep this case identical for every candidate.
Assess against criteria. Use short notes to assess faithfulness to material, usefulness, handling of uncertainty, speed and correction effort.
Explain the decision. Record which candidate fits this work case — or why none fits yet.
All materials to download — the topic overview and the worksheet:
● Members only
Read the full article and download all files with a membership.
Unlock full article + downloads → Subscribe0 comments
● Loading comments…