Local language models in an isolated environment: how to choose and evaluate one

There is no universally best local model. For an environment without internet access, choose specific weights, a specific version, and a specific license—not “Llama or Qwen” in the abstract. Then verify two things: how the model handles your documents and whether the whole system truly works without external connections. Neither a model-family name nor the word “local” proves this.

This article discusses three models from the research as examples of size classes: Qwen3-8B, Mistral Small 3.1 24B, and Llama 3.3 70B Instruct. This is not a market ranking or a list of the newest models as of October 2026. These versions were released in 2024 and 2025, and newer versions may exist by the time you read this. We did not run a comparative evaluation, so we do not publish speed or quality results.

After reading, you can create a requirements matrix, screen models by license and memory, and plan a pilot whose results apply to your specific task. Model-card facts are based on research conducted on 11 October 2026.

What a fully isolated environment means

“Isolated environment” is often taken to mean “a server without internet.” For a language model, that is not enough. A response system has several components, each of which may depend on an external resource.

Model weights. Parameter files are downloaded from the model author’s repository. In an isolated environment, transfer them in advance and verify checksums. Record the exact version and format (full precision or quantized copy): both affect quality and memory requirements.

Tokenizer and configuration. Separate files convert text into tokens and define the conversation template. If they are not stored alongside the weights, a library may try to download them on first launch. In an offline environment this fails; with partial access it may work silently and go unnoticed.

Software dependencies. The inference server, Python libraries, GPU drivers, and containers must be assembled from an offline package mirror or transferred as prepared images. The Qwen3-8B card, for example, lists local-running tools such as Ollama, LM Studio, MLX-LM, llama.cpp, and KTransformers, and serving tools such as vLLM and SGLang. Each has its own dependencies, settings, and license, which must be checked separately from the model-weight license.

Updates. Nothing updates itself in an isolated environment. That improves control but complicates security: library vulnerability fixes must also be delivered manually under a defined process.

Logs. The inference server, proxy, monitoring system, and application built around the model may record prompts and responses. Isolation does not remove the need to control access to those logs; they may contain the same confidential documents the environment was built to protect.

Network requests. Verify isolation with configured network blocks and observation during real operation: startup, first prompt, document upload, and restart. Confirm that logging covers the relevant network paths. Missing entries in incomplete logs do not prove that no requests occurred.

Three candidates as examples of size classes

Models with 8 billion and 70 billion parameters solve different tasks at different costs. “More parameters means better” is not a sound comparison: a larger model needs more memory, usually more compute for a comparable architecture and precision, and does not guarantee better results on your particular documents. The table below reports model-card information, not quality judgments.

Language support deserves special attention because it can decide the choice for a Ukrainian company. Llama 3.3’s card lists eight supported languages: English, German, French, Italian, Portuguese, Hindi, Spanish, and Thai. The card treats use in other languages as out of scope. Developers may fine-tune the model for other languages if they follow the license and acceptable-use policy; they then bear responsibility for safe use. Do not promise built-in Ukrainian support. If Ukrainian is required, plan to review the license, adapt the model, and test it separately.

A language appearing in a card is not proof of quality either. Mistral Small 3.1 listing Ukrainian and Qwen3-8B claiming more than a hundred languages only means the authors have not excluded those languages. Only a test can show how a model performs on your contracts, invoices, and customer emails.

Three use cases and what matters in each

Document question answering. An employee asks a question, the system retrieves document passages, and the model formulates an answer with a reference to the source. Two qualities matter: answer only from the supplied text and correctly say “the documents do not contain this.” A long context helps, but increases memory requirements.

Data extraction. The model reads a document and fills fields such as contract number, date, amount, and counterparty. Accuracy and a stable output format matter. A small model may be enough for a narrow task, but a test must establish that; assumption cannot.

Internal assistant. An open-ended conversation for different tasks: drafting emails, explaining policies, and helping with text. Language quality, following instructions, and serving several users at once matter more. This use case most often pushes teams toward a larger model, so calculate the load honestly.

Licenses and usage rights

A local deployment has at least two license layers: the model-weight license and the licenses for the software that runs it. Do not conflate them. An open inference server does not make its weights open, and vice versa.

The cards for these three examples state:

• Qwen3-8B is distributed under Apache 2.0.
• Mistral Small 3.1 24B is also under Apache 2.0. Its card describes the license as open for commercial and non-commercial use and modification.
• Llama 3.3 70B uses a separate commercial Llama 3.3 Community License Agreement, which includes an acceptable-use policy. Additional commercial terms apply to a licensee or its affiliates whose products or services had more than 700 million monthly active users in the calendar month before Llama 3.3’s release. In that case, separate permission from Meta is required. This is not a universal threshold that is automatically measured every month after installation.

Check four questions when reviewing a license:

1. Commercial use. Is it allowed without additional conditions for your company type and scale?
2. Distribution. May you provide the weights or a product based on them to customers, for example as a packaged solution on a customer’s server? What notices and license text are required?
3. Fine-tuning. Is modification allowed, and do the original license terms apply to a fine-tuned model?
4. Use restrictions. Is there an acceptable-use policy, and does your use case fall within a prohibited category?

Third parties often publish quantized model copies. Check whether the original license is preserved and whether the copy corresponds to the version you evaluated. Your lawyer makes the final decision on license compliance; this article explains what to look for but does not replace legal advice.

Hardware and real workload

GPU memory is used for at least two things: model weights and the KV cache. The key-value cache stores intermediate results for text already read so the model does not recalculate them for every new token. Its size increases with context length and the number of concurrent requests.

Weights. You can roughly estimate memory for weights alone by multiplying the parameter count by bytes per parameter. A 16-bit format uses 2 bytes per parameter. Four-bit quantization uses about 0.5 bytes, although real formats add metadata.

This arithmetic covers weights only; it is not a server requirement. Add KV cache, activations, inference-server overhead, and headroom. For comparison, Mistral Small 3.1’s card says about 55 GB of GPU memory is needed to run in bf16 or fp16—noticeably more than the weights alone at 48 GB. The same card claims its quantized model fits on one RTX 4090 or a MacBook with 32 GB of memory. That is the author’s claim under certain conditions, not a guarantee for your context length and user count.

Context. A large context window in a model card is a maximum, not a recommended operating mode. If you plan to feed long documents to the model, budget KV-cache memory for that length multiplied by the expected number of concurrent requests.

Concurrency. One user and twenty users at 9 a.m. create different workloads. Evaluate peak-hour use, not an average day.

Quantization. It reduces memory and may increase speed, but may lower quality on your tasks. Compare a “model + quantization” pair, and record that exact pair in the pilot results.

Evaluate with company data

Model comparisons are meaningful only if every candidate receives the same tasks under the same conditions. The test below is proposed, not completed. Its numbers define the structure, not results.

Proposed pilot: 60 tasks

Prepare 60 de-identified tasks using one fixed version of your documents:

• 20 document question-answering tasks, including questions whose answers are absent from the documents;
• 20 field-extraction tasks with a defined output format;
• 20 document summarization tasks.

For each task, record a reference answer or correctness criteria in advance. Include tasks in the languages your company actually uses, including Ukrainian if it is a working language. Subject-matter experts should score every candidate using the same instructions.

The results table should look like this. Cells are intentionally blank until measurements are taken.

“Correct refusals” show how often the model honestly says the answer is absent instead of making one up. p95 latency is the response time for 95% of requests and describes user experience better than the average. Record zero network requests only if network logs confirm it during the test, not because a product description says so.

How to measure speed

You can use vLLM to load-test an inference server. Its documentation explains latency metrics: time to first token (TTFT), time between consecutive streamed output chunks (ITL), and mean time per output token (TPOT). It also describes workload settings such as target request rate, arrival-rate variability, and maximum concurrent requests. Its sample output shows the mean, median, and P99. If you need p95 specifically, configure the report or calculate it from raw results.

The documentation warns that rerunning the same test on the same server may reuse requests in the prefix cache and overstate throughput. For a fair comparison, change the request set, restart the server between runs, or use a mode that clears the cache.

Another requirement for a fair comparison is a shared protocol: same machine, server version, generation settings, and input/output lengths. Otherwise, you are comparing configurations rather than models.

Operating without internet

The pilot is complete and the model is selected. Now comes the part teams often underestimate: maintaining the system for years without access to external repositories.

Backups. Keep copies of weights, tokenizer, configuration, container images, and dependency packages with checksums. Verify that you can restore a working server from these copies on a clean machine. A restore test matters more than merely having a backup.

Update delivery. Define who downloads a new library or model version outside the isolated environment, who verifies checksums and licensing, how files are brought inside, and how rollback works. A new model version means rerunning your 60-task test, not merely replacing files.

Network audit. Configure the firewall to block outbound connections by default and log attempts. Review logs after every update: a new library version may add a request that was not there before.

Logs and access. Decide which prompt and response text is recorded, where and for how long it is retained, and who can access it.

Responsible engineer. Assign an owner for configuration, updates, and repeated checks. Without an owner, the isolated environment gradually becomes a server whose configuration nobody remembers.

What to do next:

1. Describe one use case (search, extraction, or assistant) and its working languages.
2. Select two or three candidates and check their licenses and language limits in their model cards.
3. Calculate weight memory for the required format and estimate headroom for context and concurrent requests.
4. Prepare 60 de-identified tasks with reference answers and run a pilot under the same configuration.
5. Record the outbound-connection block and review network logs during the pilot.
6. Before launch, assign an engineer and document backup and update procedures.

Frequently asked questions