On-Premise LLMs for SMEs in Mersin: AI Without Your Data Leaving the Building

Generative AI is no longer only on the agenda of big tech companies; it matters just as much to a food exporter, a logistics company, or a manufacturer in Mersin. Language models save real time on everyday work such as preparing quotes, summarizing documents, classifying customer emails, and writing commentary for reports. Yet many business owners hesitate over the same question: 'Is it safe to send company data to a foreign cloud service?'

An on-premise LLM is the answer: an open-weight language model runs on the company's own server or in a data center under its control, and requests and documents never leave the company network. Open model families such as Llama, Qwen, Mistral, and Gemma have, in recent years, reached quality close to closed cloud models on many business tasks. That makes local deployment viable not only for large enterprises but for SMEs as well.

This article is the first part of MeSoft's Enterprise AI Series. Across the series we cover PDF reporting with in-house LLMs, inference optimization, the attention mechanism, and RAG-based knowledge assistants, grounded in the real needs of businesses in Mersin.

Cloud AI services offer real advantages: a fast start, no maintenance, and access to the largest models. For low-volume work without sensitive data, the cloud is often the most sensible choice. On-prem stands out when data includes personal information or trade secrets, when contracts or sector rules restrict data leaving the country, when usage is high and predictable, and when the system must work independently of an internet connection.

The data protection angle matters. Under Turkey's Personal Data Protection Law No. 6698 (KVKK), transferring personal data abroad is subject to specific conditions and safeguards; sending documents containing customer names, employee records, or health data to an API abroad requires legal review and additional processes. With an on-prem deployment the data is processed on the company's own infrastructure, which largely removes the transfer question, although obligations around processing records, access control, and logging still apply.

Cost depends on volume. With cloud services, cost grows linearly with the number of tokens processed; on-prem requires an upfront hardware investment, after which costs are largely fixed (power, maintenance). For a company processing thousands of documents a day, on-prem becomes more economical above a certain volume. Before deciding, we recommend a simple total cost of ownership (TCO) calculation based on current and expected usage.

Open models are usually described by parameter count: 3 to 4 billion (3B-4B), 7 to 14 billion (7B-14B), around 30 billion, and 70 billion and above. Small models are fast and cheap, and surprisingly good at well-defined tasks such as classification, information extraction, and short summaries. Large models excel at complex reasoning, long-form generation, and precise instruction following. Mixture-of-Experts (MoE) models offer a good balance by activating only part of their parameters for each token, even when the total count is large.

Turkish-language performance is a critical criterion for companies in Turkey. Models differ noticeably in Turkish grammar, terminology, and writing quality, and generic benchmarks don't always reflect those differences. We therefore select models with an evaluation set of 30 to 100 examples built from the company's real tasks. Licenses must be checked as well: some models ship under permissive licenses such as Apache 2.0, while others use custom licenses with conditions on commercial use.

For most SME scenarios our advice is: start with a good 8B to 14B model, support it with RAG and well-written instructions, measure performance, and move to a larger model only if the measurements call for it. Setups where several small task-specific models work together instead of one large model can also be cost-effective.

The most critical resource for running an LLM is GPU memory (VRAM). As a rough rule, each billion parameters needs about 2 GB at 16-bit precision and about 0.5 to 0.6 GB with 4-bit quantization. On top of that, you need headroom for the KV cache, which grows with the number of concurrent users and context length. A 4-bit 8B model runs comfortably on a single 16 to 24 GB GPU, while a 70B-class model needs 48 GB or more, or multiple GPUs.

As a typical starting point, a business with 20 to 50 employees using document summarization, email drafting, and an internal knowledge assistant is often well served by a single-GPU server with 24 to 48 GB of memory. High concurrency or large models bring in multi-GPU servers, NVLink, and tensor parallelism. On the CPU side, a strong processor, enough RAM (at least twice the model size is recommended), and fast NVMe storage speed up model loading and document processing.

We recommend making the hardware decision based on measurement rather than guesswork. A short proof of concept with samples from the real workload shows expected response times and concurrent user capacity clearly, so you neither overinvest nor frustrate users with a slow system. The inference optimization article in this series details techniques that improve hardware efficiency.

A typical on-prem LLM stack consists of GPU drivers and container infrastructure (Docker, Kubernetes where needed); an inference server such as vLLM, llama.cpp, or Ollama; a vector database for RAG; a web interface for users or an OpenAI-compatible API for integration with existing applications (ERP, CRM, email); and monitoring with tools such as Prometheus and Grafana. An OpenAI-compatible API makes it painless to switch models or servers later without touching application code.

Security doesn't end with 'data doesn't leave the building'. The model server should be reachable only from the internal network, authentication should integrate with the company's existing directory service (Active Directory, LDAP, SSO), users should only reach documents they are authorized to see, and all requests should be logged for auditing. Against LLM-specific risks such as prompt injection, the tools and data the model can access should follow the principle of least privilege.

At MeSoft we run end-to-end on-prem AI projects in Mersin covering needs analysis, model and hardware selection, deployment, integration, and team training. We also help you assess whether funding options such as KOSGEB and regional development agency programs fit your project. The next article in the series looks at one of the most tangible uses of this infrastructure: automated PDF reporting.