Securing AI: The Self-Sovereign, Privacy-Compliant LiteLLM Platform

With LiteLLM, companies can manage multiple AI models through a central API, control access and budgets, and centrally implement data protection and compliance requirements. Based on its own practical experience, dkd demonstrates how a LiteLLM gateway for OpenAI, Anthropic, Mistral, and other models can be set up and operated securely.

A gateway to AI

LiteLLM creates a central interface through which companies can deploy and manage various LLMs and AI providers in a controlled manner.

When the first people at dkd began working seriously with LLMs, the question quickly arose: who gets access? One key for OpenAI, one for Anthropic – per person or per team, each with its own billing. And once a provider key has found its way into an IDE configuration or a script, it’s practically impossible to retrieve it.

That might still work with three people, but with over thirty developers, it becomes problematic. So we set up a gateway.

What LiteLLM does

LiteLLM is an open-source proxy that aggregates models behind an OpenAI-compatible API. Applications access an endpoint; the gateway translates this to the target model’s native interface and handles authentication, routing and cost tracking. We have been running it on our own server infrastructure since 2025.

The practical result is that provider keys are now stored solely within the gateway. Instead, our end users are issued with API keys for LiteLLM – per person, team or application – each with its own budget, model selection and, where applicable, additional settings such as an expiry date. A key that is no longer required to have access is simply deleted from the gateway, and that’s it. There’s no need for anyone to rotate keys with a provider.

Monitoring the data sent

A gateway hosted on our own infrastructure does not turn a request to OpenAI into an internal request. The prompt is still sent to the provider, and their data processing terms continue to apply.

What changes is everything before and after that. Authentication, key management and logging take place on our servers, not at the provider’s. We can see which application has made which request and can intervene via guardrails before anything is sent – for example, by masking personal data or rejecting certain requests entirely.

And for cases where content must not leave the premises, a local model via Ollama or vLLM, or an Azure deployment in an EU region, can be connected to the same endpoint. The application is unaware of this; to it, it is simply a different model name. That is the real benefit: the decision on where a request is processed is a single configuration line, not a code change.

Cost control

Every call is logged – by key, user and team – and counted against a budget. Once the limit for the set period has been reached, further access is blocked. In addition, there are rate limits to manage peak loads and aliases that can be used, for example, to redirect requests to a more cost-effective model without needing to modify an application.

What we have done with this data is described in a separate article: the dkd AI usage and cost dashboard retrieves these figures in real time and displays each employee’s status and usage in a graphical format.

If a provider goes down

This happens, and usually at the worst possible moment. In the LiteLLM Gateway, you can store multiple deployments of the same model and define fallbacks. If one provider goes down, the request is routed via the next one – or perhaps even a different model – and the application is completely unaware of this.

LiteLLM at dkd

Around sixty staff members use the gateway on a daily basis. We run Open WebUI as our chat interface; alongside this, we use coding agents such as Claude Code, OpenCode and Pi, as well as plugins for the development environments. We are connected to OpenAI, Anthropic and EU providers such as TensorX and Mistral.

As a result, we’re also familiar with the pitfalls: which default settings are best changed, where prompt caching actually saves money and where it doesn’t, and how to set up tagging so that costs can be allocated to a project afterwards. For the last point, we’ve built our own plugins for coding agents, as the out-of-the-box solution wasn’t clean or was simply too complicated.

What the dkd can do for you

We set up LiteLLM on your infrastructure – whether dedicated, in the cloud or on-premises – integrate your providers, and configure virtual keys, budgets and guardrails to meet your data protection requirements. This includes integrating the tools your staff actually use: chat interfaces, coding agents, IDE plugins and existing applications.

Once up and running, we handle maintenance and updates, incorporate new models, build guardrails to meet your compliance requirements, set up monitoring and alerting systems, and train your teams. Where standard features fall short, we develop the missing components, just as we have done for ourselves.

Would you like to know what such a platform might look like in your organisation?

Request a LiteLLM consultation

Sources & further links

[1] AI transformation? We are your AI agency
[2] The path to dkd AI transformation: On adventurers, stabilisers and balancers
[3] Prompts, please!
[4] Starfruit AI