Running Open-Weight LLMs on Premises for Regulated Firms

Why regulated firms look at on-premises models

A bank, a mortgage lender or an investment adviser holds information that regulators call customer information: account details, income, identity documents, loan files, advice. Staff want to use a language model to summarize a credit file, draft a client letter or search policies. The easy way is a hosted model reached through an internet API. That can be acceptable, but it makes the model provider a service provider that handles customer information, and it means every prompt leaves the building.

The alternative is an open-weight model that the firm runs itself. Open-weight means the trained parameters are published and can be downloaded and operated locally, usually under a license that must be read, since terms vary by model. Running the weights in your own rack keeps prompts and documents inside your network, lets you control retention and logging, and removes the dependency on a vendor's availability. It also puts the whole stack in your hands: hardware, operating system, serving software, the model files and the application on top. This article covers what the rules ask of that stack and how to size it.

Our team has described one such design in an illustrative bank case on the Experience page. If you are still choosing between a local model and a frontier API, read our article on open-weight models versus frontier APIs first.

Which rule applies to you

The Gramm-Leach-Bliley Act directs regulators to set standards for safeguarding customer information. Each regulator implemented it for the firms it supervises, and the text differs in detail, not in shape. The Federal Trade Commission's Safeguards Rule, 16 CFR Part 314, applies to financial institutions over which the FTC has jurisdiction. Its own scope section lists examples: mortgage lenders, payday lenders, finance companies, mortgage brokers, account servicers, check cashers, wire transferors, collection agencies, credit counselors, tax preparation firms and non-federally insured credit unions, among others.

Banks are different. For national banks and federal savings associations, the standards are the Interagency Guidelines Establishing Information Security Standards, which the OCC publishes as Appendix B to 12 CFR Part 30, with parallel versions for other bank regulators. Their outline is familiar: involve the board, assess risk, manage and control risk, oversee service provider arrangements, adjust the program and report to the board. SEC-registered advisers and broker-dealers are covered by Regulation S-P, which our article on the Regulation S-P amendments checklist explains. The rest of this section uses the FTC text because it is the most detailed, and you should map the same ideas to the rule that governs you.

What the Safeguards Rule asks of a model server

Section 314.4 lists the elements of an information security program. Almost every one applies directly to an on-premises model. The list below connects the text to the new system.

• Qualified Individual (314.4(a)). Someone must be responsible for the security program, and the model server is in scope.
• Written risk assessment (314.4(b)). Add the model to it. The rule asks for criteria to evaluate threats, an assessment of confidentiality, integrity and availability, and requirements for how risks are mitigated or accepted.
• Access controls (314.4(c)(1)). Authenticate users and limit each to the customer information needed for the job. For a model, this includes the documents it can retrieve on a user's behalf, not just the chat window.
• Encryption (314.4(c)(3)). Customer information must be encrypted in transit over external networks and at rest, or covered by compensating controls approved by the Qualified Individual. Prompts, retrieved passages, conversation history, vector indexes and log files can all contain customer information.
• Multi-factor authentication (314.4(c)(5)) for any individual accessing any information system, unless the Qualified Individual approves in writing a reasonably equivalent control.
• Disposal and retention (314.4(c)(6)). Customer information must be disposed of securely no later than two years after last use in providing the product or service, with exceptions the text lists. Decide how long chat transcripts and embeddings live.
• Change management (314.4(c)(7)) and monitoring and logging of authorized users' activity (314.4(c)(8)). Model upgrades, prompt templates and tool permissions are changes. Logs must let you detect misuse.
• Testing (314.4(d)). Continuous monitoring or annual penetration testing, plus vulnerability assessments at least every six months. Add prompt-injection testing to the scope.
• Training and staffing (314.4(e)), service provider oversight (314.4(f)), program adjustment (314.4(g)), a written incident response plan (314.4(h)) and a written report to the board at least annually (314.4(i)).
• FTC notification (314.4(j)). A notification event involving the information of at least 500 consumers must be reported to the FTC as soon as possible and no later than 30 days after discovery.

Section 314.6 exempts firms holding information on fewer than five thousand consumers from some of these items, namely the written risk assessment, the testing, the incident response plan and the board report. The point of this list is not to treat the model as exotic. It is a server that processes customer information, and the existing program should cover it before the first user logs in.

NIST AI RMF and the generative AI profile

The NIST Artificial Intelligence Risk Management Framework, released January 26, 2023, is voluntary. It does not replace the rules above. It gives a shared vocabulary for the AI-specific work that a general security program may miss. Its Core has four functions: govern, map, measure and manage. Governance is cross-cutting and informs the other three. Map is about context: what the system is for, who is affected, what could go wrong. Measure is about testing and tracking. Manage is about acting on what you find.

NIST released a companion, NIST AI 600-1, Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile, on July 26, 2024. It is a cross-sectoral profile that helps organizations identify risks specific to generative AI and suggests actions for managing them. NIST's own pages now say the AI RMF 1.0 is being revised, so check the current version before you cite a section number in a policy.

OWASP Top 10 for LLM Applications 2025

The OWASP Gen AI Security Project publishes a list of the most critical risks in applications built on large language models. The 2025 edition has ten entries: prompt injection, sensitive information disclosure, supply chain, data and model poisoning, improper output handling, excessive agency, system prompt leakage, vector and embedding weaknesses, misinformation and unbounded consumption. It is a practical test plan for the application layer, and several entries read differently when the model runs in your own rack.

• Prompt injection. Text in a document, an email or a web page can try to change the model's instructions. If the assistant reads internal files, assume some of them are written by outsiders, such as client emails. Keep the model's tools narrow so a hijacked prompt cannot do much.
• Sensitive information disclosure. The risk is less that the model memorizes your data, since you are not training on it, and more that retrieval returns a document the asking user should not see. Enforce entitlements in the retrieval layer, not in the prompt.
• Supply chain. Model files, tokenizers, serving software and Python packages are all code or code-like inputs. Download from a source you trust, record file hashes, prefer weight formats that cannot execute code on load, and mirror what you use internally.
• Excessive agency. If the model can send mail, change records or call internal APIs, scope each tool to the least privilege that works and require human approval for actions that cannot be undone.
• Vector and embedding weaknesses. A vector database is a copy of your documents in another form. Protect and encrypt it like the source, and test whether one client's material can be retrieved for another.
• Unbounded consumption. A local system has finite GPUs. Without per-user rate limits and maximum context lengths, a single long request or a loop in an agent can starve everyone else.

Third-party risk does not disappear

A common misunderstanding is that hosting the model yourself ends third-party risk management. It narrows it and changes its shape. The Board, FDIC and OCC issued final interagency guidance on managing risks associated with third-party relationships on June 6, 2023. It describes a lifecycle of planning, due diligence and selection, contract negotiation, ongoing monitoring and termination, with governance around it. The guidance is addressed to banking organizations, and it is a sensible model for any regulated firm.

Apply it to the components. The GPU server vendor, the integrator and the support contract are third parties. So is the group that publishes the model weights and the maintainers of the serving software. You cannot audit them in the traditional sense, but you can perform due diligence on license terms, security advisories, update cadence, and the effort required to replace each part. Have an exit plan, such as the ability to swap one open-weight model for another because your application talks to a standard serving interface. For banks, the Interagency Guidelines also require due diligence in selecting service providers, contractual security obligations and monitoring where the risk assessment calls for it.

Sizing the hardware from public specifications

Two numbers drive the first sizing step: how many parameters the model has, and how many bytes each one takes. A model's weights must sit in GPU memory, or in memory the GPU can reach quickly, whenever it serves a request. In FP8, a parameter takes one byte. A model with 671 billion parameters therefore needs roughly 671 GB for its weights, plus a small allowance for scaling factors.

DeepSeek-V3, described in its technical report, is a mixture-of-experts model with 671 billion total parameters, of which 37 billion are activated for each token. The authors describe an FP8 mixed-precision training framework, and the model's public page lists FP8 and BF16 inference in serving software such as SGLang and vLLM. Mixture-of-experts matters for cost: a router sends each token through a small subset of expert networks, so the compute per token resembles a model with about 37 billion parameters. But every expert must be resident in memory because any token might need any of them. That is why memory capacity, not arithmetic, sets the minimum size of the system.

Now compare with public GPU specifications. NVIDIA lists 141 GB of HBM3e memory at 4.8 TB/s for the H200. Eight of them give 1,128 GB. NVIDIA lists 1,440 GB total GPU memory for its eight-GPU B200 system, which is 180 GB per GPU. A 671 GB weight set fits comfortably in either, but not with all the room: the KV cache, which stores attention state for every active conversation, grows with context length and the number of simultaneous users, and the serving software needs workspace. A realistic plan reserves a large share of memory for the KV cache, which is why a long-context workload with many users needs more than the weights suggest.

Smaller models change the picture. A dense model with 70 billion parameters in FP8 needs about 70 GB for weights and fits on a single 141 GB GPU. Choose the smallest model that passes your own evaluation set. The figures in our illustrative bank case, including speeds and capacity, are modeled from public specifications and benchmarks, not measured on a client system. Treat any throughput claim, including ours, as a starting hypothesis to test on your hardware with your documents.

A practical build checklist

• Encrypt prompts, history, indexes and logs; log who asked what, with restricted access, since logs are customer information too.
• Scope the use cases and data classes first, and record them in the AI risk assessment. Decide which uses are not allowed.
• Put the model servers in a segment with no direct internet access. Mirror packages and weights through a controlled path, and verify hashes.
• Require single sign-on with multi-factor authentication. Give each user their own identity so logs mean something.
• Enforce document entitlements in retrieval. Test with accounts that should not see each other's material.
• Treat model, prompt and tool changes as changes. Re-run the evaluation set and keep the previous version ready for rollback.
• Test before go-live and at least annually: penetration testing, prompt-injection tests and a tabletop incident exercise that includes the model.

None of this requires a particular vendor. Clayton Cybernetics designs and operates private AI systems for regulated clients, and is vendor-neutral about the GPUs and serving software. What matters is that the controls exist on paper, are tested, and match what the system actually does.

Sources

• 16 CFR Part 314 - Standards for Safeguarding Customer Information (eCFR)
• Appendix B to 12 CFR Part 30 - Interagency Guidelines Establishing Information Security Standards (eCFR)
• NIST - AI Risk Management Framework
• NIST AIRC - AI RMF Core (govern, map, measure, manage)
• NIST AI 600-1 - Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile
• OWASP Gen AI Security Project - Top 10 for LLM Applications 2025
• Federal Register - Interagency Guidance on Third-Party Relationships: Risk Management (June 9, 2023)
• DeepSeek-V3 Technical Report (arXiv 2412.19437)
• DeepSeek-V3 model page (Hugging Face): FP8 and BF16 inference, SGLang, vLLM, license
• NVIDIA H200 Tensor Core GPU: 141 GB HBM3e, 4.8 TB/s
• NVIDIA DGX B200: 8 Blackwell GPUs, 1,440 GB total GPU memory

This article is general information, not legal or compliance advice, and no regulator has reviewed it. We read 16 CFR Part 314 and Appendix B to 12 CFR Part 30 on the eCFR, the NIST pages, the OWASP list page (titles and summaries), the June 2023 interagency guidance on the Federal Register and public vendor and model pages on 2026-11-08. The NIST AI 600-1 PDF and the FFIEC site could not be read directly, so we make no statements about the content of the 600-1 risk categories, and NIST says the AI RMF is being revised. Which rule applies depends on your regulator and charter. Hardware sizing is arithmetic from public specifications; real memory needs and speeds depend on the serving software, context lengths and number of users, and should be measured. Check the license of any model you intend to use. Ask your counsel and your examiner about your specific obligations. Last reviewed November 8, 2026.

Keep reading

Next · InfrastructureHurricane-Proofing IT by Moving to the Cloud: A Florida PlaybookA server room on the Gulf or Atlantic coast, Florida above all, has two kinds of risk in storm season: the building and the grid. A hurricane can flood the room, cut the power for days and close the roads staff need to reach it. Moving workloads to a public cloud removes the first risk only if the design is right, and it adds new ones. This playbook explains the Atlantic hurricane season as NOAA describes it, how Ready.gov frames business continuity and IT recovery, what Florida Statutes section 501.171 asks of a business that holds personal information, and how Google Cloud regions and zones work, with a step-by-step plan for the move.10 min read

• Local, Private or Frontier? How to Use AI Without Exposing Sensitive Data6 min
• Local AI vs Frontier APIs: Cost, Privacy and Governance of Running Open-Weight Models on Your Own GPUs8 min