The demo works. Somebody wired a model to a folder of documents, the answers looked plausible, and now there is a plan to roll it out across the organization. This is the point where AI projects meet infrastructure, and most of the difficulty is not the model.
Retrieval is the product
For internal assistants and document intelligence, answer quality is dominated by retrieval, not by model choice. That means ingestion, chunking, embedding, refresh when source documents change, and permission-aware search so a user never receives content they could not otherwise open.
Permissions do not come for free
An assistant that ignores document permissions is a data breach with a friendly interface. Access control has to be carried through the retrieval layer, which shapes how documents are indexed in the first place.
Capacity is lumpy
GPU workloads do not behave like web traffic. Inference is steady and latency-sensitive; fine-tuning arrives in bursts and will happily consume everything available. Without quotas and scheduling, one experiment starves production.
Decide what must stay local
Not everything should run on local infrastructure, and claiming otherwise does customers no favours. Open-weight models handle a great deal of retrieval, classification and extraction work well. Where you need frontier capability, a hybrid design — sensitive corpus local, specific calls external, boundary documented — is usually more honest than a local-only claim nobody can sustain.