Cookies on softitland.com

We use essential storage to run this site. With your permission we would also like to use analytics cookies to see which pages help visitors, and marketing cookies to measure our ads. You can change your choice at any time with “Cookie settings” in the footer. Privacy Policy

AI & automation

RAG, fine-tuning or a long prompt: how to give an AI model your business knowledge

How to give an AI model your business knowledge: when a long prompt is enough, when you need RAG, and why fine-tuning rarely suits facts.

Cover image for the article RAG, fine-tuning or a long prompt: how to give an AI model your business knowledge
Key takeaways
  • A long prompt with caching suits a small, stable set of documents that every user may see.
  • Retrieval (RAG) suits large or changing content, answers that must cite a source, and knowledge that differs by user role.
  • Fine-tuning shapes tone and format; it is a poor home for prices and policies that change.
  • OpenAI is winding down its fine-tuning platform and no longer accepts new users, so check availability before planning around it.
  • Whatever the method, keep a fixed test set and re-run it after every change.

Which method should you use to give an AI model your business knowledge?

Start with a long prompt when your knowledge is small and stable, move to retrieval (RAG) when it is large, changes often or must be filtered by who is asking, and treat fine-tuning as a way to shape behaviour and format, not as a way to load facts. Most business assistants end up with a well-written prompt plus retrieval. Fine-tuning is the rare third step, and fewer providers offer it than a year ago.

This is a decision guide. If you want the build steps for a customer-facing bot, read our guide to website chatbots that answer from your own content.

What does each method actually change?

The three methods put your knowledge in different places, and that decides what happens when a price or a policy changes.

MethodWhere your knowledge livesBest for
Long promptIn the instructions sent with every requestA small, stable set of documents everyone may see
Retrieval (RAG)In a search index; only the relevant passages are sentLarge or changing content, citations and access by role
Fine-tuningIn a trained copy of the modelConsistent tone, format or a narrow repeated task

The key difference is upkeep. With a prompt or retrieval, you edit a document and the next answer uses it. With fine-tuning, a change means new training data, a new training run and a new round of tests.

When is a long prompt enough?

More often than people expect. Current Claude models have a 1M-token context window, and in its 2024 Contextual Retrieval article Anthropic advised that a knowledge base smaller than 200,000 tokens, about 500 pages, can simply be included in the prompt.

A long prompt suits a returns policy, a service menu, opening hours, a product spec sheet or an internal process guide. It is quick to build, easy to test and has no index to maintain.

Two limits matter. First, size is not the same as accuracy: Anthropic's context window guide notes that as the token count grows, accuracy and recall degrade, which it calls context rot. Curate what goes in rather than pasting everything you own. Second, cost: you pay for those tokens on every request unless you cache them.

Prompt caching helps here. By default the cache lasts 5 minutes and is refreshed each time it is used, and a 1-hour cache is available at extra cost. Writing to the 5-minute cache costs 1.25 times the base input price, while a cache read costs 0.1 times the base input price on most models, and less on some newer ones. For an assistant that reuses the same long instructions all day, that changes the cost picture a lot.

When do you need retrieval (RAG)?

Retrieval means your documents sit in a search index, and only the passages that match the question are sent to the model. Choose it when one of these is true:

  • The content is large: a help centre, years of product manuals or a contract library.
  • The content changes daily: stock, schedules, case notes or new tickets.
  • Different people may see different things: customers, staff and managers each have their own set of documents.
  • Answers must point to a source: staff need to open the exact page the answer came from.

Retrieval quality is where these projects succeed or fail. In the same Contextual Retrieval study, combining contextual embeddings with keyword search (BM25) reduced failed retrievals by 49%, and adding a reranking step brought the reduction to 67%. The lesson for a buyer: ask how the search is built and how it is tested, not only which model answers.

You do not always need to build the index yourself. OpenAI's file search tool searches uploaded files with semantic and keyword search, returns file citations and supports metadata filters. Amazon Bedrock Knowledge Bases manages ingestion, indexing and retrieval, can include citations in the response and offers document-level permission filtering for managed knowledge bases. A custom index makes sense when your data lives in your own database or needs rules those services do not cover. For live records such as CRM data, an MCP connection can let the assistant look records up directly instead of indexing copies.

What is fine-tuning good for?

Fine-tuning trains a copy of a model on your examples. OpenAI's model optimization guide describes it as a way to make a model consistently format responses in a certain way or handle novel inputs, or to train a smaller, cheaper model to do one task well. The same page says prompt engineering may be all you need, and OpenAI's accuracy guide calls prompt engineering the best place to start.

That is why fine-tuning is a poor home for facts. Prices, policies and stock change, and a trained model keeps the old version until you train it again. Use retrieval for what is true today, and use fine-tuning, if at all, for how the answer should look.

Check availability before you plan around it. OpenAI's guide now says it is winding down its fine-tuning platform, that the platform is no longer accessible to new users, and that existing fine-tuned models stay available until their base models are deprecated.

How do citations and access control change the choice?

Citations let a person check an answer in seconds. Claude's citations feature returns detailed citations when it answers from documents you provide, and the documentation says those citations are guaranteed to point to valid parts of the documents. That works with a long prompt and with retrieval, as long as you pass the source documents in the request.

Access control is different. With a single long prompt, everyone who uses the assistant can draw on everything in it. If staff-only pricing, HR files or client records must stay with the right people, you need retrieval that filters documents by the user's role before anything reaches the model. Keep those permissions in your own system, not in the prompt; our guide to keeping customer data safe with AI APIs covers the data side.

What does each method cost to keep running?

  • Long prompt: someone owns the source document and updates it. Re-run your tests after each change.
  • Retrieval: new and changed documents must be re-indexed, deleted ones removed, and the search checked against real questions. Plan who does this and how often.
  • Fine-tuning: every behaviour change means new examples, a new training run, a new evaluation and a switch-over.

Whatever you choose, keep a fixed test set and run it after every change, as we describe in testing an AI assistant before launch.

How do you decide for your business?

Answer five questions in order:

  • How much content is there, in pages, and could a person curate the core of it?
  • How often does it change: yearly, weekly or every hour?
  • Must answers differ by who is asking?
  • Must every answer link to its source?
  • Is the problem missing knowledge, or the style and format of the answers?

Small, stable and public points to a long prompt with caching. Large, changing or private points to retrieval. A style or format problem that survives good prompting is the only strong case for fine-tuning.

Our AI assistants and chatbots team builds customer-facing assistants on these patterns, and our AI agents and integrations team connects them to your documents, CRM and databases. For a portal or internal tool around the assistant, our custom software team builds the access rules and admin screens.

Frequently asked questions

Can we just put all our documents into the prompt?

If the core material is small, yes, and it is often the fastest start. Current Claude models accept up to 1M tokens, but accuracy and recall degrade as prompts grow, and you pay for the tokens on every request unless you cache them. Curate the documents that answer real questions instead of pasting everything you own.

Does fine-tuning teach a model our prices and policies?

It is the wrong tool for facts that change. A fine-tuned model keeps what it learned until you train it again, so an old price can stay in its answers. Put prices and policies in a prompt or a retrieval index, and keep fine-tuning, if you use it at all, for tone and format.

How do we stop the assistant showing staff-only documents to customers?

Do not put those documents in a shared prompt. Use retrieval that checks the user's role and filters documents before the search results reach the model. Keep the permission rules in your own system or in a service that supports document-level filtering, and test with accounts for each role.

Which method can show where an answer came from?

Retrieval and long prompts can both return citations when the source documents are passed in the request. Claude's citations feature and OpenAI's file search both return references to the documents used. Fine-tuning cannot: the knowledge is inside the trained model, with no page to point to.

How long does a retrieval build take?

It depends on where your content lives, how many systems feed it and whether access rules are needed. A managed service with a few document folders is a short project; several databases with role-based access take longer. A scoped quote should list sources, update frequency, roles and the test set before work starts.

Last updated

Start a project

Have a project in mind? Let’s build it.

Tell us what you need. You get a clear plan, a fixed quote and a timeline before any work starts.

Prefer email? info@softitland.com