LLM and GenAI questions, with answers

The questions below appear, in some wording, in most GenAI screens. Each answer follows the shape interviewers reward: mechanism first, trade-off second, experience third.

Setup#

pip install numpy

One question below comes with runnable proof. It needs no GPU and finishes in under a second.

"What does temperature actually do?"#

The weak answer is "it controls creativity". The strong answer: the model produces a score per candidate word, and temperature rescales those scores before they become probabilities. Low temperature sharpens the distribution; high temperature flattens it.

You can watch this happen with four fake word scores:

import numpy as np

logits = np.array([2.0, 1.0, 0.2, -1.0])   # raw scores the model gives 4 candidate words
words = ["delhi", "mumbai", "pune", "goa"]

def softmax(x, temperature):
    z = x / temperature
    z = z - z.max()            # subtracting the max avoids overflow; result is unchanged
    p = np.exp(z)
    return p / p.sum()

for t in [0.2, 1.0, 2.0]:
    probs = softmax(logits, t)
    row = "  ".join(f"{w}:{p:.3f}" for w, p in zip(words, probs))
    print(f"T={t}:  {row}")

T=0.2:  delhi:0.993  mumbai:0.007  pune:0.000  goa:0.000
T=1.0:  delhi:0.632  mumbai:0.232  pune:0.104  goa:0.031
T=2.0:  delhi:0.447  mumbai:0.271  pune:0.182  goa:0.100

At T=0.2 the top word takes 99% of the probability, so outputs barely vary. At T=2.0 the model behaves close to a random word picker.

The follow-up is always "when would you set it to zero?" Answer: extraction, classification, anything a program will parse. Then add the honest caveat: temperature 0 reduces variation, but does not guarantee bit-identical outputs across servers, batch sizes, or model updates. Saying that caveat out loud is a strong signal.

"Our chatbot hallucinates. What do you do, cheapest first?"#

Interviewers ask this to see whether you reach for an ordered plan or a silver bullet.

1. Constrain the prompt. Tell the model to answer only from provided context, and give it explicit permission to say "I don't know". Refusal must be a legal output.
2. Add retrieval, so there is something true in front of it to copy from. That is RAG.
3. Require citations. A wrong answer with a source label is catchable by a human in seconds. A wrong answer without one is invisible.
4. Build the escalation path. Low retrieval confidence should route to a human, not to a guess.
5. Measure. Build a set of real questions with known answers, and score groundedness before and after each change.

Close with the honest sentence: nothing on that list eliminates hallucination. Claiming something does is an instant credibility hit.

"RAG or fine-tuning — how do you choose?"#

The rule interviewers want to hear: retrieval for knowledge, fine-tuning for behaviour.

Facts that change — prices, policies, inventory — belong in retrieval. Updating an index is cheap and instant, and you get citations for free. Style, format, tone, and domain-specific behaviour belong to fine-tuning, usually via LoRA so the cost stays sane.

The trap inside the question: fine-tuning is an unreliable way to inject facts. The model tends to learn the style of your documents rather than their contents. Also mention that the two compose — a fine-tuned model behind a retrieval layer is a normal production setup, not a contradiction.

"Costs tripled last month. Traffic was flat. Why?"#

Billing is per token — the word-piece unit models read and write — in both directions. Flat request counts with growing token counts means the requests themselves got fatter.

The usual culprits: chat history is resent on every turn, and conversations got longer. Someone raised the number of retrieved chunks. A prompt template grew. The answer that gets hired: "I would measure tokens per request first, then cap history, summarise older turns, cache repeated prefixes, and route easy queries to a smaller model."

"The model must return JSON. It manages 95% of the time. Fix it."#

Never parse and pray. In rising order of strength:

• Validate every response against a schema, and on failure retry once with the error message pasted into the prompt.
• Use the provider's structured output mode, which constrains generation to the schema. See structured output.
• For tool arguments specifically, use function calling rather than asking for free-form JSON.

The one-line summary interviewers like: turn "usually valid" into "valid by construction, with a checked fallback".

"What is prompt injection? Does a strong system prompt stop it?"#

Prompt injection is when text the model reads — a user message, a retrieved web page, a document — contains instructions that hijack its behaviour. And no, a strong system prompt does not stop it. Instructions and data travel down the same channel, so the model cannot reliably tell them apart.

Real mitigations: give the model least-privilege tools, treat all retrieved text as untrusted, filter outputs, and put a human approval step in front of consequential actions. Full lesson: prompt injection.

Common mistakes in this round#

Brand names instead of mechanisms. "I'd use LangChain" answers nothing. Describe the moving parts; name tools only if asked.

Absolutes. "RAG eliminates hallucination" and "temperature 0 is deterministic" are both false, and interviewers know it.

Parameter-count worship. Quoting model sizes as proof of quality signals reading, not building. Task fit, latency, and cost decide.

Benchmark scores as product evidence. The strong version is: "we built an eval set from our own real tickets, and here is what moved it."

Try it yourself#

Answer "why did our bot invent a refund policy?" out loud, in ninety seconds, and record it. Then listen back and count brand names — the target is zero. Check that a mechanism appeared inside your first two sentences.

What to learn next#

• ML system design interviews — the next round in most loops.
• Hallucination — the failure behind half of these questions.
• RLHF — how preference training actually works.