If we want to build internal support chatbot pinecone style (fast, accurate, and actually useful), the winning pattern is boring and repeatable: ingest your PDF manuals into a vector database, retrieve only the relevant fragments at question-time, then force the model to answer strictly from those fragments and format the output cleanly for Slack.
Here’s the whole thing in one breath:
That’s Tier 1 support automation that doesn’t hallucinate itself into a lawsuit.
Style rules we’re following are aligned with our internal writing guidelines.
A RAG (Retrieval-Augmented Generation) support chatbot is an internal assistant that answers questions by retrieving relevant excerpts from your real documentation (PDFs, runbooks, SOPs) and generating an answer grounded in those excerpts, rather than “remembering” or inventing.
Quote-worthy line we’ve learned the hard way: If your bot can answer without retrieval, it can also hallucinate without friction.
The market meme is: “Just plug your PDFs into a chatbot and you’re done.”
No. That approach fails in the most predictable ways: stale manuals, weird PDF formatting, tables turning into soup, and the model confidently stitching unrelated paragraphs together because you never taught it what “relevant” means operationally.
What works is less sexy: retrieval discipline, metadata hygiene, and response formatting that treats Slack like a production UI, not a text box.
| Step | What we do | Why it matters in Tier 1 support |
|---|---|---|
| 1. Ingest | Extract text from PDFs, normalize, chunk, embed, upsert to Pinecone | You can’t retrieve what you didn’t structure |
| 2. Retrieve | Embed the user question, query Pinecone, apply thresholds + rerank | Stops “closest-ish” matches from poisoning answers |
| 3. Generate | Answer using retrieved passages only, with guardrails | Hallucination rate drops when context is constrained |
| 4. Deliver | Format the answer into Slack blocks with source pointers | Tier 1 is UX; ugly output doesn’t get adopted |
Pinecone isn’t “better at AI.” It’s better at being a production database for embeddings: predictable latency, namespaces for multi-tenancy, metadata filtering, and operationally sane scaling. Internal support workloads are spiky and annoying: Monday morning floods, post-release panic, “why did payout exports break again?” questions. You want boring infra.
Still, teams ask: “Why not just use a local vector store and call it a day?”
Because internal support is not a demo. You will need:
| Vector DB | Best for | What breaks first in Tier 1 support |
|---|---|---|
| Pinecone | Managed, low-ops, metadata-heavy retrieval | Cost discipline if you index everything “just in case” |
| pgvector (Postgres) | Teams already deep in Postgres ops | Recall/latency tuning under load becomes your hobby |
| Weaviate | Feature-rich retrieval + hybrid options | Operational complexity if your team isn’t owning it |
| Milvus | High-scale, self-managed vector workloads | Infra overhead and upgrades in the critical path |
We’re opinionated here: Tier 1 support bots die from maintenance fatigue, not model quality. The database choice is mostly about reducing future misery.
PDF ingestion is where “RAG” turns into “why is this returning the copyright page.”
The goal is not “extract text.” The goal is extract text with boundaries that map to how humans ask questions: feature names, UI labels, error codes, configuration keys, step sequences, and version-specific notes.
We want each stored chunk to carry enough metadata to support filtering and debugging later.
At minimum, store:
doc_id (stable identifier)titlesection (best-effort)page_start, page_endproduct / moduleversion (if you can detect it)updated_at (your ingestion time, not the PDF’s claim)content_type (manual, SOP, release_notes)A Tier 1 bot without metadata is a confident liar, because you can’t constrain it.
If your PDFs are digital, extraction is easy-ish. If they’re scanned, you’re in OCR land and you should expect worse retrieval until you clean it.
Here’s a pragmatic Python extraction path for digital PDFs:
from pathlib import Path
import re
import pypdf # pip install pypdf
def extract_pdf_pages(pdf_path: str) -> list[dict]:
reader = pypdf.PdfReader(pdf_path)
pages = []
for i, page in enumerate(reader.pages):
text = page.extract_text() or ""
text = re.sub(r"[ \t]+", " ", text).strip()
pages.append({"page": i + 1, "text": text})
return pages
pages = extract_pdf_pages("manual.pdf")
print(len(pages), pages[0]["page"], pages[0]["text"][:200])
This is deliberately boring. The fancy part comes next: chunking.
Chunking is where most “my RAG sucks” tickets come from.
What you want:
Here’s a robust baseline chunker: split by headings / blank lines, then pack into token-ish windows with overlap.
def split_into_blocks(text: str) -> list[str]:
# crude heading detection (works surprisingly often for manuals exported from docs)
lines = [ln.rstrip() for ln in text.splitlines()]
blocks, buf = [], []
for ln in lines:
if not ln.strip():
if buf:
blocks.append("\n".join(buf).strip())
buf = []
continue
buf.append(ln)
if buf:
blocks.append("\n".join(buf).strip())
return [b for b in blocks if len(b) > 30]
def pack_blocks(blocks: list[str], max_chars: int = 1800, overlap_chars: int = 250) -> list[str]:
chunks = []
cur = ""
for b in blocks:
if len(cur) + len(b) + 2 <= max_chars:
cur = (cur + "\n\n" + b).strip()
else:
if cur:
chunks.append(cur)
# overlap: carry the tail of previous chunk into next
tail = cur[-overlap_chars:] if cur else ""
cur = (tail + "\n\n" + b).strip()
if cur:
chunks.append(cur)
return chunks
def chunk_pages(pages: list[dict]) -> list[dict]:
out = []
for p in pages:
blocks = split_into_blocks(p["text"])
chunks = pack_blocks(blocks)
for idx, ch in enumerate(chunks):
out.append({
"page": p["page"],
"chunk_index": idx,
"text": ch
})
return out
This isn’t “perfect.” It’s operational. Perfect chunking is a myth; good chunking is measurable.
You generate an embedding per chunk, then upsert into Pinecone with metadata.
The one rule: store the raw chunk text in metadata (or in an external store keyed by ID). If you don’t, you’ll spend your life reconstructing context.
Below is a clean structure. The exact SDK calls vary by Pinecone client version, so treat this as a blueprint: initialize index → create vectors with id, values, metadata → upsert in batches.
import hashlib
from datetime import datetime, timezone
def stable_chunk_id(doc_id: str, page: int, chunk_index: int, text: str) -> str:
h = hashlib.sha1(text.encode("utf-8")).hexdigest()[:12]
return f"{doc_id}:p{page}:c{chunk_index}:{h}"
def build_vectors(doc_id: str, title: str, chunks: list[dict], embed_fn):
# embed_fn(texts: list[str]) -> list[list[float]]
texts = [c["text"] for c in chunks]
embs = embed_fn(texts)
now = datetime.now(timezone.utc).isoformat()
vectors = []
for c, emb in zip(chunks, embs):
vec_id = stable_chunk_id(doc_id, c["page"], c["chunk_index"], c["text"])
vectors.append({
"id": vec_id,
"values": emb,
"metadata": {
"doc_id": doc_id,
"title": title,
"page": c["page"],
"chunk_index": c["chunk_index"],
"text": c["text"],
"updated_at": now,
"content_type": "manual"
}
})
return vectors
Then upsert in batches:
def upsert_in_batches(index, vectors, batch_size=100):
for i in range(0, len(vectors), batch_size):
batch = vectors[i:i+batch_size]
index.upsert(vectors=batch)
Yes, we’re skipping a full Pinecone init snippet here because it changes across SDK generations and team auth setups. The logic does not change: you upsert vectors with metadata, and you keep your IDs stable so re-ingestion is sane.
Manuals evolve. Error codes change. UI labels get renamed. If you don’t handle versioning, your bot becomes that coworker who insists the button is called “Settings” because it was in 2022.
Fix it with one of these patterns:
docs:v3_21, docs:v3_22version >= xWe prefer namespaces when the business can name versions cleanly. You can query multiple namespaces if you must, but don’t make the model arbitrate between conflicting eras without telling it what era it’s in.
Retrieval is not “query the vector DB and paste top 5.”
Retrieval is a control system:
This is the retrieval chain we actually trust in internal support:
| Stage | What happens | Operational outcome |
|---|---|---|
| Embed question | Turn Slack question into embedding | Similarity search becomes possible |
| Query Pinecone | TopK with metadata filters | Scope control (module/version) |
| Threshold | Drop low-score matches | Prevents “kinda related” pollution |
| Deduplicate | Remove near-duplicate chunks | Less repetitive context, better answers |
| Rerank (optional) | Cross-encoder or LLM rerank | Higher precision for ambiguous queries |
| Context pack | Build a compact evidence bundle | Better grounding, lower token burn |
| Answer | Model must cite chunk IDs/pages | Auditable support output |
| Fallback | If low confidence, escalate | Stops confident nonsense |
def retrieve(index, question: str, embed_fn, top_k: int = 8, min_score: float = 0.78, filters: dict | None = None):
q_emb = embed_fn([question])[0]
res = index.query(
vector=q_emb,
top_k=top_k,
include_metadata=True,
filter=filters or {}
)
matches = []
for m in res.get("matches", []):
score = m.get("score", 0.0)
if score >= min_score and m.get("metadata", {}).get("text"):
matches.append({
"id": m["id"],
"score": score,
"metadata": m["metadata"]
})
return matches
That min_score is not universal. You tune it by measuring how often Tier 1 answers are correct vs “vaguely plausible.” In most orgs, “vaguely plausible” is the enemy because it wastes more human time than an honest “I don’t know, here’s who to ask.”
Manuals often repeat the same paragraph across pages. If you pass duplicates into the model, it learns the wrong thing: repetition equals importance.
import difflib
def dedup_by_similarity(matches, threshold=0.92):
kept = []
for m in matches:
text = m["metadata"]["text"]
is_dup = False
for k in kept:
ratio = difflib.SequenceMatcher(None, text, k["metadata"]["text"]).ratio()
if ratio >= threshold:
is_dup = True
break
if not is_dup:
kept.append(m)
return kept
def build_context_bundle(matches, max_chars=6000):
parts = []
total = 0
for m in matches:
meta = m["metadata"]
header = f"[{meta.get('title','Doc')} | p{meta.get('page')} | {m['id']} | score={m['score']:.2f}]"
body = meta["text"].strip()
chunk = header + "\n" + body
if total + len(chunk) > max_chars:
break
parts.append(chunk)
total += len(chunk)
return "\n\n".join(parts)
Now your generator gets evidence that looks like evidence, not like a random paste.
This is the part people get weirdly religious about. We’re not. We want the model to do two things:
SYSTEM = """You are an internal Tier 1 support assistant.
Rules:
- Answer ONLY using the provided CONTEXT.
- If the answer is not in CONTEXT, say you cannot confirm from docs and ask one clarifying question.
- Include a short "Sources" line with doc title + page numbers from CONTEXT headers.
- Be concise but specific. No speculation.
"""
def build_user_prompt(question: str, context_bundle: str) -> str:
return f"""CONTEXT:
{context_bundle}
QUESTION:
{question}
OUTPUT FORMAT:
Answer: <plain language answer>
Steps: <if applicable, short numbered steps>
Sources: <doc title + page numbers>
Confidence: <high/medium/low>"""
The “Confidence” field isn’t for vibes. It’s for routing. Low confidence triggers escalation or a follow-up question instead of a wrong answer.
Slack is where credibility goes to die if your bot posts wall-of-text blobs.
A Tier 1 assistant has to look like a competent teammate:
Here’s a pattern that works: one message, multiple blocks.
def slack_blocks(answer: str, steps: list[str] | None, sources: list[str], confidence: str):
steps_text = ""
if steps:
# Slack mrkdwn supports numbered lists decently
steps_text = "\n".join([f"{i+1}. {s}" for i, s in enumerate(steps)])
sources_text = ", ".join(sources[:4]) + ("" if len(sources) <= 4 else f" +{len(sources)-4} more")
conf_emoji = {"high": "🟢", "medium": "🟠", "low": "🔴"}.get(confidence.lower(), "🟠")
blocks = [
{
"type": "header",
"text": {"type": "plain_text", "text": "Tier 1 Support Answer"}
},
{
"type": "section",
"text": {"type": "mrkdwn", "text": f"*Answer*\n{answer}"}
}
]
if steps_text:
blocks.append({
"type": "section",
"text": {"type": "mrkdwn", "text": f"*Steps*\n{steps_text}"}
})
blocks.append({"type": "divider"})
blocks.append({
"type": "context",
"elements": [
{"type": "mrkdwn", "text": f"{conf_emoji} *Confidence:* {confidence.title()}"},
{"type": "mrkdwn", "text": f"📄 *Sources:* {sources_text}"}
]
})
blocks.append({
"type": "actions",
"elements": [
{"type": "button", "text": {"type": "plain_text", "text": "Show excerpts"}, "value": "show_excerpts"},
{"type": "button", "text": {"type": "plain_text", "text": "Escalate to human"}, "style": "danger", "value": "escalate"}
]
})
return blocks
Your Slack app can listen for button interactions. “Show excerpts” can respond with the top 2 retrieved chunks (verbatim, so users trust it). “Escalate” can open a ticket and attach the context bundle.
| Format | Looks good in Slack | Best for | Why it fails |
|---|---|---|---|
| Plain message text | Sometimes | Small teams, low volume | Turns into walls-of-text during incidents |
| Block Kit sections | Yes | Production Tier 1 | Needs a consistent schema or users get confused |
| Attachments | Meh | Legacy bots | Feels dated and inconsistent across clients |
| Threaded follow-ups | Yes | Evidence display | If overused, it becomes spammy |
We like Block Kit because it enforces discipline. Your bot stops rambling when you constrain the UI.
At Triumphoid, when we first built internal support bots, we expected the hardest part to be “the model.”
Wrong. The hardest part was docs reality: outdated manuals, contradictory pages, feature flags not documented, and PDF exports that chopped headings into nonsense. The bot wasn’t hallucinating because models are evil; it was hallucinating because we handed it vague retrieval and asked it to guess.
Two changes made it behave like an adult:
A practical insight: your support team becomes your labeling system if you surface sources and let them flag bad ones. That’s cheaper than building a formal evaluation pipeline on day one.
Chunk IDs matter more than you think. If IDs aren’t stable, every re-ingest duplicates your index and retrieval quietly degrades.
Tables are retrieval kryptonite. Most PDF extractors flatten them into nonsense. If your manuals contain critical tables (limits, error codes, configuration matrices), you may need a separate pass that detects tables and stores them as structured text (“Key: Value” rows) or even JSON.
You need an “unknown answer” path. Tier 1 support is not a trivia game. If the bot can’t find the answer, it should ask exactly one clarifying question or escalate. The fastest way to lose trust is one confident wrong message in a public channel.
Metadata filtering is not optional. If you have multiple products or versions, retrieval without filters will happily mix them. The model will then produce a Franken-answer that looks coherent and breaks reality.
Prompt injection is real in internal docs too. If your PDFs include “copy/paste this prompt” or user-generated content, you should sanitize and consider a content policy layer. A bot that follows instructions found inside retrieved text is a bot that can be tricked by accident.
Pro-Tip (highly technical): add a “lexical backstop” and a “retrieval audit log.”
Hybrid retrieval (vector + keyword) catches exact strings like error codes, config keys, and UI labels that embeddings sometimes under-rank.
A retrieval audit log that stores {question, filters, top_ids, scores, doc_pages} lets you debug wrong answers in minutes instead of vibes-based arguing.
Here’s how the pieces fit together in a single request cycle:
def handle_slack_question(index, question, embed_fn, llm_fn, filters=None):
matches = retrieve(index, question, embed_fn, top_k=10, min_score=0.78, filters=filters)
matches = dedup_by_similarity(matches)
if not matches:
answer = "I can’t confirm that from our manuals. Which product/module and version are you asking about?"
return slack_blocks(answer, None, [], "low")
context = build_context_bundle(matches)
llm_out = llm_fn(system=SYSTEM, user=build_user_prompt(question, context))
# assume llm_out is parsed into fields; keep parsing strict in production
answer = llm_out["answer"]
steps = llm_out.get("steps_list")
sources = llm_out.get("sources_list", [])
confidence = llm_out.get("confidence", "medium").lower()
return slack_blocks(answer, steps, sources, confidence)
That’s the spine. Everything else is improving ingestion quality, retrieval precision, and UX.
Most OCR automations fail because they OCR everything. Logos, signatures, random screenshots, someone’s cat. The…
We pulled 84,000 contact records from a client's CRM last month to feed into their…
The Triumphoid team is heading to Workflow 2026 on March 5, 2026 in San Francisco.…
Let me tell you about a Tuesday afternoon in March 2024. A client needed to…
Most automation workflows are fire-and-forget. An event happens, a sequence of steps executes, data moves…
Elizabeth Sramek from our team will be at WordCamp Madrid on March 6-7, 2026. Two…