Building a RAG chatbot on gcube docs with an RTX 5090

Introduction

In our fine-tuning series, we put knowledge into the model itself. This time we go the other way. We leave the model as it is and have it search external documents at query time, then use what it finds in its answer. This is RAG (retrieval-augmented generation).

Our source material is the official gcube documentation (gai-platform-docs). We’re building a chatbot that answers gcube questions from gcube docs, running on a gcube GPU — a self-referential exercise.

The embedding model and a 7B LLM share a single RTX 5090, and no external APIs are involved.

Note on language: We ran this experiment on the Korean version of the docs (*.ko.md) with Korean questions, so the original outputs are in Korean. Code is shown exactly as we ran it, with English translations in comments. Outputs are translated from Korean and marked as such.

Why RAG

When answers must be backed by a clear source, as with a chatbot built on internal documents, RAG has a clear advantage.

Architecture

[MkDocs docs (.ko.md)]
     │ chunking
     ▼
[BGE-M3 embeddings] ──▶ [Chroma vector DB]
                              │ search (top-k)
                              ▼
[User question] ──▶ [LangChain: retrieval + prompt assembly] ──▶ [vLLM: Qwen2.5-7B] ──▶ [Answer]

Environment

• gcube workload: 1× RTX 5090 (32 GB)
• Container image: unsloth/unsloth (JupyterLab included)

Setting up the environment

The image’s default environment already has a torch build for fine-tuning, so installing the RAG stack on top of it causes version conflicts. Instead, we create a separate venv, install everything there, and register it as a notebook kernel.

Paste the following into a notebook cell and run it (it’s a shell cell that starts with %%bash):

%%bash
pip install -q virtualenv
python -m virtualenv /workspace/rag-env

/workspace/rag-env/bin/pip install -q vllm langchain langchain-community \
  langchain-text-splitters langchain-huggingface langchain-chroma \
  langchain-openai chromadb sentence-transformers ipykernel ipywidgets

/workspace/rag-env/bin/python -m ipykernel install --user \
  --name=rag-env --display-name "Python (rag-env)"

ipywidgets is needed for progress bars. If you leave it out, later cells show TqdmWarning: IProgress not found.

Once the install finishes, switch the notebook kernel to rag-env. Use whichever method you prefer:

• Option 1: Click the current kernel name (Python 3 (ipykernel)) in the toolbar at the top right of the notebook.
• Option 2: In the top menu bar, choose Kernel → Change Kernel…

Both open the Select Kernel dialog. Pick Python (rag-env) from the dropdown in the middle and click Select Kernel at the bottom right.

After switching, the indicator at the top right changes to Python (rag-env). You can confirm with this cell:

import sys, vllm
print(sys.executable)   # /workspace/rag-env/bin/python
print(vllm.__version__) # 0.25.1

From here on, you can paste the Python code for steps 1–8 into cells and run it as is.

Finally, clean up the environment variables:

import os

# The default /data is not writable in this container
os.environ["HF_HOME"] = "/workspace/hf-cache"

# Legacy variable set in the image — leaving it triggers a deprecation warning in every cell
os.environ.pop("HF_HUB_ENABLE_HF_TRANSFER", None)

HF_HUB_ENABLE_HF_TRANSFER is set by default in the unsloth/unsloth image, but recent versions of huggingface_hub no longer use it and raise a FutureWarning. Removing it as shown above makes the warning go away.

Step 1. Collect the documents

From the official gcube docs repository, we keep only the Korean documents (.ko.md).

%%bash
cd /workspace
git clone https://github.com/data-alliance/gai-platform-docs.git 2>/dev/null || echo "already cloned"
find gai-platform-docs -name "*.ko.md" | wc -l

Result: 38 .ko.md files. They’re organized under docs/user-guide/ into folders such as node, gcube-cli and platform-guide, which makes it easy to attach per-source metadata during chunking.

Step 2. Preprocessing and chunking

We strip the MkDocs front matter (the metadata block wrapped in ---), split first by headers, then split again by character count.

from langchain_text_splitters import MarkdownHeaderTextSplitter, RecursiveCharacterTextSplitter

def clean_text(text):
    # Remove front matter (--- ... ---)
    if text.startswith('---'):
        idx = text.find('---', 3)
        if idx != -1:
            text = text[idx+3:]
    return text

headers_to_split_on = [("#", "h1"), ("##", "h2"), ("###", "h3")]
md_splitter = MarkdownHeaderTextSplitter(headers_to_split_on=headers_to_split_on)
char_splitter = RecursiveCharacterTextSplitter(chunk_size=800, chunk_overlap=100)

Read the documents and chunk them:

import glob

files = glob.glob('/workspace/gai-platform-docs/**/*.ko.md', recursive=True)
print('files found:', len(files))

all_chunks = []
for fp in files:
    with open(fp, encoding='utf-8') as f:
        raw = f.read()
    cleaned = clean_text(raw)
    md_docs = md_splitter.split_text(cleaned)
    for d in md_docs:
        d.metadata['source'] = fp  # for tracing the source document
    all_chunks.extend(char_splitter.split_documents(md_docs))

print('total chunks:', len(all_chunks))
print('avg chunk len:', sum(len(c.page_content) for c in all_chunks) // len(all_chunks))

Watch the import paths. Recent LangChain (0.3+/1.x) is split into separate packages by function. In this post we import from langchain_text_splitters (splitters), langchain_huggingface (embeddings), langchain_chroma (vector DB) and langchain_openai (LLM client). Older paths you’ll see in earlier tutorials, such as langchain.text_splitter or langchain_community.*, raise ModuleNotFoundError.

Result: 235 chunks in total, averaging 296 characters. When we opened the first chunk, we found chunks containing nothing but image Markdown links such as ![gcube_Banner image](images/...). These “noise chunks” carry almost no text, so a production setup would need a post-processing step that filters out image-only chunks or applies a minimum length.

Step 3. BGE-M3 embeddings

from langchain_huggingface import HuggingFaceEmbeddings

embeddings = HuggingFaceEmbeddings(
    model_name="BAAI/bge-m3",
    model_kwargs={"device": "cuda"},
    encode_kwargs={"normalize_embeddings": True},
)

Result: The BGE-M3 model (about 2.3 GB) downloads and all 235 chunks are embedded. The first run takes a few minutes for the download; later runs reuse the cache.

BGE-M3 uses only about 2–3 GB of VRAM, so it fits comfortably within 32 GB alongside the vLLM server we’ll start later (7B, --gpu-memory-utilization 0.6) on the same GPU.

Step 4. Indexing in Chroma

from langchain_chroma import Chroma

vectordb = Chroma.from_documents(
    documents=all_chunks,
    embedding=embeddings,
    persist_directory="/workspace/rag-lab/chroma_db",
)
print('indexed:', vectordb._collection.count())

Result: All 235 chunks indexed successfully (indexed: 235).

Step 5. Search test

Before adding the LLM, we check search quality with the index alone.

query = "gcube CLI는 어떻게 설치하나요?"  # "How do I install the gcube CLI?"
results = vectordb.similarity_search(query, k=3)

for i, doc in enumerate(results):
    print(f"--- rank{i+1} ---")
    print("SOURCE:", doc.metadata["source"])
    print(doc.page_content[:200].replace("\n", " "))
    print()

Result: cli-overview.ko.md ranks near the top, but which document comes first depends on the run. In our reproduction test, openwebui-comfyui.ko.md (a document that includes the gcube CLI install commands) came first and cli-overview.ko.md second, with similarity scores almost tied at 0.58 vs 0.60.

Output (translated from Korean):

--- rank 1 ---
SOURCE: .../platform-guide/openwebui-comfyui.ko.md
Enter the following commands in order in the gcube workload terminal. ...

--- rank 2 ---
SOURCE: .../gcube-cli/cli-overview.ko.md
gcube CLI is the official command-line tool for the gcube AI GPU cloud platform.
From the terminal, you can register and manage GPU workloads, monitor resources and stream container logs.

Since “How do I install…?” also matches a document that contains the actual install commands, both results are valid. Semantic search finds the relevant documents accurately without any tuning, which is a good sign of BGE-M3’s Korean embedding quality.

Don’t rely on top-1. When scores are this close, a small change to the documents can flip the ranking. It’s safer to retrieve several candidates with k=3 or more and pass them to the LLM together. We set it up that way in step 6.

Step 6. vLLM serving + LangChain

The vLLM server is a long-running process, so we start it in the background with Popen. With subprocess.run(), the kernel stays tied up in that cell.

import subprocess, os, sys, time, urllib.request

env = {**os.environ, "HF_HOME": "/workspace/hf-cache", "VLLM_USE_FLASHINFER_SAMPLER": "0"}
proc = subprocess.Popen(
    [sys.executable, "-m", "vllm.entrypoints.openai.api_server",
     "--model", "Qwen/Qwen2.5-7B-Instruct",
     "--host", "0.0.0.0", "--port", "8001",
     "--gpu-memory-utilization", "0.6",
     "--max-model-len", "8192"],
    stdout=open("/workspace/vllm.log", "w"), stderr=subprocess.STDOUT, env=env,
)
print("Starting server, pid:", proc.pid)

Model path: If you have a checkpoint from the fine-tuning series, change the --model value to its local path (for example /workspace/llama-3.1-8b-finetuned). That directory must contain a config.json. If you point to a path that doesn’t exist, the server dies immediately with OSError: Can't load the configuration of .... Why sys.executable****: If you write plain "python", the shell’s default python is picked up instead of the kernel’s, and the server may run outside rag-env.

In the next cell, poll until the server is ready. The first run of a 7B model needs a 15 GB download and can take 5–10 minutes.

for i in range(60):
    if proc.poll() is not None:
        print(f"Server exited (exit={proc.returncode}) — check the log below")
        break
    try:
        r = urllib.request.urlopen("http://localhost:8001/v1/models", timeout=3)
        print("READY:", r.read().decode())
        break
    except Exception as e:
        print(i, "waiting...", e)
        time.sleep(10)

If startup fails, check the log for the cause:

print(open("/workspace/vllm.log").read()[-3000:])

On RTX 5090 (Blackwell, sm_120), VLLM_USE_FLASHINFER_SAMPLER=0 is required. Without this flag, model loading goes fine, but the server crashes at the first sampling step with RuntimeError: FlashInfer requires GPUs with sm75 or higher. The RTX 5090 is a newer architecture than sm75, so this error appears to be a bug in flashinfer 0.6.13, which doesn’t recognize the latest GPU generation correctly.

Connecting LangChain:

from langchain_openai import ChatOpenAI

llm = ChatOpenAI(
    base_url="http://localhost:8001/v1",
    api_key="dummy",
    model="Qwen/Qwen2.5-7B-Instruct",  # must match the --model value above
    max_tokens=300,
)

# Guardrail — see step 7 for details
# English: "You are an assistant that answers based on the official gcube documentation.
#           Find support for your answer only in the document content provided below.
#           If the answer is not in the documents, reply 'This can't be confirmed from the documents.'"
SYSTEM_PROMPT = """당신은 gcube 공식 문서를 기반으로 답변하는 어시스턴트입니다.
아래 제공된 문서 내용에서만 근거를 찾아 답변하세요.
문서에 없는 내용이면 "문서에서 확인할 수 없습니다"라고 답하세요."""

def answer_with_rag(question, k=3):
    docs = vectordb.similarity_search(question, k=k)
    context = " ".join(d.page_content for d in docs)
    messages = [
        {"role": "system", "content": SYSTEM_PROMPT},
        {"role": "user", "content": f"문서:{context}\n질문:{question}"},  # "Documents: ... / Question: ..."
    ]
    resp = llm.invoke(messages)
    return resp.content, docs

def answer_without_rag(question):
    """For comparison — ask the model directly, without retrieval (used in DEMO 3)"""
    return llm.invoke([{"role": "user", "content": question}]).content

Instead of the RetrievalQA chain class, we wrote retrieval → prompt assembly → call as plain functions. In recent LangChain (1.x), RetrievalQA is treated as legacy, and building it directly is easier to debug. The logic is simple anyway.

Step 7. Prompt guardrail

The SYSTEM_PROMPT defined in step 6 is what guards against hallucination. The key is the instruction to say it doesn’t know when something isn’t in the documents.

You are an assistant that answers based on the official gcube documentation.
Find support for your answer only in the document content provided below.
If the answer is not in the documents, reply "This can't be confirmed from the documents."

(English translation of the Korean prompt used in the code.)

Whether the model actually follows this instruction is tested in the next step.

Step 8. Live demo

We check the effect of RAG with three questions.

# DEMO 1 — a question the docs cover
ans, docs = answer_with_rag("gcube CLI는 어떤 기능을 제공하나요?")  # "What features does the gcube CLI provide?"
print(ans)
print("Source document:", docs[0].metadata["source"])

# DEMO 2 — a question the docs don't cover (guardrail test)
ans, _ = answer_with_rag("gcube는 몇 명이 창업했나요?")  # "How many people founded gcube?"
print(ans)

# DEMO 3 — the same question without RAG (hallucination comparison)
print(answer_without_rag("gcube는 몇 명이 창업했나요?"))

DEMO 1 — a question the docs cover

Output (translated from Korean):

Q: What features does the gcube CLI provide?
A: The gcube CLI lets you register and manage GPU workloads, monitor resources and stream container logs from the terminal.
Source document: cli-overview.ko.md

The model answered accurately based on the retrieved content, and we can also trace which document it used as its source.

DEMO 2 — a question the docs don’t cover (guardrail test)

Output (translated from Korean):

Q: How many people founded gcube?
A: The documents do not provide information on how many people founded gcube. Therefore, I cannot answer this question.

The guardrail worked exactly as intended. With no supporting evidence in the retrieved context, the model declined to answer.

DEMO 3 — the same question without RAG (before/after)

Output (translated from Korean):

Q: How many people founded gcube?
A: GCube is a Korean cloud computing company founded in 2008 by three founders: CEO Kim Byeong-seon, Director Choi Jeong-hwan and Director Kim Dong-gyun.

A textbook hallucination. The model invented a founding year that doesn’t exist and three founders who don’t exist, complete with names. It’s hard to find a clearer example of why RAG is needed.

Guardrails depend on the model. We also ran the same pipeline with Qwen2.5-0.5B. In DEMO 2, it ignored the guardrail and made up an answer: “GCube was founded by 4 people.” Same prompt, same retrieval results, different outcome.

In other words, even when the RAG pipeline finds the right evidence, sticking to that evidence comes down to the LLM’s instruction-following ability. For a real service, use an instruct model of at least 7B. If a smaller model is unavoidable, it’s safer to add a code-level block that skips the LLM call entirely when the retrieval similarity falls below a threshold.

Limitations and next steps

• Chroma here is local and file-based. For production, we recommend migrating to something like Qdrant.
• Room to improve retrieval accuracy by adding a reranker (such as BGE-reranker).
• Keeping context across multi-turn conversations needs a separate implementation.
• Filtering logic for noise chunks that contain only image links still needs to be added.
• Swapping in your own fine-tuned model could improve answer quality and tone. Just change -model to the local path.