Fine-tuning Llama 3.1 on an RTX 5090

Fine-tune a Korean LLM on a single GPU

Tuning Llama 3.1 8B with Unsloth on gcube

“Don’t you need a whole farm of GPU servers to fine-tune a model?”

A few years ago, yes. Today, one GPU is enough.

In this post, we walk through fine-tuning Meta’s Llama 3.1 8B on a Korean dataset with the Unsloth library, from start to finish, on the gcube GPU cloud. By the end, you’ll be able to build your own Korean LLM.

Why fine-tune? How it differs from prompt engineering

There are two main ways to make an LLM behave the way you want.

• Prompt engineering leaves the model as it is and steers it toward the answer you want by carefully crafting the input. It’s fast and free, but it’s hard to make the model reliably handle domain knowledge it lacks or follow a specific tone or format.
• Fine-tuning further trains the model’s weights on your own data. When you need a specific domain (internal documents, law, medicine) or a consistent response style, it reaches a quality that prompts alone can’t.

If you need “a chatbot that only answers in our company’s tone” or “a model that answers questions in a specific field accurately,” fine-tuning is the answer.

Unsloth: why one GPU is now enough

Full fine-tuning of an 8B-class model used to mean retraining every parameter, which required several high-end GPUs.

This is where QLoRA comes in. The base model’s weights are quantized to 4-bit and frozen, and only small adapters inserted into the model are trained. For Llama 3.1 8B, the trainable parameters drop to about 42 million, 0.5% of the total, while keeping over 99% of the accuracy.

With hand-written Triton kernel optimizations, Unsloth trains about 2x faster and uses about 70% less VRAM than standard Hugging Face Transformers. (Sources: Unsloth official blog, Towards Data Science)

Measured VRAM on an RTX 5090

• Right after loading 4-bit Llama 3.1 8B: about 5.7 GB
• Peak during training with LoRA adapters: about 12.5 GB (a single 16 GB GPU is enough)
• Works on Colab T4 (16 GB), RTX 4090, L4, A100 and RTX 5090
• Note: Unsloth supports a single GPU only (for multi-GPU, use TRL or Axolotl)

This is exactly where gcube fits. Its promise of running and scaling GPU workloads in a few clicks, without complex AI infrastructure, matches single-GPU fine-tuning perfectly.

What you’ll need

• A gcube account (to create a workload; see STEP 0 for the settings)
• A Hugging Face account and access token (to download models and datasets)

STEP 0. Configure the gcube workload

When you create a workload in the gcube console, enter the values below as they are, and environment setup and the Unsloth install are done in one go. The unsloth/unsloth image comes with PyTorch, CUDA, Unsloth and JupyterLab preinstalled.

With HF_TOKEN set as an environment variable, models and datasets download inside the notebook without any separate login code. Once the workload is up, open JupyterLab on port 8888 and follow the steps below.

Go to gcube Docs →

※ Only if you set up the environment yourself: run pip install unsloth on Python 3.10/3.11 with CUDA 12.1 or later. With the official image, you don’t need to worry about this.

STEP 1. Load the 4-bit Llama 3.1 8B model

We load a model that Unsloth has already quantized to 4-bit. It also downloads 4x faster.

from unsloth import FastLanguageModel
import torch

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name = "unsloth/Meta-Llama-3.1-8B-bnb-4bit",
    max_seq_length = 2048,
    dtype = None,
    load_in_4bit = True,
)

The key is load_in_4bit = True. The model weights sit in VRAM in 4-bit, while activations and gradients flow at higher precision during training. That’s why the 8B model takes up only about 5.7 GB right after loading.

Note: unsloth/Meta-Llama-3.1-8B-bnb-4bit is a redistribution by Unsloth, so unlike Meta’s original, it downloads right away without a separate access approval.

STEP 2. Set up the LoRA adapters

Now we attach the adapters we’re going to train.

model = FastLanguageModel.get_peft_model(
    model,
    r = 16,
    target_modules = ["q_proj", "k_proj", "v_proj", "o_proj",
                      "gate_proj", "up_proj", "down_proj"],
    lora_alpha = 16,
    lora_dropout = 0,
    bias = "none",
    use_gradient_checkpointing = "unsloth",
    random_state = 3407,
)

• r: the adapter’s rank. Usually 8, 16, 32 or 64; start with 16.
• target_modules: attaches adapters to all attention and MLP layers.
• use_gradient_checkpointing="unsloth": saves another 30% of VRAM and handles longer contexts.

When training is done, the output isn’t a full checkpoint several gigabytes in size, but a single adapter file of a few tens of megabytes. That makes it easy to deploy.

STEP 3. Prepare the Korean dataset

This is the heart of the exercise. To adapt Llama, which was pretrained mostly on English, to Korean, you need good Korean instruction data.

We use beomi/KoAlpaca-v1.1a. It contains about 21,000 Korean Q&A pairs and downloads right away without access approval. (KoAlpaca-RealQA has more natural real-user data, but it’s a gated dataset that requires approval on Hugging Face.)

from datasets import load_dataset

dataset = load_dataset("beomi/KoAlpaca-v1.1a", split = "train")

# This dataset's columns are instruction / output.
# The template below means "### Question:" / "### Answer:" in Korean.
def formatting_func(example):
    text = f"""### 질문:
{example['instruction']}

### 답변:
{example['output']}""" + tokenizer.eos_token
    return {"text": text}

dataset = dataset.map(formatting_func)

Column names differ by dataset. After loading, check dataset.column_names first. This dataset uses output, while KoAlpaca-RealQA uses response.

STEP 4. Run fine-tuning

Now we start training, using SFTTrainer.

from trl import SFTTrainer
from transformers import TrainingArguments

trainer = SFTTrainer(
    model = model,
    tokenizer = tokenizer,
    train_dataset = dataset,
    dataset_text_field = "text",
    max_seq_length = 2048,
    args = TrainingArguments(
        per_device_train_batch_size = 2,
        gradient_accumulation_steps = 4,
        warmup_steps = 5,
        max_steps = 60,
        learning_rate = 2e-4,
        fp16 = not torch.cuda.is_bf16_supported(),
        bf16 = torch.cuda.is_bf16_supported(),
        logging_steps = 1,
        optim = "adamw_8bit",
        weight_decay = 0.01,
        lr_scheduler_type = "linear",
        seed = 3407,
        output_dir = "outputs",
        report_to = "none",
    ),
)

trainer.train()

report_to = "none" is required. In some environments, SFTTrainer tries to load its default logging integration (trackio), which conflicts with the installed huggingface_hub version and raises an ImportError. Turning logging off avoids it.

• If you hit OOM, lower per_device_train_batch_size to 1–2. This is the most common cause of memory errors.
• Training time: about 2 hours per 1,000 samples on a T4, under an hour on an RTX 4090, and faster still on an RTX 5090.

STEP 5. Before / after: comparing the results

Let’s run inference with the trained model.

FastLanguageModel.for_inference(model)

inputs = tokenizer(
    # "### Question: Explain Korea's jeonse (lump-sum deposit) rental system.### Answer:"
    ["### 질문:\n한국의 전세 제도에 대해 설명해줘.\n\n### 답변:\n"],
    return_tensors = "pt"
).to("cuda")

outputs = model.generate(**inputs, max_new_tokens = 256)
print(tokenizer.batch_decode(outputs)[0])

Before fine-tuning, the base model often answers Korean questions in English or produces stilted, translated-sounding Korean. After fine-tuning, you can see it answer in natural Korean that matches the intent of the question.

STEP 6. Export to GGUF and serve with Ollama

Once training is done, you’ll want to put the model into actual use. Unsloth supports GGUF export, so you can serve the model directly with Ollama or llama.cpp.

model.save_pretrained_gguf("korean_llama", tokenizer,
                           quantization_method = "q4_k_m")

The first run takes about 10–20 minutes because it includes merging the 16-bit weights and building llama.cpp. (Once built, later runs are faster.)

Watch the output path. Even if you pass "korean_llama", Unsloth actually saves to a korean_llama_gguf/ folder, and the file name follows the base model’s name. Check the real location with the code below.

import os
gguf_dir = "korean_llama_gguf"
print([f for f in os.listdir(gguf_dir) if f.endswith(".gguf")])
# e.g. ['Meta-Llama-3.1-8B.Q4_K_M.gguf'] (about 4.9 GB)

Register this GGUF file with Ollama, and you can call your own Korean model through an API.

That’s the full process of fine-tuning a Korean LLM on a single GPU.

Wrapping up: so where do you get the GPU?

Every step of fine-tuning comes down to one question: can you get a single GPU with enough VRAM, right when you need it?

gcube is a GPU cloud platform built to solve exactly that.

• Heterogeneous GPU pooling: from the RTX 4090 to high-end GPUs, allocated flexibly to fit your workload
• Container-based multi-tenancy: each training environment is kept safely isolated
• Start in a few clicks: focus on fine-tuning right away, without wrestling with drivers or CUDA setup

The Unsloth + Llama 3.1 8B fine-tuning we ran today needs nothing more than a single gcube workload. Start building your own LLM today.

Go to gcube →

※ gcube is a GPU cloud platform recognized with a CES 2026 Innovation Award in the AI category.