Distributed LLM fine-tuning with gcube replicas: a hands-on multi-node guide to Axolotl + DeepSpeed

We tied two RTX 5090 containers together and verified live, from start to finish, whether distributed fine-tuning actually works. It didn't go smoothly: we even hit a problem where NCCL communication froze completely, and finding the cause and working around it is the heart of this post.

Why multi-node

We've covered fine-tuning on a single GPU throughout this series, but in real work there comes a point where the model grows or the dataset expands and a single GPU's VRAM and throughput can't keep up. At that point you have two options: use one bigger GPU, or tie several of the same GPU together and split the computation. The latter is distributed training, and gcube's Replicas feature is built for exactly this scenario.

In this post, we verified live whether real multi-node distributed training works with gcube replicas, and whether adding nodes really scales training speed or the size of jobs you can handle. Tying several GPUs of the same spec together as replicas is far more stable in practice than mixing different GPUs, because compute speed and VRAM match. For this test, we deployed a Tier 2 RTX 5090 × 1 / 32 GB container from the gcube console with 2 replicas, and verified that communication between the nodes actually works and that distributed training runs to completion.

Environment

Container Image: axolotlai/axolotl-cloud-uv:main-py3.11-cu130-2.9.1

Among Axolotl's official images, we chose the -cloud-uv family for two reasons.

• cloud: built for deployment on GPU clouds like RunPod, with JupyterLab running by default and things like the HF cache volume mount preconfigured
• uv: built on the uv package manager, with the most stable support for the RTX 5090 (Blackwell, sm_120)

Container settings

Specs confirmed after deployment: CUDA 13.0, Driver 580.126.09, torch 2.9.1+cu130, verified identical on both nodes.

First attempt: running into a hang

On the first deployment, the basic network between the nodes (TCP raw sockets) was fine. We even succeeded in opening sockets directly and exchanging data. The trouble came next.

accelerate launch --config_file accelerate_config.yaml \
  -m axolotl.cli.train examples/llama-3/lora-1b.yml --num_epochs 1

When we ran this command on both nodes, rank 1 (node2) finished preparing the dataset and waited at the barrier, while rank 0 (node1) fell into a complete hang, with its CPU time not increasing at all for more than 8 minutes.

The socket state we checked with ss -tnp was the decisive clue.

CLOSE-WAIT   0   10.0.9.126:xxxxx   10.0.10.154:xxxxx   (6 different ephemeral ports)

All of the ephemeral ports node1 had opened toward node2 were stuck in CLOSE-WAIT. It looked like the socket handshake between the two nodes had gotten tangled during the bootstrap (rendezvous) stage of NCCL/torch.distributed. While we were preparing an isolation test (a pure torch.distributed all_reduce without axolotl), the web console terminal session itself disconnected, and debugging stopped there.

Lessons: this was most likely a transient network issue that wouldn't reproduce, and when something goes wrong, isolating the smallest unit first (a pure NCCL all_reduce) is a much faster way to diagnose than rerunning all of axolotl.

Second attempt: isolating the cause and fixing config issues

We deployed fresh containers with the same specs, and this time, instead of running axolotl right away, we started with a pure NCCL communication test.

import torch, torch.distributed as dist, datetime, time
t0 = time.time()
dist.init_process_group(backend='nccl', timeout=datetime.timedelta(seconds=60))
print('init done in', time.time()-t0, 's. rank', dist.get_rank(), 'world_size', dist.get_world_size())
torch.cuda.set_device(0)
x = torch.ones(1).cuda() * (dist.get_rank()+1)
dist.all_reduce(x)
print('after all_reduce', x)
dist.destroy_process_group()

# node1 (rank 0)
MASTER_ADDR=10.0.9.59 MASTER_PORT=29500 WORLD_SIZE=2 RANK=0 LOCAL_RANK=0 \
NCCL_SOCKET_IFNAME=eth0 NCCL_IB_DISABLE=1 NCCL_P2P_DISABLE=1 NCCL_DEBUG=WARN \
python3 nccl_test.py

# node2 (rank 1) — same command with RANK=1

Result:

init done in 3.30 s. rank 0 world_size 2
before all_reduce tensor([1.])
NCCL version 2.27.7+cuda13.0
after all_reduce tensor([3.])   # 1 (rank 0) + 2 (rank 1) = 3, communicated correctly
DONE OK

Communication was established on both ranks within 3 seconds. That confirmed the hang in the first attempt was a transient issue.

Three remaining config issues

Confirming NCCL communication didn't mean training ran right away. With the accelerate + DeepSpeed combination, we hit the following three config issues in order, and picked up one operational tip worth knowing during training.

1) The default launcher (pdsh) needs SSH

subprocess.CalledProcessError: Command ['deepspeed', '--hostfile', 'None', ..., '--launcher', 'pdsh', ...]

DeepSpeed's default multi-node launcher works by having rank 0 connect to the other nodes over SSH and run processes on them remotely. In gcube's axolotl-cloud image, the SSH daemon doesn't start at all unless the PUBLIC_KEY/SSH_KEY environment variables are set. The fix was to switch to a launcher that doesn't need SSH.

deepspeed_config:
  deepspeed_multinode_launcher: standard

With the standard launcher, each node starts its own processes independently and they find each other only through main_process_ip:main_process_port. No SSH is needed at all.

2) deepspeed_config_file conflicts with accelerate's top-level keys

ValueError: When using `deepspeed_config_file`, the following accelerate config variables will
be ignored: ['gradient_accumulation_steps', ..., 'mixed_precision']. Please specify them
appropriately in the DeepSpeed config file.

When you pass detailed settings through deepspeed_config_file (JSON), you must not also put values like mixed_precision at the top level of accelerate_config.yaml. bf16 is already handled in the axolotl training config (bf16: true), so remove the top-level keys.

3) A file name typo: zero2_bf16.json doesn't exist

ValueError: Expected a string path to an existing deepspeed config... Received: deepspeed_configs/zero2_bf16.json

The files you actually get from axolotl fetch deepspeed_configs are zero1.json, zero2.json, zero3.json, zero3_bf16.json and so on. There is no _bf16 version in the zero2 family. bf16 is set in the training config anyway, so just use zero2.json.

Final accelerate_config.yaml

compute_environment: LOCAL_MACHINE
distributed_type: DEEPSPEED
num_machines: 2
num_processes: 2
machine_rank: 0          # 1 on node2
main_process_ip: 10.0.9.59
main_process_port: 29500
same_network: true
rdzv_backend: static
deepspeed_config:
  deepspeed_config_file: deepspeed_configs/zero2.json
  zero3_init_flag: false
  deepspeed_multinode_launcher: standard

Note: training doesn't die when the web console terminal disconnects

Partway through training (at 69%), the web terminal session dropped with a "connection closed" message. When we refreshed the page and reconnected, training was still running inside the tmux session. The training process runs independently of the browser's WebSocket connection, so there's no need to panic if the console disconnects; just refresh.

Final results: completing training and verifying the fine-tune

Running it and the completion log

After applying all three fixes, we exported the NCCL environment variables again on both nodes and ran training.

export NCCL_SOCKET_IFNAME=eth0 NCCL_IB_DISABLE=1 NCCL_P2P_DISABLE=1

accelerate launch --config_file accelerate_config.yaml \
  -m axolotl.cli.train examples/llama-3/lora-1b.yml --num_epochs 1

[axolotl.utils.samplers.multipack] gather_len_batches: [232, 232]   # confirms the two ranks are in sync
[axolotl.utils.data.sft] Maximum number of steps set at 519
trainable params: 11,272,192 || all params: 1,247,086,592 || trainable%: 0.9039

100%|██████████| 519/519 [28:15<00:00, 3.27s/it]
{'train_loss': '1.207', 'eval_loss': '1.18', 'eval_ppl': '3.255'}
Training completed! Model successfully saved to ./outputs/lora-out

Llama-3.2-1B + LoRA completed 519 steps on two RTX 5090 nodes in about 28 minutes. Memory use per GPU was around 8.49 GB, leaving plenty of headroom out of 32 GB to scale up to bigger batches or bigger models.

Checking with a before/after comparison

Numbers alone (loss 1.4 → 1.2) don't give you a feel for it, so we compared the base model's and the LoRA model's answers to the same question directly.

Question: "Explain what a black hole is in simple terms."

Before fine-tuning (base model)

A black hole is a region of space where gravity is so strong that nothing, not even light, can escape. The gravity of a black hole is so strong that even light cannot escape. The gravity of a black hole is so strong that even light cannot escape. (the same sentence repeats endlessly)

After fine-tuning (with LoRA)

A black hole is a region of space where gravity is so strong that nothing, not even light, can escape. It is a region of space where the escape velocity from the surface of the object is greater than the speed of light. This means that nothing, not even light, can escape from the black hole. The name "black hole" comes from the fact that the object is invisible to the naked eye, and the only way to see it is through the effects of gravity.

You can see that the degenerate repetition typical of small 1B-parameter models improved noticeably with just 519 steps of fine-tuning. The answer now flows naturally into the concept of escape velocity and where the name comes from. That's the real change behind the drop in loss.

Wrapping up: a multi-node troubleshooting checklist

1. Container image: for Blackwell (RTX 50 series) GPUs, use the cloud-uv family and check for the latest cuXXX tag
2. Order for checking the network: raw TCP socket → pure NCCL all_reduce → full axolotl run (debugging in reverse widens the problem scope and wastes time)
3. deepspeed_multinode_launcher: standard is a must, assuming a container environment with no SSH daemon
4. When using deepspeed_config_file, remove accelerate's top-level keys (mixed_precision and so on)
5. The zero2 config is zero2.json (there's no bf16 version; set it separately in the training config)
6. Training in a tmux session is safe even if the web console disconnects; refresh and keep checking
7. If a hang occurs, use isolation tests to first tell whether the cause is a reproducible config problem or a transient network issue

All logs and results in this post were verified live on gcube RTX 5090 replica containers.