A 2.78-trillion-parameter model on a 121 GiB workstation

Experimental study. This is a full account of deploying Kimi-K3 —a 2.78-trillion-parameter model— on an NVIDIA DGX Spark with 121 GiB of unified memory, running a 539 GiB checkpoint: 4.5 times the physical memory available. Every figure comes from direct measurement on the machine; none is estimated or taken from third-party literature.

Joaris Angulo · R&D Lab, KIMI3 Project · 16 September 2026

Abstract

This paper documents the full deployment of Kimi-K3 —a Mixture-of-Experts model with 2,779,483,135,584 parameters— on an NVIDIA DGX Spark with 121 GiB of unified memory. The checkpoint used, quantised to IQ1_S (1.5625 bits per weight), takes up 539 GiB: 4.5 times the physical memory available. The study applied the scientific method to two rival hypotheses about the inference engine, and recorded two reproducible failures together with their root causes: an attempt to allocate 545 GB of pinned memory, and a complete system lockup caused by exhaustion of the NVIDIA driver’s memory allocator. The final system reaches 0.37 tokens/s in generation, with correct output and a clean stop. We conclude that feasibility on this class of hardware is decided not by compute capacity but by memory management policy, and that the oversubscription factor imposes a performance ceiling that no parameter tuning managed to lift.

Keywords

Mixture of Experts; unified-memory inference; extreme quantisation; mmap; cgroups v2; NVIDIA GB10; llama.cpp; memory oversubscription.

1. Introduction and problem statement

1.1 Where the research started

The starting point was an analysis of Fareed Khan’s kimi-k3-in-c repository, an inference engine written in pure C99 that claims to run Kimi-K3 on machines with as little as 8 GB of RAM by streaming directly from an NVMe SSD. The earlier research document reported a 176 KB binary, a 1.71 TB checkpoint and a latency of 32 to 33 seconds per token.

That approach moves the bottleneck from memory to secondary storage, and rests on an architectural property of the model: of its 896 experts, only 16 activate per token. The engine uses the router to work out which ones are needed and reads only those weights from disk.

1.2 Research question

Is it possible to run Kimi-K3 usefully on a DGX Spark, and what actually limits performance: compute, storage bandwidth, or memory management?

1.3 Hypotheses

H1. The DGX Spark, with 121 GiB of unified memory against the 8 GB of the reference scenario, should substantially improve on the 32 s/token latency reported in the source literature.

H2. A general-purpose engine with GPU acceleration (llama.cpp on GB10) will beat the single-threaded, CPU-oriented C99 engine, since it has a GPU able to absorb the dense layers.

H3. The limiting factor will be NVMe read bandwidth, in line with the original project’s characterisation of the problem as a bandwidth wall.

As section 5 shows, H1 and H2 were confirmed; H3 turned out to be false, and refuting it is the study’s main finding.

2. Materials and methods

2.1 Experimental platform

All specifications were obtained by direct inspection of the machine.

Component	Specification	How it was verified
Machine	NVIDIA DGX Spark (spark-6ac3)	uname -a
Architecture	aarch64 (ARM64)	uname -m
Operating system	Ubuntu 24.04.3 LTS	/etc/os-release
Kernel	6.11.0-1016-nvidia	uname -r
CPU	20 cores	nproc
GPU	NVIDIA GB10, driver 580.95.05	nvidia-smi
Memory	127,606,708 kB (121 GiB) unified CPU/GPU	/proc/meminfo
Storage	Samsung MZALC4T0HBL1-00B07 NVMe, 3.7 TB	lsblk
Compiler	GCC 13.3.0	gcc --version

Table 1. Experimental platform. Memory is unified: the GPU has no dedicated VRAM and allocates from the same LPDDR5X bank the host uses. That detail turns out to be decisive in section 4.2.

2.2 Choosing the checkpoint

Four repositories were assessed through the Hugging Face API, measuring real size by summing file bytes and checking access restrictions.

Repository	Format	Size	Access	Decision
Ryanchen911/Kimi-K3-Uncensored-GGUF	GGUF IQ1_S-XS	539 GiB	Open	SELECTED
Uniboshi/Kimi-K3-Abliterated-V1	safetensors	615 GiB	Gated (automatic)	Rejected: needs credentials
audnai/penclaw-Kimi-K3.0-abliterated	GGUF UD-Q2_K_XL	—	Gated (manual)	Rejected: human approval
moonshotai/Kimi-K3	safetensors	1.42 TiB	Open	Rejected: will not fit on disk

Table 2. Selection matrix. The deciding criterion was immediate availability combined with verified engine compatibility.

Compatibility was verified in the source code before committing to the download: llama.cpp declares the architecture identifier in src/llama-arch.cpp:151.

{ LLM_ARCH_KIMI_K3,          "kimi-k3"          },

2.3 Experimental design

The study was organised into five sequential phases, each with an explicit acceptance criterion:

• Phase I — Validate the C99 engine without weights, against a reference oracle.
• Phase II — Acquire the checkpoint and verify it byte by byte.
• Phase III — Bring up the inference service.
• Phase IV — Characterise failures and analyse root causes.
• Phase V — Sweep the memory budget and measure performance.

3. Results: validation and acquisition

3.1 Phase I — Validating the C99 engine

The kimi-k3-in-c engine compiled without errors on aarch64, producing a binary of 211,936 bytes with -O3 -std=gnu99 -mcpu=native -fopenmp. The test suite, which needs no model weights, was run against a reference oracle in Python.

GATE 1  teacher forcing : 32/32 positions match tf_pred
        generated span  : 20/20  <- must be exact
GATE 1b state reuse      : PASS  <- all logits bit-identical
GATE 2  greedy decode   : 20/20 generated tokens match full_ids
GATE 3  incremental    : 20/20 generated tokens match full_ids

VERDICT: ENGINE MATCHES THE REFERENCE EXACTLY

The engine is therefore numerically correct on this platform. Inspecting its safetensors reader, however, revealed a decisive obstacle.

The file src/io/k3_st.c accepts only the types U8, BF16, F16 and F32. It does not read GGUF. Since the only checkpoint available without credentials was in GGUF, this engine became unusable for the deployment, despite having passed every validation. It was kept as a correctness control.

Note on method: the Phase I validation was not wasted work. It established that the hardware and the compiler produce correct arithmetic, which let us rule out the platform as a source of error in later phases.

3.2 Phase II — Acquiring the checkpoint

Metric	Measured value
Shards downloaded	34 of 34
Final verified size	579,511,906,185 bytes
Total duration	2 h 45 min 04 s
Average effective throughput	≈ 56 MiB/s
Instantaneous range observed	11 – 134 MiB/s
Transport errors	2 (IncompleteMessage), retried with no loss
Exit code	rc=0

Table 3. Checkpoint acquisition. The wide variation in instantaneous throughput comes from throttling at the remote provider, not from the local link.

4. Results: characterising the failures

Two reproducible failures were recorded. Both share a common root —the unified nature of memory on GB10— but they surface through different mechanisms.

4.1 Failure I — Pinned-memory allocation

The first attempt to start aborted immediately:

E ggml_aligned_malloc: insufficient memory (attempted to allocate 520563.75 MB)
E ggml_backend_cpu_buffer_type_alloc_buffer: failed to allocate buffer of size 545850654720
E alloc_tensor_range: failed to allocate CUDA_Host buffer of size 545850654720
E llama_model_load: error loading model: unable to allocate CUDA_Host buffer

Root cause

The --cpu-moe option places the expert weights in a CUDA_Host buffer (pinned memory). That buffer type cannot be backed by mmap: it demands physical residency. The engine therefore asked for 545 GB of real RAM against the 121 GiB available.

Fix

The --no-host flag ("bypass host buffer allowing extra buffers to be used") diverts the weights to a conventional CPU buffer, which the mapping can back. Combined with --load-mode mmap, the model loaded correctly.

4.2 Failure II — Complete system lockup

During a long generation run the machine stopped responding altogether and needed a cold reboot. The kernel log from the previous boot identified the origin:

NVRM: nvAssertOkFailedNoLog: Assertion failed: Out of memory [NV_ERR_NO_MEMORY]
      (0x00000051) returned from status @ kernel_graphics_context.c:1178
NVRM: nvCheckOkFailedNoLog: Check failed: Out of memory [NV_ERR_NO_MEMORY]
      returned from kgrctxAllocMainCtxBuffer(pGpu, pKernelGraphicsContext,
      pKernelGraphics, pKernelChannel) @ kernel_graphics_context.c:1387

Root cause

The failure was not disk saturation or a scheduler problem, as hypothesis H3 would suggest. On GB10 the GPU has no dedicated VRAM and allocates from the same 121 GiB bank as the host. Three consumers coincided:

• The resident memory of the inference process.
• The page cache generated by mapping 539 GiB, which grows until it fills all free memory.
• The ollama service, which was holding 36,272 MiB of GPU memory at the same time.

The NVIDIA driver could not reserve the graphics context buffers —allocations that cannot be reclaimed— and the system became unusable.

Corrective measures applied

Measure	Implementation	Effect
Memory isolation	MemoryMax in cgroup v2	Confines reclaim to the service instead of hitting the whole system
Removing the competitor	systemctl disable ollama	Frees 36 GiB of unified memory
Pre-flight check	Guard in the start-up script	Refuses to start if the GPU is busy
No swapping	MemorySwapMax=0	Avoids further degradation from paging
Out-of-memory policy	OOMPolicy=stop, Restart=no	Stops a failure turning into a loop

Table 4. Corrective measures. The start-up guard was validated experimentally: it refused a start with 36,442 MiB of GPU memory in use.

5. Results: performance

5.1 Memory budget sweep

Performance was measured under three different memory budgets, holding the rest of the configuration constant except where noted.

Budget	Oversubscription	Load	Prefill (tok/s)	Generation (tok/s)
68 GiB	7.9 ×	11 min 33 s	0.286	0.322
60 GiB	9.0 ×	did not converge in 15 min	0.18	0.23
92 GiB	5.9 ×	15 min 21 s	0.30	0.37

Table 5. Performance against memory budget. Methodological caveat: the 60 GiB row also changes the batch size and the reasoning budget, so its comparison is not strictly controlled. The 68 and 92 GiB rows are comparable with each other.

5.2 Evidence of memory pressure

With the 60 GiB budget, the cgroup’s memory.pressure interface recorded:

some avg10=53.00 avg60=47.61 avg300=21.55
full avg10=53.00 avg60=47.61 avg300=21.55

The value full avg10 = 53.00 means that for 53 % of the time every process in the group was stalled waiting on memory reclaim. More than half the wall-clock time went on page management rather than useful compute. Memory consumption also stayed pinned exactly at the configured limit, confirming sustained saturation.

5.3 Final functional validation

The final configuration was put through a complete end-to-end inference:

CONTENT: '¡Hola! ¿En qué puedo ayudarte hoy? 😊'
FINISH: stop
gen 0.37 tok/s | prefill 0.30 tok/s | n=152

The finish_reason: stop indicator confirms the model finished of its own accord rather than running out of token budget. The output is semantically correct.

5.4 Secondary finding: the reasoning model

Kimi-K3 emits its chain of thought in a separate field, reasoning_content, before producing the answer. With low token budgets the content field comes back empty:

"content": ""
"reasoning_content": "The user says "Di hola" which is Spanish for "Say hello."..."
"finish_reason": "length"

This behaviour is easily mistaken for a service failure. The mitigation is to set max_tokens above 600, or to cap reasoning explicitly with --reasoning-budget.

6. Discussion

6.1 Testing the hypotheses

Hypothesis	Result	Evidence
H1 — Improvement on 32 s/token	CONFIRMED	2.7 s/token measured, against the 32–33 s/token reference: roughly a 12 × improvement
H2 — llama.cpp beats the C99 engine	CONFIRMED	Confirmed, though by default: the C99 engine does not accept GGUF and could not be compared directly
H3 — The limit is NVMe bandwidth	REFUTED	The limit is memory policy. Both failures originated in memory allocation; neither in disk saturation

Table 6. Hypothesis testing.

6.2 What the main finding means

Refuting H3 is the most valuable result. The original project frames the problem as a bandwidth wall, and that description holds for an architecture with separate system memory and VRAM. On a unified-memory architecture such as GB10 the constraint changes in kind: the page cache and the graphics driver compete for the same physical resource, and it is that competition —not disk speed— that determines both stability and performance.

The practical consequence is that the design variable that matters on this class of hardware is not storage speed but memory isolation between consumers. A system with an NVMe twice as fast would have suffered exactly the same lockup.

6.3 The oversubscription ceiling

Performance improved consistently as the memory budget grew —by 61 % in generation going from 60 to 92 GiB— but stayed on the order of 0.3 tokens/s in every configuration. The 539 GiB working set exceeds any budget reachable on this platform by a factor of at least 4.5. Every token forces page faults that trigger reclaim, and no parameter tuning removes that structural cost.

6.4 Limitations of the study

• Each performance point comes from a single run. No means or deviations were computed, so the figures indicate order of magnitude and direction, not statistically precise values.
• The 60 GiB condition varies three parameters at once, which makes it impossible to attribute its degradation to the memory budget alone.
• Answer quality was not assessed. Quantisation to 1.5625 bpw is extreme, and its impact on the model’s reasoning is outside the scope of this work.
• The C99 engine could not be compared directly because of the format incompatibility, so H2 is confirmed by elimination rather than by head-to-head measurement.

7. Conclusions

1. Running a 2.78-trillion-parameter model on a 121 GiB workstation is feasible. The system is operational, serves an OpenAI-compatible API and produces correct output with a clean stop.

2. It is not feasible for interactive use. At 0.37 tokens/s, a 150-token answer takes close to seven minutes. The system suits batch processing and experimentation, not conversation.

3. On unified memory the limiting factor is memory management policy, not storage bandwidth. This is the central finding, and it contradicts the earlier characterisation of the problem.

4. Isolation through cgroups is an operational safety requirement, not an optimisation. Without it, an inference process can render the whole workstation unusable by starving the graphics driver of memory.

5. Oversubscription imposes a ceiling you cannot break through. Raising the memory budget improves performance monotonically, but while the working set is 4.5 times physical memory the order of magnitude does not change.

7.1 Sizing a farm for 50 users

The measured performance is extrapolated to a business operation of 50 users. The calculation starts solely from the experimental figure of 0.37 tokens/s per machine and slot, and states every assumption.

Model assumptions

• Unit throughput: 0.37 tokens/s per DGX Spark, with a single inference slot (measured value, section 5.3).
• An effective 8-hour working day. A company’s load concentrates in office hours; it does not spread over 24.
• Target utilisation of 70 %. At 100 % the queue length diverges; the 30 % of slack absorbs variability in arrivals.
• Each request occupies one machine exclusively, since –parallel 1 is the stable configuration we verified.
• Answer length includes the chain of thought, which on this model consumes most of the token budget.

Sizing results

Scenario	Requests per user/day	Tokens per answer	Latency per answer	Machines needed
Light	5	300	13.5 min	11
Moderate	20	500	22.5 min	68
Heavy	50	800	36.0 min	269

Table 8. Machines needed for 50 users by usage intensity, at 70 % utilisation over an 8-hour day. The moderate scenario is the reference case for professional use.

Infrastructure implications

Scenario	Machines	Electrical power	Approximate investment	Total storage
Light	11	2.6 kW	USD 44,000	5.8 TiB
Moderate	68	16.3 kW	USD 272,000	35.8 TiB
Heavy	269	64.6 kW	USD 1,076,000	141.6 TiB

Table 9. Derived power draw, investment and storage. Power estimated at 240 W per machine and investment at a reference list price of USD 4,000; both figures need confirming with the vendor. Each machine needs its own copy of the 539 GiB checkpoint.

Objections to the farm approach

Latency does not improve by adding machines. A farm scales aggregate throughput, not the time of an individual answer. Even with 269 machines, a user still waits 36 minutes for their answer. No business operation tolerates that latency on interactive tasks.

The bottleneck is replicated, not solved. Every machine in the farm carries the same 4.5 × oversubscription documented in section 6.3. You would be multiplying an inefficiency instead of correcting it.

Pooling memory would be the right route, but it is not available. Five machines would be enough for the 539 GiB to sit entirely in pooled memory, removing the root cause. The DGX Spark interconnect, however, officially supports two-node clusters, which is not enough for that. Confirming this limitation with the manufacturer is a precondition for any purchase decision.

Recommendation

A DGX Spark farm is not recommended for serving Kimi-K3 in IQ1_S to 50 users. The moderate scenario demands 68 machines and USD 272,000 to deliver 22-minute answers, a worse result than cheaper alternatives.

Three alternative routes, in order of cost-effectiveness:

• Use a model that fits in one machine’s memory. With oversubscription gone, a single Spark could serve several concurrent users, replacing dozens of machines.
• Concentrate compute on data-centre GPUs with enough memory for full residency, instead of spreading it across nodes that each have too little.
• Keep this configuration for batch workloads with no latency requirement —overnight analysis, document processing— where 22 minutes per answer is acceptable and one or two machines suffice.

7.2 Future work

• Evaluate kimi-k3-in-c with a safetensors checkpoint. Its memory budget is an explicit parameter and it uses O_DIRECT, avoiding by design the page-cache pressure that caused Failure II.
• Repeat the sweep with three or more runs per point to establish confidence intervals.
• Compare answer quality against less aggressive quantisations, to determine whether IQ1_S preserves the base model’s reasoning ability.
• Characterise the expert access pattern and assess whether a router-guided prefetch policy reduces page faults.

8. Appendix: final configuration

Service start-up parameters in production:

llama-server 
    --model Kimi-K3-abliterated-IQ1_S-XS-00001-of-00034.gguf 
    --alias kimi-k3-unc 
    --host 0.0.0.0 --port 7777 
    --ctx-size 4096 
    --parallel 1 
    -b 2048 -ub 2048 
    --cache-reuse 256 
    --reasoning-budget 2048 
    --n-gpu-layers 999 
    --cpu-moe 
    --no-host 
    --load-mode mmap 
    --threads 20 
    --no-warmup

8.1 Critical parameters

Parameter	Experimental justification
--no-host	Mandatory. Without it, start-up aborts trying to allocate 545 GB pinned (Failure I)
--load-mode mmap	The only viable mode. DirectIO would require the model to be fully resident
--parallel 1	Reduced from 4. Each slot keeps its own KV cache
MemoryMax=92G	cgroup isolation. Leaves ~29 GiB to the NVIDIA driver and the system
--reasoning-budget 2048	Caps the chain of thought so that content does not come back empty

Table 7. Critical parameters and the evidence behind them.

8.2 Operational finding: PowerShell clients

In Windows PowerShell the identifier curl is an alias for Invoke-WebRequest, which changes how the JSON body is escaped. Sending it inline was experimentally shown to produce:

{"error":{"code":500,"message":"[json.exception.parse_error.101] parse error
 at line 1, column 2: syntax error while parsing object key - invalid literal;
 last read: '{c'","type":"server_error"}}

The validated solution is to invoke curl.exe explicitly and pass the body from a file with -d "@file.json". The stop-parsing operator --% was tested and does not fix the problem.

References

[1] Khan, F. kimi-k3-in-c: a C99 inference engine for Kimi-K3. GitHub repository, FareedKhan-dev/kimi-k3-in-c.

[2] Moonshot AI. Kimi-K3. Hugging Face repository, moonshotai/Kimi-K3.

[3] Ryanchen911. Kimi-K3-Uncensored-GGUF. Hugging Face repository.

[4] Gerganov, G. et al. llama.cpp, build b460-6ca4915, 27 August 2026.

[5] Linux kernel documentation. Control Group v2: memory pressure interface (PSI).