Skip to content

THOX.ai Research · Paper · v2.0.0 · 2026-11-05

Preprint, not peer reviewed

thox-edge-ops v2: fused decode kernels for small models on THOX edge hardware

Six fused inference operators for CUDA, Metal and CPU, distributed through the Hugging Face Kernel Hub, with runtime ISA dispatch on x86 and a fused QK-norm and RoPE prologue for Qwen3 and Gemma 3 class models.

Tommy Xaypanya

Chief Technology Officer & Co-Founder (corresponding author)

ORCID 0009-0000-4425-9565

THOX.ai LLC · Cedar Park, Texas · November 5, 2026

DOI 10.5281/zenodo.23971227

Abstract

Small language models on edge hardware spend their decode step in launch overhead and memory traffic, not arithmetic. thox-edge-ops v2 is a set of six fused inference operators (RMSNorm with a weight offset that covers the Llama and Gemma conventions, fused residual-add RMSNorm, SiLU and tanh-GELU gated activations, rotary embedding, and a fused per-head QK-norm plus RoPE prologue) with one source tree and three backends: CUDA, Metal and CPU. The CPU backend selects AVX2, FMA and F16C row kernels at run time from a single baseline-ISA artifact, which is what lets one Kernel Hub build load on every x86_64 host. We measure the kernels against eager PyTorch on an NVIDIA RTX 4060 Ti and an x86 CPU with 30 interleaved trials per configuration. At decode shapes the GPU kernels run 6.90x faster than eager for RMSNorm and 24.43x faster for the fused QK-norm and RoPE prologue; on the CPU, RMSNorm runs 2.42x and rotary embedding 6.75x faster. We also report where fusion does not pay: the bfloat16 gated activations at prefill sizes on an AVX-512 host are slower than eager, and we say why. The release is live on the Kernel Hub as 23 ABI-checked x86_64 builds, and the full test suite passes against builds downloaded from it. All numbers in this paper are generated from the released raw measurements.

6.90x

RMSNorm decode speedup over eager PyTorch, fp16, NVIDIA RTX 4060 Ti

Measured

24.43x

fused QK-norm + RoPE prologue decode speedup over eager PyTorch, fp16, NVIDIA RTX 4060 Ti

Measured

2.42x

RMSNorm decode speedup on x86 CPU, fp32, with runtime AVX2 dispatch (rotary embedding 6.75x)

Measured

438/438

tests passing on the CUDA and CPU paths; the full suite ran against 3 of the 23 ABI-checked x86_64 Hub builds downloaded from the Hub

Measured, Hub builds

Download the paper (PDF)Read the paper (HTML) Browser demo

Contents

  1. Executive summary
  2. 1. Introduction
  3. 2. Background and related work
  4. 3. System model and requirements
  5. 4. Deployment in the THOX stack
  6. 5. Design
  7. 6. Implementation
  8. 7. Evaluation
  9. 8. Discussion and limitations
  10. 9. Conclusion
  11. 10. Roadmap
  12. Appendix A: reproducibility
  13. Appendix B: notation
  14. Appendix C: disclosure
  15. References

Where the paper lives

Cite this paper

@techreport{xaypanya2026edgeops200,
  author      = {Tommy Xaypanya},
  title       = {{thox-edge-ops} v2: Fused Decode Kernels for Small Models on {THOX} Edge Hardware},
  institution = {THOX.ai Research, THOX.ai LLC},
  address     = {Cedar Park, Texas},
  year        = {2026},
  month       = nov,
  version     = {2.0.0},
  doi         = {10.5281/zenodo.23971227},
  url         = {https://doi.org/10.5281/zenodo.23971227},
  note        = {Preprint, not peer reviewed. Revised public edition. Zenodo concept DOI (all versions): 10.5281/zenodo.23971226. PDF SHA-256: 6ac5d3d83c2a0b06ac871784ec2360f3c1902ad8fffe02432c2ee790bc462b96.}
}

Preprint, not peer reviewed; DOI 10.5281/zenodo.23971227.

Your AI. Your Data. Your Rules.™