Nexlab

CoderAI - Multi-Backend GPU AI Server

2026-03-14 Technology

CoderAI is an OpenAI-compatible API server that supports multiple GPU backends: NVIDIA (CUDA), AMD (Vulkan), and Intel (Vulkan). It uses HuggingFace Transformers for NVIDIA GPUs and llama-cpp-python with Vulkan for AMD/Intel GPUs, making AI inference accessible regardless of your hardware.

Why CoderAI?

Running large language models locally can be challenging, especially if you don't have an NVIDIA GPU. CoderAI solves this by supporting multiple GPU vendors through different backends:

  • NVIDIA (CUDA): Via PyTorch + Transformers - Best performance on NVIDIA GPUs
  • AMD GPUs: Via llama-cpp-python + Vulkan - Works great on AMD RX 400+ series
  • Intel GPUs: Via llama-cpp-python + Vulkan - Supports Intel integrated GPUs (HD 600+) and Arc
  • CPU: Fallback for systems without GPUs

Key Features

Multi-Backend Support

One of the most innovative features of CoderAI is its ability to automatically detect and use the best available backend. Whether you have an RTX 4090, an AMD RX 7800 XT, or even just an Intel integrated GPU, CoderAI can leverage it for AI inference.

OpenAI-Compatible API

CoderAI provides endpoints that are fully compatible with the OpenAI API:

  • /v1/chat/completions - Chat completions endpoint
  • /v1/completions - Text completions
  • /v1/models - List available models

This means you can use CoderAI as a drop-in replacement for OpenAI in your existing applications!

Memory-Aware Model Loading

CoderAI automatically determines the optimal loading strategy based on your available VRAM and RAM. It can intelligently offload model layers from VRAM to RAM to disk when needed, allowing you to run larger models than your GPU would normally support.

Multi-GPU Support

For systems with multiple NVIDIA GPUs, CoderAI can automatically distribute model inference across all available devices, providing near-linear speedups.

Quantization Support

Reduce model memory requirements with 4-bit and 8-bit quantization:

  • NVIDIA: Via bitsandbytes
  • AMD/Intel: Built-in GGUF quantization support

Advanced Features

  • Flash Attention 2 for faster inference on supported NVIDIA GPUs
  • Streaming responses via Server-Sent Events (SSE)
  • Tool calling support for function use
  • Automatic GPU backend detection

Installation

The easiest way to install CoderAI is using the provided build script:

# Clone the repository
git clone git@git.nexlab.net:nexlab/coderai.git
cd coderai

# For NVIDIA GPUs (default)
./build.sh nvidia

# For AMD or Intel GPUs with Vulkan support
./build.sh vulkan

The build script will automatically create a virtual environment and install the appropriate dependencies for your GPU.

Use Cases

CoderAI is perfect for:

  • Running local LLM applications without relying on OpenAI
  • Development and testing of AI applications
  • Privacy-sensitive applications that can't send data to external APIs
  • Cost-effective AI inference with consumer hardware
  • Research and experimentation with different model architectures

Conclusion

CoderAI represents a significant step forward in making local AI inference accessible to everyone. By supporting multiple GPU vendors, it removes the NVIDIA monopoly requirement for running LLMs locally. Whether you're a developer, researcher, or AI enthusiast, CoderAI provides a flexible and powerful solution for local AI inference.

Check out the project on GitLab: https://git.nexlab.net/nexlab/coderai