Skip to content
Bagtyyarfull-stack engineer

Gonka: AI inference infrastructure

Ongoing

Specified, deployed, and supported a 24-node GPU cluster (192× RTX 4080) serving open-weight LLMs on the Gonka decentralized AI inference network, plus an OpenAI-compatible API gateway built on top of it.

GitHub
  • GPU Infrastructure
  • RTX 4080 Clusters
  • Open-Weight LLMs
  • Qwen
  • vLLM
  • Decentralized Inference
  • Express
  • TypeScript
  • Zod
  • SSE
  • Docker
  • Vitest

Outcome

24 servers / 192× RTX 4080 GPUs taken from hardware list to revenue-earning production, plus a public OpenAI-compatible API gateway.

Context

Gonka is a decentralized network for AI inference: hosts run GPU servers serving open-weight models and earn network tokens for verified compute. During the network's early phase, a host engaged me to take their operation from zero to production: hardware selection, procurement guidance (servers sourced from China), deployment, and ongoing operations, with a 3-month support guarantee. Client identity and commercial figures are confidential.

Scope Delivered

  • Hardware specification: advised on server selection for a 24-node cluster, 8× RTX 4080 per node, 192 GPUs in total.
  • Model serving: deployed open-weight LLM inference at scale, including Qwen 235B-class instruction models.
  • Operations: node setup, network onboarding, monitoring, and a 3-month support engagement covering incident response and tuning.

GonkaProvider: my open-source API gateway

On top of the infrastructure work, I built and open-sourced GonkaProvider: an OpenAI-compatible Express/TypeScript gateway that proxies chat completions to Gonka ML nodes via the signed gonka-openai client.

  • Strict TypeScript + Zod validation across both external boundaries: client requests and upstream SSE chunks.
  • OpenAI Responses API translation layer on top of chat completions, so both API families work against Gonka executors.
  • Docker and docker-compose packaging for one-command deployment.
  • Includes an upstream fix for a chunk-validation bug found while integrating: Gonka emits mid-stream errors as a bare string ({"error":"terminated"}) instead of an OpenAI-shaped object, which previously broke downstream clients' schema validation.

GonkaProvider on GitHub

Milestones

  1. Cluster specification and provisioning

    Done

    Hardware specification and procurement guidance for a 24-node cluster, 8× RTX 4080 per node (192 GPUs), followed by deployment and network onboarding on Gonka during the network's early phase.

    Completed Mar 2026

  2. First inference serving on the network

    Done

    Open-weight LLM inference (Qwen 235B-class instruction models) serving on the Gonka network from the provisioned cluster, with monitoring and operations in place.

    Completed Apr 2026

  3. GonkaProvider gateway prototype

    Done

    First public commit of GonkaProvider: a Node/Express gateway exposing an OpenAI-compatible API that proxies chat completions to Gonka ML nodes via the signed gonka-openai client.

    Completed May 2026

  4. Documented public release with Docker packaging

    Done

    Gateway layout under src/, Dockerfile and docker-compose, domain context documentation, three ADRs.

    Completed May 2026

  5. TypeScript + Zod hardening and Responses API support

    Done

    Full conversion to strict TypeScript with Zod validation at both boundaries, an upstream chunk-validation fix, 89 unit tests, and an OpenAI Responses API translation layer with vLLM/Kimi K2 reference documentation.

    Completed May 2026

  6. Gateway maintenance and upstream compatibility

    In progress

    Ongoing maintenance of the public GonkaProvider repository: tracking gonka-openai releases, vLLM executor behavior, and OpenAI API changes.

Decision records

OpenAI-compatible gateway over Gonka

One Express service exposes an OpenAI-style API and is the only client that talks to the Gonka network, so API keys and request signing live in one place.

Always stream upstream for chat completions

The gateway always streams from Gonka, then either forwards the stream or assembles one response for clients that don't stream.

Multimodal payload normalization at the gateway

The gateway reorders and inlines images and video so clients can send any common format and avoid upstream 400 errors.

Tests and deployment

Unit and integration test suites

Testing

89 unit tests across the gateway modules plus integration smoke suites for the HTTP API, SSE streaming, tool calls, and vision payloads.

View

GonkaProvider on GitHub

Repository

Full gateway source, tests, and documentation public on GitHub under bagtyyarkovusov/GonkaProvider.

View

Docker packaging with compose override for dev

Docker

Root Dockerfile and docker-compose.yml for production-like runs; docker-compose.override.yml bind-mounts src/ for local development.

View
Gonka: AI inference infrastructure

OpenAI Responses API translation layer and vLLM reference docs

Added Responses API support (schemas, SSE, compaction, translation to chat completions) so both OpenAI API families work against Gonka executors, plus a vLLM Kimi K2 reference documenting how tool-call and reasoning parsing shapes what the gateway receives upstream.

Gonka: AI inference infrastructure

TypeScript + Zod conversion with upstream chunk-validation fix

All source modules converted to strict TypeScript with Zod validation at both external boundaries. validateUpstreamChunk now catches non-OpenAI-shaped SSE chunks at the seam. This fixes the bug where Gonka emits {"error":"terminated"} as a bare string, previously forwarded verbatim and breaking downstream clients' schema validation. 89 unit tests passing across 17 suites.

Gonka: AI inference infrastructure

GonkaProvider first commit: an OpenAI-compatible proxy

First public commit of the gateway: an Express service proxying chat completions to Gonka ML nodes through the signed gonka-openai client, with an example consumer script.

Gonka: AI inference infrastructure

Gateway layout, Docker packaging, and multimodal normalization

Gateway implementation moved under src/ with a root shim, Dockerfile and compose added, and three ADRs written. Media normalization hardened for empty assistant/tool multimodal payloads from MCP/OpenCode tool rounds, eliminating a class of upstream 'content must not be empty' rejections.