SGLang

Try it in 3 steps

1. 1

   Create an isolated Python environment

   On Linux with Python 3.10 or newer, CUDA 13, and an NVIDIA sm80-or-newer GPU, activate a disposable virtual environment. Continue in the same terminal and working directory.

   python3 -m venv sglang-demo-env && . sglang-demo-env/bin/activateCopy
2. 2

   Install SGLang 0.5.21

   Install the reviewed release. The next step downloads a small public Qwen model, so it needs network access and sufficient free storage.

   python -m pip install "sglang==0.5.21"Copy
3. 3

   Send a request to local inference

   Bind the public model to 127.0.0.1 and poll readiness up to 60 times at one-second intervals, with one-second connection and request limits. If it is not ready in roughly 120 seconds, print the log, fail, and do not call the chat API. After readiness, cap the chat request at 30 seconds and require non-empty content. The subshell stops the server on success or failure without leaving its trap in the caller. This does not validate production authentication.

   ( sglang serve --model-path qwen/qwen2.5-0.5b-instruct --host 127.0.0.1 --port 30000 > sglang.log 2>&1 & server_pid=$!; trap 'kill "$server_pid" 2>/dev/null || true; wait "$server_pid" 2>/dev/null || true' EXIT; ready=0; for attempt in 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60; do if curl -fsS --connect-timeout 1 --max-time 1 http://127.0.0.1:30000/health >/dev/null 2>&1; then ready=1; break; fi; kill -0 "$server_pid" 2>/dev/null || { cat sglang.log; exit 1; }; sleep 1; done; if test "$ready" != 1; then cat sglang.log; exit 1; fi; curl -fsS --connect-timeout 2 --max-time 30 http://127.0.0.1:30000/v1/chat/completions -H 'Content-Type: application/json' -d '{"model":"qwen/qwen2.5-0.5b-instruct","messages":[{"role":"user","content":"Reply with OK"}],"temperature":0,"max_tokens":8}' > response.json && python -c 'import json; data=json.load(open("response.json")); assert data["choices"][0]["message"]["content"].strip()' )Copy

Built with

• Python

Categories and tags

GitHub data

Share a guide or use case for this OSS in Markdown. Articles are published after administrator approval.

Explore next

• Khoj37,559 Stars

  2 shared tag(s) · 1 shared category(s) · Same language

  Combine documents, the web, local/cloud LLMs, agents, and automations in a self-hosted personal AI

  Python
• Onyx32,323 Stars

  2 shared tag(s) · 1 shared category(s) · Same language

  turn internal knowledge into a context layer for AI agents

  Python
• smolagents29,670 Stars

  2 shared tag(s) · 1 shared category(s) · Same language

  Build code-writing AI agents with a compact Python library and pluggable models, tools, and execution backends

  Python
• Transformers166,931 Stars

  3 shared tag(s) · 1 shared category(s) · Same language

  Load, run, and train text, vision, audio, and multimodal models through shared AutoClass and Pipeline APIs

  Python
• DeepSeek-V3104,507 Stars

  3 shared tag(s) · 1 shared category(s) · Same language

  publish a 671B-parameter MoE LLM and reference implementation for FP8 and multi-node inference

  Python
• TRL19,450 Stars

  3 shared tag(s) · 1 shared category(s) · Same language

  Post-train foundation models with shared Python trainers and a CLI for SFT, preference optimization, and reward learning

  Python