Skip to content

feat(server): add --default-thinking-mode flag for server-wide thinking default - #476

Open
rafaeldrincon wants to merge 1 commit into
FlashML-org:mainfrom
rafaeldrincon:feat/default-thinking-mode
Open

rafaeldrincon wants to merge 1 commit into
FlashML-org:mainfrom
rafaeldrincon:feat/default-thinking-mode

Conversation

@rafaeldrincon

Copy link
Copy Markdown

Problem

OpenAI-compatible clients that never send chat_template_kwargs (Vercel AI SDK openai-compatible provider, llama-swap, LiteLLM proxies) get the model's template default, which for reasoning models like Qwen3.6 means thinking is always on. The token budget goes to reasoning_content and content comes back empty.

Measured on real hardware: a 200-token max_tokens budget produced 199 reasoning tokens and 0 content tokens; the client saw an empty answer.

Current workaround (--reasoning-parser off) just dumps thinking into content — still wasting the budget.

Solution

--default-thinking-mode {auto,chat,thinking} server flag:

  • auto (default): current behavior, the template decides
  • chat: fills enable_thinking=False into every request that does not set any explicit thinking control
  • thinking: fills enable_thinking=True the same way

An explicit per-request value (enable_thinking / thinking / thinking_mode in chat_template_kwargs, or the DeepSeek thinking.type wire toggle) always wins over the server default — the flag only fills what the request left unset.

This mirrors llama.cpp's --jinja template behavior where the server owns the default and clients can override per-request.

Implementation

  • ServerArgs.default_thinking_mode + argparse entry (server/args.py)
  • apply_default_thinking_mode() merge helper in server/openai_api.py
  • Applied at all three frontends: OpenAI chat completions, Anthropic messages, Responses API
  • 6 unit tests in tests/server/test_default_thinking_mode.py (verified all pass)

Closes #472

Test environment

ft version: 0.1.2
model: nvidia/Qwen3.6-35B-A3B-NVFP4
gpu: RTX 3080 Ti Laptop (16GB, sm_86)
cuda: 13.1, driver: 580

With --default-thinking-mode chat the same Understory agent workload that previously returned empty content produces correct answers with no client changes.

…ng default

OpenAI-compatible clients that never send chat_template_kwargs (Vercel AI
SDK openai-compatible provider, llama-swap, LiteLLM proxies) currently get
the model's template default, which for reasoning models like Qwen3.6 means
thinking is always on — the token budget goes to reasoning_content and
content comes back empty.

--default-thinking-mode {auto,chat,thinking} lets the operator pick the
server-wide default:

- auto (default): current behavior, template decides
- chat: fills enable_thinking=False into every request that does not set
  any explicit thinking control
- thinking: fills enable_thinking=True the same way

An explicit per-request value (enable_thinking / thinking / thinking_mode
in chat_template_kwargs, or the DeepSeek thinking.type wire toggle) always
wins over the server default.

Applied to all three frontends: OpenAI chat completions, Anthropic
messages, and Responses API.

Closes FlashML-org#472
gdevenyi added a commit to gdevenyi/FreeToken that referenced this pull request Sep 18, 2026
gdevenyi added a commit to gdevenyi/FreeToken that referenced this pull request Sep 19, 2026
JUNQINGV587 added a commit to JUNQINGV587/FreeToken that referenced this pull request Sep 21, 2026
…eam FlashML-org#476)

Adopts upstream PR FlashML-org#476 by @rafaeldrincon: `--default-thinking-mode {auto,chat,thinking}`
lets the server inject a thinking state for a request that carries no thinking control
of its own, for OpenAI-compatible clients that never send one.

Default is `auto`, so nothing changes until an operator opts in; `git grep` finds the
flag in no start script or config, and the production template is untouched.

Three deliberate divergences from the upstream text:

1. The default is injected AFTER effort_toggle_kwargs, not before. That helper returns
   the template kwargs unchanged as soon as any thinking key is present, so injecting
   first makes the injected key the thing that short-circuits the client's own
   reasoning_effort / thinking.type -- measured on the 5x3 matrix, the upstream order
   turned `reasoning_effort=high` into `enable_thinking=False` and inverted
   `thinking.type=enabled`, i.e. "an explicit per-request value always wins" was false
   for those two channels. Our real traffic uses them: 56 of 60 tool-carrying agent
   requests, 44 of them xhigh.
2. The injection broadcasts both spellings through thinking_toggle_kwargs, like every
   other thinking toggle in the server, instead of a bare enable_thinking.
3. An absent or unrecognised mode is a no-op rather than a second undocumented default.

Also folds the flag into /v1/cache/status geometry.reasoning.default, so the published
metadata cannot contradict what an uncontrolled request renders, and documents the flag
in docs/cli.md.

Tests: 12 new cases in tests/server/test_effort_dialect.py and test_think_gears.py
(matrix plus the metadata default). Full offline scope passes: 1798 passed, 250 skipped.
Reverting the injection order reddens exactly the two invariant tests, so they are not
vacuous. Drop this commit once the PR merges upstream (a local divergence like the rest
of `full0913-tp2`).

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat(server): server-wide default thinking mode flag (--default-thinking-mode)

1 participant