feat(server): add --default-thinking-mode flag for server-wide thinking default - #476
Open
rafaeldrincon wants to merge 1 commit into
Open
rafaeldrincon wants to merge 1 commit into
rafaeldrincon wants to merge 1 commit into
Conversation
…ng default
OpenAI-compatible clients that never send chat_template_kwargs (Vercel AI
SDK openai-compatible provider, llama-swap, LiteLLM proxies) currently get
the model's template default, which for reasoning models like Qwen3.6 means
thinking is always on — the token budget goes to reasoning_content and
content comes back empty.
--default-thinking-mode {auto,chat,thinking} lets the operator pick the
server-wide default:
- auto (default): current behavior, template decides
- chat: fills enable_thinking=False into every request that does not set
any explicit thinking control
- thinking: fills enable_thinking=True the same way
An explicit per-request value (enable_thinking / thinking / thinking_mode
in chat_template_kwargs, or the DeepSeek thinking.type wire toggle) always
wins over the server default.
Applied to all three frontends: OpenAI chat completions, Anthropic
messages, and Responses API.
Closes FlashML-org#472
4 tasks
gdevenyi
added a commit
to gdevenyi/FreeToken
that referenced
this pull request
Sep 18, 2026
… for server-wide thinking default
gdevenyi
added a commit
to gdevenyi/FreeToken
that referenced
this pull request
Sep 19, 2026
… for server-wide thinking default
JUNQINGV587
added a commit
to JUNQINGV587/FreeToken
that referenced
this pull request
Sep 21, 2026
…eam FlashML-org#476) Adopts upstream PR FlashML-org#476 by @rafaeldrincon: `--default-thinking-mode {auto,chat,thinking}` lets the server inject a thinking state for a request that carries no thinking control of its own, for OpenAI-compatible clients that never send one. Default is `auto`, so nothing changes until an operator opts in; `git grep` finds the flag in no start script or config, and the production template is untouched. Three deliberate divergences from the upstream text: 1. The default is injected AFTER effort_toggle_kwargs, not before. That helper returns the template kwargs unchanged as soon as any thinking key is present, so injecting first makes the injected key the thing that short-circuits the client's own reasoning_effort / thinking.type -- measured on the 5x3 matrix, the upstream order turned `reasoning_effort=high` into `enable_thinking=False` and inverted `thinking.type=enabled`, i.e. "an explicit per-request value always wins" was false for those two channels. Our real traffic uses them: 56 of 60 tool-carrying agent requests, 44 of them xhigh. 2. The injection broadcasts both spellings through thinking_toggle_kwargs, like every other thinking toggle in the server, instead of a bare enable_thinking. 3. An absent or unrecognised mode is a no-op rather than a second undocumented default. Also folds the flag into /v1/cache/status geometry.reasoning.default, so the published metadata cannot contradict what an uncontrolled request renders, and documents the flag in docs/cli.md. Tests: 12 new cases in tests/server/test_effort_dialect.py and test_think_gears.py (matrix plus the metadata default). Full offline scope passes: 1798 passed, 250 skipped. Reverting the injection order reddens exactly the two invariant tests, so they are not vacuous. Drop this commit once the PR merges upstream (a local divergence like the rest of `full0913-tp2`).
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
OpenAI-compatible clients that never send
chat_template_kwargs(Vercel AI SDKopenai-compatibleprovider, llama-swap, LiteLLM proxies) get the model's template default, which for reasoning models like Qwen3.6 means thinking is always on. The token budget goes toreasoning_contentandcontentcomes back empty.Measured on real hardware: a 200-token
max_tokensbudget produced 199 reasoning tokens and 0 content tokens; the client saw an empty answer.Current workaround (
--reasoning-parser off) just dumps thinking intocontent— still wasting the budget.Solution
--default-thinking-mode {auto,chat,thinking}server flag:auto(default): current behavior, the template decideschat: fillsenable_thinking=Falseinto every request that does not set any explicit thinking controlthinking: fillsenable_thinking=Truethe same wayAn explicit per-request value (
enable_thinking/thinking/thinking_modeinchat_template_kwargs, or the DeepSeekthinking.typewire toggle) always wins over the server default — the flag only fills what the request left unset.This mirrors llama.cpp's
--jinjatemplate behavior where the server owns the default and clients can override per-request.Implementation
ServerArgs.default_thinking_mode+ argparse entry (server/args.py)apply_default_thinking_mode()merge helper inserver/openai_api.pytests/server/test_default_thinking_mode.py(verified all pass)Closes #472
Test environment
With
--default-thinking-mode chatthe same Understory agent workload that previously returned emptycontentproduces correct answers with no client changes.