Claus Ibsen created CAMEL-26424:
-----------------------------------
Summary: camel-jbang - camel ask: turn thinking off with local
OpenAI-compatible servers, as with Ollama
Key: CAMEL-26424
URL: https://issues.apache.org/jira/browse/CAMEL-26424
Project: Camel
Issue Type: Improvement
Components: camel-jbang
Reporter: Claus Ibsen
Fix For: 4.24.0
camel ask turns thinking off only on Ollama's own API:
LlmClient.chatOllamaFormat sends {{"think": false}}. With
{{--api-type=openai}}, chatOpenAiFormat sends model, messages, temperature and
tools, and nothing about thinking.
The Qwen 3.5, 3.6 and 3.8 models think by default, because their chat template
turns thinking on. Local OpenAI-compatible servers such as Ollama's /v1
endpoint, llama.cpp server and vLLM render that template. So with these
servers, every answer from camel ask starts with a hidden reasoning phase.
Answers get slow, and a tool loop can spend minutes thinking. In the
local-model benchmark, steps with thinking on ran 4-8 minutes and gave no
answer (CAMEL-24886).
*Measured* on a Mac mini M4 Pro with Ollama 0.40.12, its /v1/chat/completions
endpoint, qwen3.6:35b-mlx already loaded, one short question, temperature 0.2:
||Request||Time||Completion tokens||Reasoning||
|nothing extra (what camel ask sends today)|6.5 s|504|2,114 chars|
|{{reasoning_effort}} = none|0.5 s|35|none|
|{{chat_template_kwargs.enable_thinking}} = false|5.6 s|442|1,789 chars (Ollama
ignores it)|
*Proposal*
When isLocalEndpoint() is true, the OpenAI request turns thinking off, as the
Ollama request does:
{noformat}
"reasoning_effort": "none",
"chat_template_kwargs": {"enable_thinking": false}
{noformat}
* reasoning_effort none: Ollama's /v1 honours it (see above). Rapid-MLX
documents the same.
* chat_template_kwargs: this is how vLLM and SGLang turn thinking off (Qwen's
own docs).
Requests to hosted endpoints (OpenAI, Azure, watsonx) stay as they are. There,
the extra fields can be rejected as unknown parameters. No new option: the
Ollama path has none either.
Check that Ollama /v1 and at least one other local server (llama.cpp server or
LM Studio) accept the extra fields.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
