[
https://issues.apache.org/jira/browse/CAMEL-26424?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Claus Ibsen updated CAMEL-26424:
--------------------------------
Summary: camel-jbang - turn thinking off with local models in camel ask,
explain and overview (was: camel-jbang - camel ask: turn thinking off with
local OpenAI-compatible servers, as with Ollama)
> camel-jbang - turn thinking off with local models in camel ask, explain and
> overview
> ------------------------------------------------------------------------------------
>
> Key: CAMEL-26424
> URL: https://issues.apache.org/jira/browse/CAMEL-26424
> Project: Camel
> Issue Type: Improvement
> Components: camel-jbang
> Reporter: Claus Ibsen
> Priority: Minor
> Fix For: 4.24.0
>
>
> camel ask turns thinking off only on Ollama's own API:
> LlmClient.chatOllamaFormat sends {{"think": false}}. With
> {{--api-type=openai}}, chatOpenAiFormat sends model, messages, temperature
> and tools, and nothing about thinking.
> The Qwen 3.5, 3.6 and 3.8 models think by default, because their chat
> template turns thinking on. Local OpenAI-compatible servers such as Ollama's
> /v1 endpoint, llama.cpp server and vLLM render that template. So with these
> servers, every answer from camel ask starts with a hidden reasoning phase.
> Answers get slow, and a tool loop can spend minutes thinking. In the
> local-model benchmark, steps with thinking on ran 4-8 minutes and gave no
> answer (CAMEL-24886).
> *Measured* on a Mac mini M4 Pro with Ollama 0.40.12, its /v1/chat/completions
> endpoint, qwen3.6:35b-mlx already loaded, one short question, temperature 0.2:
> ||Request||Time||Completion tokens||Reasoning||
> |nothing extra (what camel ask sends today)|6.5 s|504|2,114 chars|
> |{{reasoning_effort}} = none|0.5 s|35|none|
> |{{chat_template_kwargs.enable_thinking}} = false|5.6 s|442|1,789 chars
> (Ollama ignores it)|
> *Proposal*
> When isLocalEndpoint() is true, the OpenAI request turns thinking off, as the
> Ollama request does:
> {noformat}
> "reasoning_effort": "none",
> "chat_template_kwargs": {"enable_thinking": false}
> {noformat}
> * reasoning_effort none: Ollama's /v1 honours it (see above). Rapid-MLX
> documents the same.
> * chat_template_kwargs: this is how vLLM and SGLang turn thinking off (Qwen's
> own docs).
> Requests to hosted endpoints (OpenAI, Azure, watsonx) stay as they are.
> There, the extra fields can be rejected as unknown parameters. No new option:
> the Ollama path has none either.
> Check that Ollama /v1 and at least one other local server (llama.cpp server
> or LM Studio) accept the extra fields.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
