sohailshk opened a new issue, #41194:
URL: https://github.com/apache/beam/issues/41194
### What happened?
`message_from_string` and `message_from_conversation` in
`apache_beam/ml/inference/anthropic_inference.py` call
`inference_args.pop('max_tokens', 1024)` on the dict they receive.
`AnthropicModelHandler.run_inference` passes the same `inference_args` dict for
every batch, so the first batch removes `max_tokens` from it and every later
batch falls back to 1024 tokens:
```python
inference_args = {'max_tokens': 2048, 'temperature': 0.5}
message_from_string(model, ['a'], client, inference_args) # max_tokens=2048
message_from_string(model, ['b'], client, inference_args) # max_tokens=1024
# inference_args is now {'temperature': 0.5}
```
### Issue Priority
Priority: 2 (default / most bugs should be filed as P2)
### Issue Components
- [x] Component: Python SDK
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
