[jira] [Updated] (CAMEL-20886) Use parallel Maven build to speed up build time

2024-06-17 Thread Jira


 [ 
https://issues.apache.org/jira/browse/CAMEL-20886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aurélien Pupier updated CAMEL-20886:

Description: 
if we can use parallel Maven build, it could save a lot of time during build 
phase locally and on CIs.
Example on my locla machine with 8 cores with all dependencies downloaded:
 - 11' for mvn clean install -D quickly
 - 6'30 for mvn clean install -D quickly -T 1C

Different possibilities:
 * use it by default, configured in the pom (requires to be sure it is working 
great in all environments)
 * include parameter only for GitHub actions and provide documentation for 
end-user to allow faster nuild and let them experiment for a while
 * provide documentation for end-user to allow faster nuild and let them 
experiment for a while

  was:
if we can use parallel Maven build, it could save a lot of time during build 
phase locally and on CIs.
Example on my locla machine with 8 cores with all dependencies downloaded:
- 11' for mvn clean install -D quickly
- 6'30 for mvn clean install -D quickly -T 1C


> Use parallel Maven build to speed up build time
> ---
>
> Key: CAMEL-20886
> URL: https://issues.apache.org/jira/browse/CAMEL-20886
> Project: Camel
>  Issue Type: Task
>  Components: build system
>Reporter: Aurélien Pupier
>Priority: Minor
>
> if we can use parallel Maven build, it could save a lot of time during build 
> phase locally and on CIs.
> Example on my locla machine with 8 cores with all dependencies downloaded:
>  - 11' for mvn clean install -D quickly
>  - 6'30 for mvn clean install -D quickly -T 1C
> Different possibilities:
>  * use it by default, configured in the pom (requires to be sure it is 
> working great in all environments)
>  * include parameter only for GitHub actions and provide documentation for 
> end-user to allow faster nuild and let them experiment for a while
>  * provide documentation for end-user to allow faster nuild and let them 
> experiment for a while



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Updated] (CAMEL-20886) Use parallel Maven build to speed up build time

2024-06-17 Thread Jira


 [ 
https://issues.apache.org/jira/browse/CAMEL-20886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aurélien Pupier updated CAMEL-20886:

Summary: Use parallel Maven build to speed up build time  (was: Use 
parallel build to speed up build)

> Use parallel Maven build to speed up build time
> ---
>
> Key: CAMEL-20886
> URL: https://issues.apache.org/jira/browse/CAMEL-20886
> Project: Camel
>  Issue Type: Task
>  Components: build system
>Reporter: Aurélien Pupier
>Priority: Minor
>
> if we can use parallel Maven build, it could save a lot of time during build 
> phase locally and on CIs.
> Example on my locla machine with 8 cores with all dependencies downloaded:
> - 11' for mvn clean install -D quickly
> - 6'30 for mvn clean install -D quickly -T 1C



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Updated] (CAMEL-20886) Use parallel build to speed up build

2024-06-17 Thread Jira


 [ 
https://issues.apache.org/jira/browse/CAMEL-20886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aurélien Pupier updated CAMEL-20886:

Priority: Minor  (was: Major)

> Use parallel build to speed up build
> 
>
> Key: CAMEL-20886
> URL: https://issues.apache.org/jira/browse/CAMEL-20886
> Project: Camel
>  Issue Type: Task
>  Components: build system
>Reporter: Aurélien Pupier
>Priority: Minor
>
> if we can use parallel Maven build, it could save a lot of time during build 
> phase locally and on CIs.
> Example on my locla machine with 8 cores with all dependencies downloaded:
> - 11' for mvn clean install -D quickly
> - 6'30 for mvn clean install -D quickly -T 1C



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Created] (CAMEL-20886) Use parallel build to speed up build

2024-06-17 Thread Jira
Aurélien Pupier created CAMEL-20886:
---

 Summary: Use parallel build to speed up build
 Key: CAMEL-20886
 URL: https://issues.apache.org/jira/browse/CAMEL-20886
 Project: Camel
  Issue Type: Task
  Components: build system
Reporter: Aurélien Pupier


if we can use parallel Maven build, it could save a lot of time during build 
phase locally and on CIs.
Example on my locla machine with 8 cores with all dependencies downloaded:
- 11' for mvn clean install -D quickly
- 6'30 for mvn clean install -D quickly -T 1C



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Created] (CAMEL-20885) Deprecate kotlin

2024-06-17 Thread Jira
Aurélien Pupier created CAMEL-20885:
---

 Summary: Deprecate kotlin
 Key: CAMEL-20885
 URL: https://issues.apache.org/jira/browse/CAMEL-20885
 Project: Camel
  Issue Type: Task
  Components: camel-kotlin
Reporter: Aurélien Pupier
 Fix For: 4.7.0


kotlin API was provided as experimental but it was not picked by the community. 
The creator of it is no more active in the community. it is taking a fair part 
of the build time (without tests) between 10 and 20%.

the idea is to deprecate the Kotlin API so that it can be removed in next 
version. it will reduce codebase size, improving maintainability and decrease 
build time on each PR and each build on CIs.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Updated] (CAMEL-20885) Deprecate kotlin

2024-06-17 Thread Jira


 [ 
https://issues.apache.org/jira/browse/CAMEL-20885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aurélien Pupier updated CAMEL-20885:

Priority: Minor  (was: Major)

> Deprecate kotlin
> 
>
> Key: CAMEL-20885
> URL: https://issues.apache.org/jira/browse/CAMEL-20885
> Project: Camel
>  Issue Type: Task
>  Components: camel-kotlin
>Reporter: Aurélien Pupier
>Priority: Minor
> Fix For: 4.7.0
>
>
> kotlin API was provided as experimental but it was not picked by the 
> community. The creator of it is no more active in the community. it is taking 
> a fair part of the build time (without tests) between 10 and 20%.
> the idea is to deprecate the Kotlin API so that it can be removed in next 
> version. it will reduce codebase size, improving maintainability and decrease 
> build time on each PR and each build on CIs.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Work started] (CAMEL-20885) Deprecate kotlin

2024-06-17 Thread Jira


 [ 
https://issues.apache.org/jira/browse/CAMEL-20885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on CAMEL-20885 started by Aurélien Pupier.
---
> Deprecate kotlin
> 
>
> Key: CAMEL-20885
> URL: https://issues.apache.org/jira/browse/CAMEL-20885
> Project: Camel
>  Issue Type: Task
>  Components: camel-kotlin
>Reporter: Aurélien Pupier
>Assignee: Aurélien Pupier
>Priority: Minor
> Fix For: 4.7.0
>
>
> kotlin API was provided as experimental but it was not picked by the 
> community. The creator of it is no more active in the community. it is taking 
> a fair part of the build time (without tests) between 10 and 20%.
> the idea is to deprecate the Kotlin API so that it can be removed in next 
> version. it will reduce codebase size, improving maintainability and decrease 
> build time on each PR and each build on CIs.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Assigned] (CAMEL-20885) Deprecate kotlin

2024-06-17 Thread Jira


 [ 
https://issues.apache.org/jira/browse/CAMEL-20885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aurélien Pupier reassigned CAMEL-20885:
---

Assignee: Aurélien Pupier

> Deprecate kotlin
> 
>
> Key: CAMEL-20885
> URL: https://issues.apache.org/jira/browse/CAMEL-20885
> Project: Camel
>  Issue Type: Task
>  Components: camel-kotlin
>Reporter: Aurélien Pupier
>Assignee: Aurélien Pupier
>Priority: Minor
> Fix For: 4.7.0
>
>
> kotlin API was provided as experimental but it was not picked by the 
> community. The creator of it is no more active in the community. it is taking 
> a fair part of the build time (without tests) between 10 and 20%.
> the idea is to deprecate the Kotlin API so that it can be removed in next 
> version. it will reduce codebase size, improving maintainability and decrease 
> build time on each PR and each build on CIs.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Created] (CAMEL-20884) Improve performance of Camel Package Maven plugin

2024-06-17 Thread Jira
Aurélien Pupier created CAMEL-20884:
---

 Summary: Improve performance of Camel Package Maven plugin
 Key: CAMEL-20884
 URL: https://issues.apache.org/jira/browse/CAMEL-20884
 Project: Camel
  Issue Type: Task
  Components: camel-package-maven-plugin
Reporter: Aurélien Pupier
 Attachments: image-2024-06-17-15-23-10-200.png

The Camel package Maven plugin is used a lot during the build of Apache Camel. 
When build ding with -D quickly, it represents around 10% of the time spent. 
(on my local machine 1'11" on a 11'31" build)

!image-2024-06-17-15-23-10-200.png!



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Work started] (CAMEL-20884) Improve performance of Camel Package Maven plugin

2024-06-17 Thread Jira


 [ 
https://issues.apache.org/jira/browse/CAMEL-20884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on CAMEL-20884 started by Aurélien Pupier.
---
> Improve performance of Camel Package Maven plugin
> -
>
> Key: CAMEL-20884
> URL: https://issues.apache.org/jira/browse/CAMEL-20884
> Project: Camel
>  Issue Type: Task
>  Components: camel-package-maven-plugin
>Reporter: Aurélien Pupier
>Assignee: Aurélien Pupier
>Priority: Minor
> Attachments: image-2024-06-17-15-23-10-200.png
>
>
> The Camel package Maven plugin is used a lot during the build of Apache 
> Camel. When build ding with -D quickly, it represents around 10% of the time 
> spent. (on my local machine 1'11" on a 11'31" build)
> !image-2024-06-17-15-23-10-200.png!



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Assigned] (CAMEL-20884) Improve performance of Camel Package Maven plugin

2024-06-17 Thread Jira


 [ 
https://issues.apache.org/jira/browse/CAMEL-20884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aurélien Pupier reassigned CAMEL-20884:
---

Assignee: Aurélien Pupier

> Improve performance of Camel Package Maven plugin
> -
>
> Key: CAMEL-20884
> URL: https://issues.apache.org/jira/browse/CAMEL-20884
> Project: Camel
>  Issue Type: Task
>  Components: camel-package-maven-plugin
>Reporter: Aurélien Pupier
>Assignee: Aurélien Pupier
>Priority: Minor
> Attachments: image-2024-06-17-15-23-10-200.png
>
>
> The Camel package Maven plugin is used a lot during the build of Apache 
> Camel. When build ding with -D quickly, it represents around 10% of the time 
> spent. (on my local machine 1'11" on a 11'31" build)
> !image-2024-06-17-15-23-10-200.png!



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Resolved] (CAMEL-20414) Add CloudEvent Transformers to cloud oriented component

2024-06-17 Thread Andrea Cosentino (Jira)


 [ 
https://issues.apache.org/jira/browse/CAMEL-20414?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrea Cosentino resolved CAMEL-20414.
--
Resolution: Fixed

> Add CloudEvent Transformers to cloud oriented component
> ---
>
> Key: CAMEL-20414
> URL: https://issues.apache.org/jira/browse/CAMEL-20414
> Project: Camel
>  Issue Type: Task
>Reporter: Andrea Cosentino
>Assignee: Andrea Cosentino
>Priority: Major
> Fix For: 4.7.0
>
>




--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Resolved] (CAMEL-20880) Move AI related component to camel-ai middle folder

2024-06-17 Thread Andrea Cosentino (Jira)


 [ 
https://issues.apache.org/jira/browse/CAMEL-20880?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrea Cosentino resolved CAMEL-20880.
--
Resolution: Fixed

> Move AI related component to camel-ai middle folder
> ---
>
> Key: CAMEL-20880
> URL: https://issues.apache.org/jira/browse/CAMEL-20880
> Project: Camel
>  Issue Type: Task
>Reporter: Andrea Cosentino
>Assignee: Andrea Cosentino
>Priority: Major
> Fix For: 4.7.0
>
>




--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Created] (CAMEL-20883) Remove addPluginArtifactMetadata mojo usage

2024-06-17 Thread Jira
Aurélien Pupier created CAMEL-20883:
---

 Summary: Remove addPluginArtifactMetadata mojo usage
 Key: CAMEL-20883
 URL: https://issues.apache.org/jira/browse/CAMEL-20883
 Project: Camel
  Issue Type: Task
  Components: build system
Reporter: Aurélien Pupier


in log there is 14 times:
```
14:00:33,645 [INFO] --- plugin:3.13.1:addPluginArtifactMetadata 
(default-addPluginArtifactMetadata) @ camel-package-maven-plugin ---
14:00:33,645 [INFO] This Mojo is not used in Maven version 3.9.0 and above
```

 

In this case I think we need to upgrade the minimal Maven version (currently at 
3.5.0): 
[https://github.com/apache/camel/blob/07c502761c4d450f5b2b7bba19ce4f69f507212d/pom.xml#L915]

 

Is it fine to upgrade the minimal Maven version to 3.9.0?
I guess we are testing anyway only with the version which is defined by the 
Maven wrapper?

Advantages:
 * less message in the log
 * small time gain on the build
 * less code to maintain

Drawback:

- forcing all developers to use recent Maven (which might also be seen as an 
advantage :) )



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Resolved] (CAMEL-19972) camel-kafka manual commit configuration inconsistency

2024-06-17 Thread Otavio Rodolfo Piske (Jira)


 [ 
https://issues.apache.org/jira/browse/CAMEL-19972?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Otavio Rodolfo Piske resolved CAMEL-19972.
--
Resolution: Fixed

Likely resolved by CAMEL-20235.

> camel-kafka manual commit configuration inconsistency
> -
>
> Key: CAMEL-19972
> URL: https://issues.apache.org/jira/browse/CAMEL-19972
> Project: Camel
>  Issue Type: Bug
>  Components: camel-kafka
>Affects Versions: 3.18.5, 3.21.1
>Reporter: Sami Peltola
>Assignee: Otavio Rodolfo Piske
>Priority: Minor
>  Labels: help-wanted
> Fix For: 4.4.0
>
>
> h2. Expected behavior
>  * Should not matter if manual commits are enabled component level or 
> endpoint level, consumer should not commit automatically when 
> autoCommitEnable is set to false
> h2. Actual behavior
> The behavior for manual commits is inconsistent at the moment (Test on Camel 
> versions 3.18.5 and 3.21.1), depending on whether the KafkaComponent is 
> configured on component or endpoint level
>  * If configured on component level, will still automatically commit offsets 
> back to broker (after records from a partition have been processed)
>  * If configured on endpoint level, will never automatically commit offsets
>  * autoCommitEnable is
> h3. Configuration of manual commit for Kafka client during startup
> During startup for Kafka component, Camel context will go through the 
> following classes and methods while setting up manual commit configuration.
> h4. KafkaComponent
> [https://github.com/apache/camel/blob/camel-3.18.5/components/camel-kafka/src/main/java/org/apache/camel/component/kafka/KafkaComponent.java#L222]
> *Class:* org.apache.camel.component.kafka.KafkaComponent
> *Method:* doInit
>  
>  * When the component is being initialized, will check if 
> configuration.isAllowManualCommit() is true AND kafkaManualCommitFactory is 
> null
>  ** If they are, will initialize kafkaManualCommitFactory as 
> DefaultKafkaManualCommitFactory
>  ** If not, will leave kafkaManualCommitFactory as null
>  
> h4. CommitManagers
> [https://github.com/apache/camel/blob/camel-3.18.5/components/camel-kafka/src/main/java/org/apache/camel/component/kafka/consumer/CommitManagers.java#L32]
> *Class:* org.apache.camel.component.kafka.consumer.CommitManagers
> *Method:* createCommitManager
>  * Called when initializing Consumer
>  ** For clarity's sake, omitting Async-related logic
>  ** If configuration.isAllowManualCommit() is true AND manualCommitFactory is 
> instance of DefaultKafkaManualCommitFactory
>  *** Will initialize SyncCommitManager
>  *** Otherwise, will initialize NoopCommitManager
> h4. Summary
>  * Configuring manual commits on component level still results in automatic 
> commits when the changing partitions (or all records processed) since 
> *SyncCommitManager* is configured
>  ** allowManualCommit is true when calling *doInit* -> initializes 
> kafkaManualCommitFactory as DefaultKafkaManualCommitFactory -> initilize 
> commitManager as SyncCommitManager
>  * If allowManualCommit is only configured on endpoint level, it will be 
> false when invoking {*}doInit{*}, therefore leaving kafkaManualCommitFactory 
> as null, resulting in NoopCommitManager in *createCommitManager*
>  
> h2. Workaround
> * Configure manual commits on endpoint level



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Reopened] (CAMEL-19972) camel-kafka manual commit configuration inconsistency

2024-06-17 Thread Otavio Rodolfo Piske (Jira)


 [ 
https://issues.apache.org/jira/browse/CAMEL-19972?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Otavio Rodolfo Piske reopened CAMEL-19972:
--
  Assignee: Otavio Rodolfo Piske

> camel-kafka manual commit configuration inconsistency
> -
>
> Key: CAMEL-19972
> URL: https://issues.apache.org/jira/browse/CAMEL-19972
> Project: Camel
>  Issue Type: Bug
>  Components: camel-kafka
>Affects Versions: 3.18.5, 3.21.1
>Reporter: Sami Peltola
>Assignee: Otavio Rodolfo Piske
>Priority: Minor
>  Labels: help-wanted
>
> h2. Expected behavior
>  * Should not matter if manual commits are enabled component level or 
> endpoint level, consumer should not commit automatically when 
> autoCommitEnable is set to false
> h2. Actual behavior
> The behavior for manual commits is inconsistent at the moment (Test on Camel 
> versions 3.18.5 and 3.21.1), depending on whether the KafkaComponent is 
> configured on component or endpoint level
>  * If configured on component level, will still automatically commit offsets 
> back to broker (after records from a partition have been processed)
>  * If configured on endpoint level, will never automatically commit offsets
>  * autoCommitEnable is
> h3. Configuration of manual commit for Kafka client during startup
> During startup for Kafka component, Camel context will go through the 
> following classes and methods while setting up manual commit configuration.
> h4. KafkaComponent
> [https://github.com/apache/camel/blob/camel-3.18.5/components/camel-kafka/src/main/java/org/apache/camel/component/kafka/KafkaComponent.java#L222]
> *Class:* org.apache.camel.component.kafka.KafkaComponent
> *Method:* doInit
>  
>  * When the component is being initialized, will check if 
> configuration.isAllowManualCommit() is true AND kafkaManualCommitFactory is 
> null
>  ** If they are, will initialize kafkaManualCommitFactory as 
> DefaultKafkaManualCommitFactory
>  ** If not, will leave kafkaManualCommitFactory as null
>  
> h4. CommitManagers
> [https://github.com/apache/camel/blob/camel-3.18.5/components/camel-kafka/src/main/java/org/apache/camel/component/kafka/consumer/CommitManagers.java#L32]
> *Class:* org.apache.camel.component.kafka.consumer.CommitManagers
> *Method:* createCommitManager
>  * Called when initializing Consumer
>  ** For clarity's sake, omitting Async-related logic
>  ** If configuration.isAllowManualCommit() is true AND manualCommitFactory is 
> instance of DefaultKafkaManualCommitFactory
>  *** Will initialize SyncCommitManager
>  *** Otherwise, will initialize NoopCommitManager
> h4. Summary
>  * Configuring manual commits on component level still results in automatic 
> commits when the changing partitions (or all records processed) since 
> *SyncCommitManager* is configured
>  ** allowManualCommit is true when calling *doInit* -> initializes 
> kafkaManualCommitFactory as DefaultKafkaManualCommitFactory -> initilize 
> commitManager as SyncCommitManager
>  * If allowManualCommit is only configured on endpoint level, it will be 
> false when invoking {*}doInit{*}, therefore leaving kafkaManualCommitFactory 
> as null, resulting in NoopCommitManager in *createCommitManager*
>  
> h2. Workaround
> * Configure manual commits on endpoint level



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Updated] (CAMEL-19972) camel-kafka manual commit configuration inconsistency

2024-06-17 Thread Otavio Rodolfo Piske (Jira)


 [ 
https://issues.apache.org/jira/browse/CAMEL-19972?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Otavio Rodolfo Piske updated CAMEL-19972:
-
Fix Version/s: 4.4.0

> camel-kafka manual commit configuration inconsistency
> -
>
> Key: CAMEL-19972
> URL: https://issues.apache.org/jira/browse/CAMEL-19972
> Project: Camel
>  Issue Type: Bug
>  Components: camel-kafka
>Affects Versions: 3.18.5, 3.21.1
>Reporter: Sami Peltola
>Assignee: Otavio Rodolfo Piske
>Priority: Minor
>  Labels: help-wanted
> Fix For: 4.4.0
>
>
> h2. Expected behavior
>  * Should not matter if manual commits are enabled component level or 
> endpoint level, consumer should not commit automatically when 
> autoCommitEnable is set to false
> h2. Actual behavior
> The behavior for manual commits is inconsistent at the moment (Test on Camel 
> versions 3.18.5 and 3.21.1), depending on whether the KafkaComponent is 
> configured on component or endpoint level
>  * If configured on component level, will still automatically commit offsets 
> back to broker (after records from a partition have been processed)
>  * If configured on endpoint level, will never automatically commit offsets
>  * autoCommitEnable is
> h3. Configuration of manual commit for Kafka client during startup
> During startup for Kafka component, Camel context will go through the 
> following classes and methods while setting up manual commit configuration.
> h4. KafkaComponent
> [https://github.com/apache/camel/blob/camel-3.18.5/components/camel-kafka/src/main/java/org/apache/camel/component/kafka/KafkaComponent.java#L222]
> *Class:* org.apache.camel.component.kafka.KafkaComponent
> *Method:* doInit
>  
>  * When the component is being initialized, will check if 
> configuration.isAllowManualCommit() is true AND kafkaManualCommitFactory is 
> null
>  ** If they are, will initialize kafkaManualCommitFactory as 
> DefaultKafkaManualCommitFactory
>  ** If not, will leave kafkaManualCommitFactory as null
>  
> h4. CommitManagers
> [https://github.com/apache/camel/blob/camel-3.18.5/components/camel-kafka/src/main/java/org/apache/camel/component/kafka/consumer/CommitManagers.java#L32]
> *Class:* org.apache.camel.component.kafka.consumer.CommitManagers
> *Method:* createCommitManager
>  * Called when initializing Consumer
>  ** For clarity's sake, omitting Async-related logic
>  ** If configuration.isAllowManualCommit() is true AND manualCommitFactory is 
> instance of DefaultKafkaManualCommitFactory
>  *** Will initialize SyncCommitManager
>  *** Otherwise, will initialize NoopCommitManager
> h4. Summary
>  * Configuring manual commits on component level still results in automatic 
> commits when the changing partitions (or all records processed) since 
> *SyncCommitManager* is configured
>  ** allowManualCommit is true when calling *doInit* -> initializes 
> kafkaManualCommitFactory as DefaultKafkaManualCommitFactory -> initilize 
> commitManager as SyncCommitManager
>  * If allowManualCommit is only configured on endpoint level, it will be 
> false when invoking {*}doInit{*}, therefore leaving kafkaManualCommitFactory 
> as null, resulting in NoopCommitManager in *createCommitManager*
>  
> h2. Workaround
> * Configure manual commits on endpoint level



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Resolved] (CAMEL-20881) camel-core - Log should not need to create DefaultPollingConsumerPollStrategy

2024-06-17 Thread Claus Ibsen (Jira)


 [ 
https://issues.apache.org/jira/browse/CAMEL-20881?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Claus Ibsen resolved CAMEL-20881.
-
Resolution: Fixed

> camel-core - Log should not need to create DefaultPollingConsumerPollStrategy
> -
>
> Key: CAMEL-20881
> URL: https://issues.apache.org/jira/browse/CAMEL-20881
> Project: Camel
>  Issue Type: Improvement
>  Components: camel-core
>Reporter: Claus Ibsen
>Assignee: Claus Ibsen
>Priority: Minor
> Fix For: 4.7.0
>
>
> When using log component then its base endpoint creates an instance of 
> DefaultPollingConsumerPollStrategy which is only needed for consumers, so 
> lets optimize this



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Commented] (CAMEL-19972) camel-kafka manual commit configuration inconsistency

2024-06-17 Thread Sami Peltola (Jira)


[ 
https://issues.apache.org/jira/browse/CAMEL-19972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17855599#comment-17855599
 ] 

Sami Peltola commented on CAMEL-19972:
--

[~davsclaus]

Yes, look like the issue has been solved KafkaComponent, instead of creating a 
DefaultKafkaManualCommitFactory the conditions apply, it now prints a warning.

*+Camel 3.18.5+*
{code:java}
if (configuration.isAllowManualCommit() && kafkaManualCommitFactory == 
null) {
kafkaManualCommitFactory = new DefaultKafkaManualCommitFactory();
}
{code}


*+Camel 4.6.0+*
{code:java}
if (configuration.isAllowManualCommit() && kafkaManualCommitFactory == 
null) {
LOG.warn("The component was setup for allowing manual commits, but 
a manual commit factory was not set");
}
{code}

Seems to have been fixed by CAMEL-20235, closing this task, thanks.


> camel-kafka manual commit configuration inconsistency
> -
>
> Key: CAMEL-19972
> URL: https://issues.apache.org/jira/browse/CAMEL-19972
> Project: Camel
>  Issue Type: Bug
>  Components: camel-kafka
>Affects Versions: 3.18.5, 3.21.1
>Reporter: Sami Peltola
>Priority: Minor
>  Labels: help-wanted
>
> h2. Expected behavior
>  * Should not matter if manual commits are enabled component level or 
> endpoint level, consumer should not commit automatically when 
> autoCommitEnable is set to false
> h2. Actual behavior
> The behavior for manual commits is inconsistent at the moment (Test on Camel 
> versions 3.18.5 and 3.21.1), depending on whether the KafkaComponent is 
> configured on component or endpoint level
>  * If configured on component level, will still automatically commit offsets 
> back to broker (after records from a partition have been processed)
>  * If configured on endpoint level, will never automatically commit offsets
>  * autoCommitEnable is
> h3. Configuration of manual commit for Kafka client during startup
> During startup for Kafka component, Camel context will go through the 
> following classes and methods while setting up manual commit configuration.
> h4. KafkaComponent
> [https://github.com/apache/camel/blob/camel-3.18.5/components/camel-kafka/src/main/java/org/apache/camel/component/kafka/KafkaComponent.java#L222]
> *Class:* org.apache.camel.component.kafka.KafkaComponent
> *Method:* doInit
>  
>  * When the component is being initialized, will check if 
> configuration.isAllowManualCommit() is true AND kafkaManualCommitFactory is 
> null
>  ** If they are, will initialize kafkaManualCommitFactory as 
> DefaultKafkaManualCommitFactory
>  ** If not, will leave kafkaManualCommitFactory as null
>  
> h4. CommitManagers
> [https://github.com/apache/camel/blob/camel-3.18.5/components/camel-kafka/src/main/java/org/apache/camel/component/kafka/consumer/CommitManagers.java#L32]
> *Class:* org.apache.camel.component.kafka.consumer.CommitManagers
> *Method:* createCommitManager
>  * Called when initializing Consumer
>  ** For clarity's sake, omitting Async-related logic
>  ** If configuration.isAllowManualCommit() is true AND manualCommitFactory is 
> instance of DefaultKafkaManualCommitFactory
>  *** Will initialize SyncCommitManager
>  *** Otherwise, will initialize NoopCommitManager
> h4. Summary
>  * Configuring manual commits on component level still results in automatic 
> commits when the changing partitions (or all records processed) since 
> *SyncCommitManager* is configured
>  ** allowManualCommit is true when calling *doInit* -> initializes 
> kafkaManualCommitFactory as DefaultKafkaManualCommitFactory -> initilize 
> commitManager as SyncCommitManager
>  * If allowManualCommit is only configured on endpoint level, it will be 
> false when invoking {*}doInit{*}, therefore leaving kafkaManualCommitFactory 
> as null, resulting in NoopCommitManager in *createCommitManager*
>  
> h2. Workaround
> * Configure manual commits on endpoint level



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Closed] (CAMEL-19972) camel-kafka manual commit configuration inconsistency

2024-06-17 Thread Sami Peltola (Jira)


 [ 
https://issues.apache.org/jira/browse/CAMEL-19972?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sami Peltola closed CAMEL-19972.

Resolution: Fixed

> camel-kafka manual commit configuration inconsistency
> -
>
> Key: CAMEL-19972
> URL: https://issues.apache.org/jira/browse/CAMEL-19972
> Project: Camel
>  Issue Type: Bug
>  Components: camel-kafka
>Affects Versions: 3.18.5, 3.21.1
>Reporter: Sami Peltola
>Priority: Minor
>  Labels: help-wanted
>
> h2. Expected behavior
>  * Should not matter if manual commits are enabled component level or 
> endpoint level, consumer should not commit automatically when 
> autoCommitEnable is set to false
> h2. Actual behavior
> The behavior for manual commits is inconsistent at the moment (Test on Camel 
> versions 3.18.5 and 3.21.1), depending on whether the KafkaComponent is 
> configured on component or endpoint level
>  * If configured on component level, will still automatically commit offsets 
> back to broker (after records from a partition have been processed)
>  * If configured on endpoint level, will never automatically commit offsets
>  * autoCommitEnable is
> h3. Configuration of manual commit for Kafka client during startup
> During startup for Kafka component, Camel context will go through the 
> following classes and methods while setting up manual commit configuration.
> h4. KafkaComponent
> [https://github.com/apache/camel/blob/camel-3.18.5/components/camel-kafka/src/main/java/org/apache/camel/component/kafka/KafkaComponent.java#L222]
> *Class:* org.apache.camel.component.kafka.KafkaComponent
> *Method:* doInit
>  
>  * When the component is being initialized, will check if 
> configuration.isAllowManualCommit() is true AND kafkaManualCommitFactory is 
> null
>  ** If they are, will initialize kafkaManualCommitFactory as 
> DefaultKafkaManualCommitFactory
>  ** If not, will leave kafkaManualCommitFactory as null
>  
> h4. CommitManagers
> [https://github.com/apache/camel/blob/camel-3.18.5/components/camel-kafka/src/main/java/org/apache/camel/component/kafka/consumer/CommitManagers.java#L32]
> *Class:* org.apache.camel.component.kafka.consumer.CommitManagers
> *Method:* createCommitManager
>  * Called when initializing Consumer
>  ** For clarity's sake, omitting Async-related logic
>  ** If configuration.isAllowManualCommit() is true AND manualCommitFactory is 
> instance of DefaultKafkaManualCommitFactory
>  *** Will initialize SyncCommitManager
>  *** Otherwise, will initialize NoopCommitManager
> h4. Summary
>  * Configuring manual commits on component level still results in automatic 
> commits when the changing partitions (or all records processed) since 
> *SyncCommitManager* is configured
>  ** allowManualCommit is true when calling *doInit* -> initializes 
> kafkaManualCommitFactory as DefaultKafkaManualCommitFactory -> initilize 
> commitManager as SyncCommitManager
>  * If allowManualCommit is only configured on endpoint level, it will be 
> false when invoking {*}doInit{*}, therefore leaving kafkaManualCommitFactory 
> as null, resulting in NoopCommitManager in *createCommitManager*
>  
> h2. Workaround
> * Configure manual commits on endpoint level



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Created] (CAMEL-20882) camel-spring-boot - Upgrade to 3.2.7

2024-06-17 Thread Claus Ibsen (Jira)
Claus Ibsen created CAMEL-20882:
---

 Summary: camel-spring-boot - Upgrade to 3.2.7
 Key: CAMEL-20882
 URL: https://issues.apache.org/jira/browse/CAMEL-20882
 Project: Camel
  Issue Type: Dependency upgrade
  Components: camel-spring-boot
Reporter: Claus Ibsen
 Fix For: 4.4.3






--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Updated] (CAMEL-20835) OOM using RecipientList

2024-06-17 Thread Claus Ibsen (Jira)


 [ 
https://issues.apache.org/jira/browse/CAMEL-20835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Claus Ibsen updated CAMEL-20835:

Fix Version/s: 3.22.3

> OOM using RecipientList
> ---
>
> Key: CAMEL-20835
> URL: https://issues.apache.org/jira/browse/CAMEL-20835
> Project: Camel
>  Issue Type: Bug
>  Components: camel-core
>Affects Versions: 3.10.0, 4.6.0
>Reporter: Arthur Naseef
>Assignee: Claus Ibsen
>Priority: Major
> Fix For: 3.22.3, 4.0.6, 4.4.3, 4.7.0
>
>
> Out Of Memory (OOM) occurs when using the Recipient List with a large number 
> of dynamic URLs.  For example:
>  
>     
> .recipientList(simple("http://{{{}downstream-server{}}}/employee/${header.emplId}";))
>  
> with a large number of values for ${header.emplId} leads to the OOM.
>  
> *REPRODUCER:*
> *=*
> [https://github.com/artnaseef/camel-recipient-list-oom-reproducer]
>  
> See the README.md for instructions to reproduce and detect the problem
>  
> *DETAILS*
> *===*
>  The MulticastProcessor, which RecipientListProcessor extends, has the 
> following "unlimited" cache:
>  
>     private final ConcurrentMap errorHandlers = new 
> ConcurrentHashMap<>();
>  
> Entries are added to this map for every unique processor created - every 
> unique URL generates a unique processor.  The entries themselves are wrapped 
> processor instances for error handling IIUC (to support the custom error 
> handling used by multicast and recipient-list).  Entries are only removed 
> from this map on shutdown.  Ironically, there is an LRUCache for the 
> processors themselves, with a default maximum size of 1000, so the wrapped 
> processors may get recreated even though the error handler remains in the map 
> indefinitely.
>  
> *IMPACT VERSIONS:*
> **
> Appears to impact versions >= 3.10.0
> *COMMIT: 0d9227ff16fb00e047fdd087740c87cce01bb545*
> *===*
> It appears this commit introduced the use of the errorHandlers "unlimited" 
> cache for recipient lists.
>  
> *FOLLOW-UP*
> *==*
> I have ideas and questions for implemeting a fix:
>     - IDEA 1: We can use an LRUCache for this data structure as well.
>     - Does it make more sense to remove the entries from errorHandlers when 
> the related Processor entry is removed from it's LRUCache?
>     - IDEA 2: setting on recipient list to disable the errorHandler cache 
> (for dyamic urls with little chance of duplicates, this could be the best)



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Updated] (CAMEL-20835) OOM using RecipientList

2024-06-17 Thread Claus Ibsen (Jira)


 [ 
https://issues.apache.org/jira/browse/CAMEL-20835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Claus Ibsen updated CAMEL-20835:

Fix Version/s: 4.0.6
   4.4.3

> OOM using RecipientList
> ---
>
> Key: CAMEL-20835
> URL: https://issues.apache.org/jira/browse/CAMEL-20835
> Project: Camel
>  Issue Type: Bug
>  Components: camel-core
>Affects Versions: 3.10.0, 4.6.0
>Reporter: Arthur Naseef
>Assignee: Claus Ibsen
>Priority: Major
> Fix For: 4.0.6, 4.4.3, 4.7.0
>
>
> Out Of Memory (OOM) occurs when using the Recipient List with a large number 
> of dynamic URLs.  For example:
>  
>     
> .recipientList(simple("http://{{{}downstream-server{}}}/employee/${header.emplId}";))
>  
> with a large number of values for ${header.emplId} leads to the OOM.
>  
> *REPRODUCER:*
> *=*
> [https://github.com/artnaseef/camel-recipient-list-oom-reproducer]
>  
> See the README.md for instructions to reproduce and detect the problem
>  
> *DETAILS*
> *===*
>  The MulticastProcessor, which RecipientListProcessor extends, has the 
> following "unlimited" cache:
>  
>     private final ConcurrentMap errorHandlers = new 
> ConcurrentHashMap<>();
>  
> Entries are added to this map for every unique processor created - every 
> unique URL generates a unique processor.  The entries themselves are wrapped 
> processor instances for error handling IIUC (to support the custom error 
> handling used by multicast and recipient-list).  Entries are only removed 
> from this map on shutdown.  Ironically, there is an LRUCache for the 
> processors themselves, with a default maximum size of 1000, so the wrapped 
> processors may get recreated even though the error handler remains in the map 
> indefinitely.
>  
> *IMPACT VERSIONS:*
> **
> Appears to impact versions >= 3.10.0
> *COMMIT: 0d9227ff16fb00e047fdd087740c87cce01bb545*
> *===*
> It appears this commit introduced the use of the errorHandlers "unlimited" 
> cache for recipient lists.
>  
> *FOLLOW-UP*
> *==*
> I have ideas and questions for implemeting a fix:
>     - IDEA 1: We can use an LRUCache for this data structure as well.
>     - Does it make more sense to remove the entries from errorHandlers when 
> the related Processor entry is removed from it's LRUCache?
>     - IDEA 2: setting on recipient list to disable the errorHandler cache 
> (for dyamic urls with little chance of duplicates, this could be the best)



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Resolved] (CAMEL-20835) OOM using RecipientList

2024-06-17 Thread Claus Ibsen (Jira)


 [ 
https://issues.apache.org/jira/browse/CAMEL-20835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Claus Ibsen resolved CAMEL-20835.
-
Resolution: Fixed

> OOM using RecipientList
> ---
>
> Key: CAMEL-20835
> URL: https://issues.apache.org/jira/browse/CAMEL-20835
> Project: Camel
>  Issue Type: Bug
>  Components: camel-core
>Affects Versions: 3.10.0, 4.6.0
>Reporter: Arthur Naseef
>Assignee: Claus Ibsen
>Priority: Major
> Fix For: 4.0.6, 4.4.3, 4.7.0
>
>
> Out Of Memory (OOM) occurs when using the Recipient List with a large number 
> of dynamic URLs.  For example:
>  
>     
> .recipientList(simple("http://{{{}downstream-server{}}}/employee/${header.emplId}";))
>  
> with a large number of values for ${header.emplId} leads to the OOM.
>  
> *REPRODUCER:*
> *=*
> [https://github.com/artnaseef/camel-recipient-list-oom-reproducer]
>  
> See the README.md for instructions to reproduce and detect the problem
>  
> *DETAILS*
> *===*
>  The MulticastProcessor, which RecipientListProcessor extends, has the 
> following "unlimited" cache:
>  
>     private final ConcurrentMap errorHandlers = new 
> ConcurrentHashMap<>();
>  
> Entries are added to this map for every unique processor created - every 
> unique URL generates a unique processor.  The entries themselves are wrapped 
> processor instances for error handling IIUC (to support the custom error 
> handling used by multicast and recipient-list).  Entries are only removed 
> from this map on shutdown.  Ironically, there is an LRUCache for the 
> processors themselves, with a default maximum size of 1000, so the wrapped 
> processors may get recreated even though the error handler remains in the map 
> indefinitely.
>  
> *IMPACT VERSIONS:*
> **
> Appears to impact versions >= 3.10.0
> *COMMIT: 0d9227ff16fb00e047fdd087740c87cce01bb545*
> *===*
> It appears this commit introduced the use of the errorHandlers "unlimited" 
> cache for recipient lists.
>  
> *FOLLOW-UP*
> *==*
> I have ideas and questions for implemeting a fix:
>     - IDEA 1: We can use an LRUCache for this data structure as well.
>     - Does it make more sense to remove the entries from errorHandlers when 
> the related Processor entry is removed from it's LRUCache?
>     - IDEA 2: setting on recipient list to disable the errorHandler cache 
> (for dyamic urls with little chance of duplicates, this could be the best)



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Created] (CAMEL-20881) camel-core - Log should not need to create DefaultPollingConsumerPollStrategy

2024-06-17 Thread Claus Ibsen (Jira)
Claus Ibsen created CAMEL-20881:
---

 Summary: camel-core - Log should not need to create 
DefaultPollingConsumerPollStrategy
 Key: CAMEL-20881
 URL: https://issues.apache.org/jira/browse/CAMEL-20881
 Project: Camel
  Issue Type: Improvement
  Components: camel-core
Reporter: Claus Ibsen
Assignee: Claus Ibsen
 Fix For: 4.7.0


When using log component then its base endpoint creates an instance of 
DefaultPollingConsumerPollStrategy which is only needed for consumers, so lets 
optimize this



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Work started] (CAMEL-20870) Include PR check for documentation related to Camel website in each repository

2024-06-17 Thread Jira


 [ 
https://issues.apache.org/jira/browse/CAMEL-20870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on CAMEL-20870 started by Aurélien Pupier.
---
> Include PR check for documentation related to Camel website in each repository
> --
>
> Key: CAMEL-20870
> URL: https://issues.apache.org/jira/browse/CAMEL-20870
> Project: Camel
>  Issue Type: Task
>  Components: website
>Reporter: Aurélien Pupier
>Assignee: Aurélien Pupier
>Priority: Major
>
> it is often that the camel-website is broken due to changes in documentation 
> in other Camel repositories where the documentation is defined.
> it will be convenient to have a PR check on each of these repositories to 
> avoid being in this state.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Assigned] (CAMEL-20870) Include PR check for documentation related to Camel website in each repository

2024-06-17 Thread Jira


 [ 
https://issues.apache.org/jira/browse/CAMEL-20870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aurélien Pupier reassigned CAMEL-20870:
---

Assignee: Aurélien Pupier

> Include PR check for documentation related to Camel website in each repository
> --
>
> Key: CAMEL-20870
> URL: https://issues.apache.org/jira/browse/CAMEL-20870
> Project: Camel
>  Issue Type: Task
>  Components: website
>Reporter: Aurélien Pupier
>Assignee: Aurélien Pupier
>Priority: Major
>
> it is often that the camel-website is broken due to changes in documentation 
> in other Camel repositories where the documentation is defined.
> it will be convenient to have a PR check on each of these repositories to 
> avoid being in this state.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Commented] (CAMEL-20835) OOM using RecipientList

2024-06-17 Thread Claus Ibsen (Jira)


[ 
https://issues.apache.org/jira/browse/CAMEL-20835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1788#comment-1788
 ] 

Claus Ibsen commented on CAMEL-20835:
-

And with 1000 as default cache size

  26:          1000         224000  org.apache.camel.component.log.LogEndpoint
  53:          1000          48000  
org.apache.camel.support.processor.CamelLogProcessor
  62:          1000          32000  org.apache.camel.component.log.LogProducer
  74:          1044          25056  org.apache.camel.spi.CamelLogger
  75:          1000          24000  
org.apache.camel.support.cache.ServicePool$SinglePool
 103:          1008          16128  org.apache.camel.support.NormalizedUri
 105:          1000          16000  
org.apache.camel.support.DefaultPollingConsumerPollStrategy

> OOM using RecipientList
> ---
>
> Key: CAMEL-20835
> URL: https://issues.apache.org/jira/browse/CAMEL-20835
> Project: Camel
>  Issue Type: Bug
>  Components: camel-core
>Affects Versions: 3.10.0, 4.6.0
>Reporter: Arthur Naseef
>Assignee: Claus Ibsen
>Priority: Major
> Fix For: 4.7.0
>
>
> Out Of Memory (OOM) occurs when using the Recipient List with a large number 
> of dynamic URLs.  For example:
>  
>     
> .recipientList(simple("http://{{{}downstream-server{}}}/employee/${header.emplId}";))
>  
> with a large number of values for ${header.emplId} leads to the OOM.
>  
> *REPRODUCER:*
> *=*
> [https://github.com/artnaseef/camel-recipient-list-oom-reproducer]
>  
> See the README.md for instructions to reproduce and detect the problem
>  
> *DETAILS*
> *===*
>  The MulticastProcessor, which RecipientListProcessor extends, has the 
> following "unlimited" cache:
>  
>     private final ConcurrentMap errorHandlers = new 
> ConcurrentHashMap<>();
>  
> Entries are added to this map for every unique processor created - every 
> unique URL generates a unique processor.  The entries themselves are wrapped 
> processor instances for error handling IIUC (to support the custom error 
> handling used by multicast and recipient-list).  Entries are only removed 
> from this map on shutdown.  Ironically, there is an LRUCache for the 
> processors themselves, with a default maximum size of 1000, so the wrapped 
> processors may get recreated even though the error handler remains in the map 
> indefinitely.
>  
> *IMPACT VERSIONS:*
> **
> Appears to impact versions >= 3.10.0
> *COMMIT: 0d9227ff16fb00e047fdd087740c87cce01bb545*
> *===*
> It appears this commit introduced the use of the errorHandlers "unlimited" 
> cache for recipient lists.
>  
> *FOLLOW-UP*
> *==*
> I have ideas and questions for implemeting a fix:
>     - IDEA 1: We can use an LRUCache for this data structure as well.
>     - Does it make more sense to remove the entries from errorHandlers when 
> the related Processor entry is removed from it's LRUCache?
>     - IDEA 2: setting on recipient list to disable the errorHandler cache 
> (for dyamic urls with little chance of duplicates, this could be the best)



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Resolved] (CAMEL-19544) camel-sql: replace Thread.sleep in tests

2024-06-17 Thread Jira


 [ 
https://issues.apache.org/jira/browse/CAMEL-19544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aurélien Pupier resolved CAMEL-19544.
-
Fix Version/s: 4.7.0
   Resolution: Fixed

It remains several Thread.sleep but that are testing timeouts of the sql 
component or trying to simulate a little delay so I guess we need to keep them.

> camel-sql: replace Thread.sleep in tests
> 
>
> Key: CAMEL-19544
> URL: https://issues.apache.org/jira/browse/CAMEL-19544
> Project: Camel
>  Issue Type: Task
>  Components: camel-sql, tests
>Affects Versions: 4.0.0
>Reporter: Otavio Rodolfo Piske
>Assignee: Aurélien Pupier
>Priority: Minor
>  Labels: easy, help-wanted
> Fix For: 4.7.0
>
>
> We have many tests which use Thread.sleep for synchronization. This is bug 
> prone and can introduce flakiness when running on environments with different 
> capacities.
> Ideally we should replace these with:
>  * [Awaitility|http://www.awaitility.org/]
>  * Java's native syncronization mechanism (Latches, Phasers, Locks, etc)
>  * Nothing (i.e.; in some cases the sleep can simply be removed)
>  
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Commented] (CAMEL-20835) OOM using RecipientList

2024-06-17 Thread Claus Ibsen (Jira)


[ 
https://issues.apache.org/jira/browse/CAMEL-20835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17855548#comment-17855548
 ] 

Claus Ibsen commented on CAMEL-20835:
-

With a fix then Camel does not leak memory. The biggest objects are (after GC)

 193:           239           5736  org.apache.camel.spi.TypeConvertible

 

 

> OOM using RecipientList
> ---
>
> Key: CAMEL-20835
> URL: https://issues.apache.org/jira/browse/CAMEL-20835
> Project: Camel
>  Issue Type: Bug
>  Components: camel-core
>Affects Versions: 3.10.0, 4.6.0
>Reporter: Arthur Naseef
>Assignee: Claus Ibsen
>Priority: Major
> Fix For: 4.7.0
>
>
> Out Of Memory (OOM) occurs when using the Recipient List with a large number 
> of dynamic URLs.  For example:
>  
>     
> .recipientList(simple("http://{{{}downstream-server{}}}/employee/${header.emplId}";))
>  
> with a large number of values for ${header.emplId} leads to the OOM.
>  
> *REPRODUCER:*
> *=*
> [https://github.com/artnaseef/camel-recipient-list-oom-reproducer]
>  
> See the README.md for instructions to reproduce and detect the problem
>  
> *DETAILS*
> *===*
>  The MulticastProcessor, which RecipientListProcessor extends, has the 
> following "unlimited" cache:
>  
>     private final ConcurrentMap errorHandlers = new 
> ConcurrentHashMap<>();
>  
> Entries are added to this map for every unique processor created - every 
> unique URL generates a unique processor.  The entries themselves are wrapped 
> processor instances for error handling IIUC (to support the custom error 
> handling used by multicast and recipient-list).  Entries are only removed 
> from this map on shutdown.  Ironically, there is an LRUCache for the 
> processors themselves, with a default maximum size of 1000, so the wrapped 
> processors may get recreated even though the error handler remains in the map 
> indefinitely.
>  
> *IMPACT VERSIONS:*
> **
> Appears to impact versions >= 3.10.0
> *COMMIT: 0d9227ff16fb00e047fdd087740c87cce01bb545*
> *===*
> It appears this commit introduced the use of the errorHandlers "unlimited" 
> cache for recipient lists.
>  
> *FOLLOW-UP*
> *==*
> I have ideas and questions for implemeting a fix:
>     - IDEA 1: We can use an LRUCache for this data structure as well.
>     - Does it make more sense to remove the entries from errorHandlers when 
> the related Processor entry is removed from it's LRUCache?
>     - IDEA 2: setting on recipient list to disable the errorHandler cache 
> (for dyamic urls with little chance of duplicates, this could be the best)



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Commented] (CAMEL-20835) OOM using RecipientList

2024-06-17 Thread Claus Ibsen (Jira)


[ 
https://issues.apache.org/jira/browse/CAMEL-20835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17855544#comment-17855544
 ] 

Claus Ibsen commented on CAMEL-20835:
-

But for unlimited HTTP endpoints then use toD which has special optimizations 
for this then it should work out of the box.

> OOM using RecipientList
> ---
>
> Key: CAMEL-20835
> URL: https://issues.apache.org/jira/browse/CAMEL-20835
> Project: Camel
>  Issue Type: Bug
>  Components: camel-core
>Affects Versions: 3.10.0, 4.6.0
>Reporter: Arthur Naseef
>Assignee: Claus Ibsen
>Priority: Major
> Fix For: 4.7.0
>
>
> Out Of Memory (OOM) occurs when using the Recipient List with a large number 
> of dynamic URLs.  For example:
>  
>     
> .recipientList(simple("http://{{{}downstream-server{}}}/employee/${header.emplId}";))
>  
> with a large number of values for ${header.emplId} leads to the OOM.
>  
> *REPRODUCER:*
> *=*
> [https://github.com/artnaseef/camel-recipient-list-oom-reproducer]
>  
> See the README.md for instructions to reproduce and detect the problem
>  
> *DETAILS*
> *===*
>  The MulticastProcessor, which RecipientListProcessor extends, has the 
> following "unlimited" cache:
>  
>     private final ConcurrentMap errorHandlers = new 
> ConcurrentHashMap<>();
>  
> Entries are added to this map for every unique processor created - every 
> unique URL generates a unique processor.  The entries themselves are wrapped 
> processor instances for error handling IIUC (to support the custom error 
> handling used by multicast and recipient-list).  Entries are only removed 
> from this map on shutdown.  Ironically, there is an LRUCache for the 
> processors themselves, with a default maximum size of 1000, so the wrapped 
> processors may get recreated even though the error handler remains in the map 
> indefinitely.
>  
> *IMPACT VERSIONS:*
> **
> Appears to impact versions >= 3.10.0
> *COMMIT: 0d9227ff16fb00e047fdd087740c87cce01bb545*
> *===*
> It appears this commit introduced the use of the errorHandlers "unlimited" 
> cache for recipient lists.
>  
> *FOLLOW-UP*
> *==*
> I have ideas and questions for implemeting a fix:
>     - IDEA 1: We can use an LRUCache for this data structure as well.
>     - Does it make more sense to remove the entries from errorHandlers when 
> the related Processor entry is removed from it's LRUCache?
>     - IDEA 2: setting on recipient list to disable the errorHandler cache 
> (for dyamic urls with little chance of duplicates, this could be the best)



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Commented] (CAMEL-20835) OOM using RecipientList

2024-06-17 Thread Claus Ibsen (Jira)


[ 
https://issues.apache.org/jira/browse/CAMEL-20835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17855542#comment-17855542
 ] 

Claus Ibsen commented on CAMEL-20835:
-

But yeah we should use a LRUCache with same cache size as recipient list out of 
the box. And also allow to turn it off, if you set the size to -1

> OOM using RecipientList
> ---
>
> Key: CAMEL-20835
> URL: https://issues.apache.org/jira/browse/CAMEL-20835
> Project: Camel
>  Issue Type: Bug
>  Components: camel-core
>Affects Versions: 3.10.0, 4.6.0
>Reporter: Arthur Naseef
>Assignee: Claus Ibsen
>Priority: Major
> Fix For: 4.7.0
>
>
> Out Of Memory (OOM) occurs when using the Recipient List with a large number 
> of dynamic URLs.  For example:
>  
>     
> .recipientList(simple("http://{{{}downstream-server{}}}/employee/${header.emplId}";))
>  
> with a large number of values for ${header.emplId} leads to the OOM.
>  
> *REPRODUCER:*
> *=*
> [https://github.com/artnaseef/camel-recipient-list-oom-reproducer]
>  
> See the README.md for instructions to reproduce and detect the problem
>  
> *DETAILS*
> *===*
>  The MulticastProcessor, which RecipientListProcessor extends, has the 
> following "unlimited" cache:
>  
>     private final ConcurrentMap errorHandlers = new 
> ConcurrentHashMap<>();
>  
> Entries are added to this map for every unique processor created - every 
> unique URL generates a unique processor.  The entries themselves are wrapped 
> processor instances for error handling IIUC (to support the custom error 
> handling used by multicast and recipient-list).  Entries are only removed 
> from this map on shutdown.  Ironically, there is an LRUCache for the 
> processors themselves, with a default maximum size of 1000, so the wrapped 
> processors may get recreated even though the error handler remains in the map 
> indefinitely.
>  
> *IMPACT VERSIONS:*
> **
> Appears to impact versions >= 3.10.0
> *COMMIT: 0d9227ff16fb00e047fdd087740c87cce01bb545*
> *===*
> It appears this commit introduced the use of the errorHandlers "unlimited" 
> cache for recipient lists.
>  
> *FOLLOW-UP*
> *==*
> I have ideas and questions for implemeting a fix:
>     - IDEA 1: We can use an LRUCache for this data structure as well.
>     - Does it make more sense to remove the entries from errorHandlers when 
> the related Processor entry is removed from it's LRUCache?
>     - IDEA 2: setting on recipient list to disable the errorHandler cache 
> (for dyamic urls with little chance of duplicates, this could be the best)



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Updated] (CAMEL-19659) camel-fhir: slow tests

2024-06-17 Thread Jira


 [ 
https://issues.apache.org/jira/browse/CAMEL-19659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aurélien Pupier updated CAMEL-19659:

Fix Version/s: 4.7.0

> camel-fhir: slow tests
> --
>
> Key: CAMEL-19659
> URL: https://issues.apache.org/jira/browse/CAMEL-19659
> Project: Camel
>  Issue Type: Task
>  Components: camel-fhir, tests
>Reporter: Otavio Rodolfo Piske
>Priority: Major
>  Labels: easy, help-wanted
> Fix For: 4.7.0
>
> Attachments: screenshot-1.png
>
>
> The integration tests for camel-fhir are extremely slow and resource 
> intensive. They are constantly flaky on Apache CI and causing false negative 
> results. 
> We should investigate how to make these tests faster and less resource 
> intensive.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Resolved] (CAMEL-19659) camel-fhir: slow tests

2024-06-17 Thread Jira


 [ 
https://issues.apache.org/jira/browse/CAMEL-19659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aurélien Pupier resolved CAMEL-19659.
-
Resolution: Fixed

> camel-fhir: slow tests
> --
>
> Key: CAMEL-19659
> URL: https://issues.apache.org/jira/browse/CAMEL-19659
> Project: Camel
>  Issue Type: Task
>  Components: camel-fhir, tests
>Reporter: Otavio Rodolfo Piske
>Priority: Major
>  Labels: easy, help-wanted
> Fix For: 4.7.0
>
> Attachments: screenshot-1.png
>
>
> The integration tests for camel-fhir are extremely slow and resource 
> intensive. They are constantly flaky on Apache CI and causing false negative 
> results. 
> We should investigate how to make these tests faster and less resource 
> intensive.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Updated] (CAMEL-19659) camel-fhir: slow tests

2024-06-17 Thread Jira


 [ 
https://issues.apache.org/jira/browse/CAMEL-19659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aurélien Pupier updated CAMEL-19659:

Attachment: screenshot-1.png

> camel-fhir: slow tests
> --
>
> Key: CAMEL-19659
> URL: https://issues.apache.org/jira/browse/CAMEL-19659
> Project: Camel
>  Issue Type: Task
>  Components: camel-fhir, tests
>Reporter: Otavio Rodolfo Piske
>Priority: Major
>  Labels: easy, help-wanted
> Attachments: screenshot-1.png
>
>
> The integration tests for camel-fhir are extremely slow and resource 
> intensive. They are constantly flaky on Apache CI and causing false negative 
> results. 
> We should investigate how to make these tests faster and less resource 
> intensive.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Commented] (CAMEL-19659) camel-fhir: slow tests

2024-06-17 Thread Jira


[ 
https://issues.apache.org/jira/browse/CAMEL-19659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17855537#comment-17855537
 ] 

Aurélien Pupier commented on CAMEL-19659:
-

After upgrade, the timing seems good for Integration tests:

 !screenshot-1.png! 

> camel-fhir: slow tests
> --
>
> Key: CAMEL-19659
> URL: https://issues.apache.org/jira/browse/CAMEL-19659
> Project: Camel
>  Issue Type: Task
>  Components: camel-fhir, tests
>Reporter: Otavio Rodolfo Piske
>Priority: Major
>  Labels: easy, help-wanted
> Attachments: screenshot-1.png
>
>
> The integration tests for camel-fhir are extremely slow and resource 
> intensive. They are constantly flaky on Apache CI and causing false negative 
> results. 
> We should investigate how to make these tests faster and less resource 
> intensive.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Updated] (CAMEL-20835) OOM using RecipientList

2024-06-17 Thread Claus Ibsen (Jira)


 [ 
https://issues.apache.org/jira/browse/CAMEL-20835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Claus Ibsen updated CAMEL-20835:

Fix Version/s: 4.7.0

> OOM using RecipientList
> ---
>
> Key: CAMEL-20835
> URL: https://issues.apache.org/jira/browse/CAMEL-20835
> Project: Camel
>  Issue Type: Bug
>  Components: camel-core
>Affects Versions: 3.10.0, 4.6.0
>Reporter: Arthur Naseef
>Priority: Major
> Fix For: 4.7.0
>
>
> Out Of Memory (OOM) occurs when using the Recipient List with a large number 
> of dynamic URLs.  For example:
>  
>     
> .recipientList(simple("http://{{{}downstream-server{}}}/employee/${header.emplId}";))
>  
> with a large number of values for ${header.emplId} leads to the OOM.
>  
> *REPRODUCER:*
> *=*
> [https://github.com/artnaseef/camel-recipient-list-oom-reproducer]
>  
> See the README.md for instructions to reproduce and detect the problem
>  
> *DETAILS*
> *===*
>  The MulticastProcessor, which RecipientListProcessor extends, has the 
> following "unlimited" cache:
>  
>     private final ConcurrentMap errorHandlers = new 
> ConcurrentHashMap<>();
>  
> Entries are added to this map for every unique processor created - every 
> unique URL generates a unique processor.  The entries themselves are wrapped 
> processor instances for error handling IIUC (to support the custom error 
> handling used by multicast and recipient-list).  Entries are only removed 
> from this map on shutdown.  Ironically, there is an LRUCache for the 
> processors themselves, with a default maximum size of 1000, so the wrapped 
> processors may get recreated even though the error handler remains in the map 
> indefinitely.
>  
> *IMPACT VERSIONS:*
> **
> Appears to impact versions >= 3.10.0
> *COMMIT: 0d9227ff16fb00e047fdd087740c87cce01bb545*
> *===*
> It appears this commit introduced the use of the errorHandlers "unlimited" 
> cache for recipient lists.
>  
> *FOLLOW-UP*
> *==*
> I have ideas and questions for implemeting a fix:
>     - IDEA 1: We can use an LRUCache for this data structure as well.
>     - Does it make more sense to remove the entries from errorHandlers when 
> the related Processor entry is removed from it's LRUCache?
>     - IDEA 2: setting on recipient list to disable the errorHandler cache 
> (for dyamic urls with little chance of duplicates, this could be the best)



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Assigned] (CAMEL-20835) OOM using RecipientList

2024-06-17 Thread Claus Ibsen (Jira)


 [ 
https://issues.apache.org/jira/browse/CAMEL-20835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Claus Ibsen reassigned CAMEL-20835:
---

Assignee: Claus Ibsen

> OOM using RecipientList
> ---
>
> Key: CAMEL-20835
> URL: https://issues.apache.org/jira/browse/CAMEL-20835
> Project: Camel
>  Issue Type: Bug
>  Components: camel-core
>Affects Versions: 3.10.0, 4.6.0
>Reporter: Arthur Naseef
>Assignee: Claus Ibsen
>Priority: Major
> Fix For: 4.7.0
>
>
> Out Of Memory (OOM) occurs when using the Recipient List with a large number 
> of dynamic URLs.  For example:
>  
>     
> .recipientList(simple("http://{{{}downstream-server{}}}/employee/${header.emplId}";))
>  
> with a large number of values for ${header.emplId} leads to the OOM.
>  
> *REPRODUCER:*
> *=*
> [https://github.com/artnaseef/camel-recipient-list-oom-reproducer]
>  
> See the README.md for instructions to reproduce and detect the problem
>  
> *DETAILS*
> *===*
>  The MulticastProcessor, which RecipientListProcessor extends, has the 
> following "unlimited" cache:
>  
>     private final ConcurrentMap errorHandlers = new 
> ConcurrentHashMap<>();
>  
> Entries are added to this map for every unique processor created - every 
> unique URL generates a unique processor.  The entries themselves are wrapped 
> processor instances for error handling IIUC (to support the custom error 
> handling used by multicast and recipient-list).  Entries are only removed 
> from this map on shutdown.  Ironically, there is an LRUCache for the 
> processors themselves, with a default maximum size of 1000, so the wrapped 
> processors may get recreated even though the error handler remains in the map 
> indefinitely.
>  
> *IMPACT VERSIONS:*
> **
> Appears to impact versions >= 3.10.0
> *COMMIT: 0d9227ff16fb00e047fdd087740c87cce01bb545*
> *===*
> It appears this commit introduced the use of the errorHandlers "unlimited" 
> cache for recipient lists.
>  
> *FOLLOW-UP*
> *==*
> I have ideas and questions for implemeting a fix:
>     - IDEA 1: We can use an LRUCache for this data structure as well.
>     - Does it make more sense to remove the entries from errorHandlers when 
> the related Processor entry is removed from it's LRUCache?
>     - IDEA 2: setting on recipient list to disable the errorHandler cache 
> (for dyamic urls with little chance of duplicates, this could be the best)



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Commented] (CAMEL-20835) OOM using RecipientList

2024-06-17 Thread Claus Ibsen (Jira)


[ 
https://issues.apache.org/jira/browse/CAMEL-20835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17855530#comment-17855530
 ] 

Claus Ibsen commented on CAMEL-20835:
-

You should turn off the cache on recipient list / toD, or use a header with the 
HTTP_PATH and a static endpoint - when you essentially have unique endpoints

> OOM using RecipientList
> ---
>
> Key: CAMEL-20835
> URL: https://issues.apache.org/jira/browse/CAMEL-20835
> Project: Camel
>  Issue Type: Bug
>  Components: camel-core
>Affects Versions: 3.10.0, 4.6.0
>Reporter: Arthur Naseef
>Priority: Major
>
> Out Of Memory (OOM) occurs when using the Recipient List with a large number 
> of dynamic URLs.  For example:
>  
>     
> .recipientList(simple("http://{{{}downstream-server{}}}/employee/${header.emplId}";))
>  
> with a large number of values for ${header.emplId} leads to the OOM.
>  
> *REPRODUCER:*
> *=*
> [https://github.com/artnaseef/camel-recipient-list-oom-reproducer]
>  
> See the README.md for instructions to reproduce and detect the problem
>  
> *DETAILS*
> *===*
>  The MulticastProcessor, which RecipientListProcessor extends, has the 
> following "unlimited" cache:
>  
>     private final ConcurrentMap errorHandlers = new 
> ConcurrentHashMap<>();
>  
> Entries are added to this map for every unique processor created - every 
> unique URL generates a unique processor.  The entries themselves are wrapped 
> processor instances for error handling IIUC (to support the custom error 
> handling used by multicast and recipient-list).  Entries are only removed 
> from this map on shutdown.  Ironically, there is an LRUCache for the 
> processors themselves, with a default maximum size of 1000, so the wrapped 
> processors may get recreated even though the error handler remains in the map 
> indefinitely.
>  
> *IMPACT VERSIONS:*
> **
> Appears to impact versions >= 3.10.0
> *COMMIT: 0d9227ff16fb00e047fdd087740c87cce01bb545*
> *===*
> It appears this commit introduced the use of the errorHandlers "unlimited" 
> cache for recipient lists.
>  
> *FOLLOW-UP*
> *==*
> I have ideas and questions for implemeting a fix:
>     - IDEA 1: We can use an LRUCache for this data structure as well.
>     - Does it make more sense to remove the entries from errorHandlers when 
> the related Processor entry is removed from it's LRUCache?
>     - IDEA 2: setting on recipient list to disable the errorHandler cache 
> (for dyamic urls with little chance of duplicates, this could be the best)



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Resolved] (CAMEL-20500) camel-jbang - Move generate commands to its own plugin

2024-06-17 Thread Claus Ibsen (Jira)


 [ 
https://issues.apache.org/jira/browse/CAMEL-20500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Claus Ibsen resolved CAMEL-20500.
-
Resolution: Fixed

> camel-jbang - Move generate commands to its own plugin
> --
>
> Key: CAMEL-20500
> URL: https://issues.apache.org/jira/browse/CAMEL-20500
> Project: Camel
>  Issue Type: Improvement
>  Components: camel-jbang
>Reporter: Claus Ibsen
>Assignee: Claus Ibsen
>Priority: Major
> Fix For: 4.7.0
>
>
> Today we have a generate command that can generate openapi spec and pojo 
> classes for rest-dsl routes and whatnot. 
> There may be more generate commands in the future. We could consider moving 
> those into its own plugin. As today they have dependencies in the pom.xml 
> that is not needed for usual camel jbang run / export etc. And this will just 
> bloat up the classpath.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Comment Edited] (CAMEL-20864) camel-kafka - With confluent schema registry does not work properly.

2024-06-17 Thread Claus Ibsen (Jira)


[ 
https://issues.apache.org/jira/browse/CAMEL-20864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17855517#comment-17855517
 ] 

Claus Ibsen edited comment on CAMEL-20864 at 6/17/24 7:39 AM:
--

You are welcome to add docs, click "edit this page" in the top right

[https://camel.apache.org/components/next/kafka-component.html]

 

And send the doc changes as PR


was (Author: davsclaus):
You are welcome to add docs, click "edit this page" in the bottom

[https://camel.apache.org/components/next/kafka-component.html]

 

And send the doc changes as PR

> camel-kafka - With confluent schema registry does not work properly.
> 
>
> Key: CAMEL-20864
> URL: https://issues.apache.org/jira/browse/CAMEL-20864
> Project: Camel
>  Issue Type: Bug
>  Components: camel-kafka
>Affects Versions: 3.22.2
> Environment: Confluent Kafka
> Camel-3.22.x
> Java 17
>Reporter: Kartik
>Assignee: Claus Ibsen
>Priority: Minor
>  Labels: camel-kafka
> Fix For: 3.21.5, 3.22.3, 4.0.6, 4.4.3, 4.7.0
>
> Attachments: image-2024-06-12-11-05-12-616.png, 
> image-2024-06-12-11-06-33-915.png, image-2024-06-12-12-51-58-966.png
>
>
> In confluent kafka, we can register the topic against schema validation from 
> the schema registry. When configured the confluent document says either we 
> should have a pojo object defined in the code that is used for 
> serialization/deserialization *OR* a custom "ObjectNode" can be created from 
> their schema utils. Attaching the document below
> [https://docs.confluent.io/platform/current/schema-registry/fundamentals/serdes-develop/serdes-json.html#sending-a-jsonnode-payload]
>  
>  
> For our case, we have a different schema registered and can't have all the 
> POJO defined as schema registered at run time, so we are using the below code 
> to generate an object from the schema.
> {code:java}
> Map config = new HashMap<>();
> config.put("basic.auth.credentials.source", "USER_INFO");
> config.put("basic.auth.user.info", "");
> config.put("auto.register.schemas", false);
> config.put("use.latest.version", true);
> CachedSchemaRegistryClient registryClient = new 
> CachedSchemaRegistryClient("", 10, config);
> String schemaDoc = 
> registryClient.getLatestSchemaMetadata("topicTest-value").getSchema();
> JsonSchema schema = new JsonSchema(schemaDoc);
> ObjectMapper mapper = new ObjectMapper();
> JsonNode jsonNode = 
> mapper.readTree("{\"myField1\":123,\"myField2\":123.32,\"myField3\":\"pqr\"}");
> ObjectNode envelope = JsonSchemaUtils.envelope(schema, jsonNode);
> from("timer://foo?fixedRate=true&period=6")
> .setBody(ExpressionBuilder.constantExpression(envelope))
> .log("Sent new message")// Message to send
> .to(kafkaEndpoint); {code}
> If the "ObjectNode" payload is directly written using kafka-client library it 
> works. But when written using camel component, The "KafkaProducer" in camel 
> component does "isIterable" check and if true sends each value and this 
> doesn't work for confluent kafka as the custom 
> "{*}io.confluent.kafka.serializers.json.KafkaJsonSchemaSerializer{*}" expects 
> a whole object.
> !image-2024-06-12-11-05-12-616.png|width=733,height=459!
>  
>  The code in "
> io.confluent.kafka.serializers.json.KafkaJsonSchemaSerializer expects whole 
> object.
> !image-2024-06-12-11-06-33-915.png|width=906,height=438!
>  
> Basically, in simple words, The "envelope" object created is no longer the 
> same object but is iterated and values are iterated and sent independently 
> resulting in schema validation error.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Commented] (CAMEL-20864) camel-kafka - With confluent schema registry does not work properly.

2024-06-17 Thread Claus Ibsen (Jira)


[ 
https://issues.apache.org/jira/browse/CAMEL-20864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17855517#comment-17855517
 ] 

Claus Ibsen commented on CAMEL-20864:
-

You are welcome to add docs, click "edit this page" in the bottom

[https://camel.apache.org/components/next/kafka-component.html]

 

And send the doc changes as PR

> camel-kafka - With confluent schema registry does not work properly.
> 
>
> Key: CAMEL-20864
> URL: https://issues.apache.org/jira/browse/CAMEL-20864
> Project: Camel
>  Issue Type: Bug
>  Components: camel-kafka
>Affects Versions: 3.22.2
> Environment: Confluent Kafka
> Camel-3.22.x
> Java 17
>Reporter: Kartik
>Assignee: Claus Ibsen
>Priority: Minor
>  Labels: camel-kafka
> Fix For: 3.21.5, 3.22.3, 4.0.6, 4.4.3, 4.7.0
>
> Attachments: image-2024-06-12-11-05-12-616.png, 
> image-2024-06-12-11-06-33-915.png, image-2024-06-12-12-51-58-966.png
>
>
> In confluent kafka, we can register the topic against schema validation from 
> the schema registry. When configured the confluent document says either we 
> should have a pojo object defined in the code that is used for 
> serialization/deserialization *OR* a custom "ObjectNode" can be created from 
> their schema utils. Attaching the document below
> [https://docs.confluent.io/platform/current/schema-registry/fundamentals/serdes-develop/serdes-json.html#sending-a-jsonnode-payload]
>  
>  
> For our case, we have a different schema registered and can't have all the 
> POJO defined as schema registered at run time, so we are using the below code 
> to generate an object from the schema.
> {code:java}
> Map config = new HashMap<>();
> config.put("basic.auth.credentials.source", "USER_INFO");
> config.put("basic.auth.user.info", "");
> config.put("auto.register.schemas", false);
> config.put("use.latest.version", true);
> CachedSchemaRegistryClient registryClient = new 
> CachedSchemaRegistryClient("", 10, config);
> String schemaDoc = 
> registryClient.getLatestSchemaMetadata("topicTest-value").getSchema();
> JsonSchema schema = new JsonSchema(schemaDoc);
> ObjectMapper mapper = new ObjectMapper();
> JsonNode jsonNode = 
> mapper.readTree("{\"myField1\":123,\"myField2\":123.32,\"myField3\":\"pqr\"}");
> ObjectNode envelope = JsonSchemaUtils.envelope(schema, jsonNode);
> from("timer://foo?fixedRate=true&period=6")
> .setBody(ExpressionBuilder.constantExpression(envelope))
> .log("Sent new message")// Message to send
> .to(kafkaEndpoint); {code}
> If the "ObjectNode" payload is directly written using kafka-client library it 
> works. But when written using camel component, The "KafkaProducer" in camel 
> component does "isIterable" check and if true sends each value and this 
> doesn't work for confluent kafka as the custom 
> "{*}io.confluent.kafka.serializers.json.KafkaJsonSchemaSerializer{*}" expects 
> a whole object.
> !image-2024-06-12-11-05-12-616.png|width=733,height=459!
>  
>  The code in "
> io.confluent.kafka.serializers.json.KafkaJsonSchemaSerializer expects whole 
> object.
> !image-2024-06-12-11-06-33-915.png|width=906,height=438!
>  
> Basically, in simple words, The "envelope" object created is no longer the 
> same object but is iterated and values are iterated and sent independently 
> resulting in schema validation error.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Created] (CAMEL-20880) Move AI related component to camel-ai middle folder

2024-06-17 Thread Andrea Cosentino (Jira)
Andrea Cosentino created CAMEL-20880:


 Summary: Move AI related component to camel-ai middle folder
 Key: CAMEL-20880
 URL: https://issues.apache.org/jira/browse/CAMEL-20880
 Project: Camel
  Issue Type: Task
Reporter: Andrea Cosentino
Assignee: Andrea Cosentino
 Fix For: 4.7.0






--
This message was sent by Atlassian Jira
(v8.20.10#820010)