[jira] [Assigned] (BEAM-10183) Update links for RabbitMqIO and KuduIO IO doc to Javadoc
[ https://issues.apache.org/jira/browse/BEAM-10183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Romanenko reassigned BEAM-10183: --- Assignee: Alexey Romanenko > Update links for RabbitMqIO and KuduIO IO doc to Javadoc > > > Key: BEAM-10183 > URL: https://issues.apache.org/jira/browse/BEAM-10183 > Project: Beam > Issue Type: Improvement > Components: website >Reporter: Alexey Romanenko >Assignee: Alexey Romanenko >Priority: P4 > > [IO doc page|https://beam.apache.org/documentation/io/built-in/] - links > should be updated once Beam 2.23 will be released and Javadoc for RabbitMqIO > and KuduIO will be generated -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (BEAM-10183) Update links for RabbitMqIO and KuduIO IO doc to Javadoc
Alexey Romanenko created BEAM-10183: --- Summary: Update links for RabbitMqIO and KuduIO IO doc to Javadoc Key: BEAM-10183 URL: https://issues.apache.org/jira/browse/BEAM-10183 Project: Beam Issue Type: Improvement Components: website Reporter: Alexey Romanenko [IO doc page|https://beam.apache.org/documentation/io/built-in/] - links should be updated once Beam 2.23 will be released and Javadoc for RabbitMqIO and KuduIO will be generated -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (BEAM-9742) Add ability to pass FluentBackoff to JdbcIo.Write
[ https://issues.apache.org/jira/browse/BEAM-9742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17123706#comment-17123706 ] Alexey Romanenko edited comment on BEAM-9742 at 6/2/20, 12:22 PM: -- UPD: PRs review is in process. [~aiyangar] Could you change a status of this Jira to "IN PROGRESS"? Thanks. was (Author: aromanenko): UPD: PRs review is in process. > Add ability to pass FluentBackoff to JdbcIo.Write > - > > Key: BEAM-9742 > URL: https://issues.apache.org/jira/browse/BEAM-9742 > Project: Beam > Issue Type: Improvement > Components: io-java-jdbc >Reporter: Akshay Iyangar >Assignee: Akshay Iyangar >Priority: P3 > Time Spent: 3h 10m > Remaining Estimate: 0h > > Currently, the FluentBackoff is hardcoded with `maxRetries` and > `initialBackoff` . > It would be helpful if the client were able to pass these values. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work started] (BEAM-6515) Unable to read messages from RabbitMQ as a continuous stream
[
https://issues.apache.org/jira/browse/BEAM-6515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on BEAM-6515 started by Alexey Romanenko.
--
> Unable to read messages from RabbitMQ as a continuous stream
> -
>
> Key: BEAM-6515
> URL: https://issues.apache.org/jira/browse/BEAM-6515
> Project: Beam
> Issue Type: Bug
> Components: io-java-rabbitmq
>Reporter: Edin
>Assignee: Alexey Romanenko
>Priority: P2
> Labels: stale-assigned
>
> The RabbitMqIO connector does not work entirely as expected. Not passing
> {{withMaxReadTime}} gives a NullPointer exception. Exact error:
> {code:java}
> Exception in thread "main" java.lang.NullPointerException
> at
> org.apache.beam.runners.direct.UnboundedReadEvaluatorFactory$UnboundedReadEvaluator.processElement(UnboundedReadEvaluatorFactory.java:169)
> at
> org.apache.beam.runners.direct.DirectTransformExecutor.processElements(DirectTransformExecutor.java:160)
> at
> org.apache.beam.runners.direct.DirectTransformExecutor.run(DirectTransformExecutor.java:124)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {code}
>
> It seems RabbitMqIO does not cover the case when messages received from
> RabbitMQ have messages of which timestamps are null. On Stackoverflow the
> following workaround is proposed:
> [https://stackoverflow.com/a/54119782]
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (BEAM-9742) Add ability to pass FluentBackoff to JdbcIo.Write
[ https://issues.apache.org/jira/browse/BEAM-9742?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Romanenko updated BEAM-9742: --- Labels: (was: stale-assigned) > Add ability to pass FluentBackoff to JdbcIo.Write > - > > Key: BEAM-9742 > URL: https://issues.apache.org/jira/browse/BEAM-9742 > Project: Beam > Issue Type: Improvement > Components: io-java-jdbc >Reporter: Akshay Iyangar >Assignee: Akshay Iyangar >Priority: P3 > Time Spent: 3h 10m > Remaining Estimate: 0h > > Currently, the FluentBackoff is hardcoded with `maxRetries` and > `initialBackoff` . > It would be helpful if the client were able to pass these values. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-9742) Add ability to pass FluentBackoff to JdbcIo.Write
[ https://issues.apache.org/jira/browse/BEAM-9742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17123706#comment-17123706 ] Alexey Romanenko commented on BEAM-9742: UPD: PRs review is in process. > Add ability to pass FluentBackoff to JdbcIo.Write > - > > Key: BEAM-9742 > URL: https://issues.apache.org/jira/browse/BEAM-9742 > Project: Beam > Issue Type: Improvement > Components: io-java-jdbc >Reporter: Akshay Iyangar >Assignee: Akshay Iyangar >Priority: P3 > Time Spent: 3h 10m > Remaining Estimate: 0h > > Currently, the FluentBackoff is hardcoded with `maxRetries` and > `initialBackoff` . > It would be helpful if the client were able to pass these values. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (BEAM-5786) KafkaIO should support topic patterns
[
https://issues.apache.org/jira/browse/BEAM-5786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexey Romanenko reassigned BEAM-5786:
--
Assignee: (was: Alexey Romanenko)
> KafkaIO should support topic patterns
> -
>
> Key: BEAM-5786
> URL: https://issues.apache.org/jira/browse/BEAM-5786
> Project: Beam
> Issue Type: Improvement
> Components: io-java-kafka
>Reporter: Alexey Romanenko
>Priority: P2
>
> For the moment, it is possible to set topics for KafkaIO.Read in 2 ways - one
> single topic by using {{withTopic(String)}} or list of topics by using
> {{withTopics(List)}}.
> It would make sense to provide another way for this using patterns, based on
> regular expressions, like {{withTopics(“org_[0-9]+_source”)}}. In this case,
> Kafka consumer could discover the topics based on user provided pattern.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (BEAM-10098) Javadoc export deactivated for RabbitMqIO and KuduIO
[ https://issues.apache.org/jira/browse/BEAM-10098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Romanenko resolved BEAM-10098. - Fix Version/s: Not applicable Resolution: Fixed > Javadoc export deactivated for RabbitMqIO and KuduIO > > > Key: BEAM-10098 > URL: https://issues.apache.org/jira/browse/BEAM-10098 > Project: Beam > Issue Type: Bug > Components: website >Reporter: Ashwin Ramaswami >Assignee: Pablo Estrada >Priority: P2 > Fix For: Not applicable > > Time Spent: 10m > Remaining Estimate: 0h > > Javadoc export is deactivated for RabbitMqIO and KuduIO. We should enable > this. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-10098) Javadoc export deactivated for RabbitMqIO and KuduIO
[ https://issues.apache.org/jira/browse/BEAM-10098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Romanenko updated BEAM-10098: Fix Version/s: (was: Not applicable) 2.23.0 > Javadoc export deactivated for RabbitMqIO and KuduIO > > > Key: BEAM-10098 > URL: https://issues.apache.org/jira/browse/BEAM-10098 > Project: Beam > Issue Type: Bug > Components: website >Reporter: Ashwin Ramaswami >Assignee: Pablo Estrada >Priority: P2 > Fix For: 2.23.0 > > Time Spent: 10m > Remaining Estimate: 0h > > Javadoc export is deactivated for RabbitMqIO and KuduIO. We should enable > this. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (BEAM-9916) Update IOs documentation links
[ https://issues.apache.org/jira/browse/BEAM-9916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Romanenko resolved BEAM-9916. Fix Version/s: Not applicable Resolution: Fixed > Update IOs documentation links > -- > > Key: BEAM-9916 > URL: https://issues.apache.org/jira/browse/BEAM-9916 > Project: Beam > Issue Type: Improvement > Components: website >Reporter: Alexey Romanenko >Assignee: Ashwin Ramaswami >Priority: P3 > Fix For: Not applicable > > Time Spent: 2h > Remaining Estimate: 0h > > Currently, on the webpage > [https://beam.apache.org/documentation/io/built-in/] , we link all IOs to > their code on github, which could be quite odd for users. After a > [discussion|https://lists.apache.org/thread.html/r2c7bb9a2e13f1b8c296930fc8a36edafe731f49b5182567989afcb98%40%3Cdev.beam.apache.org%3E] > on dev@, it was decided to link them to the latest doc reference (like > [https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/FileSystems.html] > for FileIO). > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (BEAM-7770) Add ReadAll transform for SolrIO
[ https://issues.apache.org/jira/browse/BEAM-7770?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Romanenko resolved BEAM-7770. Fix Version/s: 2.23.0 Resolution: Fixed > Add ReadAll transform for SolrIO > > > Key: BEAM-7770 > URL: https://issues.apache.org/jira/browse/BEAM-7770 > Project: Beam > Issue Type: Improvement > Components: io-java-solr >Reporter: Ismaël Mejía >Assignee: Ismaël Mejía >Priority: P3 > Fix For: 2.23.0 > > Time Spent: 1.5h > Remaining Estimate: 0h > > SolrIO already uses internally a composable approach but we need to expose an > explicit ReadAll transform that allows user to create reads in the middle of > the Pipeline to improve composability (e.g. Reads in the middle of a > Pipeline). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-9977) Build Kafka Read on top of Java SplittableDoFn
[ https://issues.apache.org/jira/browse/BEAM-9977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17116859#comment-17116859 ] Alexey Romanenko commented on BEAM-9977: [~boyuanz] I see that there is a PR #11715, that was merged recently, but it doesn't affect KafkaIO directly (as I understand). Do you expect other PRs for this Jira? > Build Kafka Read on top of Java SplittableDoFn > -- > > Key: BEAM-9977 > URL: https://issues.apache.org/jira/browse/BEAM-9977 > Project: Beam > Issue Type: New Feature > Components: io-java-kafka >Reporter: Boyuan Zhang >Assignee: Boyuan Zhang >Priority: P2 > Time Spent: 4.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (BEAM-9916) Update IOs documentation links
[ https://issues.apache.org/jira/browse/BEAM-9916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Romanenko reassigned BEAM-9916: -- Assignee: Ashwin Ramaswami (was: Alexey Romanenko) > Update IOs documentation links > -- > > Key: BEAM-9916 > URL: https://issues.apache.org/jira/browse/BEAM-9916 > Project: Beam > Issue Type: Improvement > Components: website >Reporter: Alexey Romanenko >Assignee: Ashwin Ramaswami >Priority: P3 > Time Spent: 10m > Remaining Estimate: 0h > > Currently, on the webpage > [https://beam.apache.org/documentation/io/built-in/] , we link all IOs to > their code on github, which could be quite odd for users. After a > [discussion|https://lists.apache.org/thread.html/r2c7bb9a2e13f1b8c296930fc8a36edafe731f49b5182567989afcb98%40%3Cdev.beam.apache.org%3E] > on dev@, it was decided to link them to the latest doc reference (like > [https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/FileSystems.html] > for FileIO). > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (BEAM-9916) Update IOs documentation links
[ https://issues.apache.org/jira/browse/BEAM-9916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17116786#comment-17116786 ] Alexey Romanenko edited comment on BEAM-9916 at 5/26/20, 2:40 PM: -- [~epicfaace] Good point, thanks! was (Author: aromanenko): [~epicfaace] Good point, thanks! I'll check them out and add the missing. > Update IOs documentation links > -- > > Key: BEAM-9916 > URL: https://issues.apache.org/jira/browse/BEAM-9916 > Project: Beam > Issue Type: Improvement > Components: website >Reporter: Alexey Romanenko >Assignee: Alexey Romanenko >Priority: P3 > Time Spent: 10m > Remaining Estimate: 0h > > Currently, on the webpage > [https://beam.apache.org/documentation/io/built-in/] , we link all IOs to > their code on github, which could be quite odd for users. After a > [discussion|https://lists.apache.org/thread.html/r2c7bb9a2e13f1b8c296930fc8a36edafe731f49b5182567989afcb98%40%3Cdev.beam.apache.org%3E] > on dev@, it was decided to link them to the latest doc reference (like > [https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/FileSystems.html] > for FileIO). > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (BEAM-9916) Update IOs documentation links
[ https://issues.apache.org/jira/browse/BEAM-9916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Romanenko reassigned BEAM-9916: -- Assignee: Alexey Romanenko > Update IOs documentation links > -- > > Key: BEAM-9916 > URL: https://issues.apache.org/jira/browse/BEAM-9916 > Project: Beam > Issue Type: Improvement > Components: website >Reporter: Alexey Romanenko >Assignee: Alexey Romanenko >Priority: P3 > Time Spent: 10m > Remaining Estimate: 0h > > Currently, on the webpage > [https://beam.apache.org/documentation/io/built-in/] , we link all IOs to > their code on github, which could be quite odd for users. After a > [discussion|https://lists.apache.org/thread.html/r2c7bb9a2e13f1b8c296930fc8a36edafe731f49b5182567989afcb98%40%3Cdev.beam.apache.org%3E] > on dev@, it was decided to link them to the latest doc reference (like > [https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/FileSystems.html] > for FileIO). > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (BEAM-9916) Update IOs documentation links
[ https://issues.apache.org/jira/browse/BEAM-9916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Romanenko reassigned BEAM-9916: -- Assignee: (was: Alexey Romanenko) > Update IOs documentation links > -- > > Key: BEAM-9916 > URL: https://issues.apache.org/jira/browse/BEAM-9916 > Project: Beam > Issue Type: Improvement > Components: website >Reporter: Alexey Romanenko >Priority: P3 > Time Spent: 10m > Remaining Estimate: 0h > > Currently, on the webpage > [https://beam.apache.org/documentation/io/built-in/] , we link all IOs to > their code on github, which could be quite odd for users. After a > [discussion|https://lists.apache.org/thread.html/r2c7bb9a2e13f1b8c296930fc8a36edafe731f49b5182567989afcb98%40%3Cdev.beam.apache.org%3E] > on dev@, it was decided to link them to the latest doc reference (like > [https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/FileSystems.html] > for FileIO). > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-9916) Update IOs documentation links
[ https://issues.apache.org/jira/browse/BEAM-9916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17116786#comment-17116786 ] Alexey Romanenko commented on BEAM-9916: [~epicfaace] Good point, thanks! I'll check them out and add the missing. > Update IOs documentation links > -- > > Key: BEAM-9916 > URL: https://issues.apache.org/jira/browse/BEAM-9916 > Project: Beam > Issue Type: Improvement > Components: website >Reporter: Alexey Romanenko >Assignee: Alexey Romanenko >Priority: P3 > Time Spent: 10m > Remaining Estimate: 0h > > Currently, on the webpage > [https://beam.apache.org/documentation/io/built-in/] , we link all IOs to > their code on github, which could be quite odd for users. After a > [discussion|https://lists.apache.org/thread.html/r2c7bb9a2e13f1b8c296930fc8a36edafe731f49b5182567989afcb98%40%3Cdev.beam.apache.org%3E] > on dev@, it was decided to link them to the latest doc reference (like > [https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/FileSystems.html] > for FileIO). > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-9760) KafkaIO supports consumer group?
[ https://issues.apache.org/jira/browse/BEAM-9760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17105641#comment-17105641 ] Alexey Romanenko commented on BEAM-9760: Hi [~wongkawah], thanks for details. I see your point but I'm not sure that we can use "subscribe()" method in KafkaIO because of some internal Beam limitations. We had a similar discussion a while ago ([see here|https://issues.apache.org/jira/browse/BEAM-5786?focusedCommentId=16655883=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16655883]) > KafkaIO supports consumer group? > > > Key: BEAM-9760 > URL: https://issues.apache.org/jira/browse/BEAM-9760 > Project: Beam > Issue Type: Improvement > Components: io-java-kafka >Reporter: Ka Wah WONG >Priority: Minor > > It seems only assign method of Kafka Consumer class is called in > org.apache.beam.sdk.io.kafka.ConsumerSpEL class. According to documentation > of org.apache.kafka.clients.consumer.KafkaConsumer, manual topic assignment > through this assign method does not use the consumer's group management > functionality. > May I ask if KafkaIO will be enhanced to support consumer's group management > with using Kafka consumer's subscribe method? > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (BEAM-9439) KinesisReader does not report correct backlog statistics
[ https://issues.apache.org/jira/browse/BEAM-9439?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Romanenko resolved BEAM-9439. Fix Version/s: 2.22.0 Resolution: Fixed > KinesisReader does not report correct backlog statistics > - > > Key: BEAM-9439 > URL: https://issues.apache.org/jira/browse/BEAM-9439 > Project: Beam > Issue Type: Bug > Components: io-java-kinesis >Reporter: Sam Whittle >Assignee: Sebastian Graca >Priority: Major > Fix For: 2.22.0 > > Time Spent: 1h 40m > Remaining Estimate: 0h > > The KinesisReader implementing KinesisIO reports backlog by implementing the > UnboundedSource.getTotalBacklogBytes() > method as opposed to the > UnboundedSource.getSplitBacklogBytes() > This value is supposed to represent the total backlog across all shards. > This function is implemented by calling > SimplifiedKinesisClient.getBacklogBytes with the watermark of the kinesis > shards managed within the UnboundedReader instance. As this watermark may be > further ahead than the watermark across all shards, this may miss backlog > bytes. > An additional concern is that the watermark is calculated using a > WatermarkPolicy, which means that the watermark may be inconsistent to the > kinesis timestamp for querying backlog. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9916) Update IOs documentation links
[ https://issues.apache.org/jira/browse/BEAM-9916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Romanenko updated BEAM-9916: --- Description: Currently, on the webpage [https://beam.apache.org/documentation/io/built-in/] , we link all IOs to their code on github, which could be quite odd for users. After a [discussion|https://lists.apache.org/thread.html/r2c7bb9a2e13f1b8c296930fc8a36edafe731f49b5182567989afcb98%40%3Cdev.beam.apache.org%3E] on dev@, it was decided to link them to the latest doc reference (like [https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/FileSystems.html] for FileIO). was: Currently, on the webpage [https://beam.apache.org/documentation/io/built-in/] , we link all IOs to their code on github, which could be quite odd for users. After a [discussion|https://lists.apache.org/thread.html/r2c7bb9a2e13f1b8c296930fc8a36edafe731f49b5182567989afcb98%40%3Cdev.beam.apache.org%3E] on dev@, it was decided to link them to the latest doc reference. > Update IOs documentation links > -- > > Key: BEAM-9916 > URL: https://issues.apache.org/jira/browse/BEAM-9916 > Project: Beam > Issue Type: Improvement > Components: website >Reporter: Alexey Romanenko >Priority: Minor > > Currently, on the webpage > [https://beam.apache.org/documentation/io/built-in/] , we link all IOs to > their code on github, which could be quite odd for users. After a > [discussion|https://lists.apache.org/thread.html/r2c7bb9a2e13f1b8c296930fc8a36edafe731f49b5182567989afcb98%40%3Cdev.beam.apache.org%3E] > on dev@, it was decided to link them to the latest doc reference (like > [https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/FileSystems.html] > for FileIO). > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9916) Update IOs documentation links
[ https://issues.apache.org/jira/browse/BEAM-9916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Romanenko updated BEAM-9916: --- Status: Open (was: Triage Needed) > Update IOs documentation links > -- > > Key: BEAM-9916 > URL: https://issues.apache.org/jira/browse/BEAM-9916 > Project: Beam > Issue Type: Improvement > Components: website >Reporter: Alexey Romanenko >Assignee: Alexey Romanenko >Priority: Minor > > Currently, on the webpage > [https://beam.apache.org/documentation/io/built-in/] , we link all IOs to > their code on github, which could be quite odd for users. After a > [discussion|https://lists.apache.org/thread.html/r2c7bb9a2e13f1b8c296930fc8a36edafe731f49b5182567989afcb98%40%3Cdev.beam.apache.org%3E] > on dev@, it was decided to link them to the latest doc reference (like > [https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/FileSystems.html] > for FileIO). > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (BEAM-9916) Update IOs documentation links
[ https://issues.apache.org/jira/browse/BEAM-9916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Romanenko reassigned BEAM-9916: -- Assignee: Alexey Romanenko > Update IOs documentation links > -- > > Key: BEAM-9916 > URL: https://issues.apache.org/jira/browse/BEAM-9916 > Project: Beam > Issue Type: Improvement > Components: website >Reporter: Alexey Romanenko >Assignee: Alexey Romanenko >Priority: Minor > > Currently, on the webpage > [https://beam.apache.org/documentation/io/built-in/] , we link all IOs to > their code on github, which could be quite odd for users. After a > [discussion|https://lists.apache.org/thread.html/r2c7bb9a2e13f1b8c296930fc8a36edafe731f49b5182567989afcb98%40%3Cdev.beam.apache.org%3E] > on dev@, it was decided to link them to the latest doc reference (like > [https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/FileSystems.html] > for FileIO). > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9916) Update IOs documentation links
[ https://issues.apache.org/jira/browse/BEAM-9916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Romanenko updated BEAM-9916: --- Description: Currently, on the webpage [https://beam.apache.org/documentation/io/built-in/] , we link all IOs to their code on github, which could be quite odd for users. After a [discussion|https://lists.apache.org/thread.html/r2c7bb9a2e13f1b8c296930fc8a36edafe731f49b5182567989afcb98%40%3Cdev.beam.apache.org%3E] on dev@, it was decided to link them to the latest doc reference. was:Currently, on the webpage [https://beam.apache.org/documentation/io/built-in/] , we link all IOs to their code on github, which could be quite odd for users. After a discussion on dev@, it was decided to link them to the latest doc reference. > Update IOs documentation links > -- > > Key: BEAM-9916 > URL: https://issues.apache.org/jira/browse/BEAM-9916 > Project: Beam > Issue Type: Improvement > Components: website >Reporter: Alexey Romanenko >Priority: Minor > > Currently, on the webpage > [https://beam.apache.org/documentation/io/built-in/] , we link all IOs to > their code on github, which could be quite odd for users. After a > [discussion|https://lists.apache.org/thread.html/r2c7bb9a2e13f1b8c296930fc8a36edafe731f49b5182567989afcb98%40%3Cdev.beam.apache.org%3E] > on dev@, it was decided to link them to the latest doc reference. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (BEAM-9916) Update IOs documentation links
Alexey Romanenko created BEAM-9916: -- Summary: Update IOs documentation links Key: BEAM-9916 URL: https://issues.apache.org/jira/browse/BEAM-9916 Project: Beam Issue Type: Improvement Components: website Reporter: Alexey Romanenko Currently, on the webpage [https://beam.apache.org/documentation/io/built-in/] , we link all IOs to their code on github, which could be quite odd for users. After a discussion on dev@, it was decided to link them to the latest doc reference. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (BEAM-8542) Add async write to AWS SNS IO & remove retry logic
[ https://issues.apache.org/jira/browse/BEAM-8542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Romanenko resolved BEAM-8542. Fix Version/s: 2.22.0 Resolution: Fixed > Add async write to AWS SNS IO & remove retry logic > -- > > Key: BEAM-8542 > URL: https://issues.apache.org/jira/browse/BEAM-8542 > Project: Beam > Issue Type: Improvement > Components: io-java-aws >Reporter: Ajo Thomas >Assignee: Ajo Thomas >Priority: Major > Fix For: 2.22.0 > > Time Spent: 11h > Remaining Estimate: 0h > > - While working with SNS IO for one of my work-related projects, I found that > the IO uses synchronous publishes during writes. I had a simple mock pipeline > where I was reading from a kinesis stream and publishing it to SNS using > Beam's SNS IO. For comparison, I also had a lamdba which did the same using > asynchronous publishes but was about 5x faster. Changing the SNS IO to use > async publishes would improve publish latencies. > - SNS IO also has some retry logic which isn't required as SNS clients can > handle retries. The retry logic in the SNS client is user-configurable and > therefore, an explicit retry logic in SNS IO is not required > I have a working version of the IO with these changes, will create a PR > linking this ticket to it once I get some feedback here. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (BEAM-9720) Add custom AWS Http Client Configuration capability for AWS client 1.0/2.0
[ https://issues.apache.org/jira/browse/BEAM-9720?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Romanenko resolved BEAM-9720. Fix Version/s: 2.22.0 Resolution: Fixed > Add custom AWS Http Client Configuration capability for AWS client 1.0/2.0 > -- > > Key: BEAM-9720 > URL: https://issues.apache.org/jira/browse/BEAM-9720 > Project: Beam > Issue Type: Improvement > Components: io-java-aws >Reporter: Akshay Iyangar >Assignee: Akshay Iyangar >Priority: Minor > Fix For: 2.22.0 > > Time Spent: 4h 50m > Remaining Estimate: 0h > > Currently, there is no way to set custom client configuration abilities to > AWS client service. > Enable a way to pass these custom client configuration options as pipeline > options. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-8121) Messages are not distributed per machines when consuming from Kafka topic with 1 partition
[ https://issues.apache.org/jira/browse/BEAM-8121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17091688#comment-17091688 ] Alexey Romanenko commented on BEAM-8121: [~chamikara] Yes, it could be a good idea. Do you mean to have more granularity with split? Could you elaborate a bit? > Messages are not distributed per machines when consuming from Kafka topic > with 1 partition > -- > > Key: BEAM-8121 > URL: https://issues.apache.org/jira/browse/BEAM-8121 > Project: Beam > Issue Type: Bug > Components: io-java-kafka >Affects Versions: 2.14.0 >Reporter: TJ >Assignee: Alexey Romanenko >Priority: Major > Fix For: Not applicable > > Attachments: datalake-dataflow-cleaned.zip > > > Messages are consumed from Kafka using KafkaIO. Each kafka topic contains > only 1 partition. (That means that messages can be consumed only by one > Consumer per 1 consumer group) > When backlog of topic grows and system scales from 1 to X machines, all the > messages seems to be executed on the same machine on which they are read. > Due to that message throughput doesn't increase comparing X machines to 1 > machine. If one machine was reading 2K messages per s, X machines will be > reading the same amount. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-9760) KafkaIO supports consumer group?
[
https://issues.apache.org/jira/browse/BEAM-9760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17091682#comment-17091682
]
Alexey Romanenko commented on BEAM-9760:
Could you elaborate more about your proposal?
For now, consumer group could be assigned by setting a consumer config option
with \{{.withConsumerConfigUpdates(ImmutableMap.of("group.id",
"my_beam_app_1"))}} method.
> KafkaIO supports consumer group?
>
>
> Key: BEAM-9760
> URL: https://issues.apache.org/jira/browse/BEAM-9760
> Project: Beam
> Issue Type: Improvement
> Components: io-java-kafka
>Reporter: Ka Wah WONG
>Priority: Minor
>
> It seems only assign method of Kafka Consumer class is called in
> org.apache.beam.sdk.io.kafka.ConsumerSpEL class. According to documentation
> of org.apache.kafka.clients.consumer.KafkaConsumer, manual topic assignment
> through this assign method does not use the consumer's group management
> functionality.
> May I ask if KafkaIO will be enhanced to support consumer's group management
> with using Kafka consumer's subscribe method?
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (BEAM-9686) SparkCommonPipelineOptions should not depend of a child class to resolve the tmp checkpoint dir
[ https://issues.apache.org/jira/browse/BEAM-9686?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Romanenko resolved BEAM-9686. Fix Version/s: 2.21.0 Resolution: Fixed > SparkCommonPipelineOptions should not depend of a child class to resolve the > tmp checkpoint dir > --- > > Key: BEAM-9686 > URL: https://issues.apache.org/jira/browse/BEAM-9686 > Project: Beam > Issue Type: Bug > Components: runner-spark >Affects Versions: 2.19.0, 2.20.0 >Reporter: Ismaël Mejía >Assignee: Ismaël Mejía >Priority: Minor > Fix For: 2.21.0 > > Time Spent: 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (BEAM-8121) Messages are not distributed per machines when consuming from Kafka topic with 1 partition
[ https://issues.apache.org/jira/browse/BEAM-8121?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Romanenko closed BEAM-8121. -- Fix Version/s: Not applicable Resolution: Not A Problem I close it since the problem was resolved on the user's side. > Messages are not distributed per machines when consuming from Kafka topic > with 1 partition > -- > > Key: BEAM-8121 > URL: https://issues.apache.org/jira/browse/BEAM-8121 > Project: Beam > Issue Type: Bug > Components: io-java-kafka >Affects Versions: 2.14.0 >Reporter: TJ >Priority: Major > Fix For: Not applicable > > Attachments: datalake-dataflow-cleaned.zip > > > Messages are consumed from Kafka using KafkaIO. Each kafka topic contains > only 1 partition. (That means that messages can be consumed only by one > Consumer per 1 consumer group) > When backlog of topic grows and system scales from 1 to X machines, all the > messages seems to be executed on the same machine on which they are read. > Due to that message throughput doesn't increase comparing X machines to 1 > machine. If one machine was reading 2K messages per s, X machines will be > reading the same amount. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (BEAM-8121) Messages are not distributed per machines when consuming from Kafka topic with 1 partition
[ https://issues.apache.org/jira/browse/BEAM-8121?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Romanenko reassigned BEAM-8121: -- Assignee: Alexey Romanenko > Messages are not distributed per machines when consuming from Kafka topic > with 1 partition > -- > > Key: BEAM-8121 > URL: https://issues.apache.org/jira/browse/BEAM-8121 > Project: Beam > Issue Type: Bug > Components: io-java-kafka >Affects Versions: 2.14.0 >Reporter: TJ >Assignee: Alexey Romanenko >Priority: Major > Fix For: Not applicable > > Attachments: datalake-dataflow-cleaned.zip > > > Messages are consumed from Kafka using KafkaIO. Each kafka topic contains > only 1 partition. (That means that messages can be consumed only by one > Consumer per 1 consumer group) > When backlog of topic grows and system scales from 1 to X machines, all the > messages seems to be executed on the same machine on which they are read. > Due to that message throughput doesn't increase comparing X machines to 1 > machine. If one machine was reading 2K messages per s, X machines will be > reading the same amount. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (BEAM-7942) Kafka backward compatibility
[ https://issues.apache.org/jira/browse/BEAM-7942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Romanenko closed BEAM-7942. -- Fix Version/s: (was: 2.8.0) Not applicable Assignee: Alexey Romanenko Resolution: Won't Fix It was fixed on user side. > Kafka backward compatibility > - > > Key: BEAM-7942 > URL: https://issues.apache.org/jira/browse/BEAM-7942 > Project: Beam > Issue Type: Improvement > Components: io-java-kafka >Affects Versions: 2.13.0 >Reporter: Reenu Saluja >Assignee: Alexey Romanenko >Priority: Major > Fix For: Not applicable > > > getting below error using kafka 1.0 with Beam 2.13.0 > Uncaught throwable from user code: java.lang.IllegalStateException: No > TransformEvaluator registered for BOUNDED transform Read(CreateSource) > at > org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState(Preconditions.java:518) > So I had to downgrade beam version to 2.8.0 to work it. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-8121) Messages are not distributed per machines when consuming from Kafka topic with 1 partition
[ https://issues.apache.org/jira/browse/BEAM-8121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17074511#comment-17074511 ] Alexey Romanenko commented on BEAM-8121: [~TauJan] Do you have any updates on this issue? Do you think we can close it? > Messages are not distributed per machines when consuming from Kafka topic > with 1 partition > -- > > Key: BEAM-8121 > URL: https://issues.apache.org/jira/browse/BEAM-8121 > Project: Beam > Issue Type: Bug > Components: io-java-kafka >Affects Versions: 2.14.0 >Reporter: TJ >Priority: Major > Attachments: datalake-dataflow-cleaned.zip > > > Messages are consumed from Kafka using KafkaIO. Each kafka topic contains > only 1 partition. (That means that messages can be consumed only by one > Consumer per 1 consumer group) > When backlog of topic grows and system scales from 1 to X machines, all the > messages seems to be executed on the same machine on which they are read. > Due to that message throughput doesn't increase comparing X machines to 1 > machine. If one machine was reading 2K messages per s, X machines will be > reading the same amount. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (BEAM-9525) KafkaIO attempting to make connection when uploading via Dataflow Runner
[
https://issues.apache.org/jira/browse/BEAM-9525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17072936#comment-17072936
]
Alexey Romanenko edited comment on BEAM-9525 at 4/3/20, 11:25 AM:
--
Do you know what kind of request it sends to broker in the beginning?
I can assume that it tries to fetch a list of topic(s) partitions (the source
splits that are used then to distribute a work across workers). In this case
you can try to use {{withTopicPartitions()}} and provide a list of partitions.
In this case it won't fetch it from broker.
was (Author: aromanenko):
Do you know what kind of request it sends to broker in the beginning?
I can assume that it tries to fetch a list of topic(s) partitions (the source
splits that are used then to distribute a work across workers). In this case
you try to use {{withTopicPartitions()}} and provide a list of partitions. In
this case it won't fetch it from broker.
> KafkaIO attempting to make connection when uploading via Dataflow Runner
>
>
> Key: BEAM-9525
> URL: https://issues.apache.org/jira/browse/BEAM-9525
> Project: Beam
> Issue Type: Bug
> Components: io-java-kafka
>Reporter: Ross Rothenstine
>Priority: Minor
>
> Given our Kafka is running in GCP, within a VPC that is not accessible
> outside of the VPC.
> When uploading a pipeline to GCS using the Dataflow Runner via our CI/CD
> worker, it attempts to make a connection to the broker. Our broker isn't
> available outside the VPC, so this process fails. We can remedy this by
> running a pod from within the VPC to upload, but it causes an awkward step
> within our CI/CD process.
> If there is a more elegant solution to this, such as KafkaIO taking in a
> ValueProvider for bootstrap servers.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (BEAM-7299) Upgrade kafka client for KafkaIO to support new compression zstd
[ https://issues.apache.org/jira/browse/BEAM-7299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Romanenko resolved BEAM-7299. Fix Version/s: Not applicable Resolution: Won't Fix I close this one since it was fixed in other Jira issue and no more feedback was provided. Please, reopen this if there is still a problem. > Upgrade kafka client for KafkaIO to support new compression zstd > > > Key: BEAM-7299 > URL: https://issues.apache.org/jira/browse/BEAM-7299 > Project: Beam > Issue Type: Improvement > Components: io-java-kafka >Affects Versions: 2.12.0 >Reporter: Evgeny >Priority: Major > Fix For: Not applicable > > > Current io-java-kafka dependency contains kafka-clients 1.0.0 which does not > support new compression type of Kafka 2.1.1 "zstd". -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (BEAM-9476) KinesisIO DescribeStream transient errors are not retried
[
https://issues.apache.org/jira/browse/BEAM-9476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexey Romanenko resolved BEAM-9476.
Fix Version/s: 2.21.0
Resolution: Fixed
> KinesisIO DescribeStream transient errors are not retried
> -
>
> Key: BEAM-9476
> URL: https://issues.apache.org/jira/browse/BEAM-9476
> Project: Beam
> Issue Type: Bug
> Components: io-java-kinesis
>Reporter: Aaron Meihm
>Assignee: Alexey Romanenko
>Priority: Major
> Fix For: 2.21.0
>
> Time Spent: 5h
> Remaining Estimate: 0h
>
> During Kinesis stream setup {{DescribeStream}} is used in {{listShards}}.
> This API call has quota limits that can become problematic when attempting to
> configure multiple Kinesis streams in the same AWS account. AWS currently
> limits this call by default to 10 times per second for a given account. With
> multiple streams, this can be hit and results in a {{RuntimeException}} being
> thrown immediately upon the first error.
> Ideally the rate limited call can be retried a number of times before giving
> up instead of failing immediately.
> I have a work in progress PR that resolves this issue at
> https://github.com/apache/beam/pull/10973.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (BEAM-9476) KinesisIO DescribeStream transient errors are not retried
[
https://issues.apache.org/jira/browse/BEAM-9476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexey Romanenko reassigned BEAM-9476:
--
Assignee: Alexey Romanenko
> KinesisIO DescribeStream transient errors are not retried
> -
>
> Key: BEAM-9476
> URL: https://issues.apache.org/jira/browse/BEAM-9476
> Project: Beam
> Issue Type: Bug
> Components: io-java-kinesis
>Reporter: Aaron Meihm
>Assignee: Alexey Romanenko
>Priority: Major
> Time Spent: 5h
> Remaining Estimate: 0h
>
> During Kinesis stream setup {{DescribeStream}} is used in {{listShards}}.
> This API call has quota limits that can become problematic when attempting to
> configure multiple Kinesis streams in the same AWS account. AWS currently
> limits this call by default to 10 times per second for a given account. With
> multiple streams, this can be hit and results in a {{RuntimeException}} being
> thrown immediately upon the first error.
> Ideally the rate limited call can be retried a number of times before giving
> up instead of failing immediately.
> I have a work in progress PR that resolves this issue at
> https://github.com/apache/beam/pull/10973.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (BEAM-9476) KinesisIO DescribeStream transient errors are not retried
[
https://issues.apache.org/jira/browse/BEAM-9476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexey Romanenko updated BEAM-9476:
---
Status: Open (was: Triage Needed)
> KinesisIO DescribeStream transient errors are not retried
> -
>
> Key: BEAM-9476
> URL: https://issues.apache.org/jira/browse/BEAM-9476
> Project: Beam
> Issue Type: Bug
> Components: io-java-kinesis
>Reporter: Aaron Meihm
>Priority: Major
> Time Spent: 5h
> Remaining Estimate: 0h
>
> During Kinesis stream setup {{DescribeStream}} is used in {{listShards}}.
> This API call has quota limits that can become problematic when attempting to
> configure multiple Kinesis streams in the same AWS account. AWS currently
> limits this call by default to 10 times per second for a given account. With
> multiple streams, this can be hit and results in a {{RuntimeException}} being
> thrown immediately upon the first error.
> Ideally the rate limited call can be retried a number of times before giving
> up instead of failing immediately.
> I have a work in progress PR that resolves this issue at
> https://github.com/apache/beam/pull/10973.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (BEAM-7519) Improve work with SSL/TLS certificates in SDK IO, for Google Dataflow
[ https://issues.apache.org/jira/browse/BEAM-7519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17073706#comment-17073706 ] Alexey Romanenko commented on BEAM-7519: I just found this issue by chance, so the answer is quite late, but required functionality in KafkaIO can implemented by writing custom consumer client. More details can be found here BEAM-9627 > Improve work with SSL/TLS certificates in SDK IO, for Google Dataflow > - > > Key: BEAM-7519 > URL: https://issues.apache.org/jira/browse/BEAM-7519 > Project: Beam > Issue Type: Wish > Components: io-ideas, io-java-elasticsearch, io-java-gcp, > io-java-kafka >Reporter: Oleksandr Priadko >Priority: Major > > Sinks like Kafka or Elasticsearch have worked with SSL/TLS. But all of these > can read only certificates from a local machine (use > java.io.File#File(java.lang.String). > In the case of Google Dataflow, we can not easily copy files to a worker, I > see two possible variants to read certificates from the worker: > 1) somehow copy certificates to a worker from GC Storage > 2) in Kafka IO or Elasticsearch IO read certificates directly from GC Storage > Any thoughts about how I can solve this issue easily or mb other variants > will be helpful. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (BEAM-9627) KafkaIO needs better support for SSL
[ https://issues.apache.org/jira/browse/BEAM-9627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17073702#comment-17073702 ] Alexey Romanenko edited comment on BEAM-9627 at 4/2/20, 12:55 PM: -- I close this issue since required functionality can be implemented on user side by creating custom consumer client. We may reopen this issue in case if such client will be contributed back to Beam. was (Author: aromanenko): I close this issue since required functionality can be implemented on user side by creating custom consumer client. > KafkaIO needs better support for SSL > > > Key: BEAM-9627 > URL: https://issues.apache.org/jira/browse/BEAM-9627 > Project: Beam > Issue Type: Improvement > Components: io-java-kafka >Reporter: Daniel Mills >Priority: Minor > Fix For: Not applicable > > > Configuring SSL for kafka requires pointing an option at local files > containing keys and roots of trust as described here: > [https://www.confluent.io/blog/apache-kafka-security-authorization-authentication-encryption/] > Currently, it is somewhat tricky to ensure that these files are written > before KafkaIO starts reading from the source; one potential option would be > to add an init hook where the user could download keys from the keystore of > their choice and write them to local files. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9525) KafkaIO attempting to make connection when uploading via Dataflow Runner
[ https://issues.apache.org/jira/browse/BEAM-9525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Romanenko updated BEAM-9525: --- Status: Open (was: Triage Needed) > KafkaIO attempting to make connection when uploading via Dataflow Runner > > > Key: BEAM-9525 > URL: https://issues.apache.org/jira/browse/BEAM-9525 > Project: Beam > Issue Type: Bug > Components: io-java-kafka >Reporter: Ross Rothenstine >Priority: Minor > > Given our Kafka is running in GCP, within a VPC that is not accessible > outside of the VPC. > When uploading a pipeline to GCS using the Dataflow Runner via our CI/CD > worker, it attempts to make a connection to the broker. Our broker isn't > available outside the VPC, so this process fails. We can remedy this by > running a pod from within the VPC to upload, but it causes an awkward step > within our CI/CD process. > If there is a more elegant solution to this, such as KafkaIO taking in a > ValueProvider for bootstrap servers. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (BEAM-9627) KafkaIO needs better support for SSL
[ https://issues.apache.org/jira/browse/BEAM-9627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Romanenko resolved BEAM-9627. Fix Version/s: Not applicable Resolution: Not A Problem I close this issue since required functionality can be implemented on user side by creating custom consumer client. > KafkaIO needs better support for SSL > > > Key: BEAM-9627 > URL: https://issues.apache.org/jira/browse/BEAM-9627 > Project: Beam > Issue Type: Improvement > Components: io-java-kafka >Reporter: Daniel Mills >Priority: Minor > Fix For: Not applicable > > > Configuring SSL for kafka requires pointing an option at local files > containing keys and roots of trust as described here: > [https://www.confluent.io/blog/apache-kafka-security-authorization-authentication-encryption/] > Currently, it is somewhat tricky to ensure that these files are written > before KafkaIO starts reading from the source; one potential option would be > to add an init hook where the user could download keys from the keystore of > their choice and write them to local files. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-9627) KafkaIO needs better support for SSL
[
https://issues.apache.org/jira/browse/BEAM-9627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17072966#comment-17072966
]
Alexey Romanenko commented on BEAM-9627:
Actually, this hook already exists. You can try to use
{{KafkaIO.Read.withConsumerFactoryFn()}} and implement custom consumer
function, which will be called to create Kafka consumer on worker. In this
function you can copy your files to local temp directories from whatever place
you need.
Some more concrete examples about that approach:
https://stackoverflow.com/questions/42726011/truststore-and-google-cloud-dataflow/53549757
https://stackoverflow.com/questions/54337653/connect-to-kafka-with-ssl-using-kafkaio-on-google-dataflow
> KafkaIO needs better support for SSL
>
>
> Key: BEAM-9627
> URL: https://issues.apache.org/jira/browse/BEAM-9627
> Project: Beam
> Issue Type: Improvement
> Components: io-java-kafka
>Reporter: Daniel Mills
>Priority: Minor
>
> Configuring SSL for kafka requires pointing an option at local files
> containing keys and roots of trust as described here:
> [https://www.confluent.io/blog/apache-kafka-security-authorization-authentication-encryption/]
> Currently, it is somewhat tricky to ensure that these files are written
> before KafkaIO starts reading from the source; one potential option would be
> to add an init hook where the user could download keys from the keystore of
> their choice and write them to local files.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (BEAM-9525) KafkaIO attempting to make connection when uploading via Dataflow Runner
[ https://issues.apache.org/jira/browse/BEAM-9525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Romanenko updated BEAM-9525: --- Status: Triage Needed (was: Open) > KafkaIO attempting to make connection when uploading via Dataflow Runner > > > Key: BEAM-9525 > URL: https://issues.apache.org/jira/browse/BEAM-9525 > Project: Beam > Issue Type: Bug > Components: io-java-kafka >Reporter: Ross Rothenstine >Priority: Minor > > Given our Kafka is running in GCP, within a VPC that is not accessible > outside of the VPC. > When uploading a pipeline to GCS using the Dataflow Runner via our CI/CD > worker, it attempts to make a connection to the broker. Our broker isn't > available outside the VPC, so this process fails. We can remedy this by > running a pod from within the VPC to upload, but it causes an awkward step > within our CI/CD process. > If there is a more elegant solution to this, such as KafkaIO taking in a > ValueProvider for bootstrap servers. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9627) KafkaIO needs better support for SSL
[ https://issues.apache.org/jira/browse/BEAM-9627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Romanenko updated BEAM-9627: --- Status: Open (was: Triage Needed) > KafkaIO needs better support for SSL > > > Key: BEAM-9627 > URL: https://issues.apache.org/jira/browse/BEAM-9627 > Project: Beam > Issue Type: Improvement > Components: io-java-kafka >Reporter: Daniel Mills >Priority: Minor > > Configuring SSL for kafka requires pointing an option at local files > containing keys and roots of trust as described here: > [https://www.confluent.io/blog/apache-kafka-security-authorization-authentication-encryption/] > Currently, it is somewhat tricky to ensure that these files are written > before KafkaIO starts reading from the source; one potential option would be > to add an init hook where the user could download keys from the keystore of > their choice and write them to local files. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9525) KafkaIO attempting to make connection when uploading via Dataflow Runner
[ https://issues.apache.org/jira/browse/BEAM-9525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Romanenko updated BEAM-9525: --- Status: Open (was: Triage Needed) > KafkaIO attempting to make connection when uploading via Dataflow Runner > > > Key: BEAM-9525 > URL: https://issues.apache.org/jira/browse/BEAM-9525 > Project: Beam > Issue Type: Bug > Components: io-java-kafka >Reporter: Ross Rothenstine >Priority: Minor > > Given our Kafka is running in GCP, within a VPC that is not accessible > outside of the VPC. > When uploading a pipeline to GCS using the Dataflow Runner via our CI/CD > worker, it attempts to make a connection to the broker. Our broker isn't > available outside the VPC, so this process fails. We can remedy this by > running a pod from within the VPC to upload, but it causes an awkward step > within our CI/CD process. > If there is a more elegant solution to this, such as KafkaIO taking in a > ValueProvider for bootstrap servers. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-9525) KafkaIO attempting to make connection when uploading via Dataflow Runner
[
https://issues.apache.org/jira/browse/BEAM-9525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17072936#comment-17072936
]
Alexey Romanenko commented on BEAM-9525:
Do you know what kind of request it sends to broker in the beginning?
I can assume that it tries to fetch a list of topic(s) partitions (the source
splits that are used then to distribute a work across workers). In this case
you try to use {{withTopicPartitions()}} and provide a list of partitions. In
this case it won't fetch it from broker.
> KafkaIO attempting to make connection when uploading via Dataflow Runner
>
>
> Key: BEAM-9525
> URL: https://issues.apache.org/jira/browse/BEAM-9525
> Project: Beam
> Issue Type: Bug
> Components: io-java-kafka
>Reporter: Ross Rothenstine
>Priority: Minor
>
> Given our Kafka is running in GCP, within a VPC that is not accessible
> outside of the VPC.
> When uploading a pipeline to GCS using the Dataflow Runner via our CI/CD
> worker, it attempts to make a connection to the broker. Our broker isn't
> available outside the VPC, so this process fails. We can remedy this by
> running a pod from within the VPC to upload, but it causes an awkward step
> within our CI/CD process.
> If there is a more elegant solution to this, such as KafkaIO taking in a
> ValueProvider for bootstrap servers.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (BEAM-9439) KinesisReader does not report correct backlog statistics
[ https://issues.apache.org/jira/browse/BEAM-9439?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Romanenko reassigned BEAM-9439: -- Assignee: Sebastian Graca > KinesisReader does not report correct backlog statistics > - > > Key: BEAM-9439 > URL: https://issues.apache.org/jira/browse/BEAM-9439 > Project: Beam > Issue Type: Bug > Components: io-java-kinesis >Reporter: Sam Whittle >Assignee: Sebastian Graca >Priority: Major > > The KinesisReader implementing KinesisIO reports backlog by implementing the > UnboundedSource.getTotalBacklogBytes() > method as opposed to the > UnboundedSource.getSplitBacklogBytes() > This value is supposed to represent the total backlog across all shards. > This function is implemented by calling > SimplifiedKinesisClient.getBacklogBytes with the watermark of the kinesis > shards managed within the UnboundedReader instance. As this watermark may be > further ahead than the watermark across all shards, this may miss backlog > bytes. > An additional concern is that the watermark is calculated using a > WatermarkPolicy, which means that the watermark may be inconsistent to the > kinesis timestamp for querying backlog. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-9439) KinesisReader does not report correct backlog statistics
[
https://issues.apache.org/jira/browse/BEAM-9439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17071688#comment-17071688
]
Alexey Romanenko commented on BEAM-9439:
[~titmus] Thanks, sounds good to me as a trade-off solution. Perhaps we can
have an +optional+ {{getSplitBacklogBytes()}} implementation based on Enhanced
Shard-level Metrics, as Pawel suggested, but I think it will a different Jira
for that.
I'll assign this Jira to you if you mind, feel free to reassign it to
[~m.juraszek]. And I'll be happy to review a PR once it will be ready.
> KinesisReader does not report correct backlog statistics
> -
>
> Key: BEAM-9439
> URL: https://issues.apache.org/jira/browse/BEAM-9439
> Project: Beam
> Issue Type: Bug
> Components: io-java-kinesis
>Reporter: Sam Whittle
>Priority: Major
>
> The KinesisReader implementing KinesisIO reports backlog by implementing the
> UnboundedSource.getTotalBacklogBytes()
> method as opposed to the
> UnboundedSource.getSplitBacklogBytes()
> This value is supposed to represent the total backlog across all shards.
> This function is implemented by calling
> SimplifiedKinesisClient.getBacklogBytes with the watermark of the kinesis
> shards managed within the UnboundedReader instance. As this watermark may be
> further ahead than the watermark across all shards, this may miss backlog
> bytes.
> An additional concern is that the watermark is calculated using a
> WatermarkPolicy, which means that the watermark may be inconsistent to the
> kinesis timestamp for querying backlog.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (BEAM-9439) KinesisReader does not report correct backlog statistics
[
https://issues.apache.org/jira/browse/BEAM-9439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17071128#comment-17071128
]
Alexey Romanenko edited comment on BEAM-9439 at 3/30/20, 4:41 PM:
--
[~pawel.kaczmarczyk] I think now I see why {{getTotalBacklogBytes()}} was
implemented, thanks.
{quote}As I understand, to fix getTotalBacklogBytes() we would need the
watermark of the whole stream, not just current split, right?
{quote}
I think so if we can't get the number of bytes per shard and implement
\{{getSplitBacklogBytes(}}
In the same time, I doubt that implementation based on Enhanced Shard-level
Metrics should be default since, as you mentioned, it will charge additionally.
Perhaps, it could be an alternative solution for those who wants to use it.
was (Author: aromanenko):
[~pawel.kaczmarczyk] I think now I see why {{getTotalBacklogBytes()}} was
implemented, thanks.
bq. As I understand, to fix getTotalBacklogBytes() we would need the watermark
of the whole stream, not just current split, right?
I think so if we can't get the number of bytes per shard.
In the same time, I doubt that implementation based on Enhanced Shard-level
Metrics should be default since, as you mentioned, it will charge additionally.
Perhaps, it could be an alternative solution for those who wants to use it.
> KinesisReader does not report correct backlog statistics
> -
>
> Key: BEAM-9439
> URL: https://issues.apache.org/jira/browse/BEAM-9439
> Project: Beam
> Issue Type: Bug
> Components: io-java-kinesis
>Reporter: Sam Whittle
>Priority: Major
>
> The KinesisReader implementing KinesisIO reports backlog by implementing the
> UnboundedSource.getTotalBacklogBytes()
> method as opposed to the
> UnboundedSource.getSplitBacklogBytes()
> This value is supposed to represent the total backlog across all shards.
> This function is implemented by calling
> SimplifiedKinesisClient.getBacklogBytes with the watermark of the kinesis
> shards managed within the UnboundedReader instance. As this watermark may be
> further ahead than the watermark across all shards, this may miss backlog
> bytes.
> An additional concern is that the watermark is calculated using a
> WatermarkPolicy, which means that the watermark may be inconsistent to the
> kinesis timestamp for querying backlog.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (BEAM-9439) KinesisReader does not report correct backlog statistics
[
https://issues.apache.org/jira/browse/BEAM-9439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17071128#comment-17071128
]
Alexey Romanenko commented on BEAM-9439:
[~pawel.kaczmarczyk] I think now I see why {{getTotalBacklogBytes()}} was
implemented, thanks.
bq. As I understand, to fix getTotalBacklogBytes() we would need the watermark
of the whole stream, not just current split, right?
I think so if we can't get the number of bytes per shard.
In the same time, I doubt that implementation based on Enhanced Shard-level
Metrics should be default since, as you mentioned, it will charge additionally.
Perhaps, it could be an alternative solution for those who wants to use it.
> KinesisReader does not report correct backlog statistics
> -
>
> Key: BEAM-9439
> URL: https://issues.apache.org/jira/browse/BEAM-9439
> Project: Beam
> Issue Type: Bug
> Components: io-java-kinesis
>Reporter: Sam Whittle
>Priority: Major
>
> The KinesisReader implementing KinesisIO reports backlog by implementing the
> UnboundedSource.getTotalBacklogBytes()
> method as opposed to the
> UnboundedSource.getSplitBacklogBytes()
> This value is supposed to represent the total backlog across all shards.
> This function is implemented by calling
> SimplifiedKinesisClient.getBacklogBytes with the watermark of the kinesis
> shards managed within the UnboundedReader instance. As this watermark may be
> further ahead than the watermark across all shards, this may miss backlog
> bytes.
> An additional concern is that the watermark is calculated using a
> WatermarkPolicy, which means that the watermark may be inconsistent to the
> kinesis timestamp for querying backlog.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (BEAM-9439) KinesisReader does not report correct backlog statistics
[
https://issues.apache.org/jira/browse/BEAM-9439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17071120#comment-17071120
]
Alexey Romanenko commented on BEAM-9439:
[~titmus] Thank for testing this.
I'm not a Dataflow expert but its [autoscaling
documentation|https://cloud.google.com/dataflow/docs/guides/deploying-a-pipeline#autoscaling]
recommends to use total backlog bytes only in case of implementing
{{getTotalBacklogBytes()}} and I'm not sure that it will work properly if every
shard reader will return the total bytes (if I got it right your current fix).
For example, KafkaIO implements {{getSplitBacklogBytes()}} by summing up all
backlog byte sizes for every shard assigned to the reader. It does it
approximately by taking average number of records read between two offsets and
multiplied by average record size.
> KinesisReader does not report correct backlog statistics
> -
>
> Key: BEAM-9439
> URL: https://issues.apache.org/jira/browse/BEAM-9439
> Project: Beam
> Issue Type: Bug
> Components: io-java-kinesis
>Reporter: Sam Whittle
>Priority: Major
>
> The KinesisReader implementing KinesisIO reports backlog by implementing the
> UnboundedSource.getTotalBacklogBytes()
> method as opposed to the
> UnboundedSource.getSplitBacklogBytes()
> This value is supposed to represent the total backlog across all shards.
> This function is implemented by calling
> SimplifiedKinesisClient.getBacklogBytes with the watermark of the kinesis
> shards managed within the UnboundedReader instance. As this watermark may be
> further ahead than the watermark across all shards, this may miss backlog
> bytes.
> An additional concern is that the watermark is calculated using a
> WatermarkPolicy, which means that the watermark may be inconsistent to the
> kinesis timestamp for querying backlog.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (BEAM-9439) KinesisReader does not report correct backlog statistics
[ https://issues.apache.org/jira/browse/BEAM-9439?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Romanenko updated BEAM-9439: --- Status: Open (was: Triage Needed) > KinesisReader does not report correct backlog statistics > - > > Key: BEAM-9439 > URL: https://issues.apache.org/jira/browse/BEAM-9439 > Project: Beam > Issue Type: Bug > Components: io-java-kinesis >Reporter: Sam Whittle >Priority: Major > > The KinesisReader implementing KinesisIO reports backlog by implementing the > UnboundedSource.getTotalBacklogBytes() > method as opposed to the > UnboundedSource.getSplitBacklogBytes() > This value is supposed to represent the total backlog across all shards. > This function is implemented by calling > SimplifiedKinesisClient.getBacklogBytes with the watermark of the kinesis > shards managed within the UnboundedReader instance. As this watermark may be > further ahead than the watermark across all shards, this may miss backlog > bytes. > An additional concern is that the watermark is calculated using a > WatermarkPolicy, which means that the watermark may be inconsistent to the > kinesis timestamp for querying backlog. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (BEAM-9439) KinesisReader does not report correct backlog statistics
[
https://issues.apache.org/jira/browse/BEAM-9439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17068938#comment-17068938
]
Alexey Romanenko edited comment on BEAM-9439 at 3/27/20, 5:56 PM:
--
Thank you for reporting this. Do you have any fix for that or an idea how
{{getSplitBacklogBytes()}} could be implemented?
Also, I'd ask [~pawel.kaczmarczyk], who initially worked on BEAM-2455, why
{{getTotalBacklogBytes()}} was used in this case?
was (Author: aromanenko):
Thank you for reporting this. I'd ask [~pawel.kaczmarczyk], who initially
worked on BEAM-2455, why {{getTotalBacklogBytes()}} was used in this case?
> KinesisReader does not report correct backlog statistics
> -
>
> Key: BEAM-9439
> URL: https://issues.apache.org/jira/browse/BEAM-9439
> Project: Beam
> Issue Type: Bug
> Components: io-java-kinesis
>Reporter: Sam Whittle
>Priority: Major
>
> The KinesisReader implementing KinesisIO reports backlog by implementing the
> UnboundedSource.getTotalBacklogBytes()
> method as opposed to the
> UnboundedSource.getSplitBacklogBytes()
> This value is supposed to represent the total backlog across all shards.
> This function is implemented by calling
> SimplifiedKinesisClient.getBacklogBytes with the watermark of the kinesis
> shards managed within the UnboundedReader instance. As this watermark may be
> further ahead than the watermark across all shards, this may miss backlog
> bytes.
> An additional concern is that the watermark is calculated using a
> WatermarkPolicy, which means that the watermark may be inconsistent to the
> kinesis timestamp for querying backlog.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (BEAM-9439) KinesisReader does not report correct backlog statistics
[
https://issues.apache.org/jira/browse/BEAM-9439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17068938#comment-17068938
]
Alexey Romanenko commented on BEAM-9439:
Thank you for reporting this. I'd ask [~pawel.kaczmarczyk], who initially
worked on BEAM-2455, why {{getTotalBacklogBytes()}} was used in this case?
> KinesisReader does not report correct backlog statistics
> -
>
> Key: BEAM-9439
> URL: https://issues.apache.org/jira/browse/BEAM-9439
> Project: Beam
> Issue Type: Bug
> Components: io-java-kinesis
>Reporter: Sam Whittle
>Priority: Major
>
> The KinesisReader implementing KinesisIO reports backlog by implementing the
> UnboundedSource.getTotalBacklogBytes()
> method as opposed to the
> UnboundedSource.getSplitBacklogBytes()
> This value is supposed to represent the total backlog across all shards.
> This function is implemented by calling
> SimplifiedKinesisClient.getBacklogBytes with the watermark of the kinesis
> shards managed within the UnboundedReader instance. As this watermark may be
> further ahead than the watermark across all shards, this may miss backlog
> bytes.
> An additional concern is that the watermark is calculated using a
> WatermarkPolicy, which means that the watermark may be inconsistent to the
> kinesis timestamp for querying backlog.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (BEAM-9133) CassandraIOTest.classMethod test is still flaky
[
https://issues.apache.org/jira/browse/BEAM-9133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17066687#comment-17066687
]
Alexey Romanenko commented on BEAM-9133:
Is there any update on this? It's still flaky:
[https://builds.apache.org/job/beam_PreCommit_Java_Phrase/1924/testReport/junit/org.apache.beam.sdk.io.cassandra/CassandraIOTest/classMethod/]
> CassandraIOTest.classMethod test is still flaky
> ---
>
> Key: BEAM-9133
> URL: https://issues.apache.org/jira/browse/BEAM-9133
> Project: Beam
> Issue Type: Bug
> Components: io-java-cassandra, test-failures
>Affects Versions: 2.17.0
>Reporter: Alexey Romanenko
>Assignee: Etienne Chauchot
>Priority: Critical
>
> CassandraIOTest is still flaky. For example:
> https://builds.apache.org/job/beam_PreCommit_Java_Phrase/1646/
> https://builds.apache.org/job/beam_PreCommit_Java_Phrase/1625/
> {code}
> Error Message
> java.lang.RuntimeException: Unable to create embedded Cassandra cluster
> Stacktrace
> java.lang.RuntimeException: Unable to create embedded Cassandra cluster
> at
> org.apache.beam.sdk.io.cassandra.CassandraIOTest.buildCluster(CassandraIOTest.java:167)
> at
> org.apache.beam.sdk.io.cassandra.CassandraIOTest.beforeClass(CassandraIOTest.java:146)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> at
> org.junit.internal.runners.statements.RunBefores.invokeMethod(RunBefores.java:33)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:305)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:412)
> at
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.runTestClass(JUnitTestClassExecutor.java:110)
> at
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:58)
> at
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:38)
> at
> org.gradle.api.internal.tasks.testing.junit.AbstractJUnitTestClassProcessor.processTestClass(AbstractJUnitTestClassProcessor.java:62)
> at
> org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:51)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
> at
> org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
> at
> org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:32)
> at
> org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:93)
> at com.sun.proxy.$Proxy2.processTestClass(Unknown Source)
> at
> org.gradle.api.internal.tasks.testing.worker.TestWorker.processTestClass(TestWorker.java:118)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
> at
> org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
> at
> org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:175)
> at
> org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:157)
> at
>
[jira] [Resolved] (BEAM-9420) Configurable timeout for Kafka setupInitialOffset()
[ https://issues.apache.org/jira/browse/BEAM-9420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Romanenko resolved BEAM-9420. Fix Version/s: 2.21.0 Resolution: Fixed > Configurable timeout for Kafka setupInitialOffset() > --- > > Key: BEAM-9420 > URL: https://issues.apache.org/jira/browse/BEAM-9420 > Project: Beam > Issue Type: Bug > Components: io-java-kafka >Affects Versions: 2.19.0 >Reporter: Jozef Vilcek >Assignee: Jozef Vilcek >Priority: Major > Fix For: 2.21.0 > > Time Spent: 1h 50m > Remaining Estimate: 0h > > If bootstrap brokers does contain an unhealthy server, it can break the start > of a whole Beam job. During the start, `KafkaUnboundedReader` is waiting for > `setupInitialOffset()`. Wait timeout is either a double time of `request. > timeout.ms` or some default constant. In both cases, it might not be enough > time for kafka-client to initiate fallback and retry metadata discovery via > another broker from given bootstrap list. > The client should be able to specify timeout for `setupInitialOffset()` > explicitly as a setting to KafkaIO read. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-9420) Configurable timeout for Kafka setupInitialOffset()
[ https://issues.apache.org/jira/browse/BEAM-9420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17065571#comment-17065571 ] Alexey Romanenko commented on BEAM-9420: I left my comments on the PR page. Sorry for delay with response. > Configurable timeout for Kafka setupInitialOffset() > --- > > Key: BEAM-9420 > URL: https://issues.apache.org/jira/browse/BEAM-9420 > Project: Beam > Issue Type: Bug > Components: io-java-kafka >Affects Versions: 2.19.0 >Reporter: Jozef Vilcek >Assignee: Jozef Vilcek >Priority: Major > Time Spent: 1h 10m > Remaining Estimate: 0h > > If bootstrap brokers does contain an unhealthy server, it can break the start > of a whole Beam job. During the start, `KafkaUnboundedReader` is waiting for > `setupInitialOffset()`. Wait timeout is either a double time of `request. > timeout.ms` or some default constant. In both cases, it might not be enough > time for kafka-client to initiate fallback and retry metadata discovery via > another broker from given bootstrap list. > The client should be able to specify timeout for `setupInitialOffset()` > explicitly as a setting to KafkaIO read. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (BEAM-9470) :sdks:java:io:kinesis:test is flaky
[ https://issues.apache.org/jira/browse/BEAM-9470?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Romanenko resolved BEAM-9470. Fix Version/s: 2.21.0 Resolution: Fixed > :sdks:java:io:kinesis:test is flaky > --- > > Key: BEAM-9470 > URL: https://issues.apache.org/jira/browse/BEAM-9470 > Project: Beam > Issue Type: Test > Components: io-java-kinesis >Reporter: Etienne Chauchot >Assignee: Jonothan Farr >Priority: Major > Fix For: 2.21.0 > > Time Spent: 11h 20m > Remaining Estimate: 0h > > [https://scans.gradle.com/s/b4jmmu72ku5jc/console-log?task=:sdks:java:io:kinesis:test] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (BEAM-9279) Make HBase.ReadAll based on Reads instead of HBaseQuery
[ https://issues.apache.org/jira/browse/BEAM-9279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Romanenko resolved BEAM-9279. Fix Version/s: 2.21.0 Resolution: Fixed > Make HBase.ReadAll based on Reads instead of HBaseQuery > --- > > Key: BEAM-9279 > URL: https://issues.apache.org/jira/browse/BEAM-9279 > Project: Beam > Issue Type: Improvement > Components: io-java-hbase >Reporter: Ismaël Mejía >Assignee: Ismaël Mejía >Priority: Minor > Fix For: 2.21.0 > > Time Spent: 3h 20m > Remaining Estimate: 0h > > HBaseIO support for SplittableDoFn introduced a new request type HBaseQuery, > however the attributes defined in that class are already available in > HBase.Read. Allowing users to define pipelines based on HBaseIO.Read allows > to create pipelines that can read from multiple clusters because the > Configuration now is part of the request object. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9470) :sdks:java:io:kinesis:test is flaky
[ https://issues.apache.org/jira/browse/BEAM-9470?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Romanenko updated BEAM-9470: --- Status: Open (was: Triage Needed) > :sdks:java:io:kinesis:test is flaky > --- > > Key: BEAM-9470 > URL: https://issues.apache.org/jira/browse/BEAM-9470 > Project: Beam > Issue Type: Test > Components: io-java-kinesis >Reporter: Etienne Chauchot >Assignee: Jonothan Farr >Priority: Major > Time Spent: 40m > Remaining Estimate: 0h > > [https://scans.gradle.com/s/b4jmmu72ku5jc/console-log?task=:sdks:java:io:kinesis:test] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-9470) :sdks:java:io:kinesis:test is flaky
[
https://issues.apache.org/jira/browse/BEAM-9470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17055169#comment-17055169
]
Alexey Romanenko commented on BEAM-9470:
[https://builds.apache.org/job/beam_PreCommit_Java_Commit/10277/testReport/junit/org.apache.beam.sdk.io.kinesis/ShardReadersPoolTest/shouldCallRateLimitPolicy/]
[~jfarr] Could you take a look on this? I think it fails on test
{{ShardReadersPoolTest.shouldCallRateLimitPolicy}} which was added during a
work on BEAM-8382
> :sdks:java:io:kinesis:test is flaky
> ---
>
> Key: BEAM-9470
> URL: https://issues.apache.org/jira/browse/BEAM-9470
> Project: Beam
> Issue Type: Test
> Components: io-java-kinesis
>Reporter: Etienne Chauchot
>Priority: Major
>
> [https://scans.gradle.com/s/b4jmmu72ku5jc/console-log?task=:sdks:java:io:kinesis:test]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (BEAM-9420) Configurable timeout for Kafka setupInitialOffset()
[
https://issues.apache.org/jira/browse/BEAM-9420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17052385#comment-17052385
]
Alexey Romanenko edited comment on BEAM-9420 at 3/5/20, 5:55 PM:
-
Why increasing a {{request.timeout.ms}} wouldn't work in this case?
was (Author: aromanenko):
Why \{{request.timeout.ms}} wouldn't work in this case?
> Configurable timeout for Kafka setupInitialOffset()
> ---
>
> Key: BEAM-9420
> URL: https://issues.apache.org/jira/browse/BEAM-9420
> Project: Beam
> Issue Type: Bug
> Components: io-java-kafka
>Affects Versions: 2.19.0
>Reporter: Jozef Vilcek
>Assignee: Jozef Vilcek
>Priority: Major
>
> If bootstrap brokers does contain an unhealthy server, it can break the start
> of a whole Beam job. During the start, `KafkaUnboundedReader` is waiting for
> `setupInitialOffset()`. Wait timeout is either a double time of `request.
> timeout.ms` or some default constant. In both cases, it might not be enough
> time for kafka-client to initiate fallback and retry metadata discovery via
> another broker from given bootstrap list.
> The client should be able to specify timeout for `setupInitialOffset()`
> explicitly as a setting to KafkaIO read.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (BEAM-9420) Configurable timeout for Kafka setupInitialOffset()
[
https://issues.apache.org/jira/browse/BEAM-9420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17052385#comment-17052385
]
Alexey Romanenko commented on BEAM-9420:
Why \{{request.timeout.ms}} wouldn't work in this case?
> Configurable timeout for Kafka setupInitialOffset()
> ---
>
> Key: BEAM-9420
> URL: https://issues.apache.org/jira/browse/BEAM-9420
> Project: Beam
> Issue Type: Bug
> Components: io-java-kafka
>Affects Versions: 2.19.0
>Reporter: Jozef Vilcek
>Assignee: Jozef Vilcek
>Priority: Major
>
> If bootstrap brokers does contain an unhealthy server, it can break the start
> of a whole Beam job. During the start, `KafkaUnboundedReader` is waiting for
> `setupInitialOffset()`. Wait timeout is either a double time of `request.
> timeout.ms` or some default constant. In both cases, it might not be enough
> time for kafka-client to initiate fallback and retry metadata discovery via
> another broker from given bootstrap list.
> The client should be able to specify timeout for `setupInitialOffset()`
> explicitly as a setting to KafkaIO read.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (BEAM-8382) Add polling interval to KinesisIO.Read
[
https://issues.apache.org/jira/browse/BEAM-8382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexey Romanenko updated BEAM-8382:
---
Fix Version/s: (was: 2.20.0)
2.21.0
> Add polling interval to KinesisIO.Read
> --
>
> Key: BEAM-8382
> URL: https://issues.apache.org/jira/browse/BEAM-8382
> Project: Beam
> Issue Type: Improvement
> Components: io-java-kinesis
>Affects Versions: 2.13.0, 2.14.0, 2.15.0
>Reporter: Jonothan Farr
>Assignee: Jonothan Farr
>Priority: Major
> Fix For: 2.21.0
>
> Time Spent: 15h 20m
> Remaining Estimate: 0h
>
> With the current implementation we are observing Kinesis throttling due to
> ReadProvisionedThroughputExceeded on the order of hundreds of times per
> second, regardless of the actual Kinesis throughput. This is because the
> ShardReadersPool readLoop() method is polling getRecords() as fast as
> possible.
> From the KDS documentation:
> {quote}Each shard can support up to five read transactions per second.
> {quote}
> and
> {quote}For best results, sleep for at least 1 second (1,000 milliseconds)
> between calls to getRecords to avoid exceeding the limit on getRecords
> frequency.
> {quote}
> [https://docs.aws.amazon.com/streams/latest/dev/service-sizes-and-limits.html]
> [https://docs.aws.amazon.com/streams/latest/dev/developing-consumers-with-sdk.html]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (BEAM-8382) Add polling interval to KinesisIO.Read
[
https://issues.apache.org/jira/browse/BEAM-8382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexey Romanenko resolved BEAM-8382.
Fix Version/s: 2.20.0
Resolution: Fixed
> Add polling interval to KinesisIO.Read
> --
>
> Key: BEAM-8382
> URL: https://issues.apache.org/jira/browse/BEAM-8382
> Project: Beam
> Issue Type: Improvement
> Components: io-java-kinesis
>Affects Versions: 2.13.0, 2.14.0, 2.15.0
>Reporter: Jonothan Farr
>Assignee: Jonothan Farr
>Priority: Major
> Fix For: 2.20.0
>
> Time Spent: 15h 20m
> Remaining Estimate: 0h
>
> With the current implementation we are observing Kinesis throttling due to
> ReadProvisionedThroughputExceeded on the order of hundreds of times per
> second, regardless of the actual Kinesis throughput. This is because the
> ShardReadersPool readLoop() method is polling getRecords() as fast as
> possible.
> From the KDS documentation:
> {quote}Each shard can support up to five read transactions per second.
> {quote}
> and
> {quote}For best results, sleep for at least 1 second (1,000 milliseconds)
> between calls to getRecords to avoid exceeding the limit on getRecords
> frequency.
> {quote}
> [https://docs.aws.amazon.com/streams/latest/dev/service-sizes-and-limits.html]
> [https://docs.aws.amazon.com/streams/latest/dev/developing-consumers-with-sdk.html]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (BEAM-9329) Support request of schemas by version on KafkaIO + Confluent Schema Registry
[ https://issues.apache.org/jira/browse/BEAM-9329?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Romanenko resolved BEAM-9329. Fix Version/s: 2.20.0 Resolution: Fixed > Support request of schemas by version on KafkaIO + Confluent Schema Registry > > > Key: BEAM-9329 > URL: https://issues.apache.org/jira/browse/BEAM-9329 > Project: Beam > Issue Type: Improvement > Components: io-java-kafka >Reporter: Ismaël Mejía >Assignee: Ismaël Mejía >Priority: Minor > Fix For: 2.20.0 > > Time Spent: 2h 40m > Remaining Estimate: 0h > > BEAM-7310 introduced support to get Avro Schemas from Confluent Schema > Registry as part of KafkaIO. By default it always takes the latest (most > recent) version of the Schema. It would be nice to allow users to also > request specific versions of the Schema. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (BEAM-9364) Refactor KafkaIO to use DeserializerProviders
[ https://issues.apache.org/jira/browse/BEAM-9364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Romanenko resolved BEAM-9364. Resolution: Fixed > Refactor KafkaIO to use DeserializerProviders > - > > Key: BEAM-9364 > URL: https://issues.apache.org/jira/browse/BEAM-9364 > Project: Beam > Issue Type: Improvement > Components: io-java-kafka >Reporter: Ismaël Mejía >Assignee: Ismaël Mejía >Priority: Major > Fix For: 2.20.0 > > Time Spent: 3h 20m > Remaining Estimate: 0h > > BEAM-7310 introduced support for Confluent Schema Registry, even if the API > did not explicitly mention DeserializerProviders it opened the option to use > multiple providers of Deserializers. The goal of this ticket is to refactor > the KafkaIO API to reflect this new concept both for local Deserializer > resolution and Confluent Schema Registry based (notice that this one becomes > better isolated). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (BEAM-9281) Update commons-csv to version 1.8
[ https://issues.apache.org/jira/browse/BEAM-9281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Romanenko resolved BEAM-9281. Resolution: Fixed > Update commons-csv to version 1.8 > - > > Key: BEAM-9281 > URL: https://issues.apache.org/jira/browse/BEAM-9281 > Project: Beam > Issue Type: Improvement > Components: build-system >Reporter: Ismaël Mejía >Assignee: Ismaël Mejía >Priority: Trivial > Fix For: 2.20.0 > > Time Spent: 2.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9292) Provide an ability to specify additional maven repositories for published POMs
[
https://issues.apache.org/jira/browse/BEAM-9292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexey Romanenko updated BEAM-9292:
---
Description:
To support Confluent Schema Registry, KafkaIO has a dependency on
{{io.confluent:kafka-avro-serializer}} from
https://packages.confluent.io/maven/ repository. In this case, it should add
this repository into published KafkaIO POM file. Otherwise, it will fail with
the following error during building a user pipeline:
{code}
[ERROR] Failed to execute goal on project kafka-io: Could not resolve
dependencies for project org.apache.beam.issues:kafka-io:jar:1.0.0-SNAPSHOT:
Could not find artifact io.confluent:kafka-avro-serializer:jar:5.3.2 in central
(https://repo.maven.apache.org/maven2) -> [Help 1]
{code}
The repositories for publishing can be added by {{mavenRepositories}} argument
in build script for Java configuration.
For example (KafkaIO:
{code}
$ cat sdks/java/io/kafka/build.gradle
...
applyJavaNature(
...
mavenRepositories: [
[id: 'io.confluent', url: 'https://packages.confluent.io/maven/']
]
)
...
{code}
It will generate the following xml code snippet in pom file of
{{beam-sdks-java-io-kafka}} artifact after publishing:
{code}
io.confluent
https://packages.confluent.io/maven/
{code}
was:
To support Confluent Schema Registry, KafkaIO has a dependency on
{{io.confluent:kafka-avro-serializer}} from
https://packages.confluent.io/maven/ repository. In this case, it should add
this repository into published KafkaIO POM file. Otherwise, it will fail with
the following error during building a user pipeline:
{code}
[ERROR] Failed to execute goal on project kafka-io: Could not resolve
dependencies for project org.apache.beam.issues:kafka-io:jar:1.0.0-SNAPSHOT:
Could not find artifact io.confluent:kafka-avro-serializer:jar:5.3.2 in central
(https://repo.maven.apache.org/maven2) -> [Help 1]
{code}
> Provide an ability to specify additional maven repositories for published POMs
> --
>
> Key: BEAM-9292
> URL: https://issues.apache.org/jira/browse/BEAM-9292
> Project: Beam
> Issue Type: Improvement
> Components: build-system, io-java-kafka
>Reporter: Alexey Romanenko
>Assignee: Alexey Romanenko
>Priority: Major
> Fix For: 2.20.0
>
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> To support Confluent Schema Registry, KafkaIO has a dependency on
> {{io.confluent:kafka-avro-serializer}} from
> https://packages.confluent.io/maven/ repository. In this case, it should add
> this repository into published KafkaIO POM file. Otherwise, it will fail with
> the following error during building a user pipeline:
> {code}
> [ERROR] Failed to execute goal on project kafka-io: Could not resolve
> dependencies for project org.apache.beam.issues:kafka-io:jar:1.0.0-SNAPSHOT:
> Could not find artifact io.confluent:kafka-avro-serializer:jar:5.3.2 in
> central (https://repo.maven.apache.org/maven2) -> [Help 1]
> {code}
> The repositories for publishing can be added by {{mavenRepositories}}
> argument in build script for Java configuration.
> For example (KafkaIO:
> {code}
> $ cat sdks/java/io/kafka/build.gradle
> ...
> applyJavaNature(
> ...
> mavenRepositories: [
> [id: 'io.confluent', url: 'https://packages.confluent.io/maven/']
> ]
> )
> ...
> {code}
> It will generate the following xml code snippet in pom file of
> {{beam-sdks-java-io-kafka}} artifact after publishing:
> {code}
>
>
> io.confluent
> https://packages.confluent.io/maven/
>
>
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (BEAM-9292) Provide an ability to specify additional maven repositories for published POMs
[
https://issues.apache.org/jira/browse/BEAM-9292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexey Romanenko updated BEAM-9292:
---
Description:
To support Confluent Schema Registry, KafkaIO has a dependency on
{{io.confluent:kafka-avro-serializer}} from
https://packages.confluent.io/maven/ repository. In this case, it should add
this repository into published KafkaIO pom file. Otherwise, it will fail with
the following error during building a user pipeline:
{code}
[ERROR] Failed to execute goal on project kafka-io: Could not resolve
dependencies for project org.apache.beam.issues:kafka-io:jar:1.0.0-SNAPSHOT:
Could not find artifact io.confluent:kafka-avro-serializer:jar:5.3.2 in central
(https://repo.maven.apache.org/maven2) -> [Help 1]
{code}
was:
To support Confluent Schema Registry, KafkaIO has a dependency from
{{https://packages.confluent.io/maven/}} repository. In this case, it should
add this repository into published KafkaIO pom file. Otherwise, it will fail
with the following error during building a user pipeline:
{code}
[ERROR] Failed to execute goal on project kafka-io: Could not resolve
dependencies for project org.apache.beam.issues:kafka-io:jar:1.0.0-SNAPSHOT:
Could not find artifact io.confluent:kafka-avro-serializer:jar:5.3.2 in central
(https://repo.maven.apache.org/maven2) -> [Help 1]
{code}
> Provide an ability to specify additional maven repositories for published POMs
> --
>
> Key: BEAM-9292
> URL: https://issues.apache.org/jira/browse/BEAM-9292
> Project: Beam
> Issue Type: Improvement
> Components: build-system, io-java-kafka
>Reporter: Alexey Romanenko
>Assignee: Alexey Romanenko
>Priority: Major
>
> To support Confluent Schema Registry, KafkaIO has a dependency on
> {{io.confluent:kafka-avro-serializer}} from
> https://packages.confluent.io/maven/ repository. In this case, it should add
> this repository into published KafkaIO pom file. Otherwise, it will fail with
> the following error during building a user pipeline:
> {code}
> [ERROR] Failed to execute goal on project kafka-io: Could not resolve
> dependencies for project org.apache.beam.issues:kafka-io:jar:1.0.0-SNAPSHOT:
> Could not find artifact io.confluent:kafka-avro-serializer:jar:5.3.2 in
> central (https://repo.maven.apache.org/maven2) -> [Help 1]
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (BEAM-9292) Provide an ability to specify additional maven repositories for published POMs
[
https://issues.apache.org/jira/browse/BEAM-9292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexey Romanenko updated BEAM-9292:
---
Description:
To support Confluent Schema Registry, KafkaIO has a dependency on
{{io.confluent:kafka-avro-serializer}} from
https://packages.confluent.io/maven/ repository. In this case, it should add
this repository into published KafkaIO POM file. Otherwise, it will fail with
the following error during building a user pipeline:
{code}
[ERROR] Failed to execute goal on project kafka-io: Could not resolve
dependencies for project org.apache.beam.issues:kafka-io:jar:1.0.0-SNAPSHOT:
Could not find artifact io.confluent:kafka-avro-serializer:jar:5.3.2 in central
(https://repo.maven.apache.org/maven2) -> [Help 1]
{code}
was:
To support Confluent Schema Registry, KafkaIO has a dependency on
{{io.confluent:kafka-avro-serializer}} from
https://packages.confluent.io/maven/ repository. In this case, it should add
this repository into published KafkaIO pom file. Otherwise, it will fail with
the following error during building a user pipeline:
{code}
[ERROR] Failed to execute goal on project kafka-io: Could not resolve
dependencies for project org.apache.beam.issues:kafka-io:jar:1.0.0-SNAPSHOT:
Could not find artifact io.confluent:kafka-avro-serializer:jar:5.3.2 in central
(https://repo.maven.apache.org/maven2) -> [Help 1]
{code}
> Provide an ability to specify additional maven repositories for published POMs
> --
>
> Key: BEAM-9292
> URL: https://issues.apache.org/jira/browse/BEAM-9292
> Project: Beam
> Issue Type: Improvement
> Components: build-system, io-java-kafka
>Reporter: Alexey Romanenko
>Assignee: Alexey Romanenko
>Priority: Major
>
> To support Confluent Schema Registry, KafkaIO has a dependency on
> {{io.confluent:kafka-avro-serializer}} from
> https://packages.confluent.io/maven/ repository. In this case, it should add
> this repository into published KafkaIO POM file. Otherwise, it will fail with
> the following error during building a user pipeline:
> {code}
> [ERROR] Failed to execute goal on project kafka-io: Could not resolve
> dependencies for project org.apache.beam.issues:kafka-io:jar:1.0.0-SNAPSHOT:
> Could not find artifact io.confluent:kafka-avro-serializer:jar:5.3.2 in
> central (https://repo.maven.apache.org/maven2) -> [Help 1]
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (BEAM-9292) Provide an ability to specify additional maven repositories for published POMs
[
https://issues.apache.org/jira/browse/BEAM-9292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexey Romanenko updated BEAM-9292:
---
Description:
To support Confluent Schema Registry, KafkaIO has a dependency from
{{https://packages.confluent.io/maven/}} repository. In this case, it should
add this repository into published KafkaIO pom file. Otherwise, it will fail
with the following error during building a user pipeline:
{code}
[ERROR] Failed to execute goal on project kafka-io: Could not resolve
dependencies for project org.apache.beam.issues:kafka-io:jar:1.0.0-SNAPSHOT:
Could not find artifact io.confluent:kafka-avro-serializer:jar:5.3.2 in central
(https://repo.maven.apache.org/maven2) -> [Help 1]
{code}
was:
To support Confluent Schema Registry, KafkaIO has a dependency from
{{https://packages.confluent.io/maven/}} repository. In this case, it should
add this repository into published KafkaIO pom file. Otherwise, it will fail
with the following error during building a user pipeline:
{code}
[ERROR] Failed to execute goal on project kafka-io: Could not resolve
dependencies for project org.apache.beam.issues:kafka-io:jar:1.0.0-SNAPSHOT:
Could not find artifact
org.apache.beam:beam-sdks-java-io-kafka:jar:2.20.0-SNAPSHOT in snapshots
{code}
> Provide an ability to specify additional maven repositories for published POMs
> --
>
> Key: BEAM-9292
> URL: https://issues.apache.org/jira/browse/BEAM-9292
> Project: Beam
> Issue Type: Improvement
> Components: build-system, io-java-kafka
>Reporter: Alexey Romanenko
>Assignee: Alexey Romanenko
>Priority: Major
>
> To support Confluent Schema Registry, KafkaIO has a dependency from
> {{https://packages.confluent.io/maven/}} repository. In this case, it should
> add this repository into published KafkaIO pom file. Otherwise, it will fail
> with the following error during building a user pipeline:
> {code}
> [ERROR] Failed to execute goal on project kafka-io: Could not resolve
> dependencies for project org.apache.beam.issues:kafka-io:jar:1.0.0-SNAPSHOT:
> Could not find artifact io.confluent:kafka-avro-serializer:jar:5.3.2 in
> central (https://repo.maven.apache.org/maven2) -> [Help 1]
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (BEAM-9292) Provide an ability to specify additional maven repositories for published POMs
[
https://issues.apache.org/jira/browse/BEAM-9292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexey Romanenko updated BEAM-9292:
---
Description:
To support Confluent Schema Registry, KafkaIO has a dependency from
{{https://packages.confluent.io/maven/}} repository. In this case, it should
add this repository into published KafkaIO pom file. Otherwise, it will fail
with the following error during building a user pipeline:
{code}
[ERROR] Failed to execute goal on project kafka-io: Could not resolve
dependencies for project org.apache.beam.issues:kafka-io:jar:1.0.0-SNAPSHOT:
Could not find artifact
org.apache.beam:beam-sdks-java-io-kafka:jar:2.20.0-SNAPSHOT in snapshots
{code}
was:
To support Confluent Schema Registry, KafkaIO has a dependency from
{{https://packages.confluent.io/maven/}} repository. In this case, it should
add this repository into published KafkaIO pom file. Otherwise, it will fail
with the following error:
{code}
[ERROR] Failed to execute goal on project kafka-io: Could not resolve
dependencies for project org.apache.beam.issues:kafka-io:jar:1.0.0-SNAPSHOT:
Could not find artifact
org.apache.beam:beam-sdks-java-io-kafka:jar:2.20.0-SNAPSHOT in snapshots
{code}
> Provide an ability to specify additional maven repositories for published POMs
> --
>
> Key: BEAM-9292
> URL: https://issues.apache.org/jira/browse/BEAM-9292
> Project: Beam
> Issue Type: Improvement
> Components: build-system, io-java-kafka
>Reporter: Alexey Romanenko
>Assignee: Alexey Romanenko
>Priority: Major
>
> To support Confluent Schema Registry, KafkaIO has a dependency from
> {{https://packages.confluent.io/maven/}} repository. In this case, it should
> add this repository into published KafkaIO pom file. Otherwise, it will fail
> with the following error during building a user pipeline:
> {code}
> [ERROR] Failed to execute goal on project kafka-io: Could not resolve
> dependencies for project org.apache.beam.issues:kafka-io:jar:1.0.0-SNAPSHOT:
> Could not find artifact
> org.apache.beam:beam-sdks-java-io-kafka:jar:2.20.0-SNAPSHOT in snapshots
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (BEAM-9292) Provide an ability to specify additional maven repositories for published POMs
[
https://issues.apache.org/jira/browse/BEAM-9292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexey Romanenko updated BEAM-9292:
---
Description:
To support Confluent Schema Registry, KafkaIO has a dependency on
{{packages.confluent.io}} repository. In this case, it should add this
repository into published KafkaIO pom file. Otherwise, it will fail with the
following error:
{code}
[ERROR] Failed to execute goal on project kafka-io: Could not resolve
dependencies for project org.apache.beam.issues:kafka-io:jar:1.0.0-SNAPSHOT:
Could not find artifact
org.apache.beam:beam-sdks-java-io-kafka:jar:2.20.0-SNAPSHOT in snapshots
{code}
was:
To support Confluent Schema Registry KafkaIO has a dependency on
{{packages.confluent.io}} repository. In this case, it should add this
repository into published KafkaIO pom file. Otherwise, it will fail with the
following error:
{code}
[ERROR] Failed to execute goal on project kafka-io: Could not resolve
dependencies for project org.apache.beam.issues:kafka-io:jar:1.0.0-SNAPSHOT:
Could not find artifact
org.apache.beam:beam-sdks-java-io-kafka:jar:2.20.0-SNAPSHOT in snapshots
{code}
> Provide an ability to specify additional maven repositories for published POMs
> --
>
> Key: BEAM-9292
> URL: https://issues.apache.org/jira/browse/BEAM-9292
> Project: Beam
> Issue Type: Improvement
> Components: build-system, io-java-kafka
>Reporter: Alexey Romanenko
>Assignee: Alexey Romanenko
>Priority: Major
>
> To support Confluent Schema Registry, KafkaIO has a dependency on
> {{packages.confluent.io}} repository. In this case, it should add this
> repository into published KafkaIO pom file. Otherwise, it will fail with the
> following error:
> {code}
> [ERROR] Failed to execute goal on project kafka-io: Could not resolve
> dependencies for project org.apache.beam.issues:kafka-io:jar:1.0.0-SNAPSHOT:
> Could not find artifact
> org.apache.beam:beam-sdks-java-io-kafka:jar:2.20.0-SNAPSHOT in snapshots
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (BEAM-9292) Provide an ability to specify additional maven repositories for published POMs
[
https://issues.apache.org/jira/browse/BEAM-9292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexey Romanenko updated BEAM-9292:
---
Description:
To support Confluent Schema Registry, KafkaIO has a dependency from
{{https://packages.confluent.io/maven/}} repository. In this case, it should
add this repository into published KafkaIO pom file. Otherwise, it will fail
with the following error:
{code}
[ERROR] Failed to execute goal on project kafka-io: Could not resolve
dependencies for project org.apache.beam.issues:kafka-io:jar:1.0.0-SNAPSHOT:
Could not find artifact
org.apache.beam:beam-sdks-java-io-kafka:jar:2.20.0-SNAPSHOT in snapshots
{code}
was:
To support Confluent Schema Registry, KafkaIO has a dependency from
{{packages.confluent.io}} repository. In this case, it should add this
repository into published KafkaIO pom file. Otherwise, it will fail with the
following error:
{code}
[ERROR] Failed to execute goal on project kafka-io: Could not resolve
dependencies for project org.apache.beam.issues:kafka-io:jar:1.0.0-SNAPSHOT:
Could not find artifact
org.apache.beam:beam-sdks-java-io-kafka:jar:2.20.0-SNAPSHOT in snapshots
{code}
> Provide an ability to specify additional maven repositories for published POMs
> --
>
> Key: BEAM-9292
> URL: https://issues.apache.org/jira/browse/BEAM-9292
> Project: Beam
> Issue Type: Improvement
> Components: build-system, io-java-kafka
>Reporter: Alexey Romanenko
>Assignee: Alexey Romanenko
>Priority: Major
>
> To support Confluent Schema Registry, KafkaIO has a dependency from
> {{https://packages.confluent.io/maven/}} repository. In this case, it should
> add this repository into published KafkaIO pom file. Otherwise, it will fail
> with the following error:
> {code}
> [ERROR] Failed to execute goal on project kafka-io: Could not resolve
> dependencies for project org.apache.beam.issues:kafka-io:jar:1.0.0-SNAPSHOT:
> Could not find artifact
> org.apache.beam:beam-sdks-java-io-kafka:jar:2.20.0-SNAPSHOT in snapshots
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (BEAM-9292) Provide an ability to specify additional maven repositories for published POMs
[
https://issues.apache.org/jira/browse/BEAM-9292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexey Romanenko updated BEAM-9292:
---
Description:
To support Confluent Schema Registry, KafkaIO has a dependency from
{{packages.confluent.io}} repository. In this case, it should add this
repository into published KafkaIO pom file. Otherwise, it will fail with the
following error:
{code}
[ERROR] Failed to execute goal on project kafka-io: Could not resolve
dependencies for project org.apache.beam.issues:kafka-io:jar:1.0.0-SNAPSHOT:
Could not find artifact
org.apache.beam:beam-sdks-java-io-kafka:jar:2.20.0-SNAPSHOT in snapshots
{code}
was:
To support Confluent Schema Registry, KafkaIO has a dependency on
{{packages.confluent.io}} repository. In this case, it should add this
repository into published KafkaIO pom file. Otherwise, it will fail with the
following error:
{code}
[ERROR] Failed to execute goal on project kafka-io: Could not resolve
dependencies for project org.apache.beam.issues:kafka-io:jar:1.0.0-SNAPSHOT:
Could not find artifact
org.apache.beam:beam-sdks-java-io-kafka:jar:2.20.0-SNAPSHOT in snapshots
{code}
> Provide an ability to specify additional maven repositories for published POMs
> --
>
> Key: BEAM-9292
> URL: https://issues.apache.org/jira/browse/BEAM-9292
> Project: Beam
> Issue Type: Improvement
> Components: build-system, io-java-kafka
>Reporter: Alexey Romanenko
>Assignee: Alexey Romanenko
>Priority: Major
>
> To support Confluent Schema Registry, KafkaIO has a dependency from
> {{packages.confluent.io}} repository. In this case, it should add this
> repository into published KafkaIO pom file. Otherwise, it will fail with the
> following error:
> {code}
> [ERROR] Failed to execute goal on project kafka-io: Could not resolve
> dependencies for project org.apache.beam.issues:kafka-io:jar:1.0.0-SNAPSHOT:
> Could not find artifact
> org.apache.beam:beam-sdks-java-io-kafka:jar:2.20.0-SNAPSHOT in snapshots
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (BEAM-9292) Provide an ability to specify additional maven repositories for published POMs
Alexey Romanenko created BEAM-9292:
--
Summary: Provide an ability to specify additional maven
repositories for published POMs
Key: BEAM-9292
URL: https://issues.apache.org/jira/browse/BEAM-9292
Project: Beam
Issue Type: Improvement
Components: build-system, io-java-kafka
Reporter: Alexey Romanenko
To support Confluent Schema Registry KafkaIO has a dependency on
{{packages.confluent.io}} repository. In this case, it should add this
repository into published KafkaIO pom file. Otherwise, it will fail with the
following error:
{code}
[ERROR] Failed to execute goal on project kafka-io: Could not resolve
dependencies for project org.apache.beam.issues:kafka-io:jar:1.0.0-SNAPSHOT:
Could not find artifact
org.apache.beam:beam-sdks-java-io-kafka:jar:2.20.0-SNAPSHOT in snapshots
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (BEAM-9292) Provide an ability to specify additional maven repositories for published POMs
[
https://issues.apache.org/jira/browse/BEAM-9292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexey Romanenko reassigned BEAM-9292:
--
Assignee: Alexey Romanenko
> Provide an ability to specify additional maven repositories for published POMs
> --
>
> Key: BEAM-9292
> URL: https://issues.apache.org/jira/browse/BEAM-9292
> Project: Beam
> Issue Type: Improvement
> Components: build-system, io-java-kafka
>Reporter: Alexey Romanenko
>Assignee: Alexey Romanenko
>Priority: Major
>
> To support Confluent Schema Registry KafkaIO has a dependency on
> {{packages.confluent.io}} repository. In this case, it should add this
> repository into published KafkaIO pom file. Otherwise, it will fail with the
> following error:
> {code}
> [ERROR] Failed to execute goal on project kafka-io: Could not resolve
> dependencies for project org.apache.beam.issues:kafka-io:jar:1.0.0-SNAPSHOT:
> Could not find artifact
> org.apache.beam:beam-sdks-java-io-kafka:jar:2.20.0-SNAPSHOT in snapshots
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (BEAM-9253) SQL PostCommit failure: ClassCastException: java.lang.Integer cannot be cast to java.lang.Long
[
https://issues.apache.org/jira/browse/BEAM-9253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17030661#comment-17030661
]
Alexey Romanenko commented on BEAM-9253:
More details about failing test:
{code}
:sdks:java:extensions:sql:datacatalog:integrationTest »
org.apache.beam.sdk.extensions.sql.meta.provider.datacatalog.DataCatalogGCSIT »
testReadFromGCS (7.486s)
java.lang.Integer cannot be cast to java.lang.Long
java.lang.ClassCastException: java.lang.Integer cannot be cast to
java.lang.LongOpen stacktrace
[Test worker] INFO org.apache.beam.sdk.extensions.sql.impl.CalciteQueryPlanner
- SQL:
SELECT `integ_test_small_csv_test_1`.`id`,
`integ_test_small_csv_test_1`.`name`, `integ_test_small_csv_test_1`.`type`
FROM
`dc`.`datacatalog`.`entry`.`apache-beam-testing`.`us-central1`.`samples`.`integ_test_small_csv_test_1`
AS `integ_test_small_csv_test_1`
[Test worker] INFO org.apache.beam.sdk.extensions.sql.impl.CalciteQueryPlanner
- SQLPlan>
LogicalProject(id=[$0], name=[$1], type=[$2])
BeamIOSourceRel(table=[[dc, datacatalog, entry, apache-beam-testing,
us-central1, samples, integ_test_small_csv_test_1]])
[Test worker] INFO org.apache.beam.sdk.extensions.sql.impl.CalciteQueryPlanner
- BEAMPlan>
BeamCalcRel(expr#0..2=[{inputs}], proj#0..2=[{exprs}])
BeamIOSourceRel(table=[[dc, datacatalog, entry, apache-beam-testing,
us-central1, samples, integ_test_small_csv_test_1]])
{code}
> SQL PostCommit failure: ClassCastException: java.lang.Integer cannot be cast
> to java.lang.Long
> --
>
> Key: BEAM-9253
> URL: https://issues.apache.org/jira/browse/BEAM-9253
> Project: Beam
> Issue Type: Bug
> Components: test-failures
>Reporter: Tomo Suzuki
>Priority: Major
>
> SQL Postcommit check started failing since build number 3924. The first
> failure: https://builds.apache.org/job/beam_PostCommit_SQL/3924/
> {noformat}
> org.apache.beam.sdk.extensions.sql.meta.provider.datacatalog.DataCatalogGCSIT.testReadFromGCS
> java.lang.ClassCastException: java.lang.Integer cannot be cast to
> java.lang.Long
> at org.apache.beam.sdk.coders.VarLongCoder.encode(VarLongCoder.java:35)
> at
> org.apache.beam.sdk.coders.RowCoderGenerator$EncodeInstruction.encodeDelegate(RowCoderGenerator.java:239)
> at org.apache.beam.sdk.coders.Coder$ByteBuddy$WZzHUNls.encode(Unknown
> Source)
> at org.apache.beam.sdk.coders.Coder$ByteBuddy$WZzHUNls.encode(Unknown
> Source)
> at org.apache.beam.sdk.schemas.SchemaCoder.encode(SchemaCoder.java:166)
> at
> org.apache.beam.sdk.coders.IterableLikeCoder.encode(IterableLikeCoder.java:98)
> {noformat}
> History: https://builds.apache.org/job/beam_PostCommit_SQL/
> Pull request before the failure:
> https://github.com/apache/beam/pull/10563#issuecomment-582384925
> DataCatalogGCSIT does not run locally:
> https://gist.github.com/suztomo/43de4ff1f0458e801fef1a16a28da301
> DataCatalogGCSIT reads this Google Cloud Storage object:
> {code:java}
> String gcsEntryId =
> "`datacatalog`" // this is part of the resource name in DataCatalog,
> so it has to be
> + ".`entry`" // different from the table provider name ("dc" in
> this test)
> + ".`apache-beam-testing`"
> + ".`us-central1`"
> + ".`samples`"
> + ".`integ_test_small_csv_test_1`";
> {code}
> but I cannot see the content.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (BEAM-9251) :sdks:java:io:kafka:updateOfflineRepository fails
[
https://issues.apache.org/jira/browse/BEAM-9251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17030541#comment-17030541
]
Alexey Romanenko commented on BEAM-9251:
Thank you for fixing this!
> :sdks:java:io:kafka:updateOfflineRepository fails
> -
>
> Key: BEAM-9251
> URL: https://issues.apache.org/jira/browse/BEAM-9251
> Project: Beam
> Issue Type: Bug
> Components: sdk-java-core
>Reporter: Udi Meiri
>Assignee: Udi Meiri
>Priority: Major
> Fix For: Not applicable
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> I resolved the 403 here: https://github.com/apache/beam/pull/10771
> Now getting a not found error:
> {code}
> Execution failed for task ':sdks:java:io:kafka:updateOfflineRepository'.
> > Could not resolve all dependencies for configuration
> > ':sdks:java:io:kafka:detachedConfiguration14'.
>> Could not find io.confluent:kafka-avro-serializer:5.3.2.
> Searched in the following locations:
>-
> file:/usr/local/google/home/ehudm/.m2/repository/io/confluent/kafka-avro-serializer/5.3.2/kafka-avro-serializer-5.3.2.pom
>-
> file:/usr/local/google/home/ehudm/.m2/repository/io/confluent/kafka-avro-serializer/5.3.2/kafka-avro-serializer-5.3.2.jar
>-
> https://repo.maven.apache.org/maven2/io/confluent/kafka-avro-serializer/5.3.2/kafka-avro-serializer-5.3.2.pom
>-
> https://repo.maven.apache.org/maven2/io/confluent/kafka-avro-serializer/5.3.2/kafka-avro-serializer-5.3.2.jar
>-
> https://jcenter.bintray.com/io/confluent/kafka-avro-serializer/5.3.2/kafka-avro-serializer-5.3.2.pom
>-
> https://jcenter.bintray.com/io/confluent/kafka-avro-serializer/5.3.2/kafka-avro-serializer-5.3.2.jar
>-
> https://plugins.gradle.org/m2/io/confluent/kafka-avro-serializer/5.3.2/kafka-avro-serializer-5.3.2.pom
>-
> https://plugins.gradle.org/m2/io/confluent/kafka-avro-serializer/5.3.2/kafka-avro-serializer-5.3.2.jar
>-
> https://repo.spring.io/plugins-release/io/confluent/kafka-avro-serializer/5.3.2/kafka-avro-serializer-5.3.2.pom
>-
> https://repo.spring.io/plugins-release/io/confluent/kafka-avro-serializer/5.3.2/kafka-avro-serializer-5.3.2.jar
> Required by:
> project :sdks:java:io:kafka
> {code}
> I found the package here:
> https://packages.confluent.io/maven/io/confluent/kafka-avro-serializer/5.3.2/
> but I have no idea what the official location is.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (BEAM-4420) Add KafkaIO Integration Tests
[ https://issues.apache.org/jira/browse/BEAM-4420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17027701#comment-17027701 ] Alexey Romanenko edited comment on BEAM-4420 at 1/31/20 5:56 PM: - [~mwalenia] Can we close this issue since KafkaIOIT jenkins job has been already added? was (Author: aromanenko): [~mwalenia] Could we close this issue since KafkaIOIT jenkins job has been already added? > Add KafkaIO Integration Tests > - > > Key: BEAM-4420 > URL: https://issues.apache.org/jira/browse/BEAM-4420 > Project: Beam > Issue Type: Test > Components: io-java-kafka, testing >Reporter: Ismaël Mejía >Priority: Minor > Time Spent: 8h 10m > Remaining Estimate: 0h > > It is a good idea to have ITs for KafkaIO. > There are two possible issues: > 1. The tests should probably invert the pattern to be readThenWrite given > that Unbounded IOs block on Read and ... > 2. Until we have a way to do PAsserts on Unbounded sources we can rely on > withMaxNumRecords to ensure this test ends. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-4420) Add KafkaIO Integration Tests
[ https://issues.apache.org/jira/browse/BEAM-4420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17027701#comment-17027701 ] Alexey Romanenko commented on BEAM-4420: [~mwalenia] Could we close this issue since KafkaIOIT jenkins job has been already added? > Add KafkaIO Integration Tests > - > > Key: BEAM-4420 > URL: https://issues.apache.org/jira/browse/BEAM-4420 > Project: Beam > Issue Type: Test > Components: io-java-kafka, testing >Reporter: Ismaël Mejía >Priority: Minor > Time Spent: 8h 10m > Remaining Estimate: 0h > > It is a good idea to have ITs for KafkaIO. > There are two possible issues: > 1. The tests should probably invert the pattern to be readThenWrite given > that Unbounded IOs block on Read and ... > 2. Until we have a way to do PAsserts on Unbounded sources we can rely on > withMaxNumRecords to ensure this test ends. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (BEAM-9143) Add withOutputParallelization to RedisIO.Read/ReadAll
[ https://issues.apache.org/jira/browse/BEAM-9143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Romanenko resolved BEAM-9143. Fix Version/s: 2.20.0 Resolution: Fixed > Add withOutputParallelization to RedisIO.Read/ReadAll > - > > Key: BEAM-9143 > URL: https://issues.apache.org/jira/browse/BEAM-9143 > Project: Beam > Issue Type: Bug > Components: io-java-redis >Reporter: Ismaël Mejía >Assignee: Ismaël Mejía >Priority: Minor > Fix For: 2.20.0 > > Time Spent: 20m > Remaining Estimate: 0h > > Similar to the method for JdbcIO for cases where users explicitly do not care > about the output of a Read being reparallelized. By default it should be. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9133) CassandraIOTest.classMethod test is still flaky
[
https://issues.apache.org/jira/browse/BEAM-9133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexey Romanenko updated BEAM-9133:
---
Summary: CassandraIOTest.classMethod test is still flaky (was:
CassandraIOTest classMethod test is still flaky)
> CassandraIOTest.classMethod test is still flaky
> ---
>
> Key: BEAM-9133
> URL: https://issues.apache.org/jira/browse/BEAM-9133
> Project: Beam
> Issue Type: Bug
> Components: io-java-cassandra, test-failures
>Affects Versions: 2.17.0
>Reporter: Alexey Romanenko
>Assignee: Etienne Chauchot
>Priority: Critical
>
> CassandraIOTest is still flaky. For example:
> https://builds.apache.org/job/beam_PreCommit_Java_Phrase/1646/
> https://builds.apache.org/job/beam_PreCommit_Java_Phrase/1625/
> {code}
> Error Message
> java.lang.RuntimeException: Unable to create embedded Cassandra cluster
> Stacktrace
> java.lang.RuntimeException: Unable to create embedded Cassandra cluster
> at
> org.apache.beam.sdk.io.cassandra.CassandraIOTest.buildCluster(CassandraIOTest.java:167)
> at
> org.apache.beam.sdk.io.cassandra.CassandraIOTest.beforeClass(CassandraIOTest.java:146)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> at
> org.junit.internal.runners.statements.RunBefores.invokeMethod(RunBefores.java:33)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:305)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:412)
> at
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.runTestClass(JUnitTestClassExecutor.java:110)
> at
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:58)
> at
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:38)
> at
> org.gradle.api.internal.tasks.testing.junit.AbstractJUnitTestClassProcessor.processTestClass(AbstractJUnitTestClassProcessor.java:62)
> at
> org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:51)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
> at
> org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
> at
> org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:32)
> at
> org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:93)
> at com.sun.proxy.$Proxy2.processTestClass(Unknown Source)
> at
> org.gradle.api.internal.tasks.testing.worker.TestWorker.processTestClass(TestWorker.java:118)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
> at
> org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
> at
> org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:175)
> at
> org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:157)
> at
> org.gradle.internal.remote.internal.hub.MessageHub$Handler.run(MessageHub.java:404)
> at
>
[jira] [Updated] (BEAM-9133) CassandraIOTest classMethod test is still flaky
[
https://issues.apache.org/jira/browse/BEAM-9133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexey Romanenko updated BEAM-9133:
---
Summary: CassandraIOTest classMethod test is still flaky (was: Cassandra
IO classMethod test is still flaky)
> CassandraIOTest classMethod test is still flaky
> ---
>
> Key: BEAM-9133
> URL: https://issues.apache.org/jira/browse/BEAM-9133
> Project: Beam
> Issue Type: Bug
> Components: io-java-cassandra, test-failures
>Affects Versions: 2.17.0
>Reporter: Alexey Romanenko
>Assignee: Etienne Chauchot
>Priority: Critical
>
> CassandraIOTest is still flaky. For example:
> https://builds.apache.org/job/beam_PreCommit_Java_Phrase/1646/
> https://builds.apache.org/job/beam_PreCommit_Java_Phrase/1625/
> {code}
> Error Message
> java.lang.RuntimeException: Unable to create embedded Cassandra cluster
> Stacktrace
> java.lang.RuntimeException: Unable to create embedded Cassandra cluster
> at
> org.apache.beam.sdk.io.cassandra.CassandraIOTest.buildCluster(CassandraIOTest.java:167)
> at
> org.apache.beam.sdk.io.cassandra.CassandraIOTest.beforeClass(CassandraIOTest.java:146)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> at
> org.junit.internal.runners.statements.RunBefores.invokeMethod(RunBefores.java:33)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:305)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:412)
> at
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.runTestClass(JUnitTestClassExecutor.java:110)
> at
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:58)
> at
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:38)
> at
> org.gradle.api.internal.tasks.testing.junit.AbstractJUnitTestClassProcessor.processTestClass(AbstractJUnitTestClassProcessor.java:62)
> at
> org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:51)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
> at
> org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
> at
> org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:32)
> at
> org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:93)
> at com.sun.proxy.$Proxy2.processTestClass(Unknown Source)
> at
> org.gradle.api.internal.tasks.testing.worker.TestWorker.processTestClass(TestWorker.java:118)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
> at
> org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
> at
> org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:175)
> at
> org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:157)
> at
> org.gradle.internal.remote.internal.hub.MessageHub$Handler.run(MessageHub.java:404)
> at
>
[jira] [Created] (BEAM-9133) Cassandra IO classMethod test is still flaky
Alexey Romanenko created BEAM-9133:
--
Summary: Cassandra IO classMethod test is still flaky
Key: BEAM-9133
URL: https://issues.apache.org/jira/browse/BEAM-9133
Project: Beam
Issue Type: Bug
Components: io-java-cassandra, test-failures
Affects Versions: 2.17.0
Reporter: Alexey Romanenko
Assignee: Etienne Chauchot
CassandraIOTest is still flaky. For example:
https://builds.apache.org/job/beam_PreCommit_Java_Phrase/1646/
https://builds.apache.org/job/beam_PreCommit_Java_Phrase/1625/
{code}
Error Message
java.lang.RuntimeException: Unable to create embedded Cassandra cluster
Stacktrace
java.lang.RuntimeException: Unable to create embedded Cassandra cluster
at
org.apache.beam.sdk.io.cassandra.CassandraIOTest.buildCluster(CassandraIOTest.java:167)
at
org.apache.beam.sdk.io.cassandra.CassandraIOTest.beforeClass(CassandraIOTest.java:146)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
at
org.junit.internal.runners.statements.RunBefores.invokeMethod(RunBefores.java:33)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:305)
at org.junit.runners.ParentRunner.run(ParentRunner.java:412)
at
org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.runTestClass(JUnitTestClassExecutor.java:110)
at
org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:58)
at
org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:38)
at
org.gradle.api.internal.tasks.testing.junit.AbstractJUnitTestClassProcessor.processTestClass(AbstractJUnitTestClassProcessor.java:62)
at
org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:51)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
at
org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
at
org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:32)
at
org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:93)
at com.sun.proxy.$Proxy2.processTestClass(Unknown Source)
at
org.gradle.api.internal.tasks.testing.worker.TestWorker.processTestClass(TestWorker.java:118)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
at
org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
at
org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:175)
at
org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:157)
at
org.gradle.internal.remote.internal.hub.MessageHub$Handler.run(MessageHub.java:404)
at
org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:63)
at
org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:46)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at
[jira] [Commented] (BEAM-9123) HadoopResourceId returns wrong directory name
[
https://issues.apache.org/jira/browse/BEAM-9123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17016913#comment-17016913
]
Alexey Romanenko commented on BEAM-9123:
Is it really going to be included into 2.18 release (1st release candidate was
done yesterday)?
> HadoopResourceId returns wrong directory name
> -
>
> Key: BEAM-9123
> URL: https://issues.apache.org/jira/browse/BEAM-9123
> Project: Beam
> Issue Type: Bug
> Components: io-java-hadoop-file-system
>Affects Versions: 2.17.0
>Reporter: Marek Simunek
>Assignee: Marek Simunek
>Priority: Major
> Fix For: 2.18.0
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> _HadoopResourceId_ returns for directory uri wrong fileName() which doesnt
> conform _ResourceId_ interface javadoc
> {color:#629755}Returns the name of the file or directory..{color}
> String {color:#ffc66d}getFilename{color}(){color:#cc7832}; {color}
>
> {code:java}
> URI hdfsClusterBaseUri = new URI(configuration.get("fs.defaultFS") + "/");
> HadoopResourceId hadoopResourceId =
> new HadoopResourceId(hdfsClusterBaseUri.resolve("/dirA/file1"));{code}
> Expected behavior:
> _hadoopResourceId.getCurrentDirectory().getFilename()_ returns *dirA*
> Actual behavior:
> _hadoopResourceId.getCurrentDirectory().getFilename()_ returns *""*
> The problem is that uri for folder is ending with '*/*' and
> [getFileName()|https://github.com/apache/beam/blob/master/sdks/java/io/hadoop-file-system/src/main/java/org/apache/beam/sdk/io/hdfs/HadoopResourceId.java#L68]
> is using hadoop Path.getName() which returns everything after last slash.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work started] (BEAM-7310) Confluent Schema Registry support in KafkaIO
[ https://issues.apache.org/jira/browse/BEAM-7310?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on BEAM-7310 started by Alexey Romanenko. -- > Confluent Schema Registry support in KafkaIO > > > Key: BEAM-7310 > URL: https://issues.apache.org/jira/browse/BEAM-7310 > Project: Beam > Issue Type: Improvement > Components: io-java-kafka >Affects Versions: 2.12.0 >Reporter: Yohei Shimomae >Assignee: Alexey Romanenko >Priority: Minor > Time Spent: 10m > Remaining Estimate: 0h > > Confluent Schema Registry is useful when we manage Avro Schema but KafkaIO > does not support Confluent Schema Registry as discussed here. > https://stackoverflow.com/questions/56035121/unable-to-connect-from-dataflow-job-to-schema-registry-when-schema-registry-requ > https://lists.apache.org/thread.html/7695fccddebd08733b80ae1e43b79b636b63cd5fe583a2bdeecda6c4@%3Cuser.beam.apache.org%3E -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-8953) Extend ParquetIO.Read/ReadFiles.Builder to support Avro GenericData model
[ https://issues.apache.org/jira/browse/BEAM-8953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Romanenko updated BEAM-8953: --- Affects Version/s: (was: 2.16.0) > Extend ParquetIO.Read/ReadFiles.Builder to support Avro GenericData model > - > > Key: BEAM-8953 > URL: https://issues.apache.org/jira/browse/BEAM-8953 > Project: Beam > Issue Type: Improvement > Components: io-java-parquet >Reporter: Ryan Berti >Assignee: Ryan Berti >Priority: Minor > Time Spent: 6h 20m > Remaining Estimate: 0h > > When utilizing ParquetIO to deserialize objects into case classes in Scala, > we'd like to utilize a downstream converter which takes GenericRecords and > converts them to instances of our case classes, rather than relying on > ParquetIO to deserialize into the case class via reflection + implementing > the IndexedRecord interface. > The ParquetIO.Read / ParquetIO.ReadFiles Builders currently support a > filepattern + schema / schema arguments respectively. When using the Read / > ReadFiles Builders with these arguments, the underlying AvroParquetReader > object that gets created in the ParquetIO.ReadFiles.ReadFn method defaults to > utilizing an AvroReadSupport instance whose GenericData model gets set to > SpecificData. We'd like to have the the underlying AvroReadSupport utilize > the GenericData model, but there's currently no way to force this to happen > via the existing ParquetIO Read / ReadFiles builders. > I'd like to extend the ParquetIO Read / ReadFiles builders to support a new > method allowing users to define a GenericData model, which will then be > passed into the AvroParquetReader builder. I've tested and validated that > this method allows ParquetIO to generate GenericRecord instances without > requiring that the users classes can be reflectively instantiated and > initialized via the IndexedRecord interface. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (BEAM-8953) Extend ParquetIO.Read/ReadFiles.Builder to support Avro GenericData model
[ https://issues.apache.org/jira/browse/BEAM-8953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Romanenko resolved BEAM-8953. Fix Version/s: 2.19.0 Resolution: Fixed > Extend ParquetIO.Read/ReadFiles.Builder to support Avro GenericData model > - > > Key: BEAM-8953 > URL: https://issues.apache.org/jira/browse/BEAM-8953 > Project: Beam > Issue Type: Improvement > Components: io-java-parquet >Reporter: Ryan Berti >Assignee: Ryan Berti >Priority: Minor > Fix For: 2.19.0 > > Time Spent: 6h 20m > Remaining Estimate: 0h > > When utilizing ParquetIO to deserialize objects into case classes in Scala, > we'd like to utilize a downstream converter which takes GenericRecords and > converts them to instances of our case classes, rather than relying on > ParquetIO to deserialize into the case class via reflection + implementing > the IndexedRecord interface. > The ParquetIO.Read / ParquetIO.ReadFiles Builders currently support a > filepattern + schema / schema arguments respectively. When using the Read / > ReadFiles Builders with these arguments, the underlying AvroParquetReader > object that gets created in the ParquetIO.ReadFiles.ReadFn method defaults to > utilizing an AvroReadSupport instance whose GenericData model gets set to > SpecificData. We'd like to have the the underlying AvroReadSupport utilize > the GenericData model, but there's currently no way to force this to happen > via the existing ParquetIO Read / ReadFiles builders. > I'd like to extend the ParquetIO Read / ReadFiles builders to support a new > method allowing users to define a GenericData model, which will then be > passed into the AvroParquetReader builder. I've tested and validated that > this method allows ParquetIO to generate GenericRecord instances without > requiring that the users classes can be reflectively instantiated and > initialized via the IndexedRecord interface. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-8953) Extend ParquetIO.Read/ReadFiles.Builder to support Avro GenericData model
[ https://issues.apache.org/jira/browse/BEAM-8953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Romanenko updated BEAM-8953: --- Component/s: (was: examples-java) io-java-parquet > Extend ParquetIO.Read/ReadFiles.Builder to support Avro GenericData model > - > > Key: BEAM-8953 > URL: https://issues.apache.org/jira/browse/BEAM-8953 > Project: Beam > Issue Type: Improvement > Components: io-java-parquet >Affects Versions: 2.16.0 >Reporter: Ryan Berti >Assignee: Ryan Berti >Priority: Minor > Time Spent: 6h 20m > Remaining Estimate: 0h > > When utilizing ParquetIO to deserialize objects into case classes in Scala, > we'd like to utilize a downstream converter which takes GenericRecords and > converts them to instances of our case classes, rather than relying on > ParquetIO to deserialize into the case class via reflection + implementing > the IndexedRecord interface. > The ParquetIO.Read / ParquetIO.ReadFiles Builders currently support a > filepattern + schema / schema arguments respectively. When using the Read / > ReadFiles Builders with these arguments, the underlying AvroParquetReader > object that gets created in the ParquetIO.ReadFiles.ReadFn method defaults to > utilizing an AvroReadSupport instance whose GenericData model gets set to > SpecificData. We'd like to have the the underlying AvroReadSupport utilize > the GenericData model, but there's currently no way to force this to happen > via the existing ParquetIO Read / ReadFiles builders. > I'd like to extend the ParquetIO Read / ReadFiles builders to support a new > method allowing users to define a GenericData model, which will then be > passed into the AvroParquetReader builder. I've tested and validated that > this method allows ParquetIO to generate GenericRecord instances without > requiring that the users classes can be reflectively instantiated and > initialized via the IndexedRecord interface. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (BEAM-8701) Beam Dependency Update Request: commons-io:commons-io
[ https://issues.apache.org/jira/browse/BEAM-8701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Romanenko resolved BEAM-8701. Fix Version/s: 2.19.0 Resolution: Fixed > Beam Dependency Update Request: commons-io:commons-io > - > > Key: BEAM-8701 > URL: https://issues.apache.org/jira/browse/BEAM-8701 > Project: Beam > Issue Type: Sub-task > Components: dependencies >Reporter: Beam JIRA Bot >Priority: Major > Fix For: 2.19.0 > > Time Spent: 50m > Remaining Estimate: 0h > > - 2019-11-15 19:40:43.908931 > - > Please consider upgrading the dependency commons-io:commons-io. > The current version is 2.5. The latest version is 2.6 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-11-19 21:07:15.614989 > - > Please consider upgrading the dependency commons-io:commons-io. > The current version is 2.5. The latest version is 2.6 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-02 12:13:15.088671 > - > Please consider upgrading the dependency commons-io:commons-io. > The current version is 2.5. The latest version is 2.6 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-09 12:12:27.159523 > - > Please consider upgrading the dependency commons-io:commons-io. > The current version is 2.5. The latest version is 2.6 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-23 12:12:51.094732 > - > Please consider upgrading the dependency commons-io:commons-io. > The current version is 2.5. The latest version is 2.6 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-30 14:07:54.958758 > - > Please consider upgrading the dependency commons-io:commons-io. > The current version is 2.5. The latest version is 2.6 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2020-01-06 12:11:35.960966 > - > Please consider upgrading the dependency commons-io:commons-io. > The current version is 2.5. The latest version is 2.6 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-8701) Beam Dependency Update Request: commons-io:commons-io
[ https://issues.apache.org/jira/browse/BEAM-8701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Romanenko updated BEAM-8701: --- Status: Open (was: Triage Needed) > Beam Dependency Update Request: commons-io:commons-io > - > > Key: BEAM-8701 > URL: https://issues.apache.org/jira/browse/BEAM-8701 > Project: Beam > Issue Type: Sub-task > Components: dependencies >Reporter: Beam JIRA Bot >Priority: Major > Time Spent: 50m > Remaining Estimate: 0h > > - 2019-11-15 19:40:43.908931 > - > Please consider upgrading the dependency commons-io:commons-io. > The current version is 2.5. The latest version is 2.6 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-11-19 21:07:15.614989 > - > Please consider upgrading the dependency commons-io:commons-io. > The current version is 2.5. The latest version is 2.6 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-02 12:13:15.088671 > - > Please consider upgrading the dependency commons-io:commons-io. > The current version is 2.5. The latest version is 2.6 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-09 12:12:27.159523 > - > Please consider upgrading the dependency commons-io:commons-io. > The current version is 2.5. The latest version is 2.6 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-23 12:12:51.094732 > - > Please consider upgrading the dependency commons-io:commons-io. > The current version is 2.5. The latest version is 2.6 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-30 14:07:54.958758 > - > Please consider upgrading the dependency commons-io:commons-io. > The current version is 2.5. The latest version is 2.6 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2020-01-06 12:11:35.960966 > - > Please consider upgrading the dependency commons-io:commons-io. > The current version is 2.5. The latest version is 2.6 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-8749) Beam Dependency Update Request: com.datastax.cassandra:cassandra-driver-mapping
[ https://issues.apache.org/jira/browse/BEAM-8749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Romanenko updated BEAM-8749: --- Status: Open (was: Triage Needed) > Beam Dependency Update Request: > com.datastax.cassandra:cassandra-driver-mapping > --- > > Key: BEAM-8749 > URL: https://issues.apache.org/jira/browse/BEAM-8749 > Project: Beam > Issue Type: Sub-task > Components: dependencies >Reporter: Beam JIRA Bot >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > - 2019-11-19 21:02:36.897455 > - > Please consider upgrading the dependency > com.datastax.cassandra:cassandra-driver-mapping. > The current version is 3.6.0. The latest version is 3.8.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-02 12:07:24.436778 > - > Please consider upgrading the dependency > com.datastax.cassandra:cassandra-driver-mapping. > The current version is 3.6.0. The latest version is 3.8.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-09 12:06:27.390964 > - > Please consider upgrading the dependency > com.datastax.cassandra:cassandra-driver-mapping. > The current version is 3.6.0. The latest version is 3.8.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-23 12:06:37.161035 > - > Please consider upgrading the dependency > com.datastax.cassandra:cassandra-driver-mapping. > The current version is 3.6.0. The latest version is 3.8.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-30 14:02:04.416761 > - > Please consider upgrading the dependency > com.datastax.cassandra:cassandra-driver-mapping. > The current version is 3.6.0. The latest version is 3.8.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2020-01-06 12:05:37.611799 > - > Please consider upgrading the dependency > com.datastax.cassandra:cassandra-driver-mapping. > The current version is 3.6.0. The latest version is 3.8.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (BEAM-8749) Beam Dependency Update Request: com.datastax.cassandra:cassandra-driver-mapping
[ https://issues.apache.org/jira/browse/BEAM-8749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Romanenko resolved BEAM-8749. Fix Version/s: 2.19.0 Resolution: Fixed > Beam Dependency Update Request: > com.datastax.cassandra:cassandra-driver-mapping > --- > > Key: BEAM-8749 > URL: https://issues.apache.org/jira/browse/BEAM-8749 > Project: Beam > Issue Type: Sub-task > Components: dependencies >Reporter: Beam JIRA Bot >Priority: Major > Fix For: 2.19.0 > > Time Spent: 10m > Remaining Estimate: 0h > > - 2019-11-19 21:02:36.897455 > - > Please consider upgrading the dependency > com.datastax.cassandra:cassandra-driver-mapping. > The current version is 3.6.0. The latest version is 3.8.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-02 12:07:24.436778 > - > Please consider upgrading the dependency > com.datastax.cassandra:cassandra-driver-mapping. > The current version is 3.6.0. The latest version is 3.8.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-09 12:06:27.390964 > - > Please consider upgrading the dependency > com.datastax.cassandra:cassandra-driver-mapping. > The current version is 3.6.0. The latest version is 3.8.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-23 12:06:37.161035 > - > Please consider upgrading the dependency > com.datastax.cassandra:cassandra-driver-mapping. > The current version is 3.6.0. The latest version is 3.8.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-30 14:02:04.416761 > - > Please consider upgrading the dependency > com.datastax.cassandra:cassandra-driver-mapping. > The current version is 3.6.0. The latest version is 3.8.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2020-01-06 12:05:37.611799 > - > Please consider upgrading the dependency > com.datastax.cassandra:cassandra-driver-mapping. > The current version is 3.6.0. The latest version is 3.8.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-5546) Beam Dependency Update Request: commons-codec:commons-codec
[ https://issues.apache.org/jira/browse/BEAM-5546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Romanenko updated BEAM-5546: --- Fix Version/s: (was: 2.18.0) 2.19.0 > Beam Dependency Update Request: commons-codec:commons-codec > --- > > Key: BEAM-5546 > URL: https://issues.apache.org/jira/browse/BEAM-5546 > Project: Beam > Issue Type: Sub-task > Components: dependencies >Reporter: Beam JIRA Bot >Priority: Major > Fix For: 2.19.0 > > Time Spent: 40m > Remaining Estimate: 0h > > - 2018-10-01 19:30:05.437342 > - > Please consider upgrading the dependency commons-codec:commons-codec. > The current version is 1.9. The latest version is 1.11 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2018-10-08 12:16:29.372797 > - > Please consider upgrading the dependency commons-codec:commons-codec. > The current version is 1.9. The latest version is 1.11 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2020-01-06 12:11:28.511077 > - > Please consider upgrading the dependency commons-codec:commons-codec. > The current version is 1.9. The latest version is 1.14 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (BEAM-5546) Beam Dependency Update Request: commons-codec:commons-codec
[ https://issues.apache.org/jira/browse/BEAM-5546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Romanenko resolved BEAM-5546. Fix Version/s: (was: Not applicable) 2.18.0 Resolution: Fixed > Beam Dependency Update Request: commons-codec:commons-codec > --- > > Key: BEAM-5546 > URL: https://issues.apache.org/jira/browse/BEAM-5546 > Project: Beam > Issue Type: Sub-task > Components: dependencies >Reporter: Beam JIRA Bot >Priority: Major > Fix For: 2.18.0 > > Time Spent: 40m > Remaining Estimate: 0h > > - 2018-10-01 19:30:05.437342 > - > Please consider upgrading the dependency commons-codec:commons-codec. > The current version is 1.9. The latest version is 1.11 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2018-10-08 12:16:29.372797 > - > Please consider upgrading the dependency commons-codec:commons-codec. > The current version is 1.9. The latest version is 1.11 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2020-01-06 12:11:28.511077 > - > Please consider upgrading the dependency commons-codec:commons-codec. > The current version is 1.9. The latest version is 1.14 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-5546) Beam Dependency Update Request: commons-codec:commons-codec
[ https://issues.apache.org/jira/browse/BEAM-5546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Romanenko updated BEAM-5546: --- Status: Open (was: Triage Needed) > Beam Dependency Update Request: commons-codec:commons-codec > --- > > Key: BEAM-5546 > URL: https://issues.apache.org/jira/browse/BEAM-5546 > Project: Beam > Issue Type: Sub-task > Components: dependencies >Reporter: Beam JIRA Bot >Priority: Major > Fix For: Not applicable > > Time Spent: 40m > Remaining Estimate: 0h > > - 2018-10-01 19:30:05.437342 > - > Please consider upgrading the dependency commons-codec:commons-codec. > The current version is 1.9. The latest version is 1.11 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2018-10-08 12:16:29.372797 > - > Please consider upgrading the dependency commons-codec:commons-codec. > The current version is 1.9. The latest version is 1.11 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2020-01-06 12:11:28.511077 > - > Please consider upgrading the dependency commons-codec:commons-codec. > The current version is 1.9. The latest version is 1.14 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (BEAM-8978) Report saved data size from HadoopFormatIOIT
[ https://issues.apache.org/jira/browse/BEAM-8978?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Romanenko resolved BEAM-8978. Fix Version/s: 2.19.0 Resolution: Fixed > Report saved data size from HadoopFormatIOIT > > > Key: BEAM-8978 > URL: https://issues.apache.org/jira/browse/BEAM-8978 > Project: Beam > Issue Type: Sub-task > Components: io-java-hadoop-format, testing >Reporter: Michal Walenia >Assignee: Pawel Pasterz >Priority: Minor > Fix For: 2.19.0 > > Time Spent: 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
