[jira] [Work logged] (BEAM-3925) Support ValueProviders in KafkaIO
[ https://issues.apache.org/jira/browse/BEAM-3925?focusedWorklogId=650962&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-650962 ] ASF GitHub Bot logged work on BEAM-3925: Author: ASF GitHub Bot Created on: 15/Sep/21 08:23 Start Date: 15/Sep/21 08:23 Worklog Time Spent: 10m Work Description: samle-ovo commented on pull request #6636: URL: https://github.com/apache/beam/pull/6636#issuecomment-919807263 @chamikaramj yeah, I check it again. It should help. For me at first, it is difficult to follow Google documents. They did not explain how to write the template file, it makes everything a bit confused. The Docker part is no brainer. @pseudomuto : Sorry, I take everything I said back :D -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@beam.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 650962) Time Spent: 9h 40m (was: 9.5h) > Support ValueProviders in KafkaIO > - > > Key: BEAM-3925 > URL: https://issues.apache.org/jira/browse/BEAM-3925 > Project: Beam > Issue Type: Improvement > Components: io-java-kafka >Reporter: Sameer Abhyankar >Priority: P3 > Time Spent: 9h 40m > Remaining Estimate: 0h > > Add ValueProvider support for the various methods in KafkaIO. This would > allow us to use KafkaIO in reusable pipeline templates. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-3925) Support ValueProviders in KafkaIO
[ https://issues.apache.org/jira/browse/BEAM-3925?focusedWorklogId=650845&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-650845 ] ASF GitHub Bot logged work on BEAM-3925: Author: ASF GitHub Bot Created on: 14/Sep/21 22:35 Start Date: 14/Sep/21 22:35 Worklog Time Spent: 10m Work Description: chamikaramj commented on pull request #6636: URL: https://github.com/apache/beam/pull/6636#issuecomment-919559585 I agree with @pseudomuto. I think Flex templates allow using templates without adding ValueProviders by running the pipeline from a container. May be the pipeline you are trying to use is just not available as a Flex template yet ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@beam.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 650845) Time Spent: 9.5h (was: 9h 20m) > Support ValueProviders in KafkaIO > - > > Key: BEAM-3925 > URL: https://issues.apache.org/jira/browse/BEAM-3925 > Project: Beam > Issue Type: Improvement > Components: io-java-kafka >Reporter: Sameer Abhyankar >Priority: P3 > Time Spent: 9.5h > Remaining Estimate: 0h > > Add ValueProvider support for the various methods in KafkaIO. This would > allow us to use KafkaIO in reusable pipeline templates. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-3925) Support ValueProviders in KafkaIO
[ https://issues.apache.org/jira/browse/BEAM-3925?focusedWorklogId=650828&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-650828 ] ASF GitHub Bot logged work on BEAM-3925: Author: ASF GitHub Bot Created on: 14/Sep/21 22:02 Start Date: 14/Sep/21 22:02 Worklog Time Spent: 10m Work Description: samle-ovo edited a comment on pull request #6636: URL: https://github.com/apache/beam/pull/6636#issuecomment-919536064 @pseudomuto : I dont think it helps. Flex templates requires people to write description of the pipeline manually. It is not really idea situation at all. @rangadi : Can we just change the numSplits equals to numPartitions (I think it is very similar to consumers in consumer groups, isnt it ) ? Everything will be much clearer and people will not set not correct number. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@beam.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 650828) Time Spent: 9h 20m (was: 9h 10m) > Support ValueProviders in KafkaIO > - > > Key: BEAM-3925 > URL: https://issues.apache.org/jira/browse/BEAM-3925 > Project: Beam > Issue Type: Improvement > Components: io-java-kafka >Reporter: Sameer Abhyankar >Priority: P3 > Time Spent: 9h 20m > Remaining Estimate: 0h > > Add ValueProvider support for the various methods in KafkaIO. This would > allow us to use KafkaIO in reusable pipeline templates. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-3925) Support ValueProviders in KafkaIO
[ https://issues.apache.org/jira/browse/BEAM-3925?focusedWorklogId=650827&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-650827 ] ASF GitHub Bot logged work on BEAM-3925: Author: ASF GitHub Bot Created on: 14/Sep/21 22:00 Start Date: 14/Sep/21 22:00 Worklog Time Spent: 10m Work Description: samle-ovo edited a comment on pull request #6636: URL: https://github.com/apache/beam/pull/6636#issuecomment-919536064 @pseudomuto : I dont think it helps. Flex templates requires people to write description of the pipeline manually. It is not really idea situation at all. @rangadi : Can we just change the numSplits to numPartitions/numConsumers ? Everything will be much clearer and people will not set not correct number. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@beam.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 650827) Time Spent: 9h 10m (was: 9h) > Support ValueProviders in KafkaIO > - > > Key: BEAM-3925 > URL: https://issues.apache.org/jira/browse/BEAM-3925 > Project: Beam > Issue Type: Improvement > Components: io-java-kafka >Reporter: Sameer Abhyankar >Priority: P3 > Time Spent: 9h 10m > Remaining Estimate: 0h > > Add ValueProvider support for the various methods in KafkaIO. This would > allow us to use KafkaIO in reusable pipeline templates. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-3925) Support ValueProviders in KafkaIO
[ https://issues.apache.org/jira/browse/BEAM-3925?focusedWorklogId=650823&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-650823 ] ASF GitHub Bot logged work on BEAM-3925: Author: ASF GitHub Bot Created on: 14/Sep/21 21:47 Start Date: 14/Sep/21 21:47 Worklog Time Spent: 10m Work Description: samle-ovo commented on pull request #6636: URL: https://github.com/apache/beam/pull/6636#issuecomment-919536064 @pseudomuto : I dont think it helps. Flex templates requires people to write description of the pipeline manually. It is not really idea situation at all. @rangadi : Can we just change the numSplits to numPartitions ? Everything will be much clearer and people will not set not correct number. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@beam.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 650823) Time Spent: 9h (was: 8h 50m) > Support ValueProviders in KafkaIO > - > > Key: BEAM-3925 > URL: https://issues.apache.org/jira/browse/BEAM-3925 > Project: Beam > Issue Type: Improvement > Components: io-java-kafka >Reporter: Sameer Abhyankar >Priority: P3 > Time Spent: 9h > Remaining Estimate: 0h > > Add ValueProvider support for the various methods in KafkaIO. This would > allow us to use KafkaIO in reusable pipeline templates. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-3925) Support ValueProviders in KafkaIO
[ https://issues.apache.org/jira/browse/BEAM-3925?focusedWorklogId=425317&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-425317 ] ASF GitHub Bot logged work on BEAM-3925: Author: ASF GitHub Bot Created on: 20/Apr/20 13:18 Start Date: 20/Apr/20 13:18 Worklog Time Spent: 10m Work Description: pseudomuto commented on issue #6636: URL: https://github.com/apache/beam/pull/6636#issuecomment-616547340 FWIW, [Flex templates](https://cloud.google.com/dataflow/docs/guides/templates/using-flex-templates) bypass the requirement to use ValueProvider options in your pipelines (current in beta though) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 425317) Time Spent: 8h 50m (was: 8h 40m) > Support ValueProviders in KafkaIO > - > > Key: BEAM-3925 > URL: https://issues.apache.org/jira/browse/BEAM-3925 > Project: Beam > Issue Type: Improvement > Components: io-java-kafka >Reporter: Sameer Abhyankar >Assignee: Pramod Upamanyu >Priority: Major > Time Spent: 8h 50m > Remaining Estimate: 0h > > Add ValueProvider support for the various methods in KafkaIO. This would > allow us to use KafkaIO in reusable pipeline templates. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-3925) Support ValueProviders in KafkaIO
[ https://issues.apache.org/jira/browse/BEAM-3925?focusedWorklogId=424301&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-424301 ] ASF GitHub Bot logged work on BEAM-3925: Author: ASF GitHub Bot Created on: 17/Apr/20 17:27 Start Date: 17/Apr/20 17:27 Worklog Time Spent: 10m Work Description: aromanenko-dev commented on issue #6636: [BEAM-3925] [DO NOT MERGE] KafkaIO : Value provider support for reader configuration. URL: https://github.com/apache/beam/pull/6636#issuecomment-615370371 @davidscottturner I'm not aware of if anyone works on this. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 424301) Time Spent: 8h 40m (was: 8.5h) > Support ValueProviders in KafkaIO > - > > Key: BEAM-3925 > URL: https://issues.apache.org/jira/browse/BEAM-3925 > Project: Beam > Issue Type: Improvement > Components: io-java-kafka >Reporter: Sameer Abhyankar >Assignee: Pramod Upamanyu >Priority: Major > Time Spent: 8h 40m > Remaining Estimate: 0h > > Add ValueProvider support for the various methods in KafkaIO. This would > allow us to use KafkaIO in reusable pipeline templates. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-3925) Support ValueProviders in KafkaIO
[ https://issues.apache.org/jira/browse/BEAM-3925?focusedWorklogId=419153&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-419153 ] ASF GitHub Bot logged work on BEAM-3925: Author: ASF GitHub Bot Created on: 09/Apr/20 06:32 Start Date: 09/Apr/20 06:32 Worklog Time Spent: 10m Work Description: davidscottturner commented on issue #6636: [BEAM-3925] [DO NOT MERGE] KafkaIO : Value provider support for reader configuration. URL: https://github.com/apache/beam/pull/6636#issuecomment-611355305 Hi, is there any more progress on supporting Value provider? I've just hit the snag of this while trying to create a Dataflow template as well. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 419153) Time Spent: 8.5h (was: 8h 20m) > Support ValueProviders in KafkaIO > - > > Key: BEAM-3925 > URL: https://issues.apache.org/jira/browse/BEAM-3925 > Project: Beam > Issue Type: Improvement > Components: io-java-kafka >Reporter: Sameer Abhyankar >Assignee: Pramod Upamanyu >Priority: Major > Time Spent: 8.5h > Remaining Estimate: 0h > > Add ValueProvider support for the various methods in KafkaIO. This would > allow us to use KafkaIO in reusable pipeline templates. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-3925) Support ValueProviders in KafkaIO
[ https://issues.apache.org/jira/browse/BEAM-3925?focusedWorklogId=399236&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-399236 ] ASF GitHub Bot logged work on BEAM-3925: Author: ASF GitHub Bot Created on: 06/Mar/20 17:08 Start Date: 06/Mar/20 17:08 Worklog Time Spent: 10m Work Description: aromanenko-dev commented on issue #6636: [BEAM-3925] [DO NOT MERGE] KafkaIO : Value provider support for reader configuration. URL: https://github.com/apache/beam/pull/6636#issuecomment-595867006 Ok, I see. Thanks! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 399236) Time Spent: 8h 20m (was: 8h 10m) > Support ValueProviders in KafkaIO > - > > Key: BEAM-3925 > URL: https://issues.apache.org/jira/browse/BEAM-3925 > Project: Beam > Issue Type: Improvement > Components: io-java-kafka >Reporter: Sameer Abhyankar >Assignee: Pramod Upamanyu >Priority: Major > Time Spent: 8h 20m > Remaining Estimate: 0h > > Add ValueProvider support for the various methods in KafkaIO. This would > allow us to use KafkaIO in reusable pipeline templates. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-3925) Support ValueProviders in KafkaIO
[ https://issues.apache.org/jira/browse/BEAM-3925?focusedWorklogId=397964&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-397964 ] ASF GitHub Bot logged work on BEAM-3925: Author: ASF GitHub Bot Created on: 04/Mar/20 23:25 Start Date: 04/Mar/20 23:25 Worklog Time Spent: 10m Work Description: rangadi commented on issue #6636: [BEAM-3925] [DO NOT MERGE] KafkaIO : Value provider support for reader configuration. URL: https://github.com/apache/beam/pull/6636#issuecomment-594928908 It might be a bit more than than 'just' :) We can't shrink the splits since the compute graph is already constructed, I think. We would know only inside the reader during reader initialization what the total number of partitions we need to consumer. Each reader has to decide which partitions to consume based on reader id. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 397964) Time Spent: 8h 10m (was: 8h) > Support ValueProviders in KafkaIO > - > > Key: BEAM-3925 > URL: https://issues.apache.org/jira/browse/BEAM-3925 > Project: Beam > Issue Type: Improvement > Components: io-java-kafka >Reporter: Sameer Abhyankar >Assignee: Pramod Upamanyu >Priority: Major > Time Spent: 8h 10m > Remaining Estimate: 0h > > Add ValueProvider support for the various methods in KafkaIO. This would > allow us to use KafkaIO in reusable pipeline templates. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-3925) Support ValueProviders in KafkaIO
[ https://issues.apache.org/jira/browse/BEAM-3925?focusedWorklogId=397699&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-397699 ] ASF GitHub Bot logged work on BEAM-3925: Author: ASF GitHub Bot Created on: 04/Mar/20 17:14 Start Date: 04/Mar/20 17:14 Worklog Time Spent: 10m Work Description: aromanenko-dev commented on issue #6636: [BEAM-3925] [DO NOT MERGE] KafkaIO : Value provider support for reader configuration. URL: https://github.com/apache/beam/pull/6636#issuecomment-594672715 > We just need to handle that properly. @rangadi Do you mean that we just need to shrink the number of split accordingly in this case to avoid empty splits in runtime? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 397699) Time Spent: 8h (was: 7h 50m) > Support ValueProviders in KafkaIO > - > > Key: BEAM-3925 > URL: https://issues.apache.org/jira/browse/BEAM-3925 > Project: Beam > Issue Type: Improvement > Components: io-java-kafka >Reporter: Sameer Abhyankar >Assignee: Pramod Upamanyu >Priority: Major > Time Spent: 8h > Remaining Estimate: 0h > > Add ValueProvider support for the various methods in KafkaIO. This would > allow us to use KafkaIO in reusable pipeline templates. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-3925) Support ValueProviders in KafkaIO
[ https://issues.apache.org/jira/browse/BEAM-3925?focusedWorklogId=397698&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-397698 ] ASF GitHub Bot logged work on BEAM-3925: Author: ASF GitHub Bot Created on: 04/Mar/20 17:14 Start Date: 04/Mar/20 17:14 Worklog Time Spent: 10m Work Description: aromanenko-dev commented on issue #6636: [BEAM-3925] [DO NOT MERGE] KafkaIO : Value provider support for reader configuration. URL: https://github.com/apache/beam/pull/6636#issuecomment-594672715 > We just need to handle that properly. @rangadi Do you mean that we just need to shrink the number of split accordingly in this case to avoid empty splits in runtime? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 397698) Time Spent: 7h 50m (was: 7h 40m) > Support ValueProviders in KafkaIO > - > > Key: BEAM-3925 > URL: https://issues.apache.org/jira/browse/BEAM-3925 > Project: Beam > Issue Type: Improvement > Components: io-java-kafka >Reporter: Sameer Abhyankar >Assignee: Pramod Upamanyu >Priority: Major > Time Spent: 7h 50m > Remaining Estimate: 0h > > Add ValueProvider support for the various methods in KafkaIO. This would > allow us to use KafkaIO in reusable pipeline templates. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-3925) Support ValueProviders in KafkaIO
[ https://issues.apache.org/jira/browse/BEAM-3925?focusedWorklogId=397121&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-397121 ] ASF GitHub Bot logged work on BEAM-3925: Author: ASF GitHub Bot Created on: 03/Mar/20 23:14 Start Date: 03/Mar/20 23:14 Worklog Time Spent: 10m Work Description: rangadi commented on issue #6636: [BEAM-3925] [DO NOT MERGE] KafkaIO : Value provider support for reader configuration. URL: https://github.com/apache/beam/pull/6636#issuecomment-594220066 @pseudomuto @davidheryanto : The main issue we need to address is proper support for `.numSplits()` interface. If user sets `numSplits` to 20 and Kafka topic has only 5 partitions, 15 splits (`readers`) will not have any work. We just need to handle that properly. I don't think it is that much more work. cc: @aromanenko-dev. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 397121) Time Spent: 7h 40m (was: 7.5h) > Support ValueProviders in KafkaIO > - > > Key: BEAM-3925 > URL: https://issues.apache.org/jira/browse/BEAM-3925 > Project: Beam > Issue Type: Improvement > Components: io-java-kafka >Reporter: Sameer Abhyankar >Assignee: Pramod Upamanyu >Priority: Major > Time Spent: 7h 40m > Remaining Estimate: 0h > > Add ValueProvider support for the various methods in KafkaIO. This would > allow us to use KafkaIO in reusable pipeline templates. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-3925) Support ValueProviders in KafkaIO
[ https://issues.apache.org/jira/browse/BEAM-3925?focusedWorklogId=396765&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-396765 ] ASF GitHub Bot logged work on BEAM-3925: Author: ASF GitHub Bot Created on: 03/Mar/20 14:13 Start Date: 03/Mar/20 14:13 Worklog Time Spent: 10m Work Description: pseudomuto commented on issue #6636: [BEAM-3925] [DO NOT MERGE] KafkaIO : Value provider support for reader configuration. URL: https://github.com/apache/beam/pull/6636#issuecomment-593970949 @rangadi would you be able to clarify why this was closed? Similar to @davidheryanto above, we're also trying to find a way to support ValueProvider options with KafkaIO, and it would be great to not have to maintain a separate connector if possible. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 396765) Time Spent: 7.5h (was: 7h 20m) > Support ValueProviders in KafkaIO > - > > Key: BEAM-3925 > URL: https://issues.apache.org/jira/browse/BEAM-3925 > Project: Beam > Issue Type: Improvement > Components: io-java-kafka >Reporter: Sameer Abhyankar >Assignee: Pramod Upamanyu >Priority: Major > Time Spent: 7.5h > Remaining Estimate: 0h > > Add ValueProvider support for the various methods in KafkaIO. This would > allow us to use KafkaIO in reusable pipeline templates. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-3925) Support ValueProviders in KafkaIO
[ https://issues.apache.org/jira/browse/BEAM-3925?focusedWorklogId=212848&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-212848 ] ASF GitHub Bot logged work on BEAM-3925: Author: ASF GitHub Bot Created on: 14/Mar/19 02:58 Start Date: 14/Mar/19 02:58 Worklog Time Spent: 10m Work Description: davidheryanto commented on issue #6636: [BEAM-3925] [DO NOT MERGE] KafkaIO : Value provider support for reader configuration. URL: https://github.com/apache/beam/pull/6636#issuecomment-472685514 Hi @rangadi, any reason why this is not merged to the master branch? Is it because it is not fully tested or is there any other concerns? I see it has been merged here https://github.com/GoogleCloudPlatform/DataflowTemplates/blob/master/src/main/java/com/google/cloud/teleport/kafka/connector/KafkaIO.java Our deployment kind of needs this functionality of specifying Kafka brokers at runtime using Dataflow templates. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 212848) Time Spent: 7h 20m (was: 7h 10m) > Support ValueProviders in KafkaIO > - > > Key: BEAM-3925 > URL: https://issues.apache.org/jira/browse/BEAM-3925 > Project: Beam > Issue Type: Improvement > Components: io-java-kafka >Reporter: Sameer Abhyankar >Assignee: Pramod Upamanyu >Priority: Major > Labels: triaged > Time Spent: 7h 20m > Remaining Estimate: 0h > > Add ValueProvider support for the various methods in KafkaIO. This would > allow us to use KafkaIO in reusable pipeline templates. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-3925) Support ValueProviders in KafkaIO
[ https://issues.apache.org/jira/browse/BEAM-3925?focusedWorklogId=212846&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-212846 ] ASF GitHub Bot logged work on BEAM-3925: Author: ASF GitHub Bot Created on: 14/Mar/19 02:57 Start Date: 14/Mar/19 02:57 Worklog Time Spent: 10m Work Description: davidheryanto commented on issue #6636: [BEAM-3925] [DO NOT MERGE] KafkaIO : Value provider support for reader configuration. URL: https://github.com/apache/beam/pull/6636#issuecomment-472685514 Hi @rangadi, any reason why this is not merged to the master branch? Is it because it is not fully tested or is there any other concerns? I see it has been merged here https://github.com/GoogleCloudPlatform/DataflowTemplates/blob/master/src/main/java/com/google/cloud/teleport/kafka/connector/KafkaIO.java Our deployment kind of needs this functinonality of specifying Kafka brokers at runtime using Dataflow templates. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 212846) Time Spent: 7h 10m (was: 7h) > Support ValueProviders in KafkaIO > - > > Key: BEAM-3925 > URL: https://issues.apache.org/jira/browse/BEAM-3925 > Project: Beam > Issue Type: Improvement > Components: io-java-kafka >Reporter: Sameer Abhyankar >Assignee: Pramod Upamanyu >Priority: Major > Labels: triaged > Time Spent: 7h 10m > Remaining Estimate: 0h > > Add ValueProvider support for the various methods in KafkaIO. This would > allow us to use KafkaIO in reusable pipeline templates. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
