[jira] [Commented] (BEAM-9330) Add support for Protobuf and JSON Schema on KafkaIO + Confluent Schema Registry
[ https://issues.apache.org/jira/browse/BEAM-9330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17043199#comment-17043199 ] Ismaël Mejía commented on BEAM-9330: Notice that this one can be addressed only when CSR 6 is released, probably worth to check if users can use the CSR 6 client with a CSR 5 server too. > Add support for Protobuf and JSON Schema on KafkaIO + Confluent Schema > Registry > --- > > Key: BEAM-9330 > URL: https://issues.apache.org/jira/browse/BEAM-9330 > Project: Beam > Issue Type: Improvement > Components: io-java-kafka >Reporter: Ismaël Mejía >Assignee: Ismaël Mejía >Priority: Minor > > Confluent Schema Registry (CSR) 6 introduces support for other Schema types > like Protobuf and JSON Schema. We should adapt KafkaIO to obtain the type of > Schema and configure correctly its Deserializer and Coder. > Protobuf support > [https://github.com/confluentinc/schema-registry/pull/1285] > JSON Schema support > [https://github.com/confluentinc/schema-registry/pull/1289] > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9330) Add support for Protobuf and JSON Schema on KafkaIO + Confluent Schema Registry
[ https://issues.apache.org/jira/browse/BEAM-9330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismaël Mejía updated BEAM-9330: --- Description: Confluent Schema Registry (CSR) 6 introduces support for other Schema types like Protobuf and JSON Schema. We should adapt KafkaIO to obtain the type of Schema and configure correctly its Deserializer and Coder. Protobuf support [https://github.com/confluentinc/schema-registry/pull/1285] JSON Schema support [https://github.com/confluentinc/schema-registry/pull/1289] was: Confluent Schema Registry 6 introduces support for other Schema types like Protobuf and JSON Schema. We should adapt KafkaIO to obtain the type of Schema and configure correctly its Deserializer and Coder. Protobuf support https://github.com/confluentinc/schema-registry/pull/1285 JSON Schema support [https://github.com/confluentinc/schema-registry/pull/1289] > Add support for Protobuf and JSON Schema on KafkaIO + Confluent Schema > Registry > --- > > Key: BEAM-9330 > URL: https://issues.apache.org/jira/browse/BEAM-9330 > Project: Beam > Issue Type: Improvement > Components: io-java-kafka >Reporter: Ismaël Mejía >Assignee: Ismaël Mejía >Priority: Minor > > Confluent Schema Registry (CSR) 6 introduces support for other Schema types > like Protobuf and JSON Schema. We should adapt KafkaIO to obtain the type of > Schema and configure correctly its Deserializer and Coder. > Protobuf support > [https://github.com/confluentinc/schema-registry/pull/1285] > JSON Schema support > [https://github.com/confluentinc/schema-registry/pull/1289] > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9329) Support request of schemas by version on KafkaIO + Confluent Schema Registry
[ https://issues.apache.org/jira/browse/BEAM-9329?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismaël Mejía updated BEAM-9329: --- Fix Version/s: (was: 2.20.0) > Support request of schemas by version on KafkaIO + Confluent Schema Registry > > > Key: BEAM-9329 > URL: https://issues.apache.org/jira/browse/BEAM-9329 > Project: Beam > Issue Type: Improvement > Components: io-java-kafka >Reporter: Ismaël Mejía >Assignee: Ismaël Mejía >Priority: Minor > > BEAM-7310 introduced support to get Avro Schemas from Confluent Schema > Registry as part of KafkaIO. By default it always takes the latest (most > recent) version of the Schema. It would be nice to allow users to also > request specific versions of the Schema. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9364) Refactor KafkaIO to use DeserializerProviders
[ https://issues.apache.org/jira/browse/BEAM-9364?focusedWorklogId=391551&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-391551 ] ASF GitHub Bot logged work on BEAM-9364: Author: ASF GitHub Bot Created on: 24/Feb/20 07:43 Start Date: 24/Feb/20 07:43 Worklog Time Spent: 10m Work Description: iemejia commented on pull request #10947: [BEAM-9364] Refactor KafkaIO to use DeserializerProviders URL: https://github.com/apache/beam/pull/10947 This builds on your previous PR with some API improvements. Since this adjusts the public API that will be out as part of 2.20.0 is probably better if we include it as part of 2.20.0 too. R: @aromanenko-dev This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 391551) Remaining Estimate: 0h Time Spent: 10m > Refactor KafkaIO to use DeserializerProviders > - > > Key: BEAM-9364 > URL: https://issues.apache.org/jira/browse/BEAM-9364 > Project: Beam > Issue Type: Improvement > Components: io-java-kafka >Reporter: Ismaël Mejía >Assignee: Ismaël Mejía >Priority: Major > Fix For: 2.20.0 > > Time Spent: 10m > Remaining Estimate: 0h > > BEAM-7310 introduced support for Confluent Schema Registry, even if the API > did not explicitly mention DeserializerProviders it opened the option to use > multiple providers of Deserializers. The goal of this ticket is to refactor > the KafkaIO API to reflect this new concept both for local Deserializer > resolution and Confluent Schema Registry based (notice that this one becomes > better isolated). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9364) Refactor KafkaIO to use DeserializerProviders
[ https://issues.apache.org/jira/browse/BEAM-9364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismaël Mejía updated BEAM-9364: --- Fix Version/s: 2.20.0 > Refactor KafkaIO to use DeserializerProviders > - > > Key: BEAM-9364 > URL: https://issues.apache.org/jira/browse/BEAM-9364 > Project: Beam > Issue Type: Improvement > Components: io-java-kafka >Reporter: Ismaël Mejía >Assignee: Ismaël Mejía >Priority: Major > Fix For: 2.20.0 > > > BEAM-7310 introduced support for Confluent Schema Registry, even if the API > did not explicitly mention DeserializerProviders it opened the option to use > multiple providers of Deserializers. The goal of this ticket is to refactor > the KafkaIO API to reflect this new concept both for local Deserializer > resolution and Confluent Schema Registry based (notice that this one becomes > better isolated). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work started] (BEAM-9364) Refactor KafkaIO to use DeserializerProviders
[ https://issues.apache.org/jira/browse/BEAM-9364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on BEAM-9364 started by Ismaël Mejía. -- > Refactor KafkaIO to use DeserializerProviders > - > > Key: BEAM-9364 > URL: https://issues.apache.org/jira/browse/BEAM-9364 > Project: Beam > Issue Type: Improvement > Components: io-java-kafka >Reporter: Ismaël Mejía >Assignee: Ismaël Mejía >Priority: Major > Fix For: 2.20.0 > > > BEAM-7310 introduced support for Confluent Schema Registry, even if the API > did not explicitly mention DeserializerProviders it opened the option to use > multiple providers of Deserializers. The goal of this ticket is to refactor > the KafkaIO API to reflect this new concept both for local Deserializer > resolution and Confluent Schema Registry based (notice that this one becomes > better isolated). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (BEAM-9364) Refactor KafkaIO to use DeserializerProviders
Ismaël Mejía created BEAM-9364: -- Summary: Refactor KafkaIO to use DeserializerProviders Key: BEAM-9364 URL: https://issues.apache.org/jira/browse/BEAM-9364 Project: Beam Issue Type: Improvement Components: io-java-kafka Reporter: Ismaël Mejía Assignee: Ismaël Mejía BEAM-7310 introduced support for Confluent Schema Registry, even if the API did not explicitly mention DeserializerProviders it opened the option to use multiple providers of Deserializers. The goal of this ticket is to refactor the KafkaIO API to reflect this new concept both for local Deserializer resolution and Confluent Schema Registry based (notice that this one becomes better isolated). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9364) Refactor KafkaIO to use DeserializerProviders
[ https://issues.apache.org/jira/browse/BEAM-9364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismaël Mejía updated BEAM-9364: --- Description: BEAM-7310 introduced support for Confluent Schema Registry, even if the API did not explicitly mention DeserializerProviders it opened the option to use multiple providers of Deserializers. The goal of this ticket is to refactor the KafkaIO API to reflect this new concept both for local Deserializer resolution and Confluent Schema Registry based (notice that this one becomes better isolated). (was: BEAM-7310 introduced support for Confluent Schema Registry, even if the API did not explicitly mention DeserializerProviders it opened the option to use multiple providers of Deserializers. The goal of this ticket is to refactor the KafkaIO API to reflect this new concept both for local Deserializer resolution and Confluent Schema Registry based (notice that this one becomes better isolated).) > Refactor KafkaIO to use DeserializerProviders > - > > Key: BEAM-9364 > URL: https://issues.apache.org/jira/browse/BEAM-9364 > Project: Beam > Issue Type: Improvement > Components: io-java-kafka >Reporter: Ismaël Mejía >Assignee: Ismaël Mejía >Priority: Major > > BEAM-7310 introduced support for Confluent Schema Registry, even if the API > did not explicitly mention DeserializerProviders it opened the option to use > multiple providers of Deserializers. The goal of this ticket is to refactor > the KafkaIO API to reflect this new concept both for local Deserializer > resolution and Confluent Schema Registry based (notice that this one becomes > better isolated). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9364) Refactor KafkaIO to use DeserializerProviders
[ https://issues.apache.org/jira/browse/BEAM-9364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismaël Mejía updated BEAM-9364: --- Status: Open (was: Triage Needed) > Refactor KafkaIO to use DeserializerProviders > - > > Key: BEAM-9364 > URL: https://issues.apache.org/jira/browse/BEAM-9364 > Project: Beam > Issue Type: Improvement > Components: io-java-kafka >Reporter: Ismaël Mejía >Assignee: Ismaël Mejía >Priority: Major > Fix For: 2.20.0 > > > BEAM-7310 introduced support for Confluent Schema Registry, even if the API > did not explicitly mention DeserializerProviders it opened the option to use > multiple providers of Deserializers. The goal of this ticket is to refactor > the KafkaIO API to reflect this new concept both for local Deserializer > resolution and Confluent Schema Registry based (notice that this one becomes > better isolated). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9295) Add Flink 1.10 build target and Make FlinkRunner compatible with Flink 1.10
[ https://issues.apache.org/jira/browse/BEAM-9295?focusedWorklogId=391548&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-391548 ] ASF GitHub Bot logged work on BEAM-9295: Author: ASF GitHub Bot Created on: 24/Feb/20 07:18 Start Date: 24/Feb/20 07:18 Worklog Time Spent: 10m Work Description: sunjincheng121 commented on issue #10945: [BEAM-9295] Add Flink 1.10 build target and Make FlinkRunner compatible with Flink 1.10 URL: https://github.com/apache/beam/pull/10945#issuecomment-590186860 Run Python2_PVR_Flink PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 391548) Time Spent: 50m (was: 40m) > Add Flink 1.10 build target and Make FlinkRunner compatible with Flink 1.10 > --- > > Key: BEAM-9295 > URL: https://issues.apache.org/jira/browse/BEAM-9295 > Project: Beam > Issue Type: New Feature > Components: runner-flink >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Fix For: 2.20.0 > > Time Spent: 50m > Remaining Estimate: 0h > > Apache Flink 1.10 has completed the final release vote, see [1]. So, I would > like to add Flink 1.10 build target and make Flink Runner compatible with > Flink 1.10. > And I appreciate it if you can leave your suggestions or comments! > [1] > https://lists.apache.org/thread.html/r97672d4d1e47372cebf23e6643a6cc30a06bfbdf3f277b0be3695b15%40%3Cdev.flink.apache.org%3E -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9295) Add Flink 1.10 build target and Make FlinkRunner compatible with Flink 1.10
[ https://issues.apache.org/jira/browse/BEAM-9295?focusedWorklogId=391545&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-391545 ] ASF GitHub Bot logged work on BEAM-9295: Author: ASF GitHub Bot Created on: 24/Feb/20 07:06 Start Date: 24/Feb/20 07:06 Worklog Time Spent: 10m Work Description: sunjincheng121 commented on issue #10945: [BEAM-9295] Add Flink 1.10 build target and Make FlinkRunner compatible with Flink 1.10 URL: https://github.com/apache/beam/pull/10945#issuecomment-590190781 Run Python2_PVR_Flink PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 391545) Time Spent: 40m (was: 0.5h) > Add Flink 1.10 build target and Make FlinkRunner compatible with Flink 1.10 > --- > > Key: BEAM-9295 > URL: https://issues.apache.org/jira/browse/BEAM-9295 > Project: Beam > Issue Type: New Feature > Components: runner-flink >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Fix For: 2.20.0 > > Time Spent: 40m > Remaining Estimate: 0h > > Apache Flink 1.10 has completed the final release vote, see [1]. So, I would > like to add Flink 1.10 build target and make Flink Runner compatible with > Flink 1.10. > And I appreciate it if you can leave your suggestions or comments! > [1] > https://lists.apache.org/thread.html/r97672d4d1e47372cebf23e6643a6cc30a06bfbdf3f277b0be3695b15%40%3Cdev.flink.apache.org%3E -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9295) Add Flink 1.10 build target and Make FlinkRunner compatible with Flink 1.10
[ https://issues.apache.org/jira/browse/BEAM-9295?focusedWorklogId=391540&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-391540 ] ASF GitHub Bot logged work on BEAM-9295: Author: ASF GitHub Bot Created on: 24/Feb/20 06:46 Start Date: 24/Feb/20 06:46 Worklog Time Spent: 10m Work Description: sunjincheng121 commented on issue #10945: [BEAM-9295] Add Flink 1.10 build target and Make FlinkRunner compatible with Flink 1.10 URL: https://github.com/apache/beam/pull/10945#issuecomment-590186806 Run CommunityMetrics PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 391540) Time Spent: 20m (was: 10m) > Add Flink 1.10 build target and Make FlinkRunner compatible with Flink 1.10 > --- > > Key: BEAM-9295 > URL: https://issues.apache.org/jira/browse/BEAM-9295 > Project: Beam > Issue Type: New Feature > Components: runner-flink >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Fix For: 2.20.0 > > Time Spent: 20m > Remaining Estimate: 0h > > Apache Flink 1.10 has completed the final release vote, see [1]. So, I would > like to add Flink 1.10 build target and make Flink Runner compatible with > Flink 1.10. > And I appreciate it if you can leave your suggestions or comments! > [1] > https://lists.apache.org/thread.html/r97672d4d1e47372cebf23e6643a6cc30a06bfbdf3f277b0be3695b15%40%3Cdev.flink.apache.org%3E -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9295) Add Flink 1.10 build target and Make FlinkRunner compatible with Flink 1.10
[ https://issues.apache.org/jira/browse/BEAM-9295?focusedWorklogId=391541&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-391541 ] ASF GitHub Bot logged work on BEAM-9295: Author: ASF GitHub Bot Created on: 24/Feb/20 06:46 Start Date: 24/Feb/20 06:46 Worklog Time Spent: 10m Work Description: sunjincheng121 commented on issue #10945: [BEAM-9295] Add Flink 1.10 build target and Make FlinkRunner compatible with Flink 1.10 URL: https://github.com/apache/beam/pull/10945#issuecomment-590186860 Run Python2_PVR_Flink PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 391541) Time Spent: 0.5h (was: 20m) > Add Flink 1.10 build target and Make FlinkRunner compatible with Flink 1.10 > --- > > Key: BEAM-9295 > URL: https://issues.apache.org/jira/browse/BEAM-9295 > Project: Beam > Issue Type: New Feature > Components: runner-flink >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Fix For: 2.20.0 > > Time Spent: 0.5h > Remaining Estimate: 0h > > Apache Flink 1.10 has completed the final release vote, see [1]. So, I would > like to add Flink 1.10 build target and make Flink Runner compatible with > Flink 1.10. > And I appreciate it if you can leave your suggestions or comments! > [1] > https://lists.apache.org/thread.html/r97672d4d1e47372cebf23e6643a6cc30a06bfbdf3f277b0be3695b15%40%3Cdev.flink.apache.org%3E -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9363) BeamSQL windowing as TVF
[ https://issues.apache.org/jira/browse/BEAM-9363?focusedWorklogId=391535&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-391535 ] ASF GitHub Bot logged work on BEAM-9363: Author: ASF GitHub Bot Created on: 24/Feb/20 06:35 Start Date: 24/Feb/20 06:35 Worklog Time Spent: 10m Work Description: amaliujia commented on issue #10946: [BEAM-9363] TUMBLE as TVF URL: https://github.com/apache/beam/pull/10946#issuecomment-590184727 R: @kennknowles This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 391535) Time Spent: 20m (was: 10m) > BeamSQL windowing as TVF > > > Key: BEAM-9363 > URL: https://issues.apache.org/jira/browse/BEAM-9363 > Project: Beam > Issue Type: New Feature > Components: dsl-sql, dsl-sql-zetasql >Reporter: Rui Wang >Assignee: Rui Wang >Priority: Major > Time Spent: 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9363) BeamSQL windowing as TVF
[ https://issues.apache.org/jira/browse/BEAM-9363?focusedWorklogId=391534&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-391534 ] ASF GitHub Bot logged work on BEAM-9363: Author: ASF GitHub Bot Created on: 24/Feb/20 06:34 Start Date: 24/Feb/20 06:34 Worklog Time Spent: 10m Work Description: amaliujia commented on pull request #10946: [BEAM-9363] TUMBLE as TVF URL: https://github.com/apache/beam/pull/10946 Note that ZetaSQL needs an improvement to fully support TVF windowing. This PR adds a skeleton as a starter for moving streaming windowing to `FROM` clause by using TVF. Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] Update `CHANGES.md` with noteworthy changes. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier). Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) Python | [ BeamSQL windowing as TVF
Rui Wang created BEAM-9363: -- Summary: BeamSQL windowing as TVF Key: BEAM-9363 URL: https://issues.apache.org/jira/browse/BEAM-9363 Project: Beam Issue Type: New Feature Components: dsl-sql, dsl-sql-zetasql Reporter: Rui Wang Assignee: Rui Wang -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (BEAM-9362) Add support for Flink-1.10
Wenlong Lyu created BEAM-9362: - Summary: Add support for Flink-1.10 Key: BEAM-9362 URL: https://issues.apache.org/jira/browse/BEAM-9362 Project: Beam Issue Type: Improvement Components: runner-flink Reporter: Wenlong Lyu Hi, [flink-1.10.0|[https://flink.apache.org/news/2020/02/11/release-1.10.0.html#table-apisql-production-ready-hive-integration]] has been released and there are many improvements in resource managing, job managing, etc. I think it is time for beam to bump up the supporting version of flink-runners so that the user can easily integrate beam with flink-1.10 and make use of the improvements in Flink. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9298) Drop support for Flink 1.7
[ https://issues.apache.org/jira/browse/BEAM-9298?focusedWorklogId=391497&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-391497 ] ASF GitHub Bot logged work on BEAM-9298: Author: ASF GitHub Bot Created on: 24/Feb/20 03:25 Start Date: 24/Feb/20 03:25 Worklog Time Spent: 10m Work Description: sunjincheng121 commented on issue #10884: [BEAM-9298] Drop support for Flink 1.7 URL: https://github.com/apache/beam/pull/10884#issuecomment-590155795 The PR should be update, So, please review the changes after #10945 merged. :) Thank you! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 391497) Time Spent: 40m (was: 0.5h) > Drop support for Flink 1.7 > --- > > Key: BEAM-9298 > URL: https://issues.apache.org/jira/browse/BEAM-9298 > Project: Beam > Issue Type: Task > Components: runner-flink >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Fix For: 2.20.0 > > Time Spent: 40m > Remaining Estimate: 0h > > With Flink 1.10 around the corner, more detail can be found in BEAM-9295, we > should consider dropping support for Flink 1.7. Then dropping 1.7 will also > decrease the build time. > What do you think? -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9298) Drop support for Flink 1.7
[ https://issues.apache.org/jira/browse/BEAM-9298?focusedWorklogId=391495&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-391495 ] ASF GitHub Bot logged work on BEAM-9298: Author: ASF GitHub Bot Created on: 24/Feb/20 03:19 Start Date: 24/Feb/20 03:19 Worklog Time Spent: 10m Work Description: sunjincheng121 commented on pull request #10884: [BEAM-9298] Drop support for Flink 1.7 URL: https://github.com/apache/beam/pull/10884#discussion_r383073377 ## File path: website/src/documentation/runners/flink.md ## @@ -103,7 +103,7 @@ To find out which version of Flink is compatible with Beam please see the table Artifact Id Review comment: Sure，will update the PR after https://github.com/apache/beam/pull/10945 merged. :) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 391495) Time Spent: 0.5h (was: 20m) > Drop support for Flink 1.7 > --- > > Key: BEAM-9298 > URL: https://issues.apache.org/jira/browse/BEAM-9298 > Project: Beam > Issue Type: Task > Components: runner-flink >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Fix For: 2.20.0 > > Time Spent: 0.5h > Remaining Estimate: 0h > > With Flink 1.10 around the corner, more detail can be found in BEAM-9295, we > should consider dropping support for Flink 1.7. Then dropping 1.7 will also > decrease the build time. > What do you think? -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9295) Add Flink 1.10 build target and Make FlinkRunner compatible with Flink 1.10
[ https://issues.apache.org/jira/browse/BEAM-9295?focusedWorklogId=391491&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-391491 ] ASF GitHub Bot logged work on BEAM-9295: Author: ASF GitHub Bot Created on: 24/Feb/20 03:17 Start Date: 24/Feb/20 03:17 Worklog Time Spent: 10m Work Description: sunjincheng121 commented on pull request #10945: [BEAM-9295] Add Flink 1.10 build target and Make FlinkRunner compatible with Flink 1.10 URL: https://github.com/apache/beam/pull/10945 Add Flink 1.10 build target and Make FlinkRunner compatible with Flink 1.10 Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)[](https://builds.ap
[jira] [Work logged] (BEAM-7274) Protobuf Beam Schema support
[ https://issues.apache.org/jira/browse/BEAM-7274?focusedWorklogId=391486&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-391486 ] ASF GitHub Bot logged work on BEAM-7274: Author: ASF GitHub Bot Created on: 24/Feb/20 03:02 Start Date: 24/Feb/20 03:02 Worklog Time Spent: 10m Work Description: reuvenlax commented on issue #10502: [BEAM-7274] Add DynamicMessage Schema support URL: https://github.com/apache/beam/pull/10502#issuecomment-590152824 A few comments. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 391486) Time Spent: 24h 40m (was: 24.5h) > Protobuf Beam Schema support > > > Key: BEAM-7274 > URL: https://issues.apache.org/jira/browse/BEAM-7274 > Project: Beam > Issue Type: Sub-task > Components: sdk-java-core >Reporter: Alex Van Boxel >Assignee: Alex Van Boxel >Priority: Minor > Fix For: 2.20.0 > > Time Spent: 24h 40m > Remaining Estimate: 0h > > Add support for the new Beam Schema to the Protobuf extension. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-7274) Protobuf Beam Schema support
[
https://issues.apache.org/jira/browse/BEAM-7274?focusedWorklogId=391487&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-391487
]
ASF GitHub Bot logged work on BEAM-7274:
Author: ASF GitHub Bot
Created on: 24/Feb/20 03:02
Start Date: 24/Feb/20 03:02
Worklog Time Spent: 10m
Work Description: reuvenlax commented on pull request #10502: [BEAM-7274]
Add DynamicMessage Schema support
URL: https://github.com/apache/beam/pull/10502#discussion_r383070050
##
File path:
sdks/java/extensions/protobuf/src/main/java/org/apache/beam/sdk/extensions/protobuf/ProtoDomain.java
##
@@ -163,15 +163,15 @@ private void indexDescriptorByName() {
.values()
.forEach(
fileDescriptor -> {
- fileDescriptor
- .getMessageTypes()
- .forEach(
- descriptor -> {
-descriptorMap.put(descriptor.getFullName(),
descriptor);
- });
+ fileDescriptor.getMessageTypes().forEach(descriptor ->
indexDescriptor(descriptor));
Review comment:
I think you can replace the lambda with this::indexDescriptor
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 391487)
Time Spent: 24h 50m (was: 24h 40m)
> Protobuf Beam Schema support
>
>
> Key: BEAM-7274
> URL: https://issues.apache.org/jira/browse/BEAM-7274
> Project: Beam
> Issue Type: Sub-task
> Components: sdk-java-core
>Reporter: Alex Van Boxel
>Assignee: Alex Van Boxel
>Priority: Minor
> Fix For: 2.20.0
>
> Time Spent: 24h 50m
> Remaining Estimate: 0h
>
> Add support for the new Beam Schema to the Protobuf extension.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-7274) Protobuf Beam Schema support
[
https://issues.apache.org/jira/browse/BEAM-7274?focusedWorklogId=391488&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-391488
]
ASF GitHub Bot logged work on BEAM-7274:
Author: ASF GitHub Bot
Created on: 24/Feb/20 03:02
Start Date: 24/Feb/20 03:02
Worklog Time Spent: 10m
Work Description: reuvenlax commented on pull request #10502: [BEAM-7274]
Add DynamicMessage Schema support
URL: https://github.com/apache/beam/pull/10502#discussion_r383070124
##
File path:
sdks/java/extensions/protobuf/src/main/java/org/apache/beam/sdk/extensions/protobuf/ProtoDomain.java
##
@@ -163,15 +163,15 @@ private void indexDescriptorByName() {
.values()
.forEach(
fileDescriptor -> {
- fileDescriptor
- .getMessageTypes()
- .forEach(
- descriptor -> {
-descriptorMap.put(descriptor.getFullName(),
descriptor);
- });
+ fileDescriptor.getMessageTypes().forEach(descriptor ->
indexDescriptor(descriptor));
});
}
+ private void indexDescriptor(Descriptors.Descriptor descriptor) {
+descriptorMap.put(descriptor.getFullName(), descriptor);
+descriptor.getNestedTypes().forEach(nested -> indexDescriptor(nested));
Review comment:
you can do the same here - forEach(this::indexDescriptor)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 391488)
Time Spent: 25h (was: 24h 50m)
> Protobuf Beam Schema support
>
>
> Key: BEAM-7274
> URL: https://issues.apache.org/jira/browse/BEAM-7274
> Project: Beam
> Issue Type: Sub-task
> Components: sdk-java-core
>Reporter: Alex Van Boxel
>Assignee: Alex Van Boxel
>Priority: Minor
> Fix For: 2.20.0
>
> Time Spent: 25h
> Remaining Estimate: 0h
>
> Add support for the new Beam Schema to the Protobuf extension.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-7274) Protobuf Beam Schema support
[
https://issues.apache.org/jira/browse/BEAM-7274?focusedWorklogId=391489&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-391489
]
ASF GitHub Bot logged work on BEAM-7274:
Author: ASF GitHub Bot
Created on: 24/Feb/20 03:02
Start Date: 24/Feb/20 03:02
Worklog Time Spent: 10m
Work Description: reuvenlax commented on pull request #10502: [BEAM-7274]
Add DynamicMessage Schema support
URL: https://github.com/apache/beam/pull/10502#discussion_r383071152
##
File path:
sdks/java/extensions/protobuf/src/main/java/org/apache/beam/sdk/extensions/protobuf/ProtoDynamicMessageSchema.java
##
@@ -0,0 +1,852 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.protobuf;
+
+import static
org.apache.beam.sdk.extensions.protobuf.ProtoSchemaTranslator.getFieldNumber;
+import static
org.apache.beam.sdk.extensions.protobuf.ProtoSchemaTranslator.getMapKeyMessageName;
+import static
org.apache.beam.sdk.extensions.protobuf.ProtoSchemaTranslator.getMapValueMessageName;
+import static
org.apache.beam.sdk.extensions.protobuf.ProtoSchemaTranslator.getMessageName;
+import static
org.apache.beam.sdk.extensions.protobuf.ProtoSchemaTranslator.withFieldNumber;
+import static
org.apache.beam.sdk.extensions.protobuf.ProtoSchemaTranslator.withMessageName;
+
+import com.google.protobuf.ByteString;
+import com.google.protobuf.Descriptors;
+import com.google.protobuf.Descriptors.FieldDescriptor;
+import com.google.protobuf.DynamicMessage;
+import com.google.protobuf.Message;
+import java.io.Serializable;
+import java.time.Duration;
+import java.time.Instant;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import org.apache.beam.sdk.annotations.Experimental;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.schemas.SchemaCoder;
+import org.apache.beam.sdk.schemas.logicaltypes.EnumerationType;
+import org.apache.beam.sdk.schemas.logicaltypes.NanosDuration;
+import org.apache.beam.sdk.schemas.logicaltypes.NanosInstant;
+import org.apache.beam.sdk.schemas.logicaltypes.OneOfType;
+import org.apache.beam.sdk.transforms.SerializableFunction;
+import org.apache.beam.sdk.values.Row;
+import org.apache.beam.sdk.values.TypeDescriptor;
+
+@Experimental(Experimental.Kind.SCHEMAS)
+public class ProtoDynamicMessageSchema implements Serializable {
+ public static final long serialVersionUID = 1L;
+
+ /**
+ * Context of the schema, the context can be generated from a source schema
or descriptors. The
+ * ability of converting back from Row to proto depends on the type of
context.
+ */
+ private final Context context;
+
+ /** THe SchemaCoder holds the resolved schema and to/fromRow functions. */
+ private transient SchemaCoder schemaCoder;
+
+ /** List of field converters for each field in the row. */
+ private transient List converters;
+
+ private ProtoDynamicMessageSchema(String messageName, ProtoDomain domain) {
+this.context = new DescriptorContext(messageName, domain);
+readResolve();
+ }
+
+ private ProtoDynamicMessageSchema(Context context) {
+this.context = context;
+readResolve();
+ }
+
+ /**
+ * Create a new ProtoDynamicMessageSchema from a {@link ProtoDomain} and for
a message. The
+ * message need to be in the domain and needs to be the fully qualified name.
+ */
+ public static ProtoDynamicMessageSchema forDescriptor(ProtoDomain domain,
String messageName) {
+return new ProtoDynamicMessageSchema(messageName, domain);
+ }
+
+ /**
+ * Create a new ProtoDynamicMessageSchema from a {@link ProtoDomain} and for
a descriptor. The
+ * descriptor is only used for it's name, that name will be used for a
search in the domain.
+ */
+ public static ProtoDynamicMessageSchema forDescriptor(
+ ProtoDomain domain, Descriptors.Descriptor descriptor) {
+return new ProtoDynamicMessageSchema<>(descriptor.getFullName(), domain);
+ }
+
+ static ProtoDynamicMessageSchema forContext(Context context, Schema.Field
field) {
+return new Pro
[jira] [Work logged] (BEAM-7274) Protobuf Beam Schema support
[ https://issues.apache.org/jira/browse/BEAM-7274?focusedWorklogId=391475&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-391475 ] ASF GitHub Bot logged work on BEAM-7274: Author: ASF GitHub Bot Created on: 24/Feb/20 02:35 Start Date: 24/Feb/20 02:35 Worklog Time Spent: 10m Work Description: reuvenlax commented on pull request #10944: [BEAM-7274] optimize oneOf handling URL: https://github.com/apache/beam/pull/10944 When codegen support was added to enable protocol buffer schemas, oneof fields were left out and handled via map lookups. This was because ByteBuddy does not provide an easy way to generate conditional code. In this PR we use the underlying ASM framework to generate switch/case statements to handle oneofs, and remove the slower map-based code path. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 391475) Time Spent: 24.5h (was: 24h 20m) > Protobuf Beam Schema support > > > Key: BEAM-7274 > URL: https://issues.apache.org/jira/browse/BEAM-7274 > Project: Beam > Issue Type: Sub-task > Components: sdk-java-core >Reporter: Alex Van Boxel >Assignee: Alex Van Boxel >Priority: Minor > Fix For: 2.20.0 > > Time Spent: 24.5h > Remaining Estimate: 0h > > Add support for the new Beam Schema to the Protobuf extension. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9331) The Row object needs better builders
[
https://issues.apache.org/jira/browse/BEAM-9331?focusedWorklogId=391466&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-391466
]
ASF GitHub Bot logged work on BEAM-9331:
Author: ASF GitHub Bot
Created on: 24/Feb/20 01:49
Start Date: 24/Feb/20 01:49
Worklog Time Spent: 10m
Work Description: reuvenlax commented on pull request #10883: [BEAM-9331]
Add better Row builders
URL: https://github.com/apache/beam/pull/10883#discussion_r383062953
##
File path: sdks/java/core/src/test/java/org/apache/beam/sdk/values/RowTest.java
##
@@ -477,6 +477,172 @@ public void testCreateMapWithRowValue() {
assertEquals(data, row.getMap("map"));
}
+ @Test
Review comment:
The fact that getValues returns the underlying objects I think is in line
with many other things. e.g. ImmutableList in Java still returns the underlying
objects, so you can mutate them. The onus is on the user not to do that.
In particular, the elements of a PCollection are not allowed to be mutated
in a ParDo (and the main point of the Row object is to be used in ParDos). The
DirectRunner has a mutability checker that will fail your test if you mutate
the input elements.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 391466)
Time Spent: 3h 10m (was: 3h)
> The Row object needs better builders
>
>
> Key: BEAM-9331
> URL: https://issues.apache.org/jira/browse/BEAM-9331
> Project: Beam
> Issue Type: Sub-task
> Components: sdk-java-core
>Reporter: Reuven Lax
>Priority: Major
> Time Spent: 3h 10m
> Remaining Estimate: 0h
>
> Users should be able to build a Row object by specifying field names. Desired
> syntax:
>
> Row.withSchema(schema)
> .withFieldName("field1", "value)
> .withFieldName("field2.field3", value)
> .build()
>
> Users should also have a builder that allows taking an existing row and
> changing specific fields.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9276) python: create a class to encapsulate the work required to submit a pipeline to a job service
[
https://issues.apache.org/jira/browse/BEAM-9276?focusedWorklogId=391457&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-391457
]
ASF GitHub Bot logged work on BEAM-9276:
Author: ASF GitHub Bot
Created on: 23/Feb/20 23:13
Start Date: 23/Feb/20 23:13
Worklog Time Spent: 10m
Work Description: chadrik commented on issue #10811: [BEAM-9276] Create a
class to encapsulate the steps required to submit a pipeline
URL: https://github.com/apache/beam/pull/10811#issuecomment-590128062
Run Portable_Python PreCommit
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 391457)
Time Spent: 1.5h (was: 1h 20m)
> python: create a class to encapsulate the work required to submit a pipeline
> to a job service
> -
>
> Key: BEAM-9276
> URL: https://issues.apache.org/jira/browse/BEAM-9276
> Project: Beam
> Issue Type: Improvement
> Components: sdk-py-core
>Reporter: Chad Dombrova
>Assignee: Chad Dombrova
>Priority: Major
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> {{PortableRunner.run_pipeline}} is somewhat of a monolithic method for
> submitting a pipeline. It would be useful to factor out the code responsible
> for interacting with the job and artifact services (prepare, stage, run) to
> make this easier to modify this behavior in portable runner subclasses, as
> well as in tests.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-8979) protoc-gen-mypy: program not found or is not executable
[
https://issues.apache.org/jira/browse/BEAM-8979?focusedWorklogId=391442&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-391442
]
ASF GitHub Bot logged work on BEAM-8979:
Author: ASF GitHub Bot
Created on: 23/Feb/20 20:51
Start Date: 23/Feb/20 20:51
Worklog Time Spent: 10m
Work Description: chadrik commented on issue #10734: [BEAM-8979]
reintroduce mypy-protobuf stub generation
URL: https://github.com/apache/beam/pull/10734#issuecomment-590114619
@udim I used my new committer powers to push the tests along. All checks
have passed!
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 391442)
Time Spent: 10h 50m (was: 10h 40m)
> protoc-gen-mypy: program not found or is not executable
> ---
>
> Key: BEAM-8979
> URL: https://issues.apache.org/jira/browse/BEAM-8979
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core, test-failures
>Reporter: Kamil Wasilewski
>Assignee: Chad Dombrova
>Priority: Major
> Time Spent: 10h 50m
> Remaining Estimate: 0h
>
> In some tests, `:sdks:python:sdist:` task fails due to problems in finding
> protoc-gen-mypy. The following tests are affected (there might be more):
> *
> [https://builds.apache.org/job/beam_LoadTests_Python_37_ParDo_Dataflow_Batch_PR/]
> *
> [https://builds.apache.org/job/beam_BiqQueryIO_Write_Performance_Test_Python_Batch/
>
> |https://builds.apache.org/job/beam_BiqQueryIO_Write_Performance_Test_Python_Batch/]
> Relevant logs:
> {code:java}
> 10:46:32 > Task :sdks:python:sdist FAILED
> 10:46:32 Requirement already satisfied: mypy-protobuf==1.12 in
> /home/jenkins/jenkins-slave/workspace/beam_LoadTests_Python_37_ParDo_Dataflow_Batch_PR/src/build/gradleenv/192237/lib/python3.7/site-packages
> (1.12)
> 10:46:32 beam_fn_api.proto: warning: Import google/protobuf/descriptor.proto
> but not used.
> 10:46:32 beam_fn_api.proto: warning: Import google/protobuf/wrappers.proto
> but not used.
> 10:46:32 protoc-gen-mypy: program not found or is not executable
> 10:46:32 --mypy_out: protoc-gen-mypy: Plugin failed with status code 1.
> 10:46:32
> /home/jenkins/jenkins-slave/workspace/beam_LoadTests_Python_37_ParDo_Dataflow_Batch_PR/src/build/gradleenv/192237/lib/python3.7/site-packages/setuptools/dist.py:476:
> UserWarning: Normalizing '2.19.0.dev' to '2.19.0.dev0'
> 10:46:32 normalized_version,
> 10:46:32 Traceback (most recent call last):
> 10:46:32 File "setup.py", line 295, in
> 10:46:32 'mypy': generate_protos_first(mypy),
> 10:46:32 File
> "/home/jenkins/jenkins-slave/workspace/beam_LoadTests_Python_37_ParDo_Dataflow_Batch_PR/src/build/gradleenv/192237/lib/python3.7/site-packages/setuptools/__init__.py",
> line 145, in setup
> 10:46:32 return distutils.core.setup(**attrs)
> 10:46:32 File "/usr/lib/python3.7/distutils/core.py", line 148, in setup
> 10:46:32 dist.run_commands()
> 10:46:32 File "/usr/lib/python3.7/distutils/dist.py", line 966, in
> run_commands
> 10:46:32 self.run_command(cmd)
> 10:46:32 File "/usr/lib/python3.7/distutils/dist.py", line 985, in
> run_command
> 10:46:32 cmd_obj.run()
> 10:46:32 File
> "/home/jenkins/jenkins-slave/workspace/beam_LoadTests_Python_37_ParDo_Dataflow_Batch_PR/src/build/gradleenv/192237/lib/python3.7/site-packages/setuptools/command/sdist.py",
> line 44, in run
> 10:46:32 self.run_command('egg_info')
> 10:46:32 File "/usr/lib/python3.7/distutils/cmd.py", line 313, in
> run_command
> 10:46:32 self.distribution.run_command(command)
> 10:46:32 File "/usr/lib/python3.7/distutils/dist.py", line 985, in
> run_command
> 10:46:32 cmd_obj.run()
> 10:46:32 File "setup.py", line 220, in run
> 10:46:32 gen_protos.generate_proto_files(log=log)
> 10:46:32 File
> "/home/jenkins/jenkins-slave/workspace/beam_LoadTests_Python_37_ParDo_Dataflow_Batch_PR/src/sdks/python/gen_protos.py",
> line 144, in generate_proto_files
> 10:46:32 '%s' % ret_code)
> 10:46:32 RuntimeError: Protoc returned non-zero status (see logs for
> details): 1
> {code}
>
> This is what I have tried so far to resolve this (without being successful):
> * Including _--plugin=protoc-gen-mypy=\{abs_path_to_executable}_ parameter
> to the _protoc_ call ingen_protos.py:131
> * Appending protoc-gen-mypy's directory to the PATH variable
> I wasn't able to reproduce this error locally.
>
--
This message was sent by Atlassian Jira
(v
[jira] [Work logged] (BEAM-9276) python: create a class to encapsulate the work required to submit a pipeline to a job service
[
https://issues.apache.org/jira/browse/BEAM-9276?focusedWorklogId=391428&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-391428
]
ASF GitHub Bot logged work on BEAM-9276:
Author: ASF GitHub Bot
Created on: 23/Feb/20 19:46
Start Date: 23/Feb/20 19:46
Worklog Time Spent: 10m
Work Description: chadrik commented on pull request #10811: [BEAM-9276]
Create a class to encapsulate the steps required to submit a pipeline
URL: https://github.com/apache/beam/pull/10811#discussion_r383032735
##
File path: sdks/python/apache_beam/runners/portability/portable_runner.py
##
@@ -74,6 +79,149 @@
_LOGGER = logging.getLogger(__name__)
+class PortableJobServicePlan(object):
Review comment:
I went with `JobServiceHandle`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 391428)
Time Spent: 1h 20m (was: 1h 10m)
> python: create a class to encapsulate the work required to submit a pipeline
> to a job service
> -
>
> Key: BEAM-9276
> URL: https://issues.apache.org/jira/browse/BEAM-9276
> Project: Beam
> Issue Type: Improvement
> Components: sdk-py-core
>Reporter: Chad Dombrova
>Assignee: Chad Dombrova
>Priority: Major
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> {{PortableRunner.run_pipeline}} is somewhat of a monolithic method for
> submitting a pipeline. It would be useful to factor out the code responsible
> for interacting with the job and artifact services (prepare, stage, run) to
> make this easier to modify this behavior in portable runner subclasses, as
> well as in tests.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (BEAM-9361) NPE When putting Avro record with enum through SqlTransform
Niels Basjes created BEAM-9361:
--
Summary: NPE When putting Avro record with enum through
SqlTransform
Key: BEAM-9361
URL: https://issues.apache.org/jira/browse/BEAM-9361
Project: Beam
Issue Type: Bug
Components: dsl-sql
Affects Versions: 2.19.0
Reporter: Niels Basjes
I ran into this problem when trying to put my Avro records through the
SqlTransform.
I was able to reduce the reproduction path to the code below.
This code fails on my machine (using Beam 2.19.0) with the following
NullPointerException
{code:java}
org.apache.beam.sdk.extensions.sql.impl.ParseException: Unable to parse query
SELECT name, direction FROM InputStream at
org.apache.beam.sdk.extensions.sql.impl.CalciteQueryPlanner.convertToBeamRel(CalciteQueryPlanner.java:175)
at
org.apache.beam.sdk.extensions.sql.impl.BeamSqlEnv.parseQuery(BeamSqlEnv.java:103)
at
org.apache.beam.sdk.extensions.sql.SqlTransform.expand(SqlTransform.java:125)
at
org.apache.beam.sdk.extensions.sql.SqlTransform.expand(SqlTransform.java:83)
at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:539)
at org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:490)
at
org.apache.beam.sdk.values.PCollectionTuple.apply(PCollectionTuple.java:261)
at com.bol.analytics.m2.TestAvro2SQL.testAvro2SQL(TestAvro2SQL.java:99)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.apache.beam.sdk.testing.TestPipeline$1.evaluate(TestPipeline.java:319)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
at
com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
at
com.intellij.rt.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:33)
at
com.intellij.rt.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:230)
at com.intellij.rt.junit.JUnitStarter.main(JUnitStarter.java:58)
Caused by:
org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.tools.ValidationException:
java.lang.NullPointerException
at
org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.prepare.PlannerImpl.validate(PlannerImpl.java:217)
at
org.apache.beam.sdk.extensions.sql.impl.CalciteQueryPlanner.convertToBeamRel(CalciteQueryPlanner.java:144)
... 31 more
Caused by: java.lang.NullPointerException
at
org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.type.SqlTypeFactoryImpl.createSqlType(SqlTypeFactoryImpl.java:45)
at
org.apache.beam.sdk.extensions.sql.impl.utils.CalciteUtils.toRelDataType(CalciteUtils.java:280)
at
org.apache.beam.sdk.extensions.sql.impl.utils.CalciteUtils.toRelDataType(CalciteUtils.java:287)
at
org.apache.beam.sdk.extensions.sql.impl.utils.CalciteUtils.lambda$toCalciteRowType$0(CalciteUtils.java:261)
at
java.util.stream.Streams$RangeIntSpliterator.forEachRemaining(Streams.java:110)
at java.util.stream.IntPipeline$Head.forEach(IntPipeline.java:581)
at
org.apache.beam.sdk.extensions.sql.impl.utils.CalciteUtils.toCalciteRowType(CalciteUtils.java:258)
at
org.apache.beam.sdk.extensions.sql.impl.BeamCalciteTable.getRowType(BeamCalciteTable.java:71)
at
org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.validate.EmptyScope.resolve_(EmptyScope.java:159)
at
org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.validate.EmptyScope.resolveTable(EmptyScope.j
[jira] [Work logged] (BEAM-8564) Add LZO compression and decompression support
[ https://issues.apache.org/jira/browse/BEAM-8564?focusedWorklogId=391369&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-391369 ] ASF GitHub Bot logged work on BEAM-8564: Author: ASF GitHub Bot Created on: 23/Feb/20 13:43 Start Date: 23/Feb/20 13:43 Worklog Time Spent: 10m Work Description: shubham-srivastav commented on issue #10254: [BEAM-8564] Add LZO compression and decompression support URL: https://github.com/apache/beam/pull/10254#issuecomment-590069911 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 391369) Time Spent: 13h 20m (was: 13h 10m) > Add LZO compression and decompression support > - > > Key: BEAM-8564 > URL: https://issues.apache.org/jira/browse/BEAM-8564 > Project: Beam > Issue Type: New Feature > Components: sdk-java-core >Reporter: Amogh Tiwari >Assignee: Amogh Tiwari >Priority: Minor > Time Spent: 13h 20m > Remaining Estimate: 0h > > LZO is a lossless data compression algorithm which is focused on compression > and decompression speeds. > This will enable Apache Beam sdk to compress/decompress files using LZO > compression algorithm. > This will include the following functionalities: > # compress() : for compressing files into an LZO archive > # decompress() : for decompressing files archived using LZO compression > Appropriate Input and Output stream will also be added to enable working with > LZO files. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8564) Add LZO compression and decompression support
[ https://issues.apache.org/jira/browse/BEAM-8564?focusedWorklogId=391368&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-391368 ] ASF GitHub Bot logged work on BEAM-8564: Author: ASF GitHub Bot Created on: 23/Feb/20 13:39 Start Date: 23/Feb/20 13:39 Worklog Time Spent: 10m Work Description: shubham-srivastav commented on issue #10254: [BEAM-8564] Add LZO compression and decompression support URL: https://github.com/apache/beam/pull/10254#issuecomment-590069911 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 391368) Time Spent: 13h 10m (was: 13h) > Add LZO compression and decompression support > - > > Key: BEAM-8564 > URL: https://issues.apache.org/jira/browse/BEAM-8564 > Project: Beam > Issue Type: New Feature > Components: sdk-java-core >Reporter: Amogh Tiwari >Assignee: Amogh Tiwari >Priority: Minor > Time Spent: 13h 10m > Remaining Estimate: 0h > > LZO is a lossless data compression algorithm which is focused on compression > and decompression speeds. > This will enable Apache Beam sdk to compress/decompress files using LZO > compression algorithm. > This will include the following functionalities: > # compress() : for compressing files into an LZO archive > # decompress() : for decompressing files archived using LZO compression > Appropriate Input and Output stream will also be added to enable working with > LZO files. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9360) Schema FieldType should not consider metadata for equivalence
[ https://issues.apache.org/jira/browse/BEAM-9360?focusedWorklogId=391365&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-391365 ] ASF GitHub Bot logged work on BEAM-9360: Author: ASF GitHub Bot Created on: 23/Feb/20 13:21 Start Date: 23/Feb/20 13:21 Worklog Time Spent: 10m Work Description: JozoVilcek commented on pull request #10943: [BEAM-9360] Fix equivalence check for FieldType URL: https://github.com/apache/beam/pull/10943 [BEAM-9360] Fix equivalence check for FieldType Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] Update `CHANGES.md` with noteworthy changes. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier). Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)[![Build Sta
[jira] [Work logged] (BEAM-9288) Conscrypt shaded dependency
[ https://issues.apache.org/jira/browse/BEAM-9288?focusedWorklogId=391309&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-391309 ] ASF GitHub Bot logged work on BEAM-9288: Author: ASF GitHub Bot Created on: 23/Feb/20 08:41 Start Date: 23/Feb/20 08:41 Worklog Time Spent: 10m Work Description: sunjincheng121 commented on issue #10940: [BEAM-9288] Not bundle conscrypt in gRPC vendor URL: https://github.com/apache/beam/pull/10940#issuecomment-590045796 R: @lukecwik @kennknowles This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 391309) Time Spent: 1h 50m (was: 1h 40m) > Conscrypt shaded dependency > --- > > Key: BEAM-9288 > URL: https://issues.apache.org/jira/browse/BEAM-9288 > Project: Beam > Issue Type: Bug > Components: build-system >Reporter: Esun Kim >Assignee: sunjincheng >Priority: Critical > Fix For: 2.20.0 > > Time Spent: 1h 50m > Remaining Estimate: 0h > > Conscrypt is not designed to be shaded properly mainly because of so files. I > happened to see BEAM-9030 (*1) creating a new vendored gRPC shading Conscrypt > (*2) in it. I think this could make a problem when new Conscrypt is brought > by new gcsio depending on gRPC-alts (*4) in a dependency chain. (*5) In this > case, it may have a conflict when finding proper so files for Conscrypt. > *1: https://issues.apache.org/jira/browse/BEAM-9030 > *2: > [https://github.com/apache/beam/blob/e24d1e51cbabe27cb3cc381fd95b334db639c45d/buildSrc/src/main/groovy/org/apache/beam/gradle/GrpcVendoring_1_26_0.groovy#L78] > *3: https://issues.apache.org/jira/browse/BEAM-6136 > *4: [https://mvnrepository.com/artifact/io.grpc/grpc-alts/1.27.0] > *5: https://issues.apache.org/jira/browse/BEAM-8889 > -- This message was sent by Atlassian Jira (v8.3.4#803005)
