[jira] [Work logged] (BEAM-9510) Dependencies in base_image_requirements.txt are not compatible with each other
[ https://issues.apache.org/jira/browse/BEAM-9510?focusedWorklogId=404480&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-404480 ] ASF GitHub Bot logged work on BEAM-9510: Author: ASF GitHub Bot Created on: 17/Mar/20 06:52 Start Date: 17/Mar/20 06:52 Worklog Time Spent: 10m Work Description: tvalentyn commented on issue #11142: [BEAM-9510] Fixing version incompatibilities in base_image_requirements.txt URL: https://github.com/apache/beam/pull/11142#issuecomment-599906766 While Jenkins does not show test result on the PR, the tests are actually running: https://builds.apache.org/job/beam_PostCommit_Py_ValCont_PR/122/. Let's check the console logs that there are no conflicts before the merge. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 404480) Time Spent: 1h 20m (was: 1h 10m) > Dependencies in base_image_requirements.txt are not compatible with each other > -- > > Key: BEAM-9510 > URL: https://issues.apache.org/jira/browse/BEAM-9510 > Project: Beam > Issue Type: Bug > Components: sdk-py-harness >Reporter: David Yan >Assignee: Hannah Jiang >Priority: Major > Time Spent: 1h 20m > Remaining Estimate: 0h > > [https://github.com/apache/beam/blob/master/sdks/python/container/base_image_requirements.txt#L56] > says it requires google-cloud-bigquery==1.24.0, google-cloud-core==1.0.2, > google-cloud-bigtable==0.32.1, grpc-1.22.0 and tensorflow-2.1.0 > But they are incompatible with each other: > ERROR: google-cloud-bigquery 1.24.0 has requirement > google-cloud-core<2.0dev,>=1.1.0, but you'll have google-cloud-core 1.0.2 > which is incompatible. > ERROR: google-cloud-bigtable 0.32.1 has requirement > google-cloud-core<0.30dev,>=0.29.0, but you'll have google-cloud-core 1.0.2 > which is incompatible. > ERROR: tensorboard 2.1.1 has requirement grpcio>=1.24.3, but you'll have > grpcio 1.22.0 which is incompatible. > ERROR: tensorflow 2.1.0 has requirement scipy==1.4.1; python_version >= "3", > but you'll have scipy 1.2.2 which is incompatible. > > > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9510) Dependencies in base_image_requirements.txt are not compatible with each other
[ https://issues.apache.org/jira/browse/BEAM-9510?focusedWorklogId=404477&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-404477 ] ASF GitHub Bot logged work on BEAM-9510: Author: ASF GitHub Bot Created on: 17/Mar/20 06:29 Start Date: 17/Mar/20 06:29 Worklog Time Spent: 10m Work Description: tvalentyn commented on issue #11142: [BEAM-9510] Fixing version incompatibilities in base_image_requirements.txt URL: https://github.com/apache/beam/pull/11142#issuecomment-599900258 LGTM A few things out of scope of this PR: 1) This PR fixes the incompatibility in dependencies, but the list of dependencies is in base_image_requirements.txt is still not exhaustive. So, container builds may not be repeatable over time, and there is a risk that we will introduce incompatible dependencies down the road and won't notice it. 2) We can consider switching the base image to `python:slim-stretch` instead of `python:stretch` to reduce the size of the images: https://github.com/apache/beam/blob/master/sdks/python/container/Dockerfile#L20 cc: @Hannah-Jiang This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 404477) Time Spent: 1h 10m (was: 1h) > Dependencies in base_image_requirements.txt are not compatible with each other > -- > > Key: BEAM-9510 > URL: https://issues.apache.org/jira/browse/BEAM-9510 > Project: Beam > Issue Type: Bug > Components: sdk-py-harness >Reporter: David Yan >Assignee: Hannah Jiang >Priority: Major > Time Spent: 1h 10m > Remaining Estimate: 0h > > [https://github.com/apache/beam/blob/master/sdks/python/container/base_image_requirements.txt#L56] > says it requires google-cloud-bigquery==1.24.0, google-cloud-core==1.0.2, > google-cloud-bigtable==0.32.1, grpc-1.22.0 and tensorflow-2.1.0 > But they are incompatible with each other: > ERROR: google-cloud-bigquery 1.24.0 has requirement > google-cloud-core<2.0dev,>=1.1.0, but you'll have google-cloud-core 1.0.2 > which is incompatible. > ERROR: google-cloud-bigtable 0.32.1 has requirement > google-cloud-core<0.30dev,>=0.29.0, but you'll have google-cloud-core 1.0.2 > which is incompatible. > ERROR: tensorboard 2.1.1 has requirement grpcio>=1.24.3, but you'll have > grpcio 1.22.0 which is incompatible. > ERROR: tensorflow 2.1.0 has requirement scipy==1.4.1; python_version >= "3", > but you'll have scipy 1.2.2 which is incompatible. > > > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9510) Dependencies in base_image_requirements.txt are not compatible with each other
[ https://issues.apache.org/jira/browse/BEAM-9510?focusedWorklogId=404471&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-404471 ] ASF GitHub Bot logged work on BEAM-9510: Author: ASF GitHub Bot Created on: 17/Mar/20 06:14 Start Date: 17/Mar/20 06:14 Worklog Time Spent: 10m Work Description: tvalentyn commented on issue #11142: [BEAM-9510] Fixing version incompatibilities in base_image_requirements.txt URL: https://github.com/apache/beam/pull/11142#issuecomment-599896374 Thanks, @davidyan74! Added a small change to reflect that `typing` should only be installed on Python 2 (this was recently changed in Beam master). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 404471) Time Spent: 50m (was: 40m) > Dependencies in base_image_requirements.txt are not compatible with each other > -- > > Key: BEAM-9510 > URL: https://issues.apache.org/jira/browse/BEAM-9510 > Project: Beam > Issue Type: Bug > Components: sdk-py-harness >Reporter: David Yan >Assignee: Hannah Jiang >Priority: Major > Time Spent: 50m > Remaining Estimate: 0h > > [https://github.com/apache/beam/blob/master/sdks/python/container/base_image_requirements.txt#L56] > says it requires google-cloud-bigquery==1.24.0, google-cloud-core==1.0.2, > google-cloud-bigtable==0.32.1, grpc-1.22.0 and tensorflow-2.1.0 > But they are incompatible with each other: > ERROR: google-cloud-bigquery 1.24.0 has requirement > google-cloud-core<2.0dev,>=1.1.0, but you'll have google-cloud-core 1.0.2 > which is incompatible. > ERROR: google-cloud-bigtable 0.32.1 has requirement > google-cloud-core<0.30dev,>=0.29.0, but you'll have google-cloud-core 1.0.2 > which is incompatible. > ERROR: tensorboard 2.1.1 has requirement grpcio>=1.24.3, but you'll have > grpcio 1.22.0 which is incompatible. > ERROR: tensorflow 2.1.0 has requirement scipy==1.4.1; python_version >= "3", > but you'll have scipy 1.2.2 which is incompatible. > > > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9510) Dependencies in base_image_requirements.txt are not compatible with each other
[ https://issues.apache.org/jira/browse/BEAM-9510?focusedWorklogId=404472&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-404472 ] ASF GitHub Bot logged work on BEAM-9510: Author: ASF GitHub Bot Created on: 17/Mar/20 06:14 Start Date: 17/Mar/20 06:14 Worklog Time Spent: 10m Work Description: tvalentyn commented on issue #11142: [BEAM-9510] Fixing version incompatibilities in base_image_requirements.txt URL: https://github.com/apache/beam/pull/11142#issuecomment-599896422 Run Python Dataflow ValidatesContainer This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 404472) Time Spent: 1h (was: 50m) > Dependencies in base_image_requirements.txt are not compatible with each other > -- > > Key: BEAM-9510 > URL: https://issues.apache.org/jira/browse/BEAM-9510 > Project: Beam > Issue Type: Bug > Components: sdk-py-harness >Reporter: David Yan >Assignee: Hannah Jiang >Priority: Major > Time Spent: 1h > Remaining Estimate: 0h > > [https://github.com/apache/beam/blob/master/sdks/python/container/base_image_requirements.txt#L56] > says it requires google-cloud-bigquery==1.24.0, google-cloud-core==1.0.2, > google-cloud-bigtable==0.32.1, grpc-1.22.0 and tensorflow-2.1.0 > But they are incompatible with each other: > ERROR: google-cloud-bigquery 1.24.0 has requirement > google-cloud-core<2.0dev,>=1.1.0, but you'll have google-cloud-core 1.0.2 > which is incompatible. > ERROR: google-cloud-bigtable 0.32.1 has requirement > google-cloud-core<0.30dev,>=0.29.0, but you'll have google-cloud-core 1.0.2 > which is incompatible. > ERROR: tensorboard 2.1.1 has requirement grpcio>=1.24.3, but you'll have > grpcio 1.22.0 which is incompatible. > ERROR: tensorflow 2.1.0 has requirement scipy==1.4.1; python_version >= "3", > but you'll have scipy 1.2.2 which is incompatible. > > > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9136) Add LICENSES and NOTICES to docker images
[ https://issues.apache.org/jira/browse/BEAM-9136?focusedWorklogId=40&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-40 ] ASF GitHub Bot logged work on BEAM-9136: Author: ASF GitHub Bot Created on: 17/Mar/20 04:26 Start Date: 17/Mar/20 04:26 Worklog Time Spent: 10m Work Description: Hannah-Jiang commented on issue #11067: [BEAM-9136]Add licenses for dependencies URL: https://github.com/apache/beam/pull/11067#issuecomment-599871035 > In general, it's preferable to avoid checking in script-created files, preferring instead to create them on demand at build time. (There are exceptions, e.g. bootstrapping and/or difficult to obtain build tools, but I don't think that applies here.) > > As another principle, it's preferable, where possible, to enforce correctness of the repository state before things get committed/merged (e.g. at pre-commit time), rather than have a separate, asynchronous, external process to try to fix things up after the fact. > > Perhaps it's worth expanding the design doc on how we could do this? Yep, I will add it to the doc and send out for review when it is ready. Thanks for explanation! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 40) Time Spent: 2.5h (was: 2h 20m) > Add LICENSES and NOTICES to docker images > - > > Key: BEAM-9136 > URL: https://issues.apache.org/jira/browse/BEAM-9136 > Project: Beam > Issue Type: Task > Components: build-system >Reporter: Hannah Jiang >Assignee: Hannah Jiang >Priority: Major > Time Spent: 2.5h > Remaining Estimate: 0h > > Scan dependencies and add licenses and notices of the dependencies to SDK > docker images. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9494) Remove workaround for BQ transform for Dataflow
[ https://issues.apache.org/jira/browse/BEAM-9494?focusedWorklogId=404409&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-404409 ] ASF GitHub Bot logged work on BEAM-9494: Author: ASF GitHub Bot Created on: 17/Mar/20 02:34 Start Date: 17/Mar/20 02:34 Worklog Time Spent: 10m Work Description: pabloem commented on issue #5: [cherry-pick][release-2.20.0][BEAM-9494] Reifying outputs from BQ file writing URL: https://github.com/apache/beam/pull/5#issuecomment-599846684 This is no longer needed. Sorry and thanks @amaliujia This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 404409) Time Spent: 2.5h (was: 2h 20m) > Remove workaround for BQ transform for Dataflow > --- > > Key: BEAM-9494 > URL: https://issues.apache.org/jira/browse/BEAM-9494 > Project: Beam > Issue Type: Bug > Components: io-py-gcp >Reporter: Luke Cwik >Assignee: Pablo Estrada >Priority: Minor > Fix For: 2.20.0 > > Time Spent: 2.5h > Remaining Estimate: 0h > > Dataflow incorrectly uses the Flatten input PCollection coder when it > performs an optimization instead of the output PCollection coder which can > lead to issues if these coders differ. > > The workaround was introduced in [https://github.com/apache/beam/pull/11103] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9494) Remove workaround for BQ transform for Dataflow
[ https://issues.apache.org/jira/browse/BEAM-9494?focusedWorklogId=404408&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-404408 ] ASF GitHub Bot logged work on BEAM-9494: Author: ASF GitHub Bot Created on: 17/Mar/20 02:34 Start Date: 17/Mar/20 02:34 Worklog Time Spent: 10m Work Description: pabloem commented on pull request #5: [cherry-pick][release-2.20.0][BEAM-9494] Reifying outputs from BQ file writing URL: https://github.com/apache/beam/pull/5 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 404408) Time Spent: 2h 20m (was: 2h 10m) > Remove workaround for BQ transform for Dataflow > --- > > Key: BEAM-9494 > URL: https://issues.apache.org/jira/browse/BEAM-9494 > Project: Beam > Issue Type: Bug > Components: io-py-gcp >Reporter: Luke Cwik >Assignee: Pablo Estrada >Priority: Minor > Fix For: 2.20.0 > > Time Spent: 2h 20m > Remaining Estimate: 0h > > Dataflow incorrectly uses the Flatten input PCollection coder when it > performs an optimization instead of the output PCollection coder which can > lead to issues if these coders differ. > > The workaround was introduced in [https://github.com/apache/beam/pull/11103] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-9444) Shall we use GCP Libraries BOM to specify Google-related library versions?
[
https://issues.apache.org/jira/browse/BEAM-9444?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17060584#comment-17060584
]
Tomo Suzuki commented on BEAM-9444:
---
To run the build in my Mac:
{noformat}
suztomo-macbookpro44% ./gradlew -p sdks/java check -x
extensions:sql:zetasql:check -x harness:test -x io:jdbc:test -x io:kafka:test
-x io:solr:test
...
BUILD SUCCESSFUL in 1h 15m 30s
551 actionable tasks: 44 executed, 507 up-to-date
{noformat}
> Shall we use GCP Libraries BOM to specify Google-related library versions?
> --
>
> Key: BEAM-9444
> URL: https://issues.apache.org/jira/browse/BEAM-9444
> Project: Beam
> Issue Type: Task
> Components: build-system
>Reporter: Tomo Suzuki
>Assignee: Tomo Suzuki
>Priority: Major
> Attachments: Screen Shot 2020-03-13 at 13.33.01.png
>
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> Shall we use GCP Libraries BOM to specify Google-related library versions?
>
> I've been working on Beam's dependency upgrades in the past few months. I
> think it's time to consider a long-term solution to keep the libraries
> up-to-date with small maintenance effort. To achieve that, I propose Beam to
> use GCP Libraries BOM to set the Google-related library versions, rather than
> trying to make changes in each of ~30 Google libraries.
>
> h1. Background
> A BOM is pom.xml that provides dependencyManagement to importing projects.
>
> GCP Libraries BOM is a BOM that includes many Google Cloud related libraries
> + gRPC + protobuf. We (Google Cloud Java Diamond Dependency team) maintain
> the BOM so that the set of the libraries are compatible with each other.
>
> h1. Implementation
> Notes for obstacles.
> h2. BeamModulePlugin's "force" does not take BOM into account (thus fails)
> {{forcedModules}} via version resolution strategy is playing bad. This causes
> {noformat}
> A problem occurred evaluating project ':sdks:java:extensions:sql'.
> Could not resolve all dependencies for configuration
> ':sdks:java:extensions:sql:fmppTemplates'.
> Invalid format: 'com.google.cloud:google-cloud-core'. Group, name and version
> cannot be empty. Correct example: 'org.gradle:gradle-core:1.0'{noformat}
> !Screen Shot 2020-03-13 at 13.33.01.png|width=489,height=287!
>
> h2. :sdks:java:maven-archetypes:examples needs the version of
> google-http-client
> The task requires the version for the library:
> {code}
> 'google-http-client.version':
> dependencies.create(project.library.java.google_http_client).getVersion(),
> {code}
> This would generate NullPointerException. Running gradlew without the
> subproject:
>
> {code:java}
> ./gradlew -p sdks/java check -x :sdks:java:maven-archetypes:examples:check
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9444) Shall we use GCP Libraries BOM to specify Google-related library versions?
[
https://issues.apache.org/jira/browse/BEAM-9444?focusedWorklogId=404403&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-404403
]
ASF GitHub Bot logged work on BEAM-9444:
Author: ASF GitHub Bot
Created on: 17/Mar/20 02:09
Start Date: 17/Mar/20 02:09
Worklog Time Spent: 10m
Work Description: suztomo commented on pull request #11125: [BEAM-9444]
(Old) Use Google Cloud Libraries BOM to set GCP library dependencies
URL: https://github.com/apache/beam/pull/11125
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 404403)
Time Spent: 1h 20m (was: 1h 10m)
> Shall we use GCP Libraries BOM to specify Google-related library versions?
> --
>
> Key: BEAM-9444
> URL: https://issues.apache.org/jira/browse/BEAM-9444
> Project: Beam
> Issue Type: Task
> Components: build-system
>Reporter: Tomo Suzuki
>Assignee: Tomo Suzuki
>Priority: Major
> Attachments: Screen Shot 2020-03-13 at 13.33.01.png
>
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> Shall we use GCP Libraries BOM to specify Google-related library versions?
>
> I've been working on Beam's dependency upgrades in the past few months. I
> think it's time to consider a long-term solution to keep the libraries
> up-to-date with small maintenance effort. To achieve that, I propose Beam to
> use GCP Libraries BOM to set the Google-related library versions, rather than
> trying to make changes in each of ~30 Google libraries.
>
> h1. Background
> A BOM is pom.xml that provides dependencyManagement to importing projects.
>
> GCP Libraries BOM is a BOM that includes many Google Cloud related libraries
> + gRPC + protobuf. We (Google Cloud Java Diamond Dependency team) maintain
> the BOM so that the set of the libraries are compatible with each other.
>
> h1. Implementation
> Notes for obstacles.
> h2. BeamModulePlugin's "force" does not take BOM into account (thus fails)
> {{forcedModules}} via version resolution strategy is playing bad. This causes
> {noformat}
> A problem occurred evaluating project ':sdks:java:extensions:sql'.
> Could not resolve all dependencies for configuration
> ':sdks:java:extensions:sql:fmppTemplates'.
> Invalid format: 'com.google.cloud:google-cloud-core'. Group, name and version
> cannot be empty. Correct example: 'org.gradle:gradle-core:1.0'{noformat}
> !Screen Shot 2020-03-13 at 13.33.01.png|width=489,height=287!
>
> h2. :sdks:java:maven-archetypes:examples needs the version of
> google-http-client
> The task requires the version for the library:
> {code}
> 'google-http-client.version':
> dependencies.create(project.library.java.google_http_client).getVersion(),
> {code}
> This would generate NullPointerException. Running gradlew without the
> subproject:
>
> {code:java}
> ./gradlew -p sdks/java check -x :sdks:java:maven-archetypes:examples:check
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-7923) Interactive Beam
[ https://issues.apache.org/jira/browse/BEAM-7923?focusedWorklogId=404401&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-404401 ] ASF GitHub Bot logged work on BEAM-7923: Author: ASF GitHub Bot Created on: 17/Mar/20 02:02 Start Date: 17/Mar/20 02:02 Worklog Time Spent: 10m Work Description: aaltay commented on issue #11141: [BEAM-7923] Include side effects in p.run URL: https://github.com/apache/beam/pull/11141#issuecomment-599839271 Could you resolve the conflict? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 404401) Time Spent: 2h 20m (was: 2h 10m) > Interactive Beam > > > Key: BEAM-7923 > URL: https://issues.apache.org/jira/browse/BEAM-7923 > Project: Beam > Issue Type: New Feature > Components: runner-py-interactive >Reporter: Ning Kang >Assignee: Ning Kang >Priority: Major > Time Spent: 2h 20m > Remaining Estimate: 0h > > This is the top level ticket for all efforts leveraging [interactive > Beam|[https://github.com/apache/beam/tree/master/sdks/python/apache_beam/runners/interactive]] > As the development goes, blocking tickets will be added to this one. > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-3301) Go SplittableDoFn support
[ https://issues.apache.org/jira/browse/BEAM-3301?focusedWorklogId=404390&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-404390 ] ASF GitHub Bot logged work on BEAM-3301: Author: ASF GitHub Bot Created on: 17/Mar/20 01:31 Start Date: 17/Mar/20 01:31 Worklog Time Spent: 10m Work Description: youngoli commented on issue #11144: [BEAM-3301] Perform SDF validation (missing RestrictionTrackers). URL: https://github.com/apache/beam/pull/11144#issuecomment-599832332 Btw, only major thing I was unsure of. Should I be hiding this functionality behind an experiment flag of some kind? This initial one doesn't do so, and I don't expect people would randomly use it before it's ready because there's no user-facing documentation indicating its usage in the "beam" package, but it's still technically possible to use it right now. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 404390) Time Spent: 5.5h (was: 5h 20m) > Go SplittableDoFn support > - > > Key: BEAM-3301 > URL: https://issues.apache.org/jira/browse/BEAM-3301 > Project: Beam > Issue Type: Improvement > Components: sdk-go >Reporter: Henning Rohde >Assignee: Daniel Oliveira >Priority: Major > Time Spent: 5.5h > Remaining Estimate: 0h > > SDFs will be the only way to add streaming and liquid sharded IO for Go. > Design doc: https://s.apache.org/splittable-do-fn -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-3301) Go SplittableDoFn support
[ https://issues.apache.org/jira/browse/BEAM-3301?focusedWorklogId=404388&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-404388 ] ASF GitHub Bot logged work on BEAM-3301: Author: ASF GitHub Bot Created on: 17/Mar/20 01:29 Start Date: 17/Mar/20 01:29 Worklog Time Spent: 10m Work Description: youngoli commented on pull request #11144: [BEAM-3301] Perform SDF validation (missing RestrictionTrackers). URL: https://github.com/apache/beam/pull/11144 Added SDF detection and method validation when creating DoFns. This adds all the validation except for anything to do with restriction trackers, which will be added in a later change. Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [x] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [x] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] Update `CHANGES.md` with noteworthy changes. - [x] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier). Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[ Go SplittableDoFn support
[ https://issues.apache.org/jira/browse/BEAM-3301?focusedWorklogId=404389&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-404389 ] ASF GitHub Bot logged work on BEAM-3301: Author: ASF GitHub Bot Created on: 17/Mar/20 01:29 Start Date: 17/Mar/20 01:29 Worklog Time Spent: 10m Work Description: youngoli commented on issue #11144: [BEAM-3301] Perform SDF validation (missing RestrictionTrackers). URL: https://github.com/apache/beam/pull/11144#issuecomment-599831761 R: @lostluck This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 404389) Time Spent: 5h 20m (was: 5h 10m) > Go SplittableDoFn support > - > > Key: BEAM-3301 > URL: https://issues.apache.org/jira/browse/BEAM-3301 > Project: Beam > Issue Type: Improvement > Components: sdk-go >Reporter: Henning Rohde >Assignee: Daniel Oliveira >Priority: Major > Time Spent: 5h 20m > Remaining Estimate: 0h > > SDFs will be the only way to add streaming and liquid sharded IO for Go. > Design doc: https://s.apache.org/splittable-do-fn -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-3301) Go SplittableDoFn support
[ https://issues.apache.org/jira/browse/BEAM-3301?focusedWorklogId=404382&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-404382 ] ASF GitHub Bot logged work on BEAM-3301: Author: ASF GitHub Bot Created on: 17/Mar/20 01:09 Start Date: 17/Mar/20 01:09 Worklog Time Spent: 10m Work Description: youngoli commented on pull request #10991: [BEAM-3301] Refactor DoFn validation & allow specifying main inputs. URL: https://github.com/apache/beam/pull/10991 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 404382) Time Spent: 5h (was: 4h 50m) > Go SplittableDoFn support > - > > Key: BEAM-3301 > URL: https://issues.apache.org/jira/browse/BEAM-3301 > Project: Beam > Issue Type: Improvement > Components: sdk-go >Reporter: Henning Rohde >Assignee: Daniel Oliveira >Priority: Major > Time Spent: 5h > Remaining Estimate: 0h > > SDFs will be the only way to add streaming and liquid sharded IO for Go. > Design doc: https://s.apache.org/splittable-do-fn -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9498) RowJson exception for unsupported types should list the relevant fields
[
https://issues.apache.org/jira/browse/BEAM-9498?focusedWorklogId=404380&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-404380
]
ASF GitHub Bot logged work on BEAM-9498:
Author: ASF GitHub Bot
Created on: 17/Mar/20 01:06
Start Date: 17/Mar/20 01:06
Worklog Time Spent: 10m
Work Description: apilloud commented on pull request #9: [BEAM-9498]
Include descriptor and type of unsupported fields in RowJson exception
URL: https://github.com/apache/beam/pull/9#discussion_r393390344
##
File path: sdks/java/core/src/main/java/org/apache/beam/sdk/util/RowJson.java
##
@@ -84,41 +85,76 @@
private static final ImmutableSet SUPPORTED_TYPES =
ImmutableSet.of(BYTE, INT16, INT32, INT64, FLOAT, DOUBLE, BOOLEAN,
STRING, DECIMAL);
+ /**
+ * Throws {@link UnsupportedRowJsonException} if {@code schema} contains an
unsupported field
+ * type.
+ */
public static void verifySchemaSupported(Schema schema) {
-schema.getFields().forEach(RowJson::verifyFieldTypeSupported);
+ImmutableList unsupportedFields =
findUnsupportedFields(schema);
+if (!unsupportedFields.isEmpty()) {
+ throw new UnsupportedRowJsonException(
+ String.format(
+ "Field type%s %s not supported when converting between JSON and
Rows. Supported types are: %s",
+ unsupportedFields.size() > 1 ? "s" : "",
+ unsupportedFields.toString(),
+ SUPPORTED_TYPES.toString()));
+}
+ }
+
+ private static class UnsupportedField {
+final String descriptor;
+final TypeName typeName;
+
+UnsupportedField(String descriptor, TypeName typeName) {
+ this.descriptor = descriptor;
+ this.typeName = typeName;
+}
+
+@Override
+public String toString() {
+ return this.descriptor + "=" + this.typeName;
+}
+ }
+
+ private static ImmutableList findUnsupportedFields(Schema
schema) {
+return schema.getFields().stream()
+.flatMap((field) -> findUnsupportedFields(field).stream())
+.collect(toImmutableList());
}
- static void verifyFieldTypeSupported(Field field) {
+ private static ImmutableList findUnsupportedFields(Field
field) {
FieldType fieldType = field.getType();
Review comment:
nit: drop the variable, it doesn't add value.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 404380)
Time Spent: 20m (was: 10m)
> RowJson exception for unsupported types should list the relevant fields
> ---
>
> Key: BEAM-9498
> URL: https://issues.apache.org/jira/browse/BEAM-9498
> Project: Beam
> Issue Type: Improvement
> Components: dsl-sql
>Reporter: Brian Hulette
>Assignee: Brian Hulette
>Priority: Major
> Time Spent: 20m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9510) Dependencies in base_image_requirements.txt are not compatible with each other
[ https://issues.apache.org/jira/browse/BEAM-9510?focusedWorklogId=404381&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-404381 ] ASF GitHub Bot logged work on BEAM-9510: Author: ASF GitHub Bot Created on: 17/Mar/20 01:06 Start Date: 17/Mar/20 01:06 Worklog Time Spent: 10m Work Description: tvalentyn commented on issue #11142: [BEAM-9510] Fixing version incompatibilities in base_image_requirements.txt URL: https://github.com/apache/beam/pull/11142#issuecomment-599826365 Run Python Dataflow ValidatesContainer This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 404381) Time Spent: 40m (was: 0.5h) > Dependencies in base_image_requirements.txt are not compatible with each other > -- > > Key: BEAM-9510 > URL: https://issues.apache.org/jira/browse/BEAM-9510 > Project: Beam > Issue Type: Bug > Components: sdk-py-harness >Reporter: David Yan >Assignee: Hannah Jiang >Priority: Major > Time Spent: 40m > Remaining Estimate: 0h > > [https://github.com/apache/beam/blob/master/sdks/python/container/base_image_requirements.txt#L56] > says it requires google-cloud-bigquery==1.24.0, google-cloud-core==1.0.2, > google-cloud-bigtable==0.32.1, grpc-1.22.0 and tensorflow-2.1.0 > But they are incompatible with each other: > ERROR: google-cloud-bigquery 1.24.0 has requirement > google-cloud-core<2.0dev,>=1.1.0, but you'll have google-cloud-core 1.0.2 > which is incompatible. > ERROR: google-cloud-bigtable 0.32.1 has requirement > google-cloud-core<0.30dev,>=0.29.0, but you'll have google-cloud-core 1.0.2 > which is incompatible. > ERROR: tensorboard 2.1.1 has requirement grpcio>=1.24.3, but you'll have > grpcio 1.22.0 which is incompatible. > ERROR: tensorflow 2.1.0 has requirement scipy==1.4.1; python_version >= "3", > but you'll have scipy 1.2.2 which is incompatible. > > > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9498) RowJson exception for unsupported types should list the relevant fields
[
https://issues.apache.org/jira/browse/BEAM-9498?focusedWorklogId=404379&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-404379
]
ASF GitHub Bot logged work on BEAM-9498:
Author: ASF GitHub Bot
Created on: 17/Mar/20 01:06
Start Date: 17/Mar/20 01:06
Worklog Time Spent: 10m
Work Description: apilloud commented on pull request #9: [BEAM-9498]
Include descriptor and type of unsupported fields in RowJson exception
URL: https://github.com/apache/beam/pull/9#discussion_r393391148
##
File path: sdks/java/core/src/main/java/org/apache/beam/sdk/util/RowJson.java
##
@@ -84,41 +85,76 @@
private static final ImmutableSet SUPPORTED_TYPES =
ImmutableSet.of(BYTE, INT16, INT32, INT64, FLOAT, DOUBLE, BOOLEAN,
STRING, DECIMAL);
+ /**
+ * Throws {@link UnsupportedRowJsonException} if {@code schema} contains an
unsupported field
+ * type.
+ */
public static void verifySchemaSupported(Schema schema) {
-schema.getFields().forEach(RowJson::verifyFieldTypeSupported);
+ImmutableList unsupportedFields =
findUnsupportedFields(schema);
+if (!unsupportedFields.isEmpty()) {
+ throw new UnsupportedRowJsonException(
+ String.format(
+ "Field type%s %s not supported when converting between JSON and
Rows. Supported types are: %s",
+ unsupportedFields.size() > 1 ? "s" : "",
+ unsupportedFields.toString(),
+ SUPPORTED_TYPES.toString()));
+}
+ }
+
+ private static class UnsupportedField {
+final String descriptor;
+final TypeName typeName;
+
+UnsupportedField(String descriptor, TypeName typeName) {
+ this.descriptor = descriptor;
+ this.typeName = typeName;
+}
+
+@Override
+public String toString() {
+ return this.descriptor + "=" + this.typeName;
+}
+ }
+
+ private static ImmutableList findUnsupportedFields(Schema
schema) {
+return schema.getFields().stream()
+.flatMap((field) -> findUnsupportedFields(field).stream())
+.collect(toImmutableList());
}
- static void verifyFieldTypeSupported(Field field) {
+ private static ImmutableList findUnsupportedFields(Field
field) {
FieldType fieldType = field.getType();
-verifyFieldTypeSupported(fieldType);
+return findUnsupportedFields(fieldType, field.getName());
}
- static void verifyFieldTypeSupported(FieldType fieldType) {
+ private static ImmutableList findUnsupportedFields(
+ FieldType fieldType, String fieldName) {
TypeName fieldTypeName = fieldType.getTypeName();
if (fieldTypeName.isCompositeType()) {
Schema rowFieldSchema = fieldType.getRowSchema();
Review comment:
nit: drop the variable, it doesn't add value.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 404379)
Time Spent: 20m (was: 10m)
> RowJson exception for unsupported types should list the relevant fields
> ---
>
> Key: BEAM-9498
> URL: https://issues.apache.org/jira/browse/BEAM-9498
> Project: Beam
> Issue Type: Improvement
> Components: dsl-sql
>Reporter: Brian Hulette
>Assignee: Brian Hulette
>Priority: Major
> Time Spent: 20m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (BEAM-9507) Beam dependency check failing
[
https://issues.apache.org/jira/browse/BEAM-9507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17060556#comment-17060556
]
Ahmet Altay commented on BEAM-9507:
---
cc: [~yifanzou]
> Beam dependency check failing
> -
>
> Key: BEAM-9507
> URL: https://issues.apache.org/jira/browse/BEAM-9507
> Project: Beam
> Issue Type: Bug
> Components: testing
>Reporter: Michał Walenia
>Priority: Major
>
> Here are the logs:
> [https://builds.apache.org/job/beam_Dependency_Check/257/console]
>
>from grpc_tools import protoc*13:04:25* ImportError: No module
> named 'grpc_tools'*13:04:25* *13:04:25* During handling of the above
> exception, another exception occurred:*13:04:25* *13:04:25* Traceback
> (most recent call last):*13:04:25* File
> "/usr/lib/python3.5/multiprocessing/process.py", line 249, in
> _bootstrap*13:04:25* self.run()*13:04:25* File
> "/usr/lib/python3.5/multiprocessing/process.py", line 93, in run*13:04:25*
> self._target(*self._args, **self._kwargs)*13:04:25* File
> "/home/jenkins/jenkins-slave/workspace/beam_Dependency_Check/src/sdks/python/gen_protos.py",
> line 378, in _install_grpcio_tools_and_generate_proto_files*13:04:25*
> generate_proto_files(force=force)*13:04:25* File
> "/home/jenkins/jenkins-slave/workspace/beam_Dependency_Check/src/sdks/python/gen_protos.py",
> line 315, in generate_proto_files*13:04:25* protoc_gen_mypy =
> _find_protoc_gen_mypy()*13:04:25* File
> "/home/jenkins/jenkins-slave/workspace/beam_Dependency_Check/src/sdks/python/gen_protos.py",
> line 233, in _find_protoc_gen_mypy*13:04:25* (fname, ',
> '.join(search_paths)))*13:04:25* RuntimeError: Could not find
> protoc-gen-mypy in
> /home/jenkins/jenkins-slave/workspace/beam_Dependency_Check/src/sdks/python/sdks/python/bin,
>
> /home/jenkins/jenkins-slave/workspace/beam_Dependency_Check/src/sdks/python/sdks/python/bin,
> /home/jenkins/tools/java/latest1.8/bin, /usr/local/sbin, /usr/local/bin,
> /usr/sbin, /usr/bin, /sbin, /bin, /usr/games, /usr/local/games*13:04:25*
> Traceback (most recent call last):*13:04:25* File
> "/home/jenkins/jenkins-slave/workspace/beam_Dependency_Check/src/sdks/python/gen_protos.py",
> line 292, in generate_proto_files*13:04:25* from grpc_tools import
> protoc*13:04:25* ImportError: No module named 'grpc_tools'*13:04:25*
> *13:04:25* During handling of the above exception, another exception
> occurred:*13:04:25* *13:04:25* Traceback (most recent call
> last):*13:04:25* File "", line 1, in *13:04:25*
> File
> "/home/jenkins/jenkins-slave/workspace/beam_Dependency_Check/src/sdks/python/setup.py",
> line 315, in *13:04:25* 'mypy':
> generate_protos_first(mypy),*13:04:25* File
> "/home/jenkins/jenkins-slave/workspace/beam_Dependency_Check/src/sdks/python/sdks/python/lib/python3.5/site-packages/setuptools/__init__.py",
> line 144, in setup*13:04:25* return
> distutils.core.setup(**attrs)*13:04:25* File
> "/usr/lib/python3.5/distutils/core.py", line 148, in setup*13:04:25*
> dist.run_commands()*13:04:25* File
> "/usr/lib/python3.5/distutils/dist.py", line 955, in run_commands*13:04:25*
> self.run_command(cmd)*13:04:25* File
> "/usr/lib/python3.5/distutils/dist.py", line 974, in run_command*13:04:25*
> cmd_obj.run()*13:04:25* File
> "/home/jenkins/jenkins-slave/workspace/beam_Dependency_Check/src/sdks/python/setup.py",
> line 239, in run*13:04:25*
> gen_protos.generate_proto_files()*13:04:25* File
> "/home/jenkins/jenkins-slave/workspace/beam_Dependency_Check/src/sdks/python/gen_protos.py",
> line 310, in generate_proto_files*13:04:25* raise ValueError("Proto
> generation failed (see log for details).")*13:04:25* ValueError: Proto
> generation failed (see log for details).*13:04:25*
> *13:04:25* ERROR: Command errored out
> with exit status 1: python setup.py egg_info Check the logs for full command
> output.*13:04:25* *13:04:25* >
> *Task :sdks:python:dependencyUpdates*
> FAILED*13:04:25* *13:04:25* FAILURE: Build failed with an
> exception.*13:04:25* *13:04:25* * Where:*13:04:25* Build file
> '/home/jenkins/jenkins-slave/workspace/beam_Dependency_Check/src/sdks/python/build.gradle'
> line: 94*13:04:25* *13:04:25* * What went wrong:*13:04:25* Execution failed
> for task ':sdks:python:dependencyUpdates'.*13:04:25* > Process 'command 'sh''
> finished with non-zero exit value 1
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9526) Bug in top.LargestPerKey, disappearing elements.
[ https://issues.apache.org/jira/browse/BEAM-9526?focusedWorklogId=404378&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-404378 ] ASF GitHub Bot logged work on BEAM-9526: Author: ASF GitHub Bot Created on: 17/Mar/20 01:05 Start Date: 17/Mar/20 01:05 Worklog Time Spent: 10m Work Description: youngoli commented on issue #11143: [BEAM-9526] Add missing unmarshalling in top.LargestPerKey. URL: https://github.com/apache/beam/pull/11143#issuecomment-599826047 R: @lostluck This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 404378) Time Spent: 20m (was: 10m) > Bug in top.LargestPerKey, disappearing elements. > > > Key: BEAM-9526 > URL: https://issues.apache.org/jira/browse/BEAM-9526 > Project: Beam > Issue Type: Bug > Components: sdk-go >Reporter: Daniel Oliveira >Assignee: Daniel Oliveira >Priority: Minor > Time Spent: 20m > Remaining Estimate: 0h > > There's a bug in top.LargestPerKey that appears when running it on a runner > that has combiner lifting. Since it's similar to a GBK, we should be > inputting a K, V and receiving a K, []V as output. When the runner has > combiner lifting, the []V always contains an empty slice. > > I already found the fix for this, just logging this to attach it to the PR. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9526) Bug in top.LargestPerKey, disappearing elements.
[ https://issues.apache.org/jira/browse/BEAM-9526?focusedWorklogId=404377&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-404377 ] ASF GitHub Bot logged work on BEAM-9526: Author: ASF GitHub Bot Created on: 17/Mar/20 01:03 Start Date: 17/Mar/20 01:03 Worklog Time Spent: 10m Work Description: youngoli commented on pull request #11143: [BEAM-9526] Add missing unmarshalling in top.LargestPerKey. URL: https://github.com/apache/beam/pull/11143 ExtractOutput wasn't unmarshalling the accumulator first, which would cause the values to get dropped if the accumulator was marshalled when ExtractOutput was called. This fixes that and modifies the unit test slightly to catch it if it stops working in the future. Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [x] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [x] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] Update `CHANGES.md` with noteworthy changes. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier). Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCo
[jira] [Work logged] (BEAM-9510) Dependencies in base_image_requirements.txt are not compatible with each other
[ https://issues.apache.org/jira/browse/BEAM-9510?focusedWorklogId=404374&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-404374 ] ASF GitHub Bot logged work on BEAM-9510: Author: ASF GitHub Bot Created on: 17/Mar/20 01:00 Start Date: 17/Mar/20 01:00 Worklog Time Spent: 10m Work Description: tvalentyn commented on issue #11142: [BEAM-9510] Fixing version incompatibilities in base_image_requirements.txt URL: https://github.com/apache/beam/pull/11142#issuecomment-599825064 Run Dataflow Python ValidatesContainer This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 404374) Time Spent: 0.5h (was: 20m) > Dependencies in base_image_requirements.txt are not compatible with each other > -- > > Key: BEAM-9510 > URL: https://issues.apache.org/jira/browse/BEAM-9510 > Project: Beam > Issue Type: Bug > Components: sdk-py-harness >Reporter: David Yan >Assignee: Hannah Jiang >Priority: Major > Time Spent: 0.5h > Remaining Estimate: 0h > > [https://github.com/apache/beam/blob/master/sdks/python/container/base_image_requirements.txt#L56] > says it requires google-cloud-bigquery==1.24.0, google-cloud-core==1.0.2, > google-cloud-bigtable==0.32.1, grpc-1.22.0 and tensorflow-2.1.0 > But they are incompatible with each other: > ERROR: google-cloud-bigquery 1.24.0 has requirement > google-cloud-core<2.0dev,>=1.1.0, but you'll have google-cloud-core 1.0.2 > which is incompatible. > ERROR: google-cloud-bigtable 0.32.1 has requirement > google-cloud-core<0.30dev,>=0.29.0, but you'll have google-cloud-core 1.0.2 > which is incompatible. > ERROR: tensorboard 2.1.1 has requirement grpcio>=1.24.3, but you'll have > grpcio 1.22.0 which is incompatible. > ERROR: tensorflow 2.1.0 has requirement scipy==1.4.1; python_version >= "3", > but you'll have scipy 1.2.2 which is incompatible. > > > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9510) Dependencies in base_image_requirements.txt are not compatible with each other
[ https://issues.apache.org/jira/browse/BEAM-9510?focusedWorklogId=404371&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-404371 ] ASF GitHub Bot logged work on BEAM-9510: Author: ASF GitHub Bot Created on: 17/Mar/20 00:49 Start Date: 17/Mar/20 00:49 Worklog Time Spent: 10m Work Description: davidyan74 commented on issue #11142: [BEAM-9510] Fixing version incompatibilities in base_image_requirements.txt URL: https://github.com/apache/beam/pull/11142#issuecomment-599822693 R: @aaltay This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 404371) Time Spent: 20m (was: 10m) > Dependencies in base_image_requirements.txt are not compatible with each other > -- > > Key: BEAM-9510 > URL: https://issues.apache.org/jira/browse/BEAM-9510 > Project: Beam > Issue Type: Bug > Components: sdk-py-harness >Reporter: David Yan >Assignee: Hannah Jiang >Priority: Major > Time Spent: 20m > Remaining Estimate: 0h > > [https://github.com/apache/beam/blob/master/sdks/python/container/base_image_requirements.txt#L56] > says it requires google-cloud-bigquery==1.24.0, google-cloud-core==1.0.2, > google-cloud-bigtable==0.32.1, grpc-1.22.0 and tensorflow-2.1.0 > But they are incompatible with each other: > ERROR: google-cloud-bigquery 1.24.0 has requirement > google-cloud-core<2.0dev,>=1.1.0, but you'll have google-cloud-core 1.0.2 > which is incompatible. > ERROR: google-cloud-bigtable 0.32.1 has requirement > google-cloud-core<0.30dev,>=0.29.0, but you'll have google-cloud-core 1.0.2 > which is incompatible. > ERROR: tensorboard 2.1.1 has requirement grpcio>=1.24.3, but you'll have > grpcio 1.22.0 which is incompatible. > ERROR: tensorflow 2.1.0 has requirement scipy==1.4.1; python_version >= "3", > but you'll have scipy 1.2.2 which is incompatible. > > > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9510) Dependencies in base_image_requirements.txt are not compatible with each other
[ https://issues.apache.org/jira/browse/BEAM-9510?focusedWorklogId=404370&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-404370 ] ASF GitHub Bot logged work on BEAM-9510: Author: ASF GitHub Bot Created on: 17/Mar/20 00:46 Start Date: 17/Mar/20 00:46 Worklog Time Spent: 10m Work Description: davidyan74 commented on pull request #11142: [BEAM-9510] Fixing version incompatibilities in base_image_requirements.txt URL: https://github.com/apache/beam/pull/11142 Tried this out and it doesn't have any compatibility error messages any more for python3. R: @tvalentyn **Please** add a meaningful description for your change here Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] Update `CHANGES.md` with noteworthy changes. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier). Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[ Support NAN, INF, and -INF
[ https://issues.apache.org/jira/browse/BEAM-8057?focusedWorklogId=404369&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-404369 ] ASF GitHub Bot logged work on BEAM-8057: Author: ASF GitHub Bot Created on: 17/Mar/20 00:45 Start Date: 17/Mar/20 00:45 Worklog Time Spent: 10m Work Description: apilloud commented on pull request #11131: [BEAM-8057] Reject Infinite or NaN literals at parse time URL: https://github.com/apache/beam/pull/11131 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 404369) Time Spent: 0.5h (was: 20m) > Support NAN, INF, and -INF > -- > > Key: BEAM-8057 > URL: https://issues.apache.org/jira/browse/BEAM-8057 > Project: Beam > Issue Type: Sub-task > Components: dsl-sql-zetasql >Reporter: Rui Wang >Priority: Major > Time Spent: 0.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8070) Support empty array literal
[ https://issues.apache.org/jira/browse/BEAM-8070?focusedWorklogId=404368&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-404368 ] ASF GitHub Bot logged work on BEAM-8070: Author: ASF GitHub Bot Created on: 17/Mar/20 00:43 Start Date: 17/Mar/20 00:43 Worklog Time Spent: 10m Work Description: apilloud commented on pull request #11130: [BEAM-8070] Preserve type for empty array URL: https://github.com/apache/beam/pull/11130 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 404368) Time Spent: 1h 20m (was: 1h 10m) > Support empty array literal > --- > > Key: BEAM-8070 > URL: https://issues.apache.org/jira/browse/BEAM-8070 > Project: Beam > Issue Type: Sub-task > Components: dsl-sql-zetasql >Reporter: Rui Wang >Assignee: Andrew Pilloud >Priority: Major > Labels: zetasql-compliance > Time Spent: 1h 20m > Remaining Estimate: 0h > > Currently BeamSQL throws an IndexOutOfBoundsException when given a query with > an empty array literal. This happens because Calcite attempts to infer the > element types [1,2] from an empty element list. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9526) Bug in top.LargestPerKey, disappearing elements.
[ https://issues.apache.org/jira/browse/BEAM-9526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Oliveira updated BEAM-9526: -- Status: Open (was: Triage Needed) > Bug in top.LargestPerKey, disappearing elements. > > > Key: BEAM-9526 > URL: https://issues.apache.org/jira/browse/BEAM-9526 > Project: Beam > Issue Type: Bug > Components: sdk-go >Reporter: Daniel Oliveira >Assignee: Daniel Oliveira >Priority: Minor > > There's a bug in top.LargestPerKey that appears when running it on a runner > that has combiner lifting. Since it's similar to a GBK, we should be > inputting a K, V and receiving a K, []V as output. When the runner has > combiner lifting, the []V always contains an empty slice. > > I already found the fix for this, just logging this to attach it to the PR. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work started] (BEAM-9526) Bug in top.LargestPerKey, disappearing elements.
[ https://issues.apache.org/jira/browse/BEAM-9526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on BEAM-9526 started by Daniel Oliveira. - > Bug in top.LargestPerKey, disappearing elements. > > > Key: BEAM-9526 > URL: https://issues.apache.org/jira/browse/BEAM-9526 > Project: Beam > Issue Type: Bug > Components: sdk-go >Reporter: Daniel Oliveira >Assignee: Daniel Oliveira >Priority: Minor > > There's a bug in top.LargestPerKey that appears when running it on a runner > that has combiner lifting. Since it's similar to a GBK, we should be > inputting a K, V and receiving a K, []V as output. When the runner has > combiner lifting, the []V always contains an empty slice. > > I already found the fix for this, just logging this to attach it to the PR. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (BEAM-9526) Bug in top.LargestPerKey, disappearing elements.
Daniel Oliveira created BEAM-9526: - Summary: Bug in top.LargestPerKey, disappearing elements. Key: BEAM-9526 URL: https://issues.apache.org/jira/browse/BEAM-9526 Project: Beam Issue Type: Bug Components: sdk-go Reporter: Daniel Oliveira Assignee: Daniel Oliveira There's a bug in top.LargestPerKey that appears when running it on a runner that has combiner lifting. Since it's similar to a GBK, we should be inputting a K, V and receiving a K, []V as output. When the runner has combiner lifting, the []V always contains an empty slice. I already found the fix for this, just logging this to attach it to the PR. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-3301) Go SplittableDoFn support
[
https://issues.apache.org/jira/browse/BEAM-3301?focusedWorklogId=404362&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-404362
]
ASF GitHub Bot logged work on BEAM-3301:
Author: ASF GitHub Bot
Created on: 17/Mar/20 00:29
Start Date: 17/Mar/20 00:29
Worklog Time Spent: 10m
Work Description: lostluck commented on pull request #10991: [BEAM-3301]
Refactor DoFn validation & allow specifying main inputs.
URL: https://github.com/apache/beam/pull/10991#discussion_r393382209
##
File path: sdks/go/pkg/beam/core/graph/fn.go
##
@@ -239,52 +279,50 @@ func AsDoFn(fn *Fn) (*DoFn, error) {
return nil, addContext(err, fn)
}
- // Start validating DoFn. First, check that ProcessElement has a main
input.
+ // Validate ProcessElement has correct number of main inputs (as
indicated by
+ // numMainIn), and that main inputs are before side inputs.
processFn := fn.methods[processElementName]
- pos, num, ok := processFn.Inputs()
- if ok {
- first := processFn.Param[pos].Kind
- if first != funcx.FnValue {
- err := errors.New("side input parameters must follow
main input parameter")
- err = errors.SetTopLevelMsgf(err,
- "Method %v of DoFns should always have a main
input before side inputs, "+
- "but it has side inputs (as Iters or
ReIters) first in DoFn %v.",
- processElementName, fn.Name())
- err = errors.WithContextf(err, "method %v",
processElementName)
- return nil, addContext(err, fn)
- }
+ if err := validateMainInputs(fn, processFn, processElementName,
numMainIn); err != nil {
+ return nil, addContext(err, fn)
+ }
+
+ // If numMainIn is unknown, we can try inferring it from the second
input in ProcessElement.
+ // If there is none, or it's not a FnValue type, then we can safely
infer that there's only
+ // one main input.
+ pos, num, _ := processFn.Inputs()
+ if numMainIn == MainUnknown && (num == 1 || processFn.Param[pos+1].Kind
!= funcx.FnValue) {
+ numMainIn = MainSingle
}
// If the ProcessElement function includes side inputs or emit
functions those must also be
Review comment:
At most relaxed we'd be able to either not require them at all if none are
used, or isolate them by their types. All instances of a given side input or
emit with the same type would need to be listed at once, since otherwise we
have no way to distinguish them except by position. Permitting Nothing to be
set would be the most convenient, or permitting only the Side Inputs and not
requireing the Emits.
For now though, it's better to be more strict now and relax later, since the
inverse is impossible, and such variety is harder to maintain if unnecessary.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 404362)
Time Spent: 4h 50m (was: 4h 40m)
> Go SplittableDoFn support
> -
>
> Key: BEAM-3301
> URL: https://issues.apache.org/jira/browse/BEAM-3301
> Project: Beam
> Issue Type: Improvement
> Components: sdk-go
>Reporter: Henning Rohde
>Assignee: Daniel Oliveira
>Priority: Major
> Time Spent: 4h 50m
> Remaining Estimate: 0h
>
> SDFs will be the only way to add streaming and liquid sharded IO for Go.
> Design doc: https://s.apache.org/splittable-do-fn
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-8335) Add streaming support to Interactive Beam
[ https://issues.apache.org/jira/browse/BEAM-8335?focusedWorklogId=404347&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-404347 ] ASF GitHub Bot logged work on BEAM-8335: Author: ASF GitHub Bot Created on: 17/Mar/20 00:14 Start Date: 17/Mar/20 00:14 Worklog Time Spent: 10m Work Description: pabloem commented on issue #11138: [BEAM-8335] Refactor IPythonLogHandler URL: https://github.com/apache/beam/pull/11138#issuecomment-599814751 LGTM This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 404347) Time Spent: 113h 10m (was: 113h) > Add streaming support to Interactive Beam > - > > Key: BEAM-8335 > URL: https://issues.apache.org/jira/browse/BEAM-8335 > Project: Beam > Issue Type: Improvement > Components: runner-py-interactive >Reporter: Sam Rohde >Assignee: Sam Rohde >Priority: Major > Time Spent: 113h 10m > Remaining Estimate: 0h > > This issue tracks the work items to introduce streaming support to the > Interactive Beam experience. This will allow users to: > * Write and run a streaming job in IPython > * Automatically cache records from unbounded sources > * Add a replay experience that replays all cached records to simulate the > original pipeline execution > * Add controls to play/pause/stop/step individual elements from the cached > records > * Add ability to inspect/visualize unbounded PCollections -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8335) Add streaming support to Interactive Beam
[ https://issues.apache.org/jira/browse/BEAM-8335?focusedWorklogId=404346&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-404346 ] ASF GitHub Bot logged work on BEAM-8335: Author: ASF GitHub Bot Created on: 17/Mar/20 00:14 Start Date: 17/Mar/20 00:14 Worklog Time Spent: 10m Work Description: pabloem commented on pull request #11138: [BEAM-8335] Refactor IPythonLogHandler URL: https://github.com/apache/beam/pull/11138 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 404346) Time Spent: 113h (was: 112h 50m) > Add streaming support to Interactive Beam > - > > Key: BEAM-8335 > URL: https://issues.apache.org/jira/browse/BEAM-8335 > Project: Beam > Issue Type: Improvement > Components: runner-py-interactive >Reporter: Sam Rohde >Assignee: Sam Rohde >Priority: Major > Time Spent: 113h > Remaining Estimate: 0h > > This issue tracks the work items to introduce streaming support to the > Interactive Beam experience. This will allow users to: > * Write and run a streaming job in IPython > * Automatically cache records from unbounded sources > * Add a replay experience that replays all cached records to simulate the > original pipeline execution > * Add controls to play/pause/stop/step individual elements from the cached > records > * Add ability to inspect/visualize unbounded PCollections -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9421) AI Platform pipeline patterns
[
https://issues.apache.org/jira/browse/BEAM-9421?focusedWorklogId=404344&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-404344
]
ASF GitHub Bot logged work on BEAM-9421:
Author: ASF GitHub Bot
Created on: 17/Mar/20 00:12
Start Date: 17/Mar/20 00:12
Worklog Time Spent: 10m
Work Description: limingxi commented on pull request #11075: [BEAM-9421]
Website section that describes getting predictions using AI Platform Prediciton
URL: https://github.com/apache/beam/pull/11075#discussion_r393378386
##
File path: website/src/documentation/patterns/ai-platform.md
##
@@ -0,0 +1,87 @@
+---
+layout: section
+title: "AI Platform integration patterns"
+section_menu: section-menu/documentation.html
+permalink: /documentation/patterns/ai-platform/
+---
+
+
+# AI Platform integration patterns
+
+This page describes common patterns in pipelines with Google AI Platform
transforms.
+
+
+ Adapt for:
+
+Java SDK
+Python SDK
+
+
+
+## Getting predictions
+
+This section shows how to use a cloud-hosted machine learning model to make
predictions about new data using Google Cloud AI Platform Prediction within
Beam's pipeline.
+
+[tfx_bsl](https://github.com/tensorflow/tfx-bsl) is a library that provides
`RunInference` Beam's PTransform. `RunInference` is a PTransform able to
perform two types of inference. One of them can use a service endpoint. When
using a service endpoint, the transform takes a PCollection of type
`tf.train.Example` and, for each element, sends a request to Google Cloud AI
Platform Prediction service. The transform produces a PCollection of type
`PredictLog` which contains predictions.
+
+Before getting started, deploy a machine learning model to the cloud. The
cloud service manages the infrastructure needed to handle prediction requests
in both efficient and scalable way. Only Tensorflow models are supported. For
more information, see [Exporting a SavedModel for
prediction](https://cloud.google.com/ai-platform/prediction/docs/exporting-savedmodel-for-prediction).
+
+Once a machine learning model is deployed, prepare a list of instances to get
predictions for.
+
+Here is an example of a pipeline that reads input instances from the file,
converts JSON objects to `tf.train.Example` objects and sends data to the
service. The content of a file can look like this:
+
+```
+{"input": "the quick brown"}
+{"input": "la bruja le"}
+```
+
+The example creates `tf.train.BytesList` instances, thus it expects byte-like
strings as input, but other data types, like `tf.train.FloatList` and
`tf.train.Int64List`, are also supported by the transform. To send binary data,
make sure that the name of an input ends in `_bytes`.
Review comment:
I think it would be good to list all possible input formats and output
formats here or somewhere else for reference. And for the last sentence, do you
mean that we need to change l74 to something like:
feature={name+'_bytes', value} for sending binary data to endpoint?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 404344)
Time Spent: 3h 50m (was: 3h 40m)
> AI Platform pipeline patterns
> -
>
> Key: BEAM-9421
> URL: https://issues.apache.org/jira/browse/BEAM-9421
> Project: Beam
> Issue Type: Sub-task
> Components: website
>Reporter: Kamil Wasilewski
>Assignee: Kamil Wasilewski
>Priority: Major
> Labels: pipeline-patterns
> Time Spent: 3h 50m
> Remaining Estimate: 0h
>
> New pipeline patterns should be contributed to the Beam's website in order to
> demonstrate how newly implemented Google Cloud AI PTransforms can be used in
> pipelines.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9421) AI Platform pipeline patterns
[ https://issues.apache.org/jira/browse/BEAM-9421?focusedWorklogId=404327&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-404327 ] ASF GitHub Bot logged work on BEAM-9421: Author: ASF GitHub Bot Created on: 16/Mar/20 23:28 Start Date: 16/Mar/20 23:28 Worklog Time Spent: 10m Work Description: wenchenglu commented on issue #11075: [BEAM-9421] Website section that describes getting predictions using AI Platform Prediciton URL: https://github.com/apache/beam/pull/11075#issuecomment-599803048 Thanks for addressing my comments. These changes look good to me. I will ask another engineer from AI Platform prediction to do another pass. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 404327) Time Spent: 3h 40m (was: 3.5h) > AI Platform pipeline patterns > - > > Key: BEAM-9421 > URL: https://issues.apache.org/jira/browse/BEAM-9421 > Project: Beam > Issue Type: Sub-task > Components: website >Reporter: Kamil Wasilewski >Assignee: Kamil Wasilewski >Priority: Major > Labels: pipeline-patterns > Time Spent: 3h 40m > Remaining Estimate: 0h > > New pipeline patterns should be contributed to the Beam's website in order to > demonstrate how newly implemented Google Cloud AI PTransforms can be used in > pipelines. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9421) AI Platform pipeline patterns
[ https://issues.apache.org/jira/browse/BEAM-9421?focusedWorklogId=404326&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-404326 ] ASF GitHub Bot logged work on BEAM-9421: Author: ASF GitHub Bot Created on: 16/Mar/20 23:26 Start Date: 16/Mar/20 23:26 Worklog Time Spent: 10m Work Description: wenchenglu commented on pull request #11075: [BEAM-9421] Website section that describes getting predictions using AI Platform Prediciton URL: https://github.com/apache/beam/pull/11075#discussion_r393365839 ## File path: website/src/documentation/patterns/ai-platform.md ## @@ -0,0 +1,87 @@ +--- +layout: section +title: "AI Platform integration patterns" +section_menu: section-menu/documentation.html +permalink: /documentation/patterns/ai-platform/ +--- + + +# AI Platform integration patterns + +This page describes common patterns in pipelines with Google AI Platform transforms. + + + Adapt for: + +Java SDK +Python SDK + + + +## Getting predictions + +This section shows how to use a cloud-hosted machine learning model to make predictions about new data using Google Cloud AI Platform Prediction within Beam's pipeline. + +[tfx_bsl](https://github.com/tensorflow/tfx-bsl) is a library that provides `RunInference` Beam's PTransform. `RunInference` is a PTransform able to perform two types of inference. One of them can use a service endpoint. When using a service endpoint, the transform takes a PCollection of type `tf.train.Example` and, for each element, sends a request to Google Cloud AI Platform Prediction service. The transform produces a PCollection of type `PredictLog` which contains predictions. + +Before getting started, deploy a machine learning model to the cloud. The cloud service manages the infrastructure needed to handle prediction requests in both efficient and scalable way. Only Tensorflow models are supported. For more information, see [Exporting a SavedModel for prediction](https://cloud.google.com/ai-platform/prediction/docs/exporting-savedmodel-for-prediction). Review comment: LGTM This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 404326) Time Spent: 3.5h (was: 3h 20m) > AI Platform pipeline patterns > - > > Key: BEAM-9421 > URL: https://issues.apache.org/jira/browse/BEAM-9421 > Project: Beam > Issue Type: Sub-task > Components: website >Reporter: Kamil Wasilewski >Assignee: Kamil Wasilewski >Priority: Major > Labels: pipeline-patterns > Time Spent: 3.5h > Remaining Estimate: 0h > > New pipeline patterns should be contributed to the Beam's website in order to > demonstrate how newly implemented Google Cloud AI PTransforms can be used in > pipelines. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9421) AI Platform pipeline patterns
[ https://issues.apache.org/jira/browse/BEAM-9421?focusedWorklogId=404325&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-404325 ] ASF GitHub Bot logged work on BEAM-9421: Author: ASF GitHub Bot Created on: 16/Mar/20 23:23 Start Date: 16/Mar/20 23:23 Worklog Time Spent: 10m Work Description: aaltay commented on pull request #11075: [BEAM-9421] Website section that describes getting predictions using AI Platform Prediciton URL: https://github.com/apache/beam/pull/11075#discussion_r393365169 ## File path: website/src/documentation/patterns/ai-platform.md ## @@ -0,0 +1,87 @@ +--- +layout: section +title: "AI Platform integration patterns" +section_menu: section-menu/documentation.html +permalink: /documentation/patterns/ai-platform/ +--- + + +# AI Platform integration patterns + +This page describes common patterns in pipelines with Google AI Platform transforms. + + + Adapt for: + +Java SDK +Python SDK + + + +## Getting predictions + +This section shows how to use a cloud-hosted machine learning model to make predictions about new data using Google Cloud AI Platform Prediction within Beam's pipeline. + +[tfx_bsl](https://github.com/tensorflow/tfx-bsl) is a library that provides `RunInference` Beam's PTransform. `RunInference` is a PTransform able to perform two types of inference. One of them can use a service endpoint. When using a service endpoint, the transform takes a PCollection of type `tf.train.Example` and, for each element, sends a request to Google Cloud AI Platform Prediction service. The transform produces a PCollection of type `PredictLog` which contains predictions. + +Before getting started, deploy a machine learning model to the cloud. The cloud service manages the infrastructure needed to handle prediction requests in both efficient and scalable way. Only Tensorflow models are supported. For more information, see [Exporting a SavedModel for prediction](https://cloud.google.com/ai-platform/prediction/docs/exporting-savedmodel-for-prediction). Review comment: Beam does not have hooks for job start and job finish. One way to emulate those would be using an airflow wrapper to run some setup and cleanup tasks before and after a task. I do not believe we have a recommendation here but an example pattern might be useful (@rezarokni -- in case this will be a good pattern to add to the backlog.) For this version, it is probably fine to assume that the life cycle of the model will be managed by the user separately. For a next phase, we can try to build a graph similar to IOs (like BQ IO) to execute a certain setup and clean task in a single worker. (Although we cannot guarantee that it will not be retried) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 404325) Time Spent: 3h 20m (was: 3h 10m) > AI Platform pipeline patterns > - > > Key: BEAM-9421 > URL: https://issues.apache.org/jira/browse/BEAM-9421 > Project: Beam > Issue Type: Sub-task > Components: website >Reporter: Kamil Wasilewski >Assignee: Kamil Wasilewski >Priority: Major > Labels: pipeline-patterns > Time Spent: 3h 20m > Remaining Estimate: 0h > > New pipeline patterns should be contributed to the Beam's website in order to > demonstrate how newly implemented Google Cloud AI PTransforms can be used in > pipelines. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9485) Dataflow Silently drops Non implemented transform in fnapi mode.
[ https://issues.apache.org/jira/browse/BEAM-9485?focusedWorklogId=404318&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-404318 ] ASF GitHub Bot logged work on BEAM-9485: Author: ASF GitHub Bot Created on: 16/Mar/20 23:08 Start Date: 16/Mar/20 23:08 Worklog Time Spent: 10m Work Description: angoenka commented on issue #1: [cherry-pick][BEAM-9485] Raise error when transform urn is not implemented URL: https://github.com/apache/beam/pull/1#issuecomment-599797732 already merged in 2.20 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 404318) Time Spent: 3h 10m (was: 3h) > Dataflow Silently drops Non implemented transform in fnapi mode. > > > Key: BEAM-9485 > URL: https://issues.apache.org/jira/browse/BEAM-9485 > Project: Beam > Issue Type: Bug > Components: runner-dataflow, sdk-py-core >Reporter: Ankur Goenka >Assignee: Ankur Goenka >Priority: Major > Fix For: 2.20.0 > > Time Spent: 3h 10m > Remaining Estimate: 0h > > We need to raise an error here > https://github.com/apache/beam/blob/02cb8d807314a38542c9894b19483e4333d8223b/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py#L857 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9485) Dataflow Silently drops Non implemented transform in fnapi mode.
[ https://issues.apache.org/jira/browse/BEAM-9485?focusedWorklogId=404319&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-404319 ] ASF GitHub Bot logged work on BEAM-9485: Author: ASF GitHub Bot Created on: 16/Mar/20 23:08 Start Date: 16/Mar/20 23:08 Worklog Time Spent: 10m Work Description: angoenka commented on pull request #1: [cherry-pick][BEAM-9485] Raise error when transform urn is not implemented URL: https://github.com/apache/beam/pull/1 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 404319) Time Spent: 3h 20m (was: 3h 10m) > Dataflow Silently drops Non implemented transform in fnapi mode. > > > Key: BEAM-9485 > URL: https://issues.apache.org/jira/browse/BEAM-9485 > Project: Beam > Issue Type: Bug > Components: runner-dataflow, sdk-py-core >Reporter: Ankur Goenka >Assignee: Ankur Goenka >Priority: Major > Fix For: 2.20.0 > > Time Spent: 3h 20m > Remaining Estimate: 0h > > We need to raise an error here > https://github.com/apache/beam/blob/02cb8d807314a38542c9894b19483e4333d8223b/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py#L857 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9465) Reshuffle should trigger repeatedly
[ https://issues.apache.org/jira/browse/BEAM-9465?focusedWorklogId=404321&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-404321 ] ASF GitHub Bot logged work on BEAM-9465: Author: ASF GitHub Bot Created on: 16/Mar/20 23:08 Start Date: 16/Mar/20 23:08 Worklog Time Spent: 10m Work Description: angoenka commented on pull request #11071: [cherry-pick][BEAM-9465] Fire repeatedly in reshuffle URL: https://github.com/apache/beam/pull/11071 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 404321) Time Spent: 3h 50m (was: 3h 40m) > Reshuffle should trigger repeatedly > --- > > Key: BEAM-9465 > URL: https://issues.apache.org/jira/browse/BEAM-9465 > Project: Beam > Issue Type: Bug > Components: sdk-py-core >Reporter: Ankur Goenka >Assignee: Ankur Goenka >Priority: Major > Fix For: 2.20.0 > > Time Spent: 3h 50m > Remaining Estimate: 0h > > [https://github.com/apache/beam/blob/403a08f8b95d13e5381a22c1c032ad22c8848650/sdks/python/apache_beam/transforms/trigger.py#L516|https://www.google.com/url?q=https://github.com/apache/beam/blob/403a08f8b95d13e5381a22c1c032ad22c8848650/sdks/python/apache_beam/transforms/trigger.py%23L516&sa=D] > should fire repeatedly -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9465) Reshuffle should trigger repeatedly
[ https://issues.apache.org/jira/browse/BEAM-9465?focusedWorklogId=404320&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-404320 ] ASF GitHub Bot logged work on BEAM-9465: Author: ASF GitHub Bot Created on: 16/Mar/20 23:08 Start Date: 16/Mar/20 23:08 Worklog Time Spent: 10m Work Description: angoenka commented on issue #11071: [cherry-pick][BEAM-9465] Fire repeatedly in reshuffle URL: https://github.com/apache/beam/pull/11071#issuecomment-599797804 already merged in 2.20 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 404320) Time Spent: 3h 40m (was: 3.5h) > Reshuffle should trigger repeatedly > --- > > Key: BEAM-9465 > URL: https://issues.apache.org/jira/browse/BEAM-9465 > Project: Beam > Issue Type: Bug > Components: sdk-py-core >Reporter: Ankur Goenka >Assignee: Ankur Goenka >Priority: Major > Fix For: 2.20.0 > > Time Spent: 3h 40m > Remaining Estimate: 0h > > [https://github.com/apache/beam/blob/403a08f8b95d13e5381a22c1c032ad22c8848650/sdks/python/apache_beam/transforms/trigger.py#L516|https://www.google.com/url?q=https://github.com/apache/beam/blob/403a08f8b95d13e5381a22c1c032ad22c8848650/sdks/python/apache_beam/transforms/trigger.py%23L516&sa=D] > should fire repeatedly -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9485) Dataflow Silently drops Non implemented transform in fnapi mode.
[ https://issues.apache.org/jira/browse/BEAM-9485?focusedWorklogId=404317&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-404317 ] ASF GitHub Bot logged work on BEAM-9485: Author: ASF GitHub Bot Created on: 16/Mar/20 23:04 Start Date: 16/Mar/20 23:04 Worklog Time Spent: 10m Work Description: angoenka commented on issue #1: [cherry-pick][BEAM-9485] Raise error when transform urn is not implemented URL: https://github.com/apache/beam/pull/1#issuecomment-599796689 Successful Python precommit https://builds.apache.org/job/beam_PreCommit_Python_Commit/11740/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 404317) Time Spent: 3h (was: 2h 50m) > Dataflow Silently drops Non implemented transform in fnapi mode. > > > Key: BEAM-9485 > URL: https://issues.apache.org/jira/browse/BEAM-9485 > Project: Beam > Issue Type: Bug > Components: runner-dataflow, sdk-py-core >Reporter: Ankur Goenka >Assignee: Ankur Goenka >Priority: Major > Fix For: 2.20.0 > > Time Spent: 3h > Remaining Estimate: 0h > > We need to raise an error here > https://github.com/apache/beam/blob/02cb8d807314a38542c9894b19483e4333d8223b/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py#L857 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-9496) Add a Dataframe API for Python
[ https://issues.apache.org/jira/browse/BEAM-9496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17060524#comment-17060524 ] Ismaël Mejía commented on BEAM-9496: Awesome (y)! > Add a Dataframe API for Python > -- > > Key: BEAM-9496 > URL: https://issues.apache.org/jira/browse/BEAM-9496 > Project: Beam > Issue Type: New Feature > Components: sdk-py-core >Reporter: Robert Bradshaw >Assignee: Robert Bradshaw >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > This is an umbrella bug for the dataframes work. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-9496) Add a Dataframe API for Python
[ https://issues.apache.org/jira/browse/BEAM-9496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17060523#comment-17060523 ] Robert Bradshaw commented on BEAM-9496: --- Sorry. The doc should be publicly available now. > Add a Dataframe API for Python > -- > > Key: BEAM-9496 > URL: https://issues.apache.org/jira/browse/BEAM-9496 > Project: Beam > Issue Type: New Feature > Components: sdk-py-core >Reporter: Robert Bradshaw >Assignee: Robert Bradshaw >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > This is an umbrella bug for the dataframes work. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (BEAM-9525) KafkaIO attempting to make connection when uploading via Dataflow Runner
Ross Rothenstine created BEAM-9525: -- Summary: KafkaIO attempting to make connection when uploading via Dataflow Runner Key: BEAM-9525 URL: https://issues.apache.org/jira/browse/BEAM-9525 Project: Beam Issue Type: Bug Components: io-java-kafka Reporter: Ross Rothenstine Given our Kafka is running in GCP, within a VPC that is not accessible outside of the VPC. When uploading a pipeline to GCS using the Dataflow Runner via our CI/CD worker, it attempts to make a connection to the broker. Our broker isn't available outside the VPC, so this process fails. We can remedy this by running a pod from within the VPC to upload, but it causes an awkward step within our CI/CD process. If there is a more elegant solution to this, such as KafkaIO taking in a ValueProvider for bootstrap servers. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-7923) Interactive Beam
[ https://issues.apache.org/jira/browse/BEAM-7923?focusedWorklogId=404314&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-404314 ] ASF GitHub Bot logged work on BEAM-7923: Author: ASF GitHub Bot Created on: 16/Mar/20 22:51 Start Date: 16/Mar/20 22:51 Worklog Time Spent: 10m Work Description: KevinGG commented on pull request #11141: [BEAM-7923] Include side effects in p.run URL: https://github.com/apache/beam/pull/11141#discussion_r393352768 ## File path: sdks/python/apache_beam/runners/interactive/pipeline_instrument.py ## @@ -418,10 +420,16 @@ def visit_transform(self, transform_node): tuple(ie.current_env().options.capturable_sources)): unbounded_source_pcolls.update(transform_node.outputs.values()) cacheable_inputs.update(self._pin._cacheable_inputs(transform_node)) +ins, outs = self._pin._all_inputs_outputs(transform_node) +all_inputs.update(ins) +all_outputs.update(outs) v = InstrumentVisitor(self) self._pipeline.visit(v) +# Every output PCollection that is never used as an input PCollection is +# considered as a side effect of the pipeline run and should be included. +self._extended_targets.update(all_outputs.difference(all_inputs)) Review comment: It's not necessary. The intended behavior is not ambiguous: When the user uses `show`, `head`, `collect` APIs, these PCollections are excluded completely as the user explicitly wishes. And when the user invokes `p.run()`, all transforms in the pipeline should be executed as expected. This change is only to make sure that the prune logic doesn't affect the above intended behavior. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 404314) Time Spent: 2h 10m (was: 2h) > Interactive Beam > > > Key: BEAM-7923 > URL: https://issues.apache.org/jira/browse/BEAM-7923 > Project: Beam > Issue Type: New Feature > Components: runner-py-interactive >Reporter: Ning Kang >Assignee: Ning Kang >Priority: Major > Time Spent: 2h 10m > Remaining Estimate: 0h > > This is the top level ticket for all efforts leveraging [interactive > Beam|[https://github.com/apache/beam/tree/master/sdks/python/apache_beam/runners/interactive]] > As the development goes, blocking tickets will be added to this one. > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (BEAM-9485) Dataflow Silently drops Non implemented transform in fnapi mode.
[ https://issues.apache.org/jira/browse/BEAM-9485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Wang resolved BEAM-9485. Resolution: Fixed > Dataflow Silently drops Non implemented transform in fnapi mode. > > > Key: BEAM-9485 > URL: https://issues.apache.org/jira/browse/BEAM-9485 > Project: Beam > Issue Type: Bug > Components: runner-dataflow, sdk-py-core >Reporter: Ankur Goenka >Assignee: Ankur Goenka >Priority: Major > Fix For: 2.20.0 > > Time Spent: 2h 50m > Remaining Estimate: 0h > > We need to raise an error here > https://github.com/apache/beam/blob/02cb8d807314a38542c9894b19483e4333d8223b/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py#L857 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (BEAM-9465) Reshuffle should trigger repeatedly
[ https://issues.apache.org/jira/browse/BEAM-9465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Wang resolved BEAM-9465. Resolution: Fixed > Reshuffle should trigger repeatedly > --- > > Key: BEAM-9465 > URL: https://issues.apache.org/jira/browse/BEAM-9465 > Project: Beam > Issue Type: Bug > Components: sdk-py-core >Reporter: Ankur Goenka >Assignee: Ankur Goenka >Priority: Major > Fix For: 2.20.0 > > Time Spent: 3.5h > Remaining Estimate: 0h > > [https://github.com/apache/beam/blob/403a08f8b95d13e5381a22c1c032ad22c8848650/sdks/python/apache_beam/transforms/trigger.py#L516|https://www.google.com/url?q=https://github.com/apache/beam/blob/403a08f8b95d13e5381a22c1c032ad22c8848650/sdks/python/apache_beam/transforms/trigger.py%23L516&sa=D] > should fire repeatedly -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9503) SyntaxError in process worker startup
[
https://issues.apache.org/jira/browse/BEAM-9503?focusedWorklogId=404310&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-404310
]
ASF GitHub Bot logged work on BEAM-9503:
Author: ASF GitHub Bot
Created on: 16/Mar/20 22:45
Start Date: 16/Mar/20 22:45
Worklog Time Spent: 10m
Work Description: ibzib commented on issue #11123: [BEAM-9503] Insert
missing comma in process worker script.
URL: https://github.com/apache/beam/pull/11123#issuecomment-599791452
Run Python 3.7 PostCommit
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 404310)
Time Spent: 4h 20m (was: 4h 10m)
> SyntaxError in process worker startup
> -
>
> Key: BEAM-9503
> URL: https://issues.apache.org/jira/browse/BEAM-9503
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-harness
>Reporter: Kyle Weaver
>Assignee: Kyle Weaver
>Priority: Major
> Fix For: 2.20.0
>
> Time Spent: 4h 20m
> Remaining Estimate: 0h
>
> WARNING:apache_beam.runners.worker.worker_pool_main:Starting worker with
> command ['python', '-c', 'from apache_beam.runners.worker.sdk_worker import
> SdkHarness;
> SdkHarness("localhost:57103",worker_id="1-1",state_cache_size=0data_buffer_time_limit_ms=0).run()']
> Note that 'state_cache_size=0data_buffer_time_limit_ms=0' is all mashed
> together. Looks like we're missing a comma:
> https://github.com/apache/beam/blob/feefaca793d8358d5386d0725863c03e4e37b5b1/sdks/python/apache_beam/runners/worker/worker_pool_main.py#L116
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-7923) Interactive Beam
[ https://issues.apache.org/jira/browse/BEAM-7923?focusedWorklogId=404311&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-404311 ] ASF GitHub Bot logged work on BEAM-7923: Author: ASF GitHub Bot Created on: 16/Mar/20 22:45 Start Date: 16/Mar/20 22:45 Worklog Time Spent: 10m Work Description: aaltay commented on pull request #11141: [BEAM-7923] Include side effects in p.run URL: https://github.com/apache/beam/pull/11141#discussion_r393351076 ## File path: sdks/python/apache_beam/runners/interactive/pipeline_instrument.py ## @@ -418,10 +420,16 @@ def visit_transform(self, transform_node): tuple(ie.current_env().options.capturable_sources)): unbounded_source_pcolls.update(transform_node.outputs.values()) cacheable_inputs.update(self._pin._cacheable_inputs(transform_node)) +ins, outs = self._pin._all_inputs_outputs(transform_node) +all_inputs.update(ins) +all_outputs.update(outs) v = InstrumentVisitor(self) self._pipeline.visit(v) +# Every output PCollection that is never used as an input PCollection is +# considered as a side effect of the pipeline run and should be included. +self._extended_targets.update(all_outputs.difference(all_inputs)) Review comment: Do you want to track, mark side effects differently? Does users want to specifically track these pcollections? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 404311) Time Spent: 2h (was: 1h 50m) > Interactive Beam > > > Key: BEAM-7923 > URL: https://issues.apache.org/jira/browse/BEAM-7923 > Project: Beam > Issue Type: New Feature > Components: runner-py-interactive >Reporter: Ning Kang >Assignee: Ning Kang >Priority: Major > Time Spent: 2h > Remaining Estimate: 0h > > This is the top level ticket for all efforts leveraging [interactive > Beam|[https://github.com/apache/beam/tree/master/sdks/python/apache_beam/runners/interactive]] > As the development goes, blocking tickets will be added to this one. > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9503) SyntaxError in process worker startup
[
https://issues.apache.org/jira/browse/BEAM-9503?focusedWorklogId=404308&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-404308
]
ASF GitHub Bot logged work on BEAM-9503:
Author: ASF GitHub Bot
Created on: 16/Mar/20 22:45
Start Date: 16/Mar/20 22:45
Worklog Time Spent: 10m
Work Description: ibzib commented on issue #11123: [BEAM-9503] Insert
missing comma in process worker script.
URL: https://github.com/apache/beam/pull/11123#issuecomment-599791417
Run Python 3.5 PostCommit
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 404308)
Time Spent: 4h 10m (was: 4h)
> SyntaxError in process worker startup
> -
>
> Key: BEAM-9503
> URL: https://issues.apache.org/jira/browse/BEAM-9503
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-harness
>Reporter: Kyle Weaver
>Assignee: Kyle Weaver
>Priority: Major
> Fix For: 2.20.0
>
> Time Spent: 4h 10m
> Remaining Estimate: 0h
>
> WARNING:apache_beam.runners.worker.worker_pool_main:Starting worker with
> command ['python', '-c', 'from apache_beam.runners.worker.sdk_worker import
> SdkHarness;
> SdkHarness("localhost:57103",worker_id="1-1",state_cache_size=0data_buffer_time_limit_ms=0).run()']
> Note that 'state_cache_size=0data_buffer_time_limit_ms=0' is all mashed
> together. Looks like we're missing a comma:
> https://github.com/apache/beam/blob/feefaca793d8358d5386d0725863c03e4e37b5b1/sdks/python/apache_beam/runners/worker/worker_pool_main.py#L116
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-7923) Interactive Beam
[ https://issues.apache.org/jira/browse/BEAM-7923?focusedWorklogId=404305&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-404305 ] ASF GitHub Bot logged work on BEAM-7923: Author: ASF GitHub Bot Created on: 16/Mar/20 22:35 Start Date: 16/Mar/20 22:35 Worklog Time Spent: 10m Work Description: KevinGG commented on issue #11141: [BEAM-7923] Include side effects in p.run URL: https://github.com/apache/beam/pull/11141#issuecomment-599788607 Formatted with yapf. Lint passed locally. R: @aaltay R: @davidyan74 R: @rohdesamuel PTAL, thx! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 404305) Time Spent: 1h 50m (was: 1h 40m) > Interactive Beam > > > Key: BEAM-7923 > URL: https://issues.apache.org/jira/browse/BEAM-7923 > Project: Beam > Issue Type: New Feature > Components: runner-py-interactive >Reporter: Ning Kang >Assignee: Ning Kang >Priority: Major > Time Spent: 1h 50m > Remaining Estimate: 0h > > This is the top level ticket for all efforts leveraging [interactive > Beam|[https://github.com/apache/beam/tree/master/sdks/python/apache_beam/runners/interactive]] > As the development goes, blocking tickets will be added to this one. > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-7923) Interactive Beam
[ https://issues.apache.org/jira/browse/BEAM-7923?focusedWorklogId=404304&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-404304 ] ASF GitHub Bot logged work on BEAM-7923: Author: ASF GitHub Bot Created on: 16/Mar/20 22:33 Start Date: 16/Mar/20 22:33 Worklog Time Spent: 10m Work Description: KevinGG commented on pull request #11141: [BEAM-7923] Include side effects in p.run URL: https://github.com/apache/beam/pull/11141#discussion_r393347005 ## File path: sdks/python/apache_beam/runners/interactive/utils_test.py ## @@ -56,7 +56,7 @@ def test_parse_windowedvalue_with_window_info(self): [['a', 2, int(1e6), els[0].windows, els[0].pane_info], ['b', 3, int(1e6), els[1].windows, els[1].pane_info]], columns=[0, 1, 'event_time', 'windows', 'pane_info']) -pd.testing.assert_frame_equal(actual_df, expected_df) +pd.testing.assert_frame_equal(actual_df, expected_df, check_like=True) Review comment: This test was flaky because the dataframe columns can be built in arbitrary orders. This option makes sure it doesn't take column positioning into consideration since we only care about the equivalence of data. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 404304) Time Spent: 1h 40m (was: 1.5h) > Interactive Beam > > > Key: BEAM-7923 > URL: https://issues.apache.org/jira/browse/BEAM-7923 > Project: Beam > Issue Type: New Feature > Components: runner-py-interactive >Reporter: Ning Kang >Assignee: Ning Kang >Priority: Major > Time Spent: 1h 40m > Remaining Estimate: 0h > > This is the top level ticket for all efforts leveraging [interactive > Beam|[https://github.com/apache/beam/tree/master/sdks/python/apache_beam/runners/interactive]] > As the development goes, blocking tickets will be added to this one. > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9503) SyntaxError in process worker startup
[
https://issues.apache.org/jira/browse/BEAM-9503?focusedWorklogId=404301&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-404301
]
ASF GitHub Bot logged work on BEAM-9503:
Author: ASF GitHub Bot
Created on: 16/Mar/20 22:29
Start Date: 16/Mar/20 22:29
Worklog Time Spent: 10m
Work Description: ibzib commented on issue #11124:
[cherry-pick][release-2.20.0][BEAM-9503] Insert missing comma in process worker
script.
URL: https://github.com/apache/beam/pull/11124#issuecomment-599786492
Run Python PreCommit
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 404301)
Time Spent: 4h (was: 3h 50m)
> SyntaxError in process worker startup
> -
>
> Key: BEAM-9503
> URL: https://issues.apache.org/jira/browse/BEAM-9503
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-harness
>Reporter: Kyle Weaver
>Assignee: Kyle Weaver
>Priority: Major
> Fix For: 2.20.0
>
> Time Spent: 4h
> Remaining Estimate: 0h
>
> WARNING:apache_beam.runners.worker.worker_pool_main:Starting worker with
> command ['python', '-c', 'from apache_beam.runners.worker.sdk_worker import
> SdkHarness;
> SdkHarness("localhost:57103",worker_id="1-1",state_cache_size=0data_buffer_time_limit_ms=0).run()']
> Note that 'state_cache_size=0data_buffer_time_limit_ms=0' is all mashed
> together. Looks like we're missing a comma:
> https://github.com/apache/beam/blob/feefaca793d8358d5386d0725863c03e4e37b5b1/sdks/python/apache_beam/runners/worker/worker_pool_main.py#L116
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-7923) Interactive Beam
[ https://issues.apache.org/jira/browse/BEAM-7923?focusedWorklogId=404302&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-404302 ] ASF GitHub Bot logged work on BEAM-7923: Author: ASF GitHub Bot Created on: 16/Mar/20 22:29 Start Date: 16/Mar/20 22:29 Worklog Time Spent: 10m Work Description: KevinGG commented on pull request #11141: [BEAM-7923] Include side effects in p.run URL: https://github.com/apache/beam/pull/11141 1. PCollections never used as inputs and not watched, such as sinks without being assigned to variables will be pruned before `p.run()`. The change makes sure that these side effect PCollections are now considered as extended targets and will be executed on `p.run()`. 2. Note the change will not affect `show`, `head` and `collect` because they have an additional pipeline fragment logic that already prunes everything unrelated before the instrumenting and the prune logic inside instrumenting. **Please** add a meaningful description for your change here Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] Update `CHANGES.md` with noteworthy changes. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier). Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [ ArrayScanToUncollectConverter: ResolvedParameter cannot be cast to ResolvedLiteral
[
https://issues.apache.org/jira/browse/BEAM-9511?focusedWorklogId=404299&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-404299
]
ASF GitHub Bot logged work on BEAM-9511:
Author: ASF GitHub Bot
Created on: 16/Mar/20 22:24
Start Date: 16/Mar/20 22:24
Worklog Time Spent: 10m
Work Description: robinyqiu commented on issue #11133:
[BEAM-9511][BEAM-9515][BEAM-9516] Uncollect takes arbitrary expressions
URL: https://github.com/apache/beam/pull/11133#issuecomment-599784806
Would be great to add the query in
https://issues.apache.org/jira/browse/BEAM-9511 as a unit test.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 404299)
Remaining Estimate: 0h
Time Spent: 10m
> ArrayScanToUncollectConverter: ResolvedParameter cannot be cast to
> ResolvedLiteral
> --
>
> Key: BEAM-9511
> URL: https://issues.apache.org/jira/browse/BEAM-9511
> Project: Beam
> Issue Type: Bug
> Components: dsl-sql-zetasql
>Reporter: Andrew Pilloud
>Assignee: Andrew Pilloud
>Priority: Major
> Labels: zetasql-compliance
> Time Spent: 10m
> Remaining Estimate: 0h
>
>
> {code:java}
> Mar 16, 2020 12:57:42 PM
> cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
> executeQuery
> INFO: Processing Sql statement: SELECT elem FROM UNNEST(@string_arr) AS elem
> Mar 16, 2020 12:57:42 PM
> com.google.zetasql.io.grpc.internal.SerializingExecutor run
> SEVERE: Exception while executing runnable
> com.google.zetasql.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed@2f5b49be
> java.lang.ClassCastException:
> com.google.zetasql.resolvedast.ResolvedNodes$ResolvedParameter cannot be cast
> to com.google.zetasql.resolvedast.ResolvedNodes$ResolvedLiteral
> at
> org.apache.beam.sdk.extensions.sql.zetasql.translation.ArrayScanToUncollectConverter.convert(ArrayScanToUncollectConverter.java:45)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.translation.ArrayScanToUncollectConverter.convert(ArrayScanToUncollectConverter.java:31)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.translation.QueryStatementConverter.convertNode(QueryStatementConverter.java:97)
> at
> java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
> at java.util.Collections$2.tryAdvance(Collections.java:4717)
> at java.util.Collections$2.forEachRemaining(Collections.java:4725)
> at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
> at
> java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
> at
> java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
> at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
> at
> java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.translation.QueryStatementConverter.convertNode(QueryStatementConverter.java:96)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.translation.QueryStatementConverter.convert(QueryStatementConverter.java:84)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.translation.QueryStatementConverter.convertRootQuery(QueryStatementConverter.java:51)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.ZetaSQLPlannerImpl.rel(ZetaSQLPlannerImpl.java:160)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.ZetaSQLQueryPlanner.convertToBeamRelInternal(ZetaSQLQueryPlanner.java:131)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.ZetaSQLQueryPlanner.convertToBeamRel(ZetaSQLQueryPlanner.java:115)
> at
> cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl.executeQuery(ExecuteQueryServiceServer.java:242)
> at
> com.google.zetasql.testing.SqlComplianceServiceGrpc$MethodHandlers.invoke(SqlComplianceServiceGrpc.java:423)
> at
> com.google.zetasql.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171)
> at
> com.google.zetasql.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283)
> at
> com.google.zetasql.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:711)
> at
> com.google.zetasql.io.grpc.internal.ContextRunnable.
[jira] [Work logged] (BEAM-8866) portableWordCount Flink/Spark - flaky post commits
[
https://issues.apache.org/jira/browse/BEAM-8866?focusedWorklogId=404297&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-404297
]
ASF GitHub Bot logged work on BEAM-8866:
Author: ASF GitHub Bot
Created on: 16/Mar/20 22:23
Start Date: 16/Mar/20 22:23
Worklog Time Spent: 10m
Work Description: ibzib commented on issue #11140: [BEAM-8866] Use unique
temp dir for output of portable word count tests.
URL: https://github.com/apache/beam/pull/11140#issuecomment-599784705
Run Python 3.7 PostCommit
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 404297)
Time Spent: 20m (was: 10m)
> portableWordCount Flink/Spark - flaky post commits
> --
>
> Key: BEAM-8866
> URL: https://issues.apache.org/jira/browse/BEAM-8866
> Project: Beam
> Issue Type: Sub-task
> Components: runner-flink
>Reporter: Ahmet Altay
>Assignee: Kyle Weaver
>Priority: Critical
> Labels: portability-flink
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Logs:
> [https://scans.gradle.com/s/rkdiftvzvr7cy/console-log?task=:sdks:python:test-suites:portable:py36:portableWordCountFlinkRunnerStreaming]
> Error:
> ..
> File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python36/src/sdks/python/apache_beam/io/localfilesystem.py",
> line 335, in delete File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python36/src/sdks/python/apache_beam/io/localfilesystem.py",
> line 335, in delete raise BeamIOError("Delete operation failed",
> exceptions) apache_beam.io.filesystem.BeamIOError: Delete operation failed
> with exceptions \{'/tmp/py-wordcount-direct-1-of-2': OSError('No
> files found to delete under: /tmp/py-wordcount-direct-1-of-2',),
> '/tmp/py-wordcount-direct-0-of-2': OSError('No files found to delete
> under: /tmp/py-wordcount-direct-0-of-2',)} During handling of the
> above exception, another exception occurred:
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (BEAM-9510) Dependencies in base_image_requirements.txt are not compatible with each other
[ https://issues.apache.org/jira/browse/BEAM-9510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17060503#comment-17060503 ] Valentyn Tymofieiev commented on BEAM-9510: --- Related: https://issues.apache.org/jira/browse/BEAM-8551 > Dependencies in base_image_requirements.txt are not compatible with each other > -- > > Key: BEAM-9510 > URL: https://issues.apache.org/jira/browse/BEAM-9510 > Project: Beam > Issue Type: Bug > Components: sdk-py-harness >Reporter: David Yan >Assignee: Hannah Jiang >Priority: Major > > [https://github.com/apache/beam/blob/master/sdks/python/container/base_image_requirements.txt#L56] > says it requires google-cloud-bigquery==1.24.0, google-cloud-core==1.0.2, > google-cloud-bigtable==0.32.1, grpc-1.22.0 and tensorflow-2.1.0 > But they are incompatible with each other: > ERROR: google-cloud-bigquery 1.24.0 has requirement > google-cloud-core<2.0dev,>=1.1.0, but you'll have google-cloud-core 1.0.2 > which is incompatible. > ERROR: google-cloud-bigtable 0.32.1 has requirement > google-cloud-core<0.30dev,>=0.29.0, but you'll have google-cloud-core 1.0.2 > which is incompatible. > ERROR: tensorboard 2.1.1 has requirement grpcio>=1.24.3, but you'll have > grpcio 1.22.0 which is incompatible. > ERROR: tensorflow 2.1.0 has requirement scipy==1.4.1; python_version >= "3", > but you'll have scipy 1.2.2 which is incompatible. > > > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-8551) Beam Python containers should include all Beam SDK dependencies, and do not have conflicting dependencies
[ https://issues.apache.org/jira/browse/BEAM-8551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17060501#comment-17060501 ] Valentyn Tymofieiev commented on BEAM-8551: --- cc: [~hannahjiang] > Beam Python containers should include all Beam SDK dependencies, and do not > have conflicting dependencies > - > > Key: BEAM-8551 > URL: https://issues.apache.org/jira/browse/BEAM-8551 > Project: Beam > Issue Type: Bug > Components: sdk-py-core >Reporter: Valentyn Tymofieiev >Priority: Major > > Checks could be introduced during container creation, and be enforced by > ValidatesContainer test suites. We could: > - Check pip output or status code for incompatible dependency errors. > - Remove internet access when installing apache-beam in the container, to > makes sure all dependencies are installed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9524) ib.show() spins forever when cells are re-executed
[ https://issues.apache.org/jira/browse/BEAM-9524?focusedWorklogId=404290&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-404290 ] ASF GitHub Bot logged work on BEAM-9524: Author: ASF GitHub Bot Created on: 16/Mar/20 22:12 Start Date: 16/Mar/20 22:12 Worklog Time Spent: 10m Work Description: rohdesamuel commented on issue #11128: [BEAM-9524] Fix for ib.show() executing indefinitely URL: https://github.com/apache/beam/pull/11128#issuecomment-599781031 R: @pabloem This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 404290) Remaining Estimate: 0h Time Spent: 10m > ib.show() spins forever when cells are re-executed > -- > > Key: BEAM-9524 > URL: https://issues.apache.org/jira/browse/BEAM-9524 > Project: Beam > Issue Type: Bug > Components: runner-py-interactive >Reporter: Sam Rohde >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8070) Support empty array literal
[ https://issues.apache.org/jira/browse/BEAM-8070?focusedWorklogId=404278&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-404278 ] ASF GitHub Bot logged work on BEAM-8070: Author: ASF GitHub Bot Created on: 16/Mar/20 22:02 Start Date: 16/Mar/20 22:02 Worklog Time Spent: 10m Work Description: robinyqiu commented on issue #11130: [BEAM-8070] Preserve type for empty array URL: https://github.com/apache/beam/pull/11130#issuecomment-599777966 LGTM. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 404278) Time Spent: 1h 10m (was: 1h) > Support empty array literal > --- > > Key: BEAM-8070 > URL: https://issues.apache.org/jira/browse/BEAM-8070 > Project: Beam > Issue Type: Sub-task > Components: dsl-sql-zetasql >Reporter: Rui Wang >Assignee: Andrew Pilloud >Priority: Major > Labels: zetasql-compliance > Time Spent: 1h 10m > Remaining Estimate: 0h > > Currently BeamSQL throws an IndexOutOfBoundsException when given a query with > an empty array literal. This happens because Calcite attempts to infer the > element types [1,2] from an empty element list. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8070) Support empty array literal
[
https://issues.apache.org/jira/browse/BEAM-8070?focusedWorklogId=404277&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-404277
]
ASF GitHub Bot logged work on BEAM-8070:
Author: ASF GitHub Bot
Created on: 16/Mar/20 22:01
Start Date: 16/Mar/20 22:01
Worklog Time Spent: 10m
Work Description: apilloud commented on pull request #11130: [BEAM-8070]
Preserve type for empty array
URL: https://github.com/apache/beam/pull/11130#discussion_r393334075
##
File path:
sdks/java/extensions/sql/zetasql/src/test/java/org/apache/beam/sdk/extensions/sql/zetasql/ZetaSQLDialectSpecTest.java
##
@@ -658,7 +658,6 @@ public void testIfNullNegative() {
}
@Test
- @Ignore("Throws IndexOutOfBoundsException")
public void testConstructEmptyArrayLiteral() {
Review comment:
Done.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 404277)
Time Spent: 1h (was: 50m)
> Support empty array literal
> ---
>
> Key: BEAM-8070
> URL: https://issues.apache.org/jira/browse/BEAM-8070
> Project: Beam
> Issue Type: Sub-task
> Components: dsl-sql-zetasql
>Reporter: Rui Wang
>Assignee: Andrew Pilloud
>Priority: Major
> Labels: zetasql-compliance
> Time Spent: 1h
> Remaining Estimate: 0h
>
> Currently BeamSQL throws an IndexOutOfBoundsException when given a query with
> an empty array literal. This happens because Calcite attempts to infer the
> element types [1,2] from an empty element list.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (BEAM-9524) ib.show() spins forever when cells are re-executed
Sam Rohde created BEAM-9524: --- Summary: ib.show() spins forever when cells are re-executed Key: BEAM-9524 URL: https://issues.apache.org/jira/browse/BEAM-9524 Project: Beam Issue Type: Bug Components: runner-py-interactive Reporter: Sam Rohde -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-9496) Add a Dataframe API for Python
[ https://issues.apache.org/jira/browse/BEAM-9496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17060491#comment-17060491 ] Ismaël Mejía commented on BEAM-9496: Great, seems the doc is not public yet. cc: [~chadrik] who I suppose could be interested on this and the typings for it :) > Add a Dataframe API for Python > -- > > Key: BEAM-9496 > URL: https://issues.apache.org/jira/browse/BEAM-9496 > Project: Beam > Issue Type: New Feature > Components: sdk-py-core >Reporter: Robert Bradshaw >Assignee: Robert Bradshaw >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > This is an umbrella bug for the dataframes work. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8070) Support empty array literal
[
https://issues.apache.org/jira/browse/BEAM-8070?focusedWorklogId=404275&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-404275
]
ASF GitHub Bot logged work on BEAM-8070:
Author: ASF GitHub Bot
Created on: 16/Mar/20 21:54
Start Date: 16/Mar/20 21:54
Worklog Time Spent: 10m
Work Description: robinyqiu commented on pull request #11130: [BEAM-8070]
Preserve type for empty array
URL: https://github.com/apache/beam/pull/11130#discussion_r393330866
##
File path:
sdks/java/extensions/sql/zetasql/src/test/java/org/apache/beam/sdk/extensions/sql/zetasql/ZetaSQLDialectSpecTest.java
##
@@ -658,7 +658,6 @@ public void testIfNullNegative() {
}
@Test
- @Ignore("Throws IndexOutOfBoundsException")
public void testConstructEmptyArrayLiteral() {
Review comment:
Since a similar new test is added, can we rename this one to
`testEmptyArrayParameter`? That makes more sense to me.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 404275)
Time Spent: 50m (was: 40m)
> Support empty array literal
> ---
>
> Key: BEAM-8070
> URL: https://issues.apache.org/jira/browse/BEAM-8070
> Project: Beam
> Issue Type: Sub-task
> Components: dsl-sql-zetasql
>Reporter: Rui Wang
>Assignee: Andrew Pilloud
>Priority: Major
> Labels: zetasql-compliance
> Time Spent: 50m
> Remaining Estimate: 0h
>
> Currently BeamSQL throws an IndexOutOfBoundsException when given a query with
> an empty array literal. This happens because Calcite attempts to infer the
> element types [1,2] from an empty element list.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-8070) Support empty array literal
[ https://issues.apache.org/jira/browse/BEAM-8070?focusedWorklogId=404269&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-404269 ] ASF GitHub Bot logged work on BEAM-8070: Author: ASF GitHub Bot Created on: 16/Mar/20 21:35 Start Date: 16/Mar/20 21:35 Worklog Time Spent: 10m Work Description: apilloud commented on issue #11130: [BEAM-8070] Preserve type for empty array URL: https://github.com/apache/beam/pull/11130#issuecomment-599768814 Done. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 404269) Time Spent: 40m (was: 0.5h) > Support empty array literal > --- > > Key: BEAM-8070 > URL: https://issues.apache.org/jira/browse/BEAM-8070 > Project: Beam > Issue Type: Sub-task > Components: dsl-sql-zetasql >Reporter: Rui Wang >Assignee: Andrew Pilloud >Priority: Major > Labels: zetasql-compliance > Time Spent: 40m > Remaining Estimate: 0h > > Currently BeamSQL throws an IndexOutOfBoundsException when given a query with > an empty array literal. This happens because Calcite attempts to infer the > element types [1,2] from an empty element list. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-2572) Implement an S3 filesystem for Python SDK
[ https://issues.apache.org/jira/browse/BEAM-2572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17060475#comment-17060475 ] Badrul Chowdhury commented on BEAM-2572: [~pabloem], the table for built-in IO transforms needs to be updated: [https://beam.apache.org/documentation/io/built-in/] > Implement an S3 filesystem for Python SDK > - > > Key: BEAM-2572 > URL: https://issues.apache.org/jira/browse/BEAM-2572 > Project: Beam > Issue Type: Task > Components: sdk-py-core >Reporter: Dmitry Demeshchuk >Priority: Minor > Labels: GSoC2019, gsoc, gsoc2019, mentor, outreachy19dec > Fix For: 2.19.0 > > Time Spent: 5h 10m > Remaining Estimate: 0h > > There are two paths worth exploring, to my understanding: > 1. Sticking to the HDFS-based approach (like it's done in Java). > 2. Using boto/boto3 for accessing S3 through its common API endpoints. > I personally prefer the second approach, for a few reasons: > 1. In real life, HDFS and S3 have different consistency guarantees, therefore > their behaviors may contradict each other in some edge cases (say, we write > something to S3, but it's not immediately accessible for reading from another > end). > 2. There are other AWS-based sources and sinks we may want to create in the > future: DynamoDB, Kinesis, SQS, etc. > 3. boto3 already provides somewhat good logic for basic things like > reattempting. > Whatever path we choose, there's another problem related to this: we > currently cannot pass any global settings (say, pipeline options, or just an > arbitrary kwarg) to a filesystem. Because of that, we'd have to setup the > runner nodes to have AWS keys set up in the environment, which is not trivial > to achieve and doesn't look too clean either (I'd rather see one single place > for configuring the runner options). > Also, it's worth mentioning that I already have a janky S3 filesystem > implementation that only supports DirectRunner at the moment (because of the > previous paragraph). I'm perfectly fine finishing it myself, with some > guidance from the maintainers. > Where should I move on from here, and whose input should I be looking for? > Thanks! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9468) Add Google Cloud Healthcare API IO Connectors
[ https://issues.apache.org/jira/browse/BEAM-9468?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismaël Mejía updated BEAM-9468: --- Status: Open (was: Triage Needed) > Add Google Cloud Healthcare API IO Connectors > - > > Key: BEAM-9468 > URL: https://issues.apache.org/jira/browse/BEAM-9468 > Project: Beam > Issue Type: New Feature > Components: io-java-gcp >Reporter: Jacob Ferriero >Assignee: Jacob Ferriero >Priority: Minor > Time Spent: 1h 20m > Remaining Estimate: 0h > > Add IO Transforms for the HL7v2, FHIR and DICOM stores in the [Google Cloud > Healthcare API|https://cloud.google.com/healthcare/docs/] > HL7v2IO > FHIRIO > DICOM -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9506) _CustomBigQuerySource value provider parameter gcs_location is wrongly evaluated
[ https://issues.apache.org/jira/browse/BEAM-9506?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismaël Mejía updated BEAM-9506: --- Status: Open (was: Triage Needed) > _CustomBigQuerySource value provider parameter gcs_location is wrongly > evaluated > > > Key: BEAM-9506 > URL: https://issues.apache.org/jira/browse/BEAM-9506 > Project: Beam > Issue Type: Bug > Components: io-py-gcp >Affects Versions: 2.19.0 >Reporter: Elias Djurfeldt >Assignee: Kamil Wasilewski >Priority: Minor > > The `gcs_location` parameter is wrongly evaluated at pipeline construction > time, resulting in calling value_provider.get() from a non-runtime context > when using a value provider for the `gcs_location`. > See discussion at > [https://github.com/apache/beam/pull/11040#issuecomment-597872563] > The code in question is at: > [https://github.com/apache/beam/blob/master/sdks/python/apache_beam/io/gcp/bigquery.py#L1575] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (BEAM-9472) Remove excessive logging in python fn_api_runner
[ https://issues.apache.org/jira/browse/BEAM-9472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismaël Mejía resolved BEAM-9472. Resolution: Fixed > Remove excessive logging in python fn_api_runner > > > Key: BEAM-9472 > URL: https://issues.apache.org/jira/browse/BEAM-9472 > Project: Beam > Issue Type: Task > Components: sdk-py-harness >Reporter: Rui Wang >Assignee: Robert Bradshaw >Priority: Major > Fix For: 2.20.0 > > Time Spent: 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9510) Dependencies in base_image_requirements.txt are not compatible with each other
[ https://issues.apache.org/jira/browse/BEAM-9510?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismaël Mejía updated BEAM-9510: --- Status: Open (was: Triage Needed) > Dependencies in base_image_requirements.txt are not compatible with each other > -- > > Key: BEAM-9510 > URL: https://issues.apache.org/jira/browse/BEAM-9510 > Project: Beam > Issue Type: Bug > Components: sdk-py-harness >Reporter: David Yan >Assignee: Hannah Jiang >Priority: Major > > [https://github.com/apache/beam/blob/master/sdks/python/container/base_image_requirements.txt#L56] > says it requires google-cloud-bigquery==1.24.0, google-cloud-core==1.0.2, > google-cloud-bigtable==0.32.1, grpc-1.22.0 and tensorflow-2.1.0 > But they are incompatible with each other: > ERROR: google-cloud-bigquery 1.24.0 has requirement > google-cloud-core<2.0dev,>=1.1.0, but you'll have google-cloud-core 1.0.2 > which is incompatible. > ERROR: google-cloud-bigtable 0.32.1 has requirement > google-cloud-core<0.30dev,>=0.29.0, but you'll have google-cloud-core 1.0.2 > which is incompatible. > ERROR: tensorboard 2.1.1 has requirement grpcio>=1.24.3, but you'll have > grpcio 1.22.0 which is incompatible. > ERROR: tensorflow 2.1.0 has requirement scipy==1.4.1; python_version >= "3", > but you'll have scipy 1.2.2 which is incompatible. > > > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-8866) portableWordCount Flink/Spark - flaky post commits
[
https://issues.apache.org/jira/browse/BEAM-8866?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kyle Weaver updated BEAM-8866:
--
Labels: portability-flink (was: )
> portableWordCount Flink/Spark - flaky post commits
> --
>
> Key: BEAM-8866
> URL: https://issues.apache.org/jira/browse/BEAM-8866
> Project: Beam
> Issue Type: Sub-task
> Components: runner-flink
>Reporter: Ahmet Altay
>Assignee: Kyle Weaver
>Priority: Critical
> Labels: portability-flink
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Logs:
> [https://scans.gradle.com/s/rkdiftvzvr7cy/console-log?task=:sdks:python:test-suites:portable:py36:portableWordCountFlinkRunnerStreaming]
> Error:
> ..
> File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python36/src/sdks/python/apache_beam/io/localfilesystem.py",
> line 335, in delete File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python36/src/sdks/python/apache_beam/io/localfilesystem.py",
> line 335, in delete raise BeamIOError("Delete operation failed",
> exceptions) apache_beam.io.filesystem.BeamIOError: Delete operation failed
> with exceptions \{'/tmp/py-wordcount-direct-1-of-2': OSError('No
> files found to delete under: /tmp/py-wordcount-direct-1-of-2',),
> '/tmp/py-wordcount-direct-0-of-2': OSError('No files found to delete
> under: /tmp/py-wordcount-direct-0-of-2',)} During handling of the
> above exception, another exception occurred:
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (BEAM-9472) Remove excessive logging in python fn_api_runner
[ https://issues.apache.org/jira/browse/BEAM-9472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismaël Mejía updated BEAM-9472: --- Status: Open (was: Triage Needed) > Remove excessive logging in python fn_api_runner > > > Key: BEAM-9472 > URL: https://issues.apache.org/jira/browse/BEAM-9472 > Project: Beam > Issue Type: Task > Components: sdk-py-harness >Reporter: Rui Wang >Assignee: Robert Bradshaw >Priority: Major > Fix For: 2.20.0 > > Time Spent: 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-8070) Support empty array literal
[ https://issues.apache.org/jira/browse/BEAM-8070?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Pilloud updated BEAM-8070: - Labels: zetasql-compliance (was: ) > Support empty array literal > --- > > Key: BEAM-8070 > URL: https://issues.apache.org/jira/browse/BEAM-8070 > Project: Beam > Issue Type: Sub-task > Components: dsl-sql-zetasql >Reporter: Rui Wang >Assignee: Andrew Pilloud >Priority: Major > Labels: zetasql-compliance > Time Spent: 0.5h > Remaining Estimate: 0h > > Currently BeamSQL throws an IndexOutOfBoundsException when given a query with > an empty array literal. This happens because Calcite attempts to infer the > element types [1,2] from an empty element list. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8866) portableWordCount Flink/Spark - flaky post commits
[ https://issues.apache.org/jira/browse/BEAM-8866?focusedWorklogId=404257&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-404257 ] ASF GitHub Bot logged work on BEAM-8866: Author: ASF GitHub Bot Created on: 16/Mar/20 21:11 Start Date: 16/Mar/20 21:11 Worklog Time Spent: 10m Work Description: ibzib commented on pull request #11140: [BEAM-8866] Use unique temp dir for output of portable word count tests. URL: https://github.com/apache/beam/pull/11140 **Please** add a meaningful description for your change here Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] Update `CHANGES.md` with noteworthy changes. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier). Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_J
[jira] [Work logged] (BEAM-9509) Subprocess job server treats missing local file as remote URL
[ https://issues.apache.org/jira/browse/BEAM-9509?focusedWorklogId=404256&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-404256 ] ASF GitHub Bot logged work on BEAM-9509: Author: ASF GitHub Bot Created on: 16/Mar/20 21:06 Start Date: 16/Mar/20 21:06 Worklog Time Spent: 10m Work Description: ibzib commented on pull request #11139: [BEAM-9509] Improve error message for bad job server URL. URL: https://github.com/apache/beam/pull/11139 **Please** add a meaningful description for your change here Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] Update `CHANGES.md` with noteworthy changes. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier). Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRu
[jira] [Assigned] (BEAM-9511) ArrayScanToUncollectConverter: ResolvedParameter cannot be cast to ResolvedLiteral
[
https://issues.apache.org/jira/browse/BEAM-9511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Pilloud reassigned BEAM-9511:
Assignee: Andrew Pilloud
> ArrayScanToUncollectConverter: ResolvedParameter cannot be cast to
> ResolvedLiteral
> --
>
> Key: BEAM-9511
> URL: https://issues.apache.org/jira/browse/BEAM-9511
> Project: Beam
> Issue Type: Bug
> Components: dsl-sql-zetasql
>Reporter: Andrew Pilloud
>Assignee: Andrew Pilloud
>Priority: Major
> Labels: zetasql-compliance
>
>
> {code:java}
> Mar 16, 2020 12:57:42 PM
> cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
> executeQuery
> INFO: Processing Sql statement: SELECT elem FROM UNNEST(@string_arr) AS elem
> Mar 16, 2020 12:57:42 PM
> com.google.zetasql.io.grpc.internal.SerializingExecutor run
> SEVERE: Exception while executing runnable
> com.google.zetasql.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed@2f5b49be
> java.lang.ClassCastException:
> com.google.zetasql.resolvedast.ResolvedNodes$ResolvedParameter cannot be cast
> to com.google.zetasql.resolvedast.ResolvedNodes$ResolvedLiteral
> at
> org.apache.beam.sdk.extensions.sql.zetasql.translation.ArrayScanToUncollectConverter.convert(ArrayScanToUncollectConverter.java:45)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.translation.ArrayScanToUncollectConverter.convert(ArrayScanToUncollectConverter.java:31)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.translation.QueryStatementConverter.convertNode(QueryStatementConverter.java:97)
> at
> java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
> at java.util.Collections$2.tryAdvance(Collections.java:4717)
> at java.util.Collections$2.forEachRemaining(Collections.java:4725)
> at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
> at
> java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
> at
> java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
> at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
> at
> java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.translation.QueryStatementConverter.convertNode(QueryStatementConverter.java:96)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.translation.QueryStatementConverter.convert(QueryStatementConverter.java:84)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.translation.QueryStatementConverter.convertRootQuery(QueryStatementConverter.java:51)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.ZetaSQLPlannerImpl.rel(ZetaSQLPlannerImpl.java:160)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.ZetaSQLQueryPlanner.convertToBeamRelInternal(ZetaSQLQueryPlanner.java:131)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.ZetaSQLQueryPlanner.convertToBeamRel(ZetaSQLQueryPlanner.java:115)
> at
> cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl.executeQuery(ExecuteQueryServiceServer.java:242)
> at
> com.google.zetasql.testing.SqlComplianceServiceGrpc$MethodHandlers.invoke(SqlComplianceServiceGrpc.java:423)
> at
> com.google.zetasql.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171)
> at
> com.google.zetasql.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283)
> at
> com.google.zetasql.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:711)
> at
> com.google.zetasql.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
> at
> com.google.zetasql.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748) {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (BEAM-9516) ArrayScanToUncollectConverter: ResolvedSubqueryExpr cannot be cast to ResolvedLiteral
[
https://issues.apache.org/jira/browse/BEAM-9516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Pilloud reassigned BEAM-9516:
Assignee: Andrew Pilloud
> ArrayScanToUncollectConverter: ResolvedSubqueryExpr cannot be cast to
> ResolvedLiteral
> -
>
> Key: BEAM-9516
> URL: https://issues.apache.org/jira/browse/BEAM-9516
> Project: Beam
> Issue Type: Bug
> Components: dsl-sql-zetasql
>Reporter: Andrew Pilloud
>Assignee: Andrew Pilloud
>Priority: Major
> Labels: zetasql-compliance
>
> {code:java}
> Mar 16, 2020 12:58:49 PM
> com.google.zetasql.io.grpc.internal.SerializingExecutor run
> SEVERE: Exception while executing runnable
> com.google.zetasql.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed@7c1d4ea6
> java.lang.ClassCastException:
> com.google.zetasql.resolvedast.ResolvedNodes$ResolvedSubqueryExpr cannot be
> cast to com.google.zetasql.resolvedast.ResolvedNodes$ResolvedLiteral
> at
> org.apache.beam.sdk.extensions.sql.zetasql.translation.ArrayScanToUncollectConverter.convert(ArrayScanToUncollectConverter.java:45)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.translation.ArrayScanToUncollectConverter.convert(ArrayScanToUncollectConverter.java:31)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.translation.QueryStatementConverter.convertNode(QueryStatementConverter.java:97)
> at
> java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
> at java.util.Collections$2.tryAdvance(Collections.java:4717)
> at java.util.Collections$2.forEachRemaining(Collections.java:4725)
> at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
> at
> java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
> at
> java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
> at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
> at
> java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.translation.QueryStatementConverter.convertNode(QueryStatementConverter.java:96)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.translation.QueryStatementConverter.convert(QueryStatementConverter.java:84)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.translation.QueryStatementConverter.convertRootQuery(QueryStatementConverter.java:51)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.ZetaSQLPlannerImpl.rel(ZetaSQLPlannerImpl.java:160)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.ZetaSQLQueryPlanner.convertToBeamRelInternal(ZetaSQLQueryPlanner.java:131)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.ZetaSQLQueryPlanner.convertToBeamRel(ZetaSQLQueryPlanner.java:115)
> at
> cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl.executeQuery(ExecuteQueryServiceServer.java:242)
> at
> com.google.zetasql.testing.SqlComplianceServiceGrpc$MethodHandlers.invoke(SqlComplianceServiceGrpc.java:423)
> at
> com.google.zetasql.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171)
> at
> com.google.zetasql.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283)
> at
> com.google.zetasql.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:711)
> at
> com.google.zetasql.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
> at
> com.google.zetasql.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9503) SyntaxError in process worker startup
[
https://issues.apache.org/jira/browse/BEAM-9503?focusedWorklogId=404253&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-404253
]
ASF GitHub Bot logged work on BEAM-9503:
Author: ASF GitHub Bot
Created on: 16/Mar/20 20:57
Start Date: 16/Mar/20 20:57
Worklog Time Spent: 10m
Work Description: amaliujia commented on issue #11124:
[cherry-pick][release-2.20.0][BEAM-9503] Insert missing comma in process worker
script.
URL: https://github.com/apache/beam/pull/11124#issuecomment-599754982
Run Python PreCommit
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 404253)
Time Spent: 3h 50m (was: 3h 40m)
> SyntaxError in process worker startup
> -
>
> Key: BEAM-9503
> URL: https://issues.apache.org/jira/browse/BEAM-9503
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-harness
>Reporter: Kyle Weaver
>Assignee: Kyle Weaver
>Priority: Major
> Fix For: 2.20.0
>
> Time Spent: 3h 50m
> Remaining Estimate: 0h
>
> WARNING:apache_beam.runners.worker.worker_pool_main:Starting worker with
> command ['python', '-c', 'from apache_beam.runners.worker.sdk_worker import
> SdkHarness;
> SdkHarness("localhost:57103",worker_id="1-1",state_cache_size=0data_buffer_time_limit_ms=0).run()']
> Note that 'state_cache_size=0data_buffer_time_limit_ms=0' is all mashed
> together. Looks like we're missing a comma:
> https://github.com/apache/beam/blob/feefaca793d8358d5386d0725863c03e4e37b5b1/sdks/python/apache_beam/runners/worker/worker_pool_main.py#L116
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (BEAM-9515) ResolvedSubqueryExpr can not be cast to ResolvedLiteral
[
https://issues.apache.org/jira/browse/BEAM-9515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Pilloud reassigned BEAM-9515:
Assignee: Andrew Pilloud
> ResolvedSubqueryExpr can not be cast to ResolvedLiteral
> ---
>
> Key: BEAM-9515
> URL: https://issues.apache.org/jira/browse/BEAM-9515
> Project: Beam
> Issue Type: Bug
> Components: dsl-sql-zetasql
>Reporter: Andrew Pilloud
>Assignee: Andrew Pilloud
>Priority: Major
> Labels: zetasql-compliance
>
> {code:java}
> Mar 16, 2020 1:00:02 PM
> cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
> executeQuery
> INFO: Processing Sql statement: SELECT * FROM UNNEST(ARRAY(
> SELECT bool_val FROM AllTypesTable t
> ORDER BY bool_val ASC
> )) x WITH OFFSET POS
> Mar 16, 2020 1:00:02 PM
> com.google.zetasql.io.grpc.internal.SerializingExecutor run
> SEVERE: Exception while executing runnable
> com.google.zetasql.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed@7b42f9e7
> java.lang.ClassCastException:
> com.google.zetasql.resolvedast.ResolvedNodes$ResolvedSubqueryExpr cannot be
> cast to com.google.zetasql.resolvedast.ResolvedNodes$ResolvedLiteral
> at
> org.apache.beam.sdk.extensions.sql.zetasql.translation.ArrayScanToUncollectConverter.convert(ArrayScanToUncollectConverter.java:45)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.translation.ArrayScanToUncollectConverter.convert(ArrayScanToUncollectConverter.java:31)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.translation.QueryStatementConverter.convertNode(QueryStatementConverter.java:97)
> at
> java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
> at java.util.Collections$2.tryAdvance(Collections.java:4717)
> at java.util.Collections$2.forEachRemaining(Collections.java:4725)
> at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
> at
> java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
> at
> java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
> at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
> at
> java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.translation.QueryStatementConverter.convertNode(QueryStatementConverter.java:96)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.translation.QueryStatementConverter.convert(QueryStatementConverter.java:84)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.translation.QueryStatementConverter.convertRootQuery(QueryStatementConverter.java:51)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.ZetaSQLPlannerImpl.rel(ZetaSQLPlannerImpl.java:160)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.ZetaSQLQueryPlanner.convertToBeamRelInternal(ZetaSQLQueryPlanner.java:131)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.ZetaSQLQueryPlanner.convertToBeamRel(ZetaSQLQueryPlanner.java:115)
> at
> cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl.executeQuery(ExecuteQueryServiceServer.java:242)
> at
> com.google.zetasql.testing.SqlComplianceServiceGrpc$MethodHandlers.invoke(SqlComplianceServiceGrpc.java:423)
> at
> com.google.zetasql.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171)
> at
> com.google.zetasql.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283)
> at
> com.google.zetasql.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:711)
> at
> com.google.zetasql.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
> at
> com.google.zetasql.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (BEAM-9513) NullPointerException in convertRexNodeFromResolvedExprWithRefScan
[
https://issues.apache.org/jira/browse/BEAM-9513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Pilloud reassigned BEAM-9513:
Assignee: Andrew Pilloud
> NullPointerException in convertRexNodeFromResolvedExprWithRefScan
> -
>
> Key: BEAM-9513
> URL: https://issues.apache.org/jira/browse/BEAM-9513
> Project: Beam
> Issue Type: Bug
> Components: dsl-sql-zetasql
>Reporter: Andrew Pilloud
>Assignee: Andrew Pilloud
>Priority: Major
> Labels: zetasql-compliance
>
> {code:java}
> Mar 16, 2020 12:58:26 PM
> cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
> executeQuery
> INFO: Processing Sql statement: WITH
> R AS (SELECT 2 a),
> S AS (SELECT a-1 x, a+1 y FROM R)
> SELECT a, x, y FROM R, S
> Mar 16, 2020 12:58:26 PM
> com.google.zetasql.io.grpc.internal.SerializingExecutor run
> SEVERE: Exception while executing runnable
> com.google.zetasql.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed@5341cc1c
> java.lang.NullPointerException
> at
> org.apache.beam.sdk.extensions.sql.zetasql.translation.ExpressionConverter.convertRexNodeFromResolvedExprWithRefScan(ExpressionConverter.java:374)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.translation.JoinScanWithRefConverter.convert(JoinScanWithRefConverter.java:63)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.translation.JoinScanWithRefConverter.convert(JoinScanWithRefConverter.java:35)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.translation.QueryStatementConverter.convertNode(QueryStatementConverter.java:97)
> at
> java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
> at java.util.Collections$2.tryAdvance(Collections.java:4717)
> at java.util.Collections$2.forEachRemaining(Collections.java:4725)
> at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
> at
> java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
> at
> java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
> at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
> at
> java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.translation.QueryStatementConverter.convertNode(QueryStatementConverter.java:96)
> at
> java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
> at java.util.Collections$2.tryAdvance(Collections.java:4717)
> at java.util.Collections$2.forEachRemaining(Collections.java:4725)
> at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
> at
> java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
> at
> java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
> at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
> at
> java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.translation.QueryStatementConverter.convertNode(QueryStatementConverter.java:96)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.translation.QueryStatementConverter.convert(QueryStatementConverter.java:84)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.translation.QueryStatementConverter.convertRootQuery(QueryStatementConverter.java:51)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.ZetaSQLPlannerImpl.rel(ZetaSQLPlannerImpl.java:160)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.ZetaSQLQueryPlanner.convertToBeamRelInternal(ZetaSQLQueryPlanner.java:131)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.ZetaSQLQueryPlanner.convertToBeamRel(ZetaSQLQueryPlanner.java:115)
> at
> cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl.executeQuery(ExecuteQueryServiceServer.java:242)
> at
> com.google.zetasql.testing.SqlComplianceServiceGrpc$MethodHandlers.invoke(SqlComplianceServiceGrpc.java:423)
> at
> com.google.zetasql.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171)
> at
> com.google.zetasql.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283)
> at
> com.google.zetasql.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:711)
> at
> com.google.zetasql.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
> at
> com.google.zetasql.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker
[jira] [Assigned] (BEAM-9521) NullPointerException in convertRexNodeFromResolvedExpr
[
https://issues.apache.org/jira/browse/BEAM-9521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Pilloud reassigned BEAM-9521:
Assignee: Andrew Pilloud
> NullPointerException in convertRexNodeFromResolvedExpr
> --
>
> Key: BEAM-9521
> URL: https://issues.apache.org/jira/browse/BEAM-9521
> Project: Beam
> Issue Type: Bug
> Components: dsl-sql-zetasql
>Reporter: Andrew Pilloud
>Assignee: Andrew Pilloud
>Priority: Major
> Labels: zetasql-compliance
>
> {code:java}
> Mar 16, 2020 12:57:10 PM
> cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
> executeQuery
> INFO: Processing Sql statement: SELECT (SELECT AS STRUCT
> (SELECT AS STRUCT l.float_val lf, r.float_val rf),
> l.float_val = r.float_val),
>(SELECT AS STRUCT
> (SELECT AS STRUCT l.double_val ld, r.double_val rd),
> l.double_val = r.double_val)
> FROM TestTableZero l, TestTableZero r
> WHERE l.row_id > r.row_id
> Mar 16, 2020 12:57:10 PM
> com.google.zetasql.io.grpc.internal.SerializingExecutor run
> SEVERE: Exception while executing runnable
> com.google.zetasql.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed@710f8c0d
> java.lang.NullPointerException
> at
> org.apache.beam.sdk.extensions.sql.zetasql.translation.ExpressionConverter.convertRexNodeFromResolvedExpr(ExpressionConverter.java:301)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.translation.JoinScanConverter.convert(JoinScanConverter.java:83)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.translation.JoinScanConverter.convert(JoinScanConverter.java:36)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.translation.QueryStatementConverter.convertNode(QueryStatementConverter.java:97)
> at
> java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
> at java.util.Collections$2.tryAdvance(Collections.java:4717)
> at java.util.Collections$2.forEachRemaining(Collections.java:4725)
> at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
> at
> java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
> at
> java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
> at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
> at
> java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.translation.QueryStatementConverter.convertNode(QueryStatementConverter.java:96)
> at
> java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
> at java.util.Collections$2.tryAdvance(Collections.java:4717)
> at java.util.Collections$2.forEachRemaining(Collections.java:4725)
> at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
> at
> java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
> at
> java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
> at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
> at
> java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.translation.QueryStatementConverter.convertNode(QueryStatementConverter.java:96)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.translation.QueryStatementConverter.convert(QueryStatementConverter.java:84)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.translation.QueryStatementConverter.convertRootQuery(QueryStatementConverter.java:51)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.ZetaSQLPlannerImpl.rel(ZetaSQLPlannerImpl.java:160)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.ZetaSQLQueryPlanner.convertToBeamRelInternal(ZetaSQLQueryPlanner.java:131)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.ZetaSQLQueryPlanner.convertToBeamRel(ZetaSQLQueryPlanner.java:115)
> at
> cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl.executeQuery(ExecuteQueryServiceServer.java:242)
> at
> com.google.zetasql.testing.SqlComplianceServiceGrpc$MethodHandlers.invoke(SqlComplianceServiceGrpc.java:423)
> at
> com.google.zetasql.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171)
> at
> com.google.zetasql.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283)
> at
> com.google.zetasql.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:711)
> at
> com.google.zetasql.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
[jira] [Updated] (BEAM-9511) ArrayScanToUncollectConverter: ResolvedParameter cannot be cast to ResolvedLiteral
[
https://issues.apache.org/jira/browse/BEAM-9511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Pilloud updated BEAM-9511:
-
Status: Open (was: Triage Needed)
> ArrayScanToUncollectConverter: ResolvedParameter cannot be cast to
> ResolvedLiteral
> --
>
> Key: BEAM-9511
> URL: https://issues.apache.org/jira/browse/BEAM-9511
> Project: Beam
> Issue Type: Bug
> Components: dsl-sql-zetasql
>Reporter: Andrew Pilloud
>Priority: Major
> Labels: zetasql-compliance
>
>
> {code:java}
> Mar 16, 2020 12:57:42 PM
> cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
> executeQuery
> INFO: Processing Sql statement: SELECT elem FROM UNNEST(@string_arr) AS elem
> Mar 16, 2020 12:57:42 PM
> com.google.zetasql.io.grpc.internal.SerializingExecutor run
> SEVERE: Exception while executing runnable
> com.google.zetasql.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed@2f5b49be
> java.lang.ClassCastException:
> com.google.zetasql.resolvedast.ResolvedNodes$ResolvedParameter cannot be cast
> to com.google.zetasql.resolvedast.ResolvedNodes$ResolvedLiteral
> at
> org.apache.beam.sdk.extensions.sql.zetasql.translation.ArrayScanToUncollectConverter.convert(ArrayScanToUncollectConverter.java:45)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.translation.ArrayScanToUncollectConverter.convert(ArrayScanToUncollectConverter.java:31)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.translation.QueryStatementConverter.convertNode(QueryStatementConverter.java:97)
> at
> java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
> at java.util.Collections$2.tryAdvance(Collections.java:4717)
> at java.util.Collections$2.forEachRemaining(Collections.java:4725)
> at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
> at
> java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
> at
> java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
> at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
> at
> java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.translation.QueryStatementConverter.convertNode(QueryStatementConverter.java:96)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.translation.QueryStatementConverter.convert(QueryStatementConverter.java:84)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.translation.QueryStatementConverter.convertRootQuery(QueryStatementConverter.java:51)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.ZetaSQLPlannerImpl.rel(ZetaSQLPlannerImpl.java:160)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.ZetaSQLQueryPlanner.convertToBeamRelInternal(ZetaSQLQueryPlanner.java:131)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.ZetaSQLQueryPlanner.convertToBeamRel(ZetaSQLQueryPlanner.java:115)
> at
> cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl.executeQuery(ExecuteQueryServiceServer.java:242)
> at
> com.google.zetasql.testing.SqlComplianceServiceGrpc$MethodHandlers.invoke(SqlComplianceServiceGrpc.java:423)
> at
> com.google.zetasql.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171)
> at
> com.google.zetasql.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283)
> at
> com.google.zetasql.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:711)
> at
> com.google.zetasql.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
> at
> com.google.zetasql.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748) {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (BEAM-9513) NullPointerException in convertRexNodeFromResolvedExprWithRefScan
[
https://issues.apache.org/jira/browse/BEAM-9513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Pilloud updated BEAM-9513:
-
Status: Open (was: Triage Needed)
> NullPointerException in convertRexNodeFromResolvedExprWithRefScan
> -
>
> Key: BEAM-9513
> URL: https://issues.apache.org/jira/browse/BEAM-9513
> Project: Beam
> Issue Type: Bug
> Components: dsl-sql-zetasql
>Reporter: Andrew Pilloud
>Priority: Major
> Labels: zetasql-compliance
>
> {code:java}
> Mar 16, 2020 12:58:26 PM
> cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
> executeQuery
> INFO: Processing Sql statement: WITH
> R AS (SELECT 2 a),
> S AS (SELECT a-1 x, a+1 y FROM R)
> SELECT a, x, y FROM R, S
> Mar 16, 2020 12:58:26 PM
> com.google.zetasql.io.grpc.internal.SerializingExecutor run
> SEVERE: Exception while executing runnable
> com.google.zetasql.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed@5341cc1c
> java.lang.NullPointerException
> at
> org.apache.beam.sdk.extensions.sql.zetasql.translation.ExpressionConverter.convertRexNodeFromResolvedExprWithRefScan(ExpressionConverter.java:374)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.translation.JoinScanWithRefConverter.convert(JoinScanWithRefConverter.java:63)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.translation.JoinScanWithRefConverter.convert(JoinScanWithRefConverter.java:35)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.translation.QueryStatementConverter.convertNode(QueryStatementConverter.java:97)
> at
> java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
> at java.util.Collections$2.tryAdvance(Collections.java:4717)
> at java.util.Collections$2.forEachRemaining(Collections.java:4725)
> at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
> at
> java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
> at
> java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
> at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
> at
> java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.translation.QueryStatementConverter.convertNode(QueryStatementConverter.java:96)
> at
> java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
> at java.util.Collections$2.tryAdvance(Collections.java:4717)
> at java.util.Collections$2.forEachRemaining(Collections.java:4725)
> at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
> at
> java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
> at
> java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
> at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
> at
> java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.translation.QueryStatementConverter.convertNode(QueryStatementConverter.java:96)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.translation.QueryStatementConverter.convert(QueryStatementConverter.java:84)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.translation.QueryStatementConverter.convertRootQuery(QueryStatementConverter.java:51)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.ZetaSQLPlannerImpl.rel(ZetaSQLPlannerImpl.java:160)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.ZetaSQLQueryPlanner.convertToBeamRelInternal(ZetaSQLQueryPlanner.java:131)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.ZetaSQLQueryPlanner.convertToBeamRel(ZetaSQLQueryPlanner.java:115)
> at
> cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl.executeQuery(ExecuteQueryServiceServer.java:242)
> at
> com.google.zetasql.testing.SqlComplianceServiceGrpc$MethodHandlers.invoke(SqlComplianceServiceGrpc.java:423)
> at
> com.google.zetasql.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171)
> at
> com.google.zetasql.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283)
> at
> com.google.zetasql.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:711)
> at
> com.google.zetasql.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
> at
> com.google.zetasql.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>
[jira] [Updated] (BEAM-9516) ArrayScanToUncollectConverter: ResolvedSubqueryExpr cannot be cast to ResolvedLiteral
[
https://issues.apache.org/jira/browse/BEAM-9516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Pilloud updated BEAM-9516:
-
Status: Open (was: Triage Needed)
> ArrayScanToUncollectConverter: ResolvedSubqueryExpr cannot be cast to
> ResolvedLiteral
> -
>
> Key: BEAM-9516
> URL: https://issues.apache.org/jira/browse/BEAM-9516
> Project: Beam
> Issue Type: Bug
> Components: dsl-sql-zetasql
>Reporter: Andrew Pilloud
>Priority: Major
> Labels: zetasql-compliance
>
> {code:java}
> Mar 16, 2020 12:58:49 PM
> com.google.zetasql.io.grpc.internal.SerializingExecutor run
> SEVERE: Exception while executing runnable
> com.google.zetasql.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed@7c1d4ea6
> java.lang.ClassCastException:
> com.google.zetasql.resolvedast.ResolvedNodes$ResolvedSubqueryExpr cannot be
> cast to com.google.zetasql.resolvedast.ResolvedNodes$ResolvedLiteral
> at
> org.apache.beam.sdk.extensions.sql.zetasql.translation.ArrayScanToUncollectConverter.convert(ArrayScanToUncollectConverter.java:45)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.translation.ArrayScanToUncollectConverter.convert(ArrayScanToUncollectConverter.java:31)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.translation.QueryStatementConverter.convertNode(QueryStatementConverter.java:97)
> at
> java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
> at java.util.Collections$2.tryAdvance(Collections.java:4717)
> at java.util.Collections$2.forEachRemaining(Collections.java:4725)
> at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
> at
> java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
> at
> java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
> at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
> at
> java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.translation.QueryStatementConverter.convertNode(QueryStatementConverter.java:96)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.translation.QueryStatementConverter.convert(QueryStatementConverter.java:84)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.translation.QueryStatementConverter.convertRootQuery(QueryStatementConverter.java:51)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.ZetaSQLPlannerImpl.rel(ZetaSQLPlannerImpl.java:160)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.ZetaSQLQueryPlanner.convertToBeamRelInternal(ZetaSQLQueryPlanner.java:131)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.ZetaSQLQueryPlanner.convertToBeamRel(ZetaSQLQueryPlanner.java:115)
> at
> cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl.executeQuery(ExecuteQueryServiceServer.java:242)
> at
> com.google.zetasql.testing.SqlComplianceServiceGrpc$MethodHandlers.invoke(SqlComplianceServiceGrpc.java:423)
> at
> com.google.zetasql.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171)
> at
> com.google.zetasql.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283)
> at
> com.google.zetasql.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:711)
> at
> com.google.zetasql.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
> at
> com.google.zetasql.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (BEAM-9522) BeamJoinRel.extractJoinRexNode RexLiteral cannot be cast to RexCall
[
https://issues.apache.org/jira/browse/BEAM-9522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Pilloud updated BEAM-9522:
-
Status: Open (was: Triage Needed)
> BeamJoinRel.extractJoinRexNode RexLiteral cannot be cast to RexCall
> ---
>
> Key: BEAM-9522
> URL: https://issues.apache.org/jira/browse/BEAM-9522
> Project: Beam
> Issue Type: Bug
> Components: dsl-sql-zetasql
>Reporter: Andrew Pilloud
>Priority: Minor
> Labels: zetasql-compliance
>
> {code:java}
> Mar 16, 2020 1:26:41 PM
> cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
> executeQuery
> INFO: Processing Sql statement: SELECT r.*, s.* FROM
> (SELECT x, x FROM (SELECT 2 x)) r FULL JOIN (SELECT 3) s ON false
> Mar 16, 2020 1:26:41 PM
> com.google.zetasql.io.grpc.internal.SerializingExecutor run
> SEVERE: Exception while executing runnable
> com.google.zetasql.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed@16f4bbdb
> java.lang.ClassCastException:
> org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rex.RexLiteral
> cannot be cast to
> org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rex.RexCall
> at
> org.apache.beam.sdk.extensions.sql.impl.rel.BeamJoinRel.extractJoinRexNodes(BeamJoinRel.java:304)
> at
> org.apache.beam.sdk.extensions.sql.impl.rel.BeamJoinRel$ExtractJoinKeys.expand(BeamJoinRel.java:194)
> at
> org.apache.beam.sdk.extensions.sql.impl.rel.BeamJoinRel$ExtractJoinKeys.expand(BeamJoinRel.java:177)
> at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:542)
> at org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:476)
> at
> org.apache.beam.sdk.values.PCollectionList.apply(PCollectionList.java:178)
> at
> org.apache.beam.sdk.extensions.sql.impl.rel.BeamCoGBKJoinRel$StandardJoin.expand(BeamCoGBKJoinRel.java:93)
> at
> org.apache.beam.sdk.extensions.sql.impl.rel.BeamCoGBKJoinRel$StandardJoin.expand(BeamCoGBKJoinRel.java:86)
> at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:542)
> at org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:493)
> at
> org.apache.beam.sdk.extensions.sql.impl.rel.BeamSqlRelUtils.toPCollection(BeamSqlRelUtils.java:69)
> at
> org.apache.beam.sdk.extensions.sql.impl.rel.BeamSqlRelUtils.lambda$buildPCollectionList$0(BeamSqlRelUtils.java:50)
> at
> java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
> at java.util.Iterator.forEachRemaining(Iterator.java:116)
> at
> java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801)
> at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
> at
> java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
> at
> java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
> at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
> at
> java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
> at
> org.apache.beam.sdk.extensions.sql.impl.rel.BeamSqlRelUtils.buildPCollectionList(BeamSqlRelUtils.java:51)
> at
> org.apache.beam.sdk.extensions.sql.impl.rel.BeamSqlRelUtils.toPCollection(BeamSqlRelUtils.java:67)
> at
> org.apache.beam.sdk.extensions.sql.impl.rel.BeamSqlRelUtils.toPCollection(BeamSqlRelUtils.java:39)
> at
> org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.runCollector(BeamEnumerableConverter.java:199)
> at
> org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.collectRows(BeamEnumerableConverter.java:218)
> at
> org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.toRowList(BeamEnumerableConverter.java:150)
> at
> org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.toRowList(BeamEnumerableConverter.java:127)
> at
> cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl.executeQuery(ExecuteQueryServiceServer.java:244)
> at
> com.google.zetasql.testing.SqlComplianceServiceGrpc$MethodHandlers.invoke(SqlComplianceServiceGrpc.java:423)
> at
> com.google.zetasql.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171)
> at
> com.google.zetasql.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283)
> at
> com.google.zetasql.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:711)
> at
> com.google.zetasql.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
> at
> com.google.zetasql.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:
[jira] [Updated] (BEAM-9518) BeamZetaSqlCalcRel$CalcFn.processElement Unsupported Beam fieldType: LOGICAL_TYPE
[ https://issues.apache.org/jira/browse/BEAM-9518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Pilloud updated BEAM-9518: - Status: Open (was: Triage Needed) > BeamZetaSqlCalcRel$CalcFn.processElement Unsupported Beam fieldType: > LOGICAL_TYPE > - > > Key: BEAM-9518 > URL: https://issues.apache.org/jira/browse/BEAM-9518 > Project: Beam > Issue Type: Bug > Components: dsl-sql-zetasql >Reporter: Andrew Pilloud >Priority: Minor > Labels: zetasql-compliance > > Mar 16, 2020 1:23:15 PM > com.google.zetasql.io.grpc.internal.SerializingExecutor run > SEVERE: Exception while executing runnable > com.google.zetasql.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed@61d0fcb4 > org.apache.beam.sdk.Pipeline$PipelineExecutionException: > java.lang.UnsupportedOperationException: Unsupported Beam fieldType: > LOGICAL_TYPE > at > org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:348) > at > org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:318) > at org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:213) > at org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:67) > at org.apache.beam.sdk.Pipeline.run(Pipeline.java:317) > at org.apache.beam.sdk.Pipeline.run(Pipeline.java:303) > at > org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.runCollector(BeamEnumerableConverter.java:201) > at > org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.collectRows(BeamEnumerableConverter.java:218) > at > org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.toRowList(BeamEnumerableConverter.java:150) > at > org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.toRowList(BeamEnumerableConverter.java:127) > at > cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl.executeQuery(ExecuteQueryServiceServer.java:244) > at > com.google.zetasql.testing.SqlComplianceServiceGrpc$MethodHandlers.invoke(SqlComplianceServiceGrpc.java:423) > at > com.google.zetasql.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) > at > com.google.zetasql.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) > at > com.google.zetasql.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:711) > at > com.google.zetasql.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) > at > com.google.zetasql.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > Caused by: java.lang.UnsupportedOperationException: Unsupported Beam > fieldType: LOGICAL_TYPE > at > org.apache.beam.sdk.extensions.sql.zetasql.ZetaSqlUtils.zetaSqlValueToJavaObject(ZetaSqlUtils.java:219) > at > org.apache.beam.sdk.extensions.sql.zetasql.BeamZetaSqlCalcRel$CalcFn.processElement(BeamZetaSqlCalcRel.java:193) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9519) BeamZetaSqlCalcRel$CalcFn.processElement field is not nullable in Array field $arrayx
[
https://issues.apache.org/jira/browse/BEAM-9519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Pilloud updated BEAM-9519:
-
Status: Open (was: Triage Needed)
> BeamZetaSqlCalcRel$CalcFn.processElement field is not nullable in Array field
> $arrayx
> -
>
> Key: BEAM-9519
> URL: https://issues.apache.org/jira/browse/BEAM-9519
> Project: Beam
> Issue Type: Bug
> Components: dsl-sql-zetasql
>Reporter: Andrew Pilloud
>Priority: Minor
> Labels: zetasql-compliance
>
> {code:java}
> Mar 16, 2020 1:27:47 PM
> cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
> executeQuery
> INFO: Processing Sql statement: SELECT x FROM UNNEST([3, NULL]) x;
> Mar 16, 2020 1:27:48 PM
> com.google.zetasql.io.grpc.internal.SerializingExecutor run
> SEVERE: Exception while executing runnable
> com.google.zetasql.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed@39086547
> org.apache.beam.sdk.Pipeline$PipelineExecutionException:
> java.lang.IllegalArgumentException: FieldType{typeName=INT64, nullable=false,
> logicalType=null, collectionElementType=null, mapKeyType=null,
> mapValueType=null, rowSchema=null, metadata={}} is not nullable in Array
> field $arrayx
> at
> org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:348)
> at
> org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:318)
> at
> org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:213)
> at org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:67)
> at org.apache.beam.sdk.Pipeline.run(Pipeline.java:317)
> at org.apache.beam.sdk.Pipeline.run(Pipeline.java:303)
> at
> org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.runCollector(BeamEnumerableConverter.java:201)
> at
> org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.collectRows(BeamEnumerableConverter.java:218)
> at
> org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.toRowList(BeamEnumerableConverter.java:150)
> at
> org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.toRowList(BeamEnumerableConverter.java:127)
> at
> cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl.executeQuery(ExecuteQueryServiceServer.java:244)
> at
> com.google.zetasql.testing.SqlComplianceServiceGrpc$MethodHandlers.invoke(SqlComplianceServiceGrpc.java:423)
> at
> com.google.zetasql.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171)
> at
> com.google.zetasql.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283)
> at
> com.google.zetasql.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:711)
> at
> com.google.zetasql.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
> at
> com.google.zetasql.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.IllegalArgumentException: FieldType{typeName=INT64,
> nullable=false, logicalType=null, collectionElementType=null,
> mapKeyType=null, mapValueType=null, rowSchema=null, metadata={}} is not
> nullable in Array field $arrayx
> at org.apache.beam.sdk.values.Row$Builder.verifyArray(Row.java:682)
> at org.apache.beam.sdk.values.Row$Builder.verify(Row.java:647)
> at org.apache.beam.sdk.values.Row$Builder.verify(Row.java:639)
> at org.apache.beam.sdk.values.Row$Builder.build(Row.java:844)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.BeamZetaSqlCalcRel$CalcFn.processElement(BeamZetaSqlCalcRel.java:195)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (BEAM-9512) Anonymous structs have name collision in schema
[
https://issues.apache.org/jira/browse/BEAM-9512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Pilloud updated BEAM-9512:
-
Status: Open (was: Triage Needed)
> Anonymous structs have name collision in schema
> ---
>
> Key: BEAM-9512
> URL: https://issues.apache.org/jira/browse/BEAM-9512
> Project: Beam
> Issue Type: Bug
> Components: dsl-sql-zetasql
>Reporter: Andrew Pilloud
>Priority: Major
> Labels: zetasql-compliance
>
> {code:java}
> Mar 16, 2020 12:57:42 PM
> cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
> executeQuery
> INFO: Processing Sql statement: SELECT STRUCT(ARRAY INT64>>[(11, 12), (21, 22)])
> Mar 16, 2020 12:57:42 PM
> com.google.zetasql.io.grpc.internal.SerializingExecutor run
> SEVERE: Exception while executing runnable
> com.google.zetasql.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed@c73b08a
> java.lang.IllegalArgumentException: Duplicate field added to schema
> at org.apache.beam.sdk.schemas.Schema.(Schema.java:228)
> at org.apache.beam.sdk.schemas.Schema.fromFields(Schema.java:966)
> at
> java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:503)
> at
> org.apache.beam.sdk.extensions.sql.impl.utils.CalciteUtils.toSchema(CalciteUtils.java:194)
> at
> org.apache.beam.sdk.extensions.sql.impl.utils.CalciteUtils.toFieldType(CalciteUtils.java:251)
> at
> org.apache.beam.sdk.extensions.sql.impl.utils.CalciteUtils.toFieldType(CalciteUtils.java:246)
> at
> org.apache.beam.sdk.extensions.sql.impl.utils.CalciteUtils.toField(CalciteUtils.java:239)
> at
> org.apache.beam.sdk.extensions.sql.impl.utils.CalciteUtils.toField(CalciteUtils.java:235)
> at
> java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
> at java.util.Iterator.forEachRemaining(Iterator.java:116)
> at
> java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801)
> at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
> at
> java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
> at
> java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
> at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
> at
> java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
> at
> org.apache.beam.sdk.extensions.sql.impl.utils.CalciteUtils.toSchema(CalciteUtils.java:194)
> at
> org.apache.beam.sdk.extensions.sql.impl.utils.CalciteUtils.toFieldType(CalciteUtils.java:251)
> at
> org.apache.beam.sdk.extensions.sql.impl.utils.CalciteUtils.toField(CalciteUtils.java:239)
> at
> org.apache.beam.sdk.extensions.sql.impl.utils.CalciteUtils.toField(CalciteUtils.java:235)
> at
> java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
> at java.util.Iterator.forEachRemaining(Iterator.java:116)
> at
> java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801)
> at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
> at
> java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
> at
> java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
> at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
> at
> java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
> at
> org.apache.beam.sdk.extensions.sql.impl.utils.CalciteUtils.toSchema(CalciteUtils.java:194)
> at
> cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl.executeQuery(ExecuteQueryServiceServer.java:243)
> at
> com.google.zetasql.testing.SqlComplianceServiceGrpc$MethodHandlers.invoke(SqlComplianceServiceGrpc.java:423)
> at
> com.google.zetasql.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171)
> at
> com.google.zetasql.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283)
> at
> com.google.zetasql.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:711)
> at
> com.google.zetasql.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
> at
> com.google.zetasql.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (BEAM-9520) Forbidden IOException when reading from InputStream
[
https://issues.apache.org/jira/browse/BEAM-9520?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Pilloud updated BEAM-9520:
-
Status: Open (was: Triage Needed)
> Forbidden IOException when reading from InputStream
> ---
>
> Key: BEAM-9520
> URL: https://issues.apache.org/jira/browse/BEAM-9520
> Project: Beam
> Issue Type: Bug
> Components: dsl-sql-zetasql
>Reporter: Andrew Pilloud
>Priority: Minor
> Labels: zetasql-compliance
>
> {code:java}
> Mar 16, 2020 1:26:17 PM
> cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
> executeQuery
> INFO: Processing Sql statement: (SELECT 1 UNION ALL SELECT NULL UNION ALL
> SELECT 1 UNION ALL SELECT NULL)
> INTERSECT DISTINCT
> (SELECT 1 UNION ALL SELECT NULL UNION ALL SELECT NULL)
> Mar 16, 2020 1:26:18 PM
> com.google.zetasql.io.grpc.internal.SerializingExecutor run
> SEVERE: Exception while executing runnable
> com.google.zetasql.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed@7f643e84
> org.apache.beam.sdk.Pipeline$PipelineExecutionException:
> java.lang.IllegalArgumentException: Forbidden IOException when reading from
> InputStream
> at
> org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:348)
> at
> org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:318)
> at
> org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:213)
> at org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:67)
> at org.apache.beam.sdk.Pipeline.run(Pipeline.java:317)
> at org.apache.beam.sdk.Pipeline.run(Pipeline.java:303)
> at
> org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.runCollector(BeamEnumerableConverter.java:201)
> at
> org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.collectRows(BeamEnumerableConverter.java:218)
> at
> org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.toRowList(BeamEnumerableConverter.java:150)
> at
> org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.toRowList(BeamEnumerableConverter.java:127)
> at
> cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl.executeQuery(ExecuteQueryServiceServer.java:244)
> at
> com.google.zetasql.testing.SqlComplianceServiceGrpc$MethodHandlers.invoke(SqlComplianceServiceGrpc.java:423)
> at
> com.google.zetasql.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171)
> at
> com.google.zetasql.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283)
> at
> com.google.zetasql.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:711)
> at
> com.google.zetasql.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
> at
> com.google.zetasql.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.IllegalArgumentException: Forbidden IOException when
> reading from InputStream
> at
> org.apache.beam.sdk.util.CoderUtils.decodeFromSafeStream(CoderUtils.java:118)
> at
> org.apache.beam.sdk.util.CoderUtils.decodeFromByteArray(CoderUtils.java:98)
> at
> org.apache.beam.sdk.util.CoderUtils.decodeFromByteArray(CoderUtils.java:92)
> at org.apache.beam.sdk.util.CoderUtils.clone(CoderUtils.java:141)
> at
> org.apache.beam.sdk.util.MutationDetectors$CodedValueMutationDetector.(MutationDetectors.java:115)
> at
> org.apache.beam.sdk.util.MutationDetectors.forValueWithCoder(MutationDetectors.java:46)
> at
> org.apache.beam.runners.direct.ImmutabilityCheckingBundleFactory$ImmutabilityEnforcingBundle.add(ImmutabilityCheckingBundleFactory.java:112)
> at
> org.apache.beam.runners.direct.ParDoEvaluator$BundleOutputManager.output(ParDoEvaluator.java:299)
> at
> org.apache.beam.repackaged.direct_java.runners.core.SimpleDoFnRunner.outputWindowedValue(SimpleDoFnRunner.java:258)
> at
> org.apache.beam.repackaged.direct_java.runners.core.SimpleDoFnRunner.access$800(SimpleDoFnRunner.java:78)
> at
> org.apache.beam.repackaged.direct_java.runners.core.SimpleDoFnRunner$DoFnProcessContext.output(SimpleDoFnRunner.java:409)
> at
> org.apache.beam.repackaged.direct_java.runners.core.SimpleDoFnRunner$DoFnProcessContext.output(SimpleDoFnRunner.java:397)
> at
> org.apache.
[jira] [Updated] (BEAM-9523) GROUP BY DOUBLE fails at runtime instead of planning time
[
https://issues.apache.org/jira/browse/BEAM-9523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Pilloud updated BEAM-9523:
-
Status: Open (was: Triage Needed)
> GROUP BY DOUBLE fails at runtime instead of planning time
> -
>
> Key: BEAM-9523
> URL: https://issues.apache.org/jira/browse/BEAM-9523
> Project: Beam
> Issue Type: Bug
> Components: dsl-sql-zetasql
>Reporter: Andrew Pilloud
>Priority: Minor
> Labels: zetasql-compliance
>
> This should fail earlier.
> {code:java}
> Mar 16, 2020 1:27:32 PM
> cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
> executeQuery
> INFO: Processing Sql statement: SELECT COUNT(a) FROM (
> SELECT a FROM (SELECT 1.2 a UNION ALL SELECT 2.3 UNION ALL SELECT 3.4) LIMIT
> 1)
> Mar 16, 2020 1:27:33 PM
> com.google.zetasql.io.grpc.internal.SerializingExecutor run
> SEVERE: Exception while executing runnable
> com.google.zetasql.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed@7946abe5
> java.lang.IllegalStateException: the keyCoder of a GroupByKey must be
> deterministic
> at org.apache.beam.sdk.transforms.GroupByKey.expand(GroupByKey.java:234)
> at org.apache.beam.sdk.transforms.GroupByKey.expand(GroupByKey.java:110)
> at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:542)
> at org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:493)
> at org.apache.beam.sdk.values.PCollection.apply(PCollection.java:368)
> at
> org.apache.beam.sdk.transforms.join.CoGroupByKey.expand(CoGroupByKey.java:118)
> at
> org.apache.beam.sdk.transforms.join.CoGroupByKey.expand(CoGroupByKey.java:71)
> at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:542)
> at org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:476)
> at
> org.apache.beam.sdk.transforms.join.KeyedPCollectionTuple.apply(KeyedPCollectionTuple.java:108)
> at
> org.apache.beam.sdk.extensions.sql.impl.rel.BeamSetOperatorRelBase.expand(BeamSetOperatorRelBase.java:96)
> at
> org.apache.beam.sdk.extensions.sql.impl.rel.BeamSetOperatorRelBase.expand(BeamSetOperatorRelBase.java:41)
> at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:542)
> at org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:493)
> at
> org.apache.beam.sdk.extensions.sql.impl.rel.BeamSqlRelUtils.toPCollection(BeamSqlRelUtils.java:69)
> at
> org.apache.beam.sdk.extensions.sql.impl.rel.BeamSqlRelUtils.lambda$buildPCollectionList$0(BeamSqlRelUtils.java:50)
> at
> java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
> at java.util.Iterator.forEachRemaining(Iterator.java:116)
> at
> java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801)
> at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
> at
> java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
> at
> java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
> at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
> at
> java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
> at
> org.apache.beam.sdk.extensions.sql.impl.rel.BeamSqlRelUtils.buildPCollectionList(BeamSqlRelUtils.java:51)
> at
> org.apache.beam.sdk.extensions.sql.impl.rel.BeamSqlRelUtils.toPCollection(BeamSqlRelUtils.java:67)
> at
> org.apache.beam.sdk.extensions.sql.impl.rel.BeamSqlRelUtils.lambda$buildPCollectionList$0(BeamSqlRelUtils.java:50)
> at
> java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
> at java.util.Iterator.forEachRemaining(Iterator.java:116)
> at
> java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801)
> at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
> at
> java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
> at
> java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
> at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
> at
> java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
> at
> org.apache.beam.sdk.extensions.sql.impl.rel.BeamSqlRelUtils.buildPCollectionList(BeamSqlRelUtils.java:51)
> at
> org.apache.beam.sdk.extensions.sql.impl.rel.BeamSqlRelUtils.toPCollection(BeamSqlRelUtils.java:67)
> at
> org.apache.beam.sdk.extensions.sql.impl.rel.BeamSqlRelUtils.lambda$buildPCollectionList$0(BeamSqlRelUtils.java:50)
> at
> java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
> at java.util.Iterator.forEachRemaining(Iterator.java:116)
>
[jira] [Updated] (BEAM-9517) BeamZetaSqlCalcRel$CalcFn.processElement Field is not nullable
[
https://issues.apache.org/jira/browse/BEAM-9517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Pilloud updated BEAM-9517:
-
Status: Open (was: Triage Needed)
> BeamZetaSqlCalcRel$CalcFn.processElement Field is not nullable
> --
>
> Key: BEAM-9517
> URL: https://issues.apache.org/jira/browse/BEAM-9517
> Project: Beam
> Issue Type: Bug
> Components: dsl-sql-zetasql
>Reporter: Andrew Pilloud
>Priority: Minor
> Labels: zetasql-compliance
>
> {code:java}
> Mar 16, 2020 1:24:42 PM
> cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
> executeQuery
> INFO: Processing Sql statement: SELECT timestamp_diff(@p0, @p1, MICROSECOND)
> AS ColA
> Mar 16, 2020 1:24:43 PM
> com.google.zetasql.io.grpc.internal.SerializingExecutor run
> SEVERE: Exception while executing runnable
> com.google.zetasql.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed@20ca06
> org.apache.beam.sdk.Pipeline$PipelineExecutionException:
> java.lang.IllegalArgumentException: Field ColA is not nullable
> at
> org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:348)
> at
> org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:318)
> at
> org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:213)
> at org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:67)
> at org.apache.beam.sdk.Pipeline.run(Pipeline.java:317)
> at org.apache.beam.sdk.Pipeline.run(Pipeline.java:303)
> at
> org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.runCollector(BeamEnumerableConverter.java:201)
> at
> org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.collectRows(BeamEnumerableConverter.java:218)
> at
> org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.toRowList(BeamEnumerableConverter.java:150)
> at
> org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.toRowList(BeamEnumerableConverter.java:127)
> at
> cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl.executeQuery(ExecuteQueryServiceServer.java:244)
> at
> com.google.zetasql.testing.SqlComplianceServiceGrpc$MethodHandlers.invoke(SqlComplianceServiceGrpc.java:423)
> at
> com.google.zetasql.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171)
> at
> com.google.zetasql.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283)
> at
> com.google.zetasql.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:711)
> at
> com.google.zetasql.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
> at
> com.google.zetasql.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.IllegalArgumentException: Field ColA is not nullable
> at org.apache.beam.sdk.values.Row$Builder.verify(Row.java:635)
> at org.apache.beam.sdk.values.Row$Builder.build(Row.java:844)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.BeamZetaSqlCalcRel$CalcFn.processElement(BeamZetaSqlCalcRel.java:195)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (BEAM-9514) AssertionError type mismatch from SUM
[
https://issues.apache.org/jira/browse/BEAM-9514?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Pilloud updated BEAM-9514:
-
Status: Open (was: Triage Needed)
> AssertionError type mismatch from SUM
> -
>
> Key: BEAM-9514
> URL: https://issues.apache.org/jira/browse/BEAM-9514
> Project: Beam
> Issue Type: Bug
> Components: dsl-sql-zetasql
>Reporter: Andrew Pilloud
>Priority: Major
> Labels: zetasql-compliance
>
> {code:java}
> Mar 16, 2020 12:59:49 PM
> cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
> executeQuery
> INFO: Processing Sql statement: select sum(distinct_4) from TableDistincts
> group by distinct_2
> having false
> Exception in thread "pool-1-thread-1" java.lang.AssertionError: Type mismatch:
> rowtype of new rel:
> RecordType(BIGINT distinct_2, BIGINT $col1) NOT NULL
> rowtype of set:
> RecordType(BIGINT distinct_2, BIGINT NOT NULL $col1) NOT NULL
> at
> org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.util.Litmus$1.fail(Litmus.java:31)
> at
> org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelOptUtil.equal(RelOptUtil.java:1984)
> at
> org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.volcano.RelSubset.add(RelSubset.java:284)
> at
> org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.volcano.RelSet.add(RelSet.java:148)
> at
> org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.volcano.VolcanoPlanner.addRelToSet(VolcanoPlanner.java:1806)
> at
> org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.volcano.VolcanoPlanner.reregister(VolcanoPlanner.java:1480)
> at
> org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.volcano.RelSet.mergeWith(RelSet.java:331)
> at
> org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.volcano.VolcanoPlanner.merge(VolcanoPlanner.java:1571)
> at
> org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.volcano.VolcanoPlanner.ensureRegistered(VolcanoPlanner.java:863)
> at
> org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.volcano.VolcanoPlanner.ensureRegistered(VolcanoPlanner.java:1927)
> at
> org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.volcano.VolcanoRuleCall.transformTo(VolcanoRuleCall.java:129)
> at
> org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelOptRuleCall.transformTo(RelOptRuleCall.java:236)
> at
> org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.rules.AggregateRemoveRule.onMatch(AggregateRemoveRule.java:126)
> at
> org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:208)
> at
> org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp(VolcanoPlanner.java:631)
> at
> org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.tools.Programs$RuleSetProgram.run(Programs.java:328)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.ZetaSQLPlannerImpl.transform(ZetaSQLPlannerImpl.java:180)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.ZetaSQLQueryPlanner.convertToBeamRelInternal(ZetaSQLQueryPlanner.java:150)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.ZetaSQLQueryPlanner.convertToBeamRel(ZetaSQLQueryPlanner.java:115)
> at
> cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl.executeQuery(ExecuteQueryServiceServer.java:242)
> at
> com.google.zetasql.testing.SqlComplianceServiceGrpc$MethodHandlers.invoke(SqlComplianceServiceGrpc.java:423)
> at
> com.google.zetasql.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171)
> at
> com.google.zetasql.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283)
> at
> com.google.zetasql.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:711)
> at
> com.google.zetasql.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
> at
> com.google.zetasql.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748) {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (BEAM-9521) NullPointerException in convertRexNodeFromResolvedExpr
[
https://issues.apache.org/jira/browse/BEAM-9521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Pilloud updated BEAM-9521:
-
Status: Open (was: Triage Needed)
> NullPointerException in convertRexNodeFromResolvedExpr
> --
>
> Key: BEAM-9521
> URL: https://issues.apache.org/jira/browse/BEAM-9521
> Project: Beam
> Issue Type: Bug
> Components: dsl-sql-zetasql
>Reporter: Andrew Pilloud
>Priority: Major
> Labels: zetasql-compliance
>
> {code:java}
> Mar 16, 2020 12:57:10 PM
> cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
> executeQuery
> INFO: Processing Sql statement: SELECT (SELECT AS STRUCT
> (SELECT AS STRUCT l.float_val lf, r.float_val rf),
> l.float_val = r.float_val),
>(SELECT AS STRUCT
> (SELECT AS STRUCT l.double_val ld, r.double_val rd),
> l.double_val = r.double_val)
> FROM TestTableZero l, TestTableZero r
> WHERE l.row_id > r.row_id
> Mar 16, 2020 12:57:10 PM
> com.google.zetasql.io.grpc.internal.SerializingExecutor run
> SEVERE: Exception while executing runnable
> com.google.zetasql.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed@710f8c0d
> java.lang.NullPointerException
> at
> org.apache.beam.sdk.extensions.sql.zetasql.translation.ExpressionConverter.convertRexNodeFromResolvedExpr(ExpressionConverter.java:301)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.translation.JoinScanConverter.convert(JoinScanConverter.java:83)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.translation.JoinScanConverter.convert(JoinScanConverter.java:36)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.translation.QueryStatementConverter.convertNode(QueryStatementConverter.java:97)
> at
> java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
> at java.util.Collections$2.tryAdvance(Collections.java:4717)
> at java.util.Collections$2.forEachRemaining(Collections.java:4725)
> at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
> at
> java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
> at
> java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
> at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
> at
> java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.translation.QueryStatementConverter.convertNode(QueryStatementConverter.java:96)
> at
> java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
> at java.util.Collections$2.tryAdvance(Collections.java:4717)
> at java.util.Collections$2.forEachRemaining(Collections.java:4725)
> at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
> at
> java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
> at
> java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
> at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
> at
> java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.translation.QueryStatementConverter.convertNode(QueryStatementConverter.java:96)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.translation.QueryStatementConverter.convert(QueryStatementConverter.java:84)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.translation.QueryStatementConverter.convertRootQuery(QueryStatementConverter.java:51)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.ZetaSQLPlannerImpl.rel(ZetaSQLPlannerImpl.java:160)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.ZetaSQLQueryPlanner.convertToBeamRelInternal(ZetaSQLQueryPlanner.java:131)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.ZetaSQLQueryPlanner.convertToBeamRel(ZetaSQLQueryPlanner.java:115)
> at
> cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl.executeQuery(ExecuteQueryServiceServer.java:242)
> at
> com.google.zetasql.testing.SqlComplianceServiceGrpc$MethodHandlers.invoke(SqlComplianceServiceGrpc.java:423)
> at
> com.google.zetasql.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171)
> at
> com.google.zetasql.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283)
> at
> com.google.zetasql.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:711)
> at
> com.google.zetasql.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
> at
> com.google.zetasql.io
[jira] [Updated] (BEAM-9515) ResolvedSubqueryExpr can not be cast to ResolvedLiteral
[
https://issues.apache.org/jira/browse/BEAM-9515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Pilloud updated BEAM-9515:
-
Status: Open (was: Triage Needed)
> ResolvedSubqueryExpr can not be cast to ResolvedLiteral
> ---
>
> Key: BEAM-9515
> URL: https://issues.apache.org/jira/browse/BEAM-9515
> Project: Beam
> Issue Type: Bug
> Components: dsl-sql-zetasql
>Reporter: Andrew Pilloud
>Priority: Major
> Labels: zetasql-compliance
>
> {code:java}
> Mar 16, 2020 1:00:02 PM
> cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
> executeQuery
> INFO: Processing Sql statement: SELECT * FROM UNNEST(ARRAY(
> SELECT bool_val FROM AllTypesTable t
> ORDER BY bool_val ASC
> )) x WITH OFFSET POS
> Mar 16, 2020 1:00:02 PM
> com.google.zetasql.io.grpc.internal.SerializingExecutor run
> SEVERE: Exception while executing runnable
> com.google.zetasql.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed@7b42f9e7
> java.lang.ClassCastException:
> com.google.zetasql.resolvedast.ResolvedNodes$ResolvedSubqueryExpr cannot be
> cast to com.google.zetasql.resolvedast.ResolvedNodes$ResolvedLiteral
> at
> org.apache.beam.sdk.extensions.sql.zetasql.translation.ArrayScanToUncollectConverter.convert(ArrayScanToUncollectConverter.java:45)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.translation.ArrayScanToUncollectConverter.convert(ArrayScanToUncollectConverter.java:31)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.translation.QueryStatementConverter.convertNode(QueryStatementConverter.java:97)
> at
> java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
> at java.util.Collections$2.tryAdvance(Collections.java:4717)
> at java.util.Collections$2.forEachRemaining(Collections.java:4725)
> at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
> at
> java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
> at
> java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
> at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
> at
> java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.translation.QueryStatementConverter.convertNode(QueryStatementConverter.java:96)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.translation.QueryStatementConverter.convert(QueryStatementConverter.java:84)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.translation.QueryStatementConverter.convertRootQuery(QueryStatementConverter.java:51)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.ZetaSQLPlannerImpl.rel(ZetaSQLPlannerImpl.java:160)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.ZetaSQLQueryPlanner.convertToBeamRelInternal(ZetaSQLQueryPlanner.java:131)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.ZetaSQLQueryPlanner.convertToBeamRel(ZetaSQLQueryPlanner.java:115)
> at
> cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl.executeQuery(ExecuteQueryServiceServer.java:242)
> at
> com.google.zetasql.testing.SqlComplianceServiceGrpc$MethodHandlers.invoke(SqlComplianceServiceGrpc.java:423)
> at
> com.google.zetasql.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171)
> at
> com.google.zetasql.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283)
> at
> com.google.zetasql.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:711)
> at
> com.google.zetasql.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
> at
> com.google.zetasql.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (BEAM-9523) GROUP BY DOUBLE fails at runtime instead of planning time
Andrew Pilloud created BEAM-9523:
Summary: GROUP BY DOUBLE fails at runtime instead of planning time
Key: BEAM-9523
URL: https://issues.apache.org/jira/browse/BEAM-9523
Project: Beam
Issue Type: Bug
Components: dsl-sql-zetasql
Reporter: Andrew Pilloud
This should fail earlier.
{code:java}
Mar 16, 2020 1:27:32 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: SELECT COUNT(a) FROM (
SELECT a FROM (SELECT 1.2 a UNION ALL SELECT 2.3 UNION ALL SELECT 3.4) LIMIT 1)
Mar 16, 2020 1:27:33 PM com.google.zetasql.io.grpc.internal.SerializingExecutor
run
SEVERE: Exception while executing runnable
com.google.zetasql.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed@7946abe5
java.lang.IllegalStateException: the keyCoder of a GroupByKey must be
deterministic
at org.apache.beam.sdk.transforms.GroupByKey.expand(GroupByKey.java:234)
at org.apache.beam.sdk.transforms.GroupByKey.expand(GroupByKey.java:110)
at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:542)
at org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:493)
at org.apache.beam.sdk.values.PCollection.apply(PCollection.java:368)
at
org.apache.beam.sdk.transforms.join.CoGroupByKey.expand(CoGroupByKey.java:118)
at
org.apache.beam.sdk.transforms.join.CoGroupByKey.expand(CoGroupByKey.java:71)
at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:542)
at org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:476)
at
org.apache.beam.sdk.transforms.join.KeyedPCollectionTuple.apply(KeyedPCollectionTuple.java:108)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamSetOperatorRelBase.expand(BeamSetOperatorRelBase.java:96)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamSetOperatorRelBase.expand(BeamSetOperatorRelBase.java:41)
at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:542)
at org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:493)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamSqlRelUtils.toPCollection(BeamSqlRelUtils.java:69)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamSqlRelUtils.lambda$buildPCollectionList$0(BeamSqlRelUtils.java:50)
at
java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
at java.util.Iterator.forEachRemaining(Iterator.java:116)
at
java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801)
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
at
java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
at
java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at
java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamSqlRelUtils.buildPCollectionList(BeamSqlRelUtils.java:51)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamSqlRelUtils.toPCollection(BeamSqlRelUtils.java:67)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamSqlRelUtils.lambda$buildPCollectionList$0(BeamSqlRelUtils.java:50)
at
java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
at java.util.Iterator.forEachRemaining(Iterator.java:116)
at
java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801)
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
at
java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
at
java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at
java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamSqlRelUtils.buildPCollectionList(BeamSqlRelUtils.java:51)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamSqlRelUtils.toPCollection(BeamSqlRelUtils.java:67)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamSqlRelUtils.lambda$buildPCollectionList$0(BeamSqlRelUtils.java:50)
at
java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
at java.util.Iterator.forEachRemaining(Iterator.java:116)
at
java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801)
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
at
java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
at
java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
[jira] [Created] (BEAM-9522) BeamJoinRel.extractJoinRexNode RexLiteral cannot be cast to RexCall
Andrew Pilloud created BEAM-9522:
Summary: BeamJoinRel.extractJoinRexNode RexLiteral cannot be cast
to RexCall
Key: BEAM-9522
URL: https://issues.apache.org/jira/browse/BEAM-9522
Project: Beam
Issue Type: Bug
Components: dsl-sql-zetasql
Reporter: Andrew Pilloud
{code:java}
Mar 16, 2020 1:26:41 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: SELECT r.*, s.* FROM
(SELECT x, x FROM (SELECT 2 x)) r FULL JOIN (SELECT 3) s ON false
Mar 16, 2020 1:26:41 PM com.google.zetasql.io.grpc.internal.SerializingExecutor
run
SEVERE: Exception while executing runnable
com.google.zetasql.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed@16f4bbdb
java.lang.ClassCastException:
org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rex.RexLiteral cannot
be cast to org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rex.RexCall
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamJoinRel.extractJoinRexNodes(BeamJoinRel.java:304)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamJoinRel$ExtractJoinKeys.expand(BeamJoinRel.java:194)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamJoinRel$ExtractJoinKeys.expand(BeamJoinRel.java:177)
at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:542)
at org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:476)
at
org.apache.beam.sdk.values.PCollectionList.apply(PCollectionList.java:178)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamCoGBKJoinRel$StandardJoin.expand(BeamCoGBKJoinRel.java:93)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamCoGBKJoinRel$StandardJoin.expand(BeamCoGBKJoinRel.java:86)
at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:542)
at org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:493)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamSqlRelUtils.toPCollection(BeamSqlRelUtils.java:69)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamSqlRelUtils.lambda$buildPCollectionList$0(BeamSqlRelUtils.java:50)
at
java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
at java.util.Iterator.forEachRemaining(Iterator.java:116)
at
java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801)
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
at
java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
at
java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at
java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamSqlRelUtils.buildPCollectionList(BeamSqlRelUtils.java:51)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamSqlRelUtils.toPCollection(BeamSqlRelUtils.java:67)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamSqlRelUtils.toPCollection(BeamSqlRelUtils.java:39)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.runCollector(BeamEnumerableConverter.java:199)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.collectRows(BeamEnumerableConverter.java:218)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.toRowList(BeamEnumerableConverter.java:150)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.toRowList(BeamEnumerableConverter.java:127)
at
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl.executeQuery(ExecuteQueryServiceServer.java:244)
at
com.google.zetasql.testing.SqlComplianceServiceGrpc$MethodHandlers.invoke(SqlComplianceServiceGrpc.java:423)
at
com.google.zetasql.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171)
at
com.google.zetasql.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283)
at
com.google.zetasql.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:711)
at
com.google.zetasql.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at
com.google.zetasql.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (BEAM-9521) NullPointerException in convertRexNodeFromResolvedExpr
Andrew Pilloud created BEAM-9521:
Summary: NullPointerException in convertRexNodeFromResolvedExpr
Key: BEAM-9521
URL: https://issues.apache.org/jira/browse/BEAM-9521
Project: Beam
Issue Type: Bug
Components: dsl-sql-zetasql
Reporter: Andrew Pilloud
{code:java}
Mar 16, 2020 12:57:10 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: SELECT (SELECT AS STRUCT
(SELECT AS STRUCT l.float_val lf, r.float_val rf),
l.float_val = r.float_val),
(SELECT AS STRUCT
(SELECT AS STRUCT l.double_val ld, r.double_val rd),
l.double_val = r.double_val)
FROM TestTableZero l, TestTableZero r
WHERE l.row_id > r.row_id
Mar 16, 2020 12:57:10 PM
com.google.zetasql.io.grpc.internal.SerializingExecutor run
SEVERE: Exception while executing runnable
com.google.zetasql.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed@710f8c0d
java.lang.NullPointerException
at
org.apache.beam.sdk.extensions.sql.zetasql.translation.ExpressionConverter.convertRexNodeFromResolvedExpr(ExpressionConverter.java:301)
at
org.apache.beam.sdk.extensions.sql.zetasql.translation.JoinScanConverter.convert(JoinScanConverter.java:83)
at
org.apache.beam.sdk.extensions.sql.zetasql.translation.JoinScanConverter.convert(JoinScanConverter.java:36)
at
org.apache.beam.sdk.extensions.sql.zetasql.translation.QueryStatementConverter.convertNode(QueryStatementConverter.java:97)
at
java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
at java.util.Collections$2.tryAdvance(Collections.java:4717)
at java.util.Collections$2.forEachRemaining(Collections.java:4725)
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
at
java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
at
java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at
java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
at
org.apache.beam.sdk.extensions.sql.zetasql.translation.QueryStatementConverter.convertNode(QueryStatementConverter.java:96)
at
java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
at java.util.Collections$2.tryAdvance(Collections.java:4717)
at java.util.Collections$2.forEachRemaining(Collections.java:4725)
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
at
java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
at
java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at
java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
at
org.apache.beam.sdk.extensions.sql.zetasql.translation.QueryStatementConverter.convertNode(QueryStatementConverter.java:96)
at
org.apache.beam.sdk.extensions.sql.zetasql.translation.QueryStatementConverter.convert(QueryStatementConverter.java:84)
at
org.apache.beam.sdk.extensions.sql.zetasql.translation.QueryStatementConverter.convertRootQuery(QueryStatementConverter.java:51)
at
org.apache.beam.sdk.extensions.sql.zetasql.ZetaSQLPlannerImpl.rel(ZetaSQLPlannerImpl.java:160)
at
org.apache.beam.sdk.extensions.sql.zetasql.ZetaSQLQueryPlanner.convertToBeamRelInternal(ZetaSQLQueryPlanner.java:131)
at
org.apache.beam.sdk.extensions.sql.zetasql.ZetaSQLQueryPlanner.convertToBeamRel(ZetaSQLQueryPlanner.java:115)
at
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl.executeQuery(ExecuteQueryServiceServer.java:242)
at
com.google.zetasql.testing.SqlComplianceServiceGrpc$MethodHandlers.invoke(SqlComplianceServiceGrpc.java:423)
at
com.google.zetasql.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171)
at
com.google.zetasql.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283)
at
com.google.zetasql.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:711)
at
com.google.zetasql.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at
com.google.zetasql.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#80
[jira] [Created] (BEAM-9520) Forbidden IOException when reading from InputStream
Andrew Pilloud created BEAM-9520:
Summary: Forbidden IOException when reading from InputStream
Key: BEAM-9520
URL: https://issues.apache.org/jira/browse/BEAM-9520
Project: Beam
Issue Type: Bug
Components: dsl-sql-zetasql
Reporter: Andrew Pilloud
{code:java}
Mar 16, 2020 1:26:17 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: (SELECT 1 UNION ALL SELECT NULL UNION ALL
SELECT 1 UNION ALL SELECT NULL)
INTERSECT DISTINCT
(SELECT 1 UNION ALL SELECT NULL UNION ALL SELECT NULL)
Mar 16, 2020 1:26:18 PM com.google.zetasql.io.grpc.internal.SerializingExecutor
run
SEVERE: Exception while executing runnable
com.google.zetasql.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed@7f643e84
org.apache.beam.sdk.Pipeline$PipelineExecutionException:
java.lang.IllegalArgumentException: Forbidden IOException when reading from
InputStream
at
org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:348)
at
org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:318)
at
org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:213)
at org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:67)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:317)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:303)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.runCollector(BeamEnumerableConverter.java:201)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.collectRows(BeamEnumerableConverter.java:218)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.toRowList(BeamEnumerableConverter.java:150)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.toRowList(BeamEnumerableConverter.java:127)
at
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl.executeQuery(ExecuteQueryServiceServer.java:244)
at
com.google.zetasql.testing.SqlComplianceServiceGrpc$MethodHandlers.invoke(SqlComplianceServiceGrpc.java:423)
at
com.google.zetasql.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171)
at
com.google.zetasql.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283)
at
com.google.zetasql.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:711)
at
com.google.zetasql.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at
com.google.zetasql.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalArgumentException: Forbidden IOException when
reading from InputStream
at
org.apache.beam.sdk.util.CoderUtils.decodeFromSafeStream(CoderUtils.java:118)
at
org.apache.beam.sdk.util.CoderUtils.decodeFromByteArray(CoderUtils.java:98)
at
org.apache.beam.sdk.util.CoderUtils.decodeFromByteArray(CoderUtils.java:92)
at org.apache.beam.sdk.util.CoderUtils.clone(CoderUtils.java:141)
at
org.apache.beam.sdk.util.MutationDetectors$CodedValueMutationDetector.(MutationDetectors.java:115)
at
org.apache.beam.sdk.util.MutationDetectors.forValueWithCoder(MutationDetectors.java:46)
at
org.apache.beam.runners.direct.ImmutabilityCheckingBundleFactory$ImmutabilityEnforcingBundle.add(ImmutabilityCheckingBundleFactory.java:112)
at
org.apache.beam.runners.direct.ParDoEvaluator$BundleOutputManager.output(ParDoEvaluator.java:299)
at
org.apache.beam.repackaged.direct_java.runners.core.SimpleDoFnRunner.outputWindowedValue(SimpleDoFnRunner.java:258)
at
org.apache.beam.repackaged.direct_java.runners.core.SimpleDoFnRunner.access$800(SimpleDoFnRunner.java:78)
at
org.apache.beam.repackaged.direct_java.runners.core.SimpleDoFnRunner$DoFnProcessContext.output(SimpleDoFnRunner.java:409)
at
org.apache.beam.repackaged.direct_java.runners.core.SimpleDoFnRunner$DoFnProcessContext.output(SimpleDoFnRunner.java:397)
at
org.apache.beam.sdk.transforms.join.CoGroupByKey$ConstructUnionTableFn.processElement(CoGroupByKey.java:175)
Caused by: java.io.EOFException: reached end of stream after reading 0 bytes; 1
bytes expected
at
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.io.ByteStreams.readFully(ByteStreams.java:780)
at
org.apache.beam.vendor.guava.v26_0_jre.com.go
[jira] [Created] (BEAM-9519) BeamZetaSqlCalcRel$CalcFn.processElement field is not nullable in Array field $arrayx
Andrew Pilloud created BEAM-9519:
Summary: BeamZetaSqlCalcRel$CalcFn.processElement field is not
nullable in Array field $arrayx
Key: BEAM-9519
URL: https://issues.apache.org/jira/browse/BEAM-9519
Project: Beam
Issue Type: Bug
Components: dsl-sql-zetasql
Reporter: Andrew Pilloud
{code:java}
Mar 16, 2020 1:27:47 PM
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
executeQuery
INFO: Processing Sql statement: SELECT x FROM UNNEST([3, NULL]) x;
Mar 16, 2020 1:27:48 PM com.google.zetasql.io.grpc.internal.SerializingExecutor
run
SEVERE: Exception while executing runnable
com.google.zetasql.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed@39086547
org.apache.beam.sdk.Pipeline$PipelineExecutionException:
java.lang.IllegalArgumentException: FieldType{typeName=INT64, nullable=false,
logicalType=null, collectionElementType=null, mapKeyType=null,
mapValueType=null, rowSchema=null, metadata={}} is not nullable in Array field
$arrayx
at
org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:348)
at
org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:318)
at
org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:213)
at org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:67)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:317)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:303)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.runCollector(BeamEnumerableConverter.java:201)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.collectRows(BeamEnumerableConverter.java:218)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.toRowList(BeamEnumerableConverter.java:150)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter.toRowList(BeamEnumerableConverter.java:127)
at
cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl.executeQuery(ExecuteQueryServiceServer.java:244)
at
com.google.zetasql.testing.SqlComplianceServiceGrpc$MethodHandlers.invoke(SqlComplianceServiceGrpc.java:423)
at
com.google.zetasql.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171)
at
com.google.zetasql.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283)
at
com.google.zetasql.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:711)
at
com.google.zetasql.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at
com.google.zetasql.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalArgumentException: FieldType{typeName=INT64,
nullable=false, logicalType=null, collectionElementType=null, mapKeyType=null,
mapValueType=null, rowSchema=null, metadata={}} is not nullable in Array field
$arrayx
at org.apache.beam.sdk.values.Row$Builder.verifyArray(Row.java:682)
at org.apache.beam.sdk.values.Row$Builder.verify(Row.java:647)
at org.apache.beam.sdk.values.Row$Builder.verify(Row.java:639)
at org.apache.beam.sdk.values.Row$Builder.build(Row.java:844)
at
org.apache.beam.sdk.extensions.sql.zetasql.BeamZetaSqlCalcRel$CalcFn.processElement(BeamZetaSqlCalcRel.java:195)
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (BEAM-9510) Dependencies in base_image_requirements.txt are not compatible with each other
[ https://issues.apache.org/jira/browse/BEAM-9510?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ahmet Altay reassigned BEAM-9510: - Assignee: Hannah Jiang > Dependencies in base_image_requirements.txt are not compatible with each other > -- > > Key: BEAM-9510 > URL: https://issues.apache.org/jira/browse/BEAM-9510 > Project: Beam > Issue Type: Bug > Components: sdk-py-harness >Reporter: David Yan >Assignee: Hannah Jiang >Priority: Major > > [https://github.com/apache/beam/blob/master/sdks/python/container/base_image_requirements.txt#L56] > says it requires google-cloud-bigquery==1.24.0, google-cloud-core==1.0.2, > google-cloud-bigtable==0.32.1, grpc-1.22.0 and tensorflow-2.1.0 > But they are incompatible with each other: > ERROR: google-cloud-bigquery 1.24.0 has requirement > google-cloud-core<2.0dev,>=1.1.0, but you'll have google-cloud-core 1.0.2 > which is incompatible. > ERROR: google-cloud-bigtable 0.32.1 has requirement > google-cloud-core<0.30dev,>=0.29.0, but you'll have google-cloud-core 1.0.2 > which is incompatible. > ERROR: tensorboard 2.1.1 has requirement grpcio>=1.24.3, but you'll have > grpcio 1.22.0 which is incompatible. > ERROR: tensorflow 2.1.0 has requirement scipy==1.4.1; python_version >= "3", > but you'll have scipy 1.2.2 which is incompatible. > > > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-9510) Dependencies in base_image_requirements.txt are not compatible with each other
[ https://issues.apache.org/jira/browse/BEAM-9510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17060457#comment-17060457 ] Ahmet Altay commented on BEAM-9510: --- This file looks out of date. We can update it to match (https://cloud.google.com/dataflow/docs/concepts/sdk-worker-dependencies#sdk-for-python) > Dependencies in base_image_requirements.txt are not compatible with each other > -- > > Key: BEAM-9510 > URL: https://issues.apache.org/jira/browse/BEAM-9510 > Project: Beam > Issue Type: Bug > Components: sdk-py-harness >Reporter: David Yan >Priority: Major > > [https://github.com/apache/beam/blob/master/sdks/python/container/base_image_requirements.txt#L56] > says it requires google-cloud-bigquery==1.24.0, google-cloud-core==1.0.2, > google-cloud-bigtable==0.32.1, grpc-1.22.0 and tensorflow-2.1.0 > But they are incompatible with each other: > ERROR: google-cloud-bigquery 1.24.0 has requirement > google-cloud-core<2.0dev,>=1.1.0, but you'll have google-cloud-core 1.0.2 > which is incompatible. > ERROR: google-cloud-bigtable 0.32.1 has requirement > google-cloud-core<0.30dev,>=0.29.0, but you'll have google-cloud-core 1.0.2 > which is incompatible. > ERROR: tensorboard 2.1.1 has requirement grpcio>=1.24.3, but you'll have > grpcio 1.22.0 which is incompatible. > ERROR: tensorflow 2.1.0 has requirement scipy==1.4.1; python_version >= "3", > but you'll have scipy 1.2.2 which is incompatible. > > > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
