[jira] [Work logged] (BEAM-9533) Replace *-gcp/*-aws tox suites with *-cloud suites to run unit tests for both
[ https://issues.apache.org/jira/browse/BEAM-9533?focusedWorklogId=405953=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-405953 ] ASF GitHub Bot logged work on BEAM-9533: Author: ASF GitHub Bot Created on: 19/Mar/20 04:57 Start Date: 19/Mar/20 04:57 Worklog Time Spent: 10m Work Description: pabloem commented on pull request #11158: [BEAM-9533] Fixing tox.ini variants URL: https://github.com/apache/beam/pull/11158 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 405953) Time Spent: 1h 20m (was: 1h 10m) > Replace *-gcp/*-aws tox suites with *-cloud suites to run unit tests for both > - > > Key: BEAM-9533 > URL: https://issues.apache.org/jira/browse/BEAM-9533 > Project: Beam > Issue Type: Bug > Components: testing >Reporter: Pablo Estrada >Assignee: Pablo Estrada >Priority: Minor > Time Spent: 1h 20m > Remaining Estimate: 0h > > Currently there are `py37-gcp`, py37-aws test suites. Let's consolidate all > of them into py37-cloud, along with other py35-gcp, py27-gcp, etc. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-3301) Go SplittableDoFn support
[
https://issues.apache.org/jira/browse/BEAM-3301?focusedWorklogId=405941=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-405941
]
ASF GitHub Bot logged work on BEAM-3301:
Author: ASF GitHub Bot
Created on: 19/Mar/20 04:23
Start Date: 19/Mar/20 04:23
Worklog Time Spent: 10m
Work Description: youngoli commented on pull request #11144: [BEAM-3301]
Perform SDF validation (missing RestrictionTrackers).
URL: https://github.com/apache/beam/pull/11144#discussion_r394783417
##
File path: sdks/go/pkg/beam/core/graph/fn.go
##
@@ -196,13 +217,31 @@ func (f *DoFn) Name() string {
// IsSplittable returns whether the DoFn is a valid Splittable DoFn.
func (f *DoFn) IsSplittable() bool {
- return false // TODO(BEAM-3301): Implement this when we add SDFs.
+ isSdf, _ := validateSdfMethodsPresent((*Fn)(f))
Review comment:
Good point, I'll keep it in mind for the future. For this one, though, I
realized it's better just to check for the presence of a single SDF method
anyway, rather than calling that validation method.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 405941)
Time Spent: 6h (was: 5h 50m)
> Go SplittableDoFn support
> -
>
> Key: BEAM-3301
> URL: https://issues.apache.org/jira/browse/BEAM-3301
> Project: Beam
> Issue Type: Improvement
> Components: sdk-go
>Reporter: Henning Rohde
>Assignee: Daniel Oliveira
>Priority: Major
> Time Spent: 6h
> Remaining Estimate: 0h
>
> SDFs will be the only way to add streaming and liquid sharded IO for Go.
> Design doc: https://s.apache.org/splittable-do-fn
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-3301) Go SplittableDoFn support
[
https://issues.apache.org/jira/browse/BEAM-3301?focusedWorklogId=405943=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-405943
]
ASF GitHub Bot logged work on BEAM-3301:
Author: ASF GitHub Bot
Created on: 19/Mar/20 04:23
Start Date: 19/Mar/20 04:23
Worklog Time Spent: 10m
Work Description: youngoli commented on pull request #11144: [BEAM-3301]
Perform SDF validation (missing RestrictionTrackers).
URL: https://github.com/apache/beam/pull/11144#discussion_r394783450
##
File path: sdks/go/pkg/beam/core/graph/fn.go
##
@@ -569,6 +622,188 @@ func validateSideInputsNumUnknown(processFnInputs
[]funcx.FnParam, method *funcx
return nil
}
+// validateSdfMethods validates that all SDF methods are either present or
+// missing in a Fn, and then returns true if they're present and false
+// otherwise. If some are present and some are missing, it returns an error.
+func validateSdfMethodsPresent(fn *Fn) (bool, error) {
+ // Check if first sdf method is present or not, and compare all
subsequent
+ // methods to that result. If there's a mismatch, then we only fail
after
+ // finishing the loop so we can output all the missing methods.
+ missing := make([]string, 0)
+ var first, fail bool
+
+ for i, name := range sdfNames {
+ _, ok := fn.methods[name]
+ if !ok {
+ missing = append(missing, name)
+ }
+
+ if i == 0 {
+ first = ok
+ } else if ok != first {
+ fail = true
+ }
+ }
+
+ if fail {
+ err := errors.Errorf("not all SplittableDoFn methods are
present. Missing methods: %v", missing)
+ return false, err
+ }
+
+ return first, nil
Review comment:
That looks much more readable and easy to follow than the current one. Done.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 405943)
Time Spent: 6h 20m (was: 6h 10m)
> Go SplittableDoFn support
> -
>
> Key: BEAM-3301
> URL: https://issues.apache.org/jira/browse/BEAM-3301
> Project: Beam
> Issue Type: Improvement
> Components: sdk-go
>Reporter: Henning Rohde
>Assignee: Daniel Oliveira
>Priority: Major
> Time Spent: 6h 20m
> Remaining Estimate: 0h
>
> SDFs will be the only way to add streaming and liquid sharded IO for Go.
> Design doc: https://s.apache.org/splittable-do-fn
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-3301) Go SplittableDoFn support
[
https://issues.apache.org/jira/browse/BEAM-3301?focusedWorklogId=405944=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-405944
]
ASF GitHub Bot logged work on BEAM-3301:
Author: ASF GitHub Bot
Created on: 19/Mar/20 04:23
Start Date: 19/Mar/20 04:23
Worklog Time Spent: 10m
Work Description: youngoli commented on pull request #11144: [BEAM-3301]
Perform SDF validation (missing RestrictionTrackers).
URL: https://github.com/apache/beam/pull/11144#discussion_r394783477
##
File path: sdks/go/pkg/beam/core/graph/fn_test.go
##
@@ -470,6 +542,169 @@ func (fn *BadDoFnAmbiguousSideInput) StartBundle(bool) {
func (fn *BadDoFnAmbiguousSideInput) FinishBundle(bool) {
}
+// Examples of correct SplittableDoFn signatures
+
+type RestT struct{}
+
+type GoodSdf struct {
Review comment:
I was just skipping it because this commit hasn't added restriction trackers
yet, but adding a TODO to remind me to change it later is a good idea. Done.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 405944)
Time Spent: 6.5h (was: 6h 20m)
> Go SplittableDoFn support
> -
>
> Key: BEAM-3301
> URL: https://issues.apache.org/jira/browse/BEAM-3301
> Project: Beam
> Issue Type: Improvement
> Components: sdk-go
>Reporter: Henning Rohde
>Assignee: Daniel Oliveira
>Priority: Major
> Time Spent: 6.5h
> Remaining Estimate: 0h
>
> SDFs will be the only way to add streaming and liquid sharded IO for Go.
> Design doc: https://s.apache.org/splittable-do-fn
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-3301) Go SplittableDoFn support
[
https://issues.apache.org/jira/browse/BEAM-3301?focusedWorklogId=405942=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-405942
]
ASF GitHub Bot logged work on BEAM-3301:
Author: ASF GitHub Bot
Created on: 19/Mar/20 04:23
Start Date: 19/Mar/20 04:23
Worklog Time Spent: 10m
Work Description: youngoli commented on pull request #11144: [BEAM-3301]
Perform SDF validation (missing RestrictionTrackers).
URL: https://github.com/apache/beam/pull/11144#discussion_r394783430
##
File path: sdks/go/pkg/beam/core/graph/fn.go
##
@@ -569,6 +622,188 @@ func validateSideInputsNumUnknown(processFnInputs
[]funcx.FnParam, method *funcx
return nil
}
+// validateSdfMethods validates that all SDF methods are either present or
+// missing in a Fn, and then returns true if they're present and false
+// otherwise. If some are present and some are missing, it returns an error.
+func validateSdfMethodsPresent(fn *Fn) (bool, error) {
+ // Check if first sdf method is present or not, and compare all
subsequent
+ // methods to that result. If there's a mismatch, then we only fail
after
+ // finishing the loop so we can output all the missing methods.
+ missing := make([]string, 0)
Review comment:
Done. I forgot that append works on nil slices.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 405942)
Time Spent: 6h 10m (was: 6h)
> Go SplittableDoFn support
> -
>
> Key: BEAM-3301
> URL: https://issues.apache.org/jira/browse/BEAM-3301
> Project: Beam
> Issue Type: Improvement
> Components: sdk-go
>Reporter: Henning Rohde
>Assignee: Daniel Oliveira
>Priority: Major
> Time Spent: 6h 10m
> Remaining Estimate: 0h
>
> SDFs will be the only way to add streaming and liquid sharded IO for Go.
> Design doc: https://s.apache.org/splittable-do-fn
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9533) Replace *-gcp/*-aws tox suites with *-cloud suites to run unit tests for both
[ https://issues.apache.org/jira/browse/BEAM-9533?focusedWorklogId=405935=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-405935 ] ASF GitHub Bot logged work on BEAM-9533: Author: ASF GitHub Bot Created on: 19/Mar/20 03:59 Start Date: 19/Mar/20 03:59 Worklog Time Spent: 10m Work Description: pabloem commented on issue #11158: [BEAM-9533] Fixing tox.ini variants URL: https://github.com/apache/beam/pull/11158#issuecomment-600976187 Run Portable_Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 405935) Time Spent: 1h 10m (was: 1h) > Replace *-gcp/*-aws tox suites with *-cloud suites to run unit tests for both > - > > Key: BEAM-9533 > URL: https://issues.apache.org/jira/browse/BEAM-9533 > Project: Beam > Issue Type: Bug > Components: testing >Reporter: Pablo Estrada >Assignee: Pablo Estrada >Priority: Minor > Time Spent: 1h 10m > Remaining Estimate: 0h > > Currently there are `py37-gcp`, py37-aws test suites. Let's consolidate all > of them into py37-cloud, along with other py35-gcp, py27-gcp, etc. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9470) :sdks:java:io:kinesis:test is flaky
[ https://issues.apache.org/jira/browse/BEAM-9470?focusedWorklogId=405908=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-405908 ] ASF GitHub Bot logged work on BEAM-9470: Author: ASF GitHub Bot Created on: 19/Mar/20 02:14 Start Date: 19/Mar/20 02:14 Worklog Time Spent: 10m Work Description: jfarr commented on issue #11090: [BEAM-9470] :sdks:java:io:kinesis:test is flaky URL: https://github.com/apache/beam/pull/11090#issuecomment-600950516 > I think it's almost LGTM, just requires some Javadoc updates on using rate limiter in multiple threads. Thanks @aromanenko-dev, that's done now. And thank you @suztomo for pushing me on this and keeping me honest. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 405908) Time Spent: 8h 50m (was: 8h 40m) > :sdks:java:io:kinesis:test is flaky > --- > > Key: BEAM-9470 > URL: https://issues.apache.org/jira/browse/BEAM-9470 > Project: Beam > Issue Type: Test > Components: io-java-kinesis >Reporter: Etienne Chauchot >Assignee: Jonothan Farr >Priority: Major > Time Spent: 8h 50m > Remaining Estimate: 0h > > [https://scans.gradle.com/s/b4jmmu72ku5jc/console-log?task=:sdks:java:io:kinesis:test] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9542) Where the BeamModulePlugin's force is needed?
[
https://issues.apache.org/jira/browse/BEAM-9542?focusedWorklogId=405898=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-405898
]
ASF GitHub Bot logged work on BEAM-9542:
Author: ASF GitHub Bot
Created on: 19/Mar/20 01:36
Start Date: 19/Mar/20 01:36
Worklog Time Spent: 10m
Work Description: suztomo commented on pull request #11168: [BEAM-9542]
Limit and clarify the effect of "force" in Java build
URL: https://github.com/apache/beam/pull/11168#discussion_r394739168
##
File path:
buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
##
@@ -1258,23 +1258,20 @@ class BeamModulePlugin implements Plugin {
exclude group: "org.hamcrest", module: "hamcrest-all"
}
- // Force usage of the libraries defined within our common set found in
the root
+ // For tests, force usage of the libraries defined within our common set
found in the root
// build.gradle instead of using Gradles default dependency resolution
mechanism
// which chooses the latest version available.
//
// TODO: Figure out whether we should force all dependency conflict
resolution
- // to occur in the "shadow" and "shadowTest" configurations.
+ // to occur in the "shadowTest" configurations.
project.configurations.all { config ->
-// When running beam_Dependency_Check, resolutionStrategy should not
be used; otherwise
-// gradle-versions-plugin does not report the latest versions of the
dependencies.
-def startTasks = project.gradle.startParameter.taskNames
-def inDependencyUpdates = 'dependencyUpdates' in startTasks ||
'runBeamDependencyCheck' in startTasks
Review comment:
I confirmed `./gradlew runBeamDependencyCheck` works as expected with this
PR.
https://gist.github.com/suztomo/7077fcea856fba6761f17da3977b6c82
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 405898)
Time Spent: 1h 20m (was: 1h 10m)
> Where the BeamModulePlugin's force is needed?
> -
>
> Key: BEAM-9542
> URL: https://issues.apache.org/jira/browse/BEAM-9542
> Project: Beam
> Issue Type: Task
> Components: build-system
>Reporter: Tomo Suzuki
>Assignee: Tomo Suzuki
>Priority: Major
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> Followup of https://github.com/apache/beam/pull/11156#discussion_r394408735
> {noformat}
> > Task :sdks:java:core:compileTestJava FAILED
> /Users/suztomo/beam/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/DeduplicateTest.java:21:
> error: cannot find symbol
> import static org.junit.Assert.assertThrows;
> ^
> symbol: static assertThrows
> location: class
> /Users/suztomo/beam/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/splittabledofn/WatermarkEstimatorsTest.java:21:
> error: cannot find symbol
> import static org.junit.Assert.assertThrows;
> ^
> symbol: static assertThrows
> location: class
> Note: Some input files use or override a deprecated API.
> Note: Recompile with -Xlint:deprecation for details.
> 2 errors
> <-> 36% EXECUTING [19m 37s]
> {noformat}
> Memo for my Mac:
> {noformat}
> suztomo-macbookpro44% ./gradlew -p sdks/java check -x
> extensions:sql:zetasql:check -x harness:test -x io:jdbc:test -x
> io:kafka:test -x io:solr:test -x core:test
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9542) Where the BeamModulePlugin's force is needed?
[
https://issues.apache.org/jira/browse/BEAM-9542?focusedWorklogId=405900=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-405900
]
ASF GitHub Bot logged work on BEAM-9542:
Author: ASF GitHub Bot
Created on: 19/Mar/20 01:36
Start Date: 19/Mar/20 01:36
Worklog Time Spent: 10m
Work Description: suztomo commented on pull request #11168: [BEAM-9542]
Limit and clarify the effect of "force" in Java build
URL: https://github.com/apache/beam/pull/11168#discussion_r394739488
##
File path:
buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
##
@@ -1258,23 +1258,20 @@ class BeamModulePlugin implements Plugin {
exclude group: "org.hamcrest", module: "hamcrest-all"
}
- // Force usage of the libraries defined within our common set found in
the root
+ // For tests, force usage of the libraries defined within our common set
found in the root
// build.gradle instead of using Gradles default dependency resolution
mechanism
// which chooses the latest version available.
//
// TODO: Figure out whether we should force all dependency conflict
resolution
- // to occur in the "shadow" and "shadowTest" configurations.
+ // to occur in the "shadowTest" configurations.
project.configurations.all { config ->
-// When running beam_Dependency_Check, resolutionStrategy should not
be used; otherwise
-// gradle-versions-plugin does not report the latest versions of the
dependencies.
-def startTasks = project.gradle.startParameter.taskNames
-def inDependencyUpdates = 'dependencyUpdates' in startTasks ||
'runBeamDependencyCheck' in startTasks
-
-// The "errorprone" configuration controls the classpath used by
errorprone static analysis, which
-// has different dependencies than our project.
-if (config.getName() != "errorprone" && !inDependencyUpdates) {
Review comment:
Before this PR, the effect of `force` was unclear; this file had to
special-case "errorprone" and "runBeamDependencyCheck" when unexpected behavior
happened.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 405900)
Time Spent: 1.5h (was: 1h 20m)
> Where the BeamModulePlugin's force is needed?
> -
>
> Key: BEAM-9542
> URL: https://issues.apache.org/jira/browse/BEAM-9542
> Project: Beam
> Issue Type: Task
> Components: build-system
>Reporter: Tomo Suzuki
>Assignee: Tomo Suzuki
>Priority: Major
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> Followup of https://github.com/apache/beam/pull/11156#discussion_r394408735
> {noformat}
> > Task :sdks:java:core:compileTestJava FAILED
> /Users/suztomo/beam/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/DeduplicateTest.java:21:
> error: cannot find symbol
> import static org.junit.Assert.assertThrows;
> ^
> symbol: static assertThrows
> location: class
> /Users/suztomo/beam/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/splittabledofn/WatermarkEstimatorsTest.java:21:
> error: cannot find symbol
> import static org.junit.Assert.assertThrows;
> ^
> symbol: static assertThrows
> location: class
> Note: Some input files use or override a deprecated API.
> Note: Recompile with -Xlint:deprecation for details.
> 2 errors
> <-> 36% EXECUTING [19m 37s]
> {noformat}
> Memo for my Mac:
> {noformat}
> suztomo-macbookpro44% ./gradlew -p sdks/java check -x
> extensions:sql:zetasql:check -x harness:test -x io:jdbc:test -x
> io:kafka:test -x io:solr:test -x core:test
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9542) Where the BeamModulePlugin's force is needed?
[
https://issues.apache.org/jira/browse/BEAM-9542?focusedWorklogId=405897=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-405897
]
ASF GitHub Bot logged work on BEAM-9542:
Author: ASF GitHub Bot
Created on: 19/Mar/20 01:36
Start Date: 19/Mar/20 01:36
Worklog Time Spent: 10m
Work Description: suztomo commented on pull request #11168: [BEAM-9542]
Limit and clarify the effect of "force" in Java build
URL: https://github.com/apache/beam/pull/11168#discussion_r394739600
##
File path: sdks/java/core/build.gradle
##
@@ -87,6 +87,7 @@ dependencies {
provided 'io.airlift:aircompressor:0.16'
provided 'com.facebook.presto.hadoop:hadoop-apache2:3.2.0-1'
shadowTest library.java.jackson_dataformat_yaml
+ shadowTest library.java.junit
Review comment:
This module uses JUnit.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 405897)
Time Spent: 1h 20m (was: 1h 10m)
> Where the BeamModulePlugin's force is needed?
> -
>
> Key: BEAM-9542
> URL: https://issues.apache.org/jira/browse/BEAM-9542
> Project: Beam
> Issue Type: Task
> Components: build-system
>Reporter: Tomo Suzuki
>Assignee: Tomo Suzuki
>Priority: Major
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> Followup of https://github.com/apache/beam/pull/11156#discussion_r394408735
> {noformat}
> > Task :sdks:java:core:compileTestJava FAILED
> /Users/suztomo/beam/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/DeduplicateTest.java:21:
> error: cannot find symbol
> import static org.junit.Assert.assertThrows;
> ^
> symbol: static assertThrows
> location: class
> /Users/suztomo/beam/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/splittabledofn/WatermarkEstimatorsTest.java:21:
> error: cannot find symbol
> import static org.junit.Assert.assertThrows;
> ^
> symbol: static assertThrows
> location: class
> Note: Some input files use or override a deprecated API.
> Note: Recompile with -Xlint:deprecation for details.
> 2 errors
> <-> 36% EXECUTING [19m 37s]
> {noformat}
> Memo for my Mac:
> {noformat}
> suztomo-macbookpro44% ./gradlew -p sdks/java check -x
> extensions:sql:zetasql:check -x harness:test -x io:jdbc:test -x
> io:kafka:test -x io:solr:test -x core:test
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9542) Where the BeamModulePlugin's force is needed?
[
https://issues.apache.org/jira/browse/BEAM-9542?focusedWorklogId=405899=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-405899
]
ASF GitHub Bot logged work on BEAM-9542:
Author: ASF GitHub Bot
Created on: 19/Mar/20 01:36
Start Date: 19/Mar/20 01:36
Worklog Time Spent: 10m
Work Description: suztomo commented on pull request #11168: [BEAM-9542]
Limit and clarify the effect of "force" in Java build
URL: https://github.com/apache/beam/pull/11168#discussion_r394739760
##
File path: sdks/java/io/google-cloud-platform/build.gradle
##
@@ -75,6 +75,7 @@ dependencies {
compile library.java.proto_google_common_protos
compile library.java.protobuf_java
compile library.java.slf4j_api
+ compileOnly library.java.hamcrest_library // should not appear in users'
class path
Review comment:
This module's `org.apache.beam.sdk.io.gcp.pubsub.TestPubsub` in "main" (not
test) uses Hamcrest's matchers. Therefore declaring the dependency.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 405899)
Time Spent: 1.5h (was: 1h 20m)
> Where the BeamModulePlugin's force is needed?
> -
>
> Key: BEAM-9542
> URL: https://issues.apache.org/jira/browse/BEAM-9542
> Project: Beam
> Issue Type: Task
> Components: build-system
>Reporter: Tomo Suzuki
>Assignee: Tomo Suzuki
>Priority: Major
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> Followup of https://github.com/apache/beam/pull/11156#discussion_r394408735
> {noformat}
> > Task :sdks:java:core:compileTestJava FAILED
> /Users/suztomo/beam/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/DeduplicateTest.java:21:
> error: cannot find symbol
> import static org.junit.Assert.assertThrows;
> ^
> symbol: static assertThrows
> location: class
> /Users/suztomo/beam/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/splittabledofn/WatermarkEstimatorsTest.java:21:
> error: cannot find symbol
> import static org.junit.Assert.assertThrows;
> ^
> symbol: static assertThrows
> location: class
> Note: Some input files use or override a deprecated API.
> Note: Recompile with -Xlint:deprecation for details.
> 2 errors
> <-> 36% EXECUTING [19m 37s]
> {noformat}
> Memo for my Mac:
> {noformat}
> suztomo-macbookpro44% ./gradlew -p sdks/java check -x
> extensions:sql:zetasql:check -x harness:test -x io:jdbc:test -x
> io:kafka:test -x io:solr:test -x core:test
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9553) portableWordCount test using incorrect job server
[ https://issues.apache.org/jira/browse/BEAM-9553?focusedWorklogId=405894=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-405894 ] ASF GitHub Bot logged work on BEAM-9553: Author: ASF GitHub Bot Created on: 19/Mar/20 01:24 Start Date: 19/Mar/20 01:24 Worklog Time Spent: 10m Work Description: ibzib commented on pull request #11170: [BEAM-9553] Use latest Flink job server image as default. URL: https://github.com/apache/beam/pull/11170 cc @lukecwik Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] Update `CHANGES.md` with noteworthy changes. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier). Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/)
[jira] [Work logged] (BEAM-9542) Where the BeamModulePlugin's force is needed?
[
https://issues.apache.org/jira/browse/BEAM-9542?focusedWorklogId=405888=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-405888
]
ASF GitHub Bot logged work on BEAM-9542:
Author: ASF GitHub Bot
Created on: 19/Mar/20 01:08
Start Date: 19/Mar/20 01:08
Worklog Time Spent: 10m
Work Description: aaltay commented on issue #11168: [BEAM-9542] Limit and
clarify the effect of "force" in Java build
URL: https://github.com/apache/beam/pull/11168#issuecomment-600933708
Run Java HadoopFormatIO Performance Test
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 405888)
Time Spent: 0.5h (was: 20m)
> Where the BeamModulePlugin's force is needed?
> -
>
> Key: BEAM-9542
> URL: https://issues.apache.org/jira/browse/BEAM-9542
> Project: Beam
> Issue Type: Task
> Components: build-system
>Reporter: Tomo Suzuki
>Assignee: Tomo Suzuki
>Priority: Major
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Followup of https://github.com/apache/beam/pull/11156#discussion_r394408735
> {noformat}
> > Task :sdks:java:core:compileTestJava FAILED
> /Users/suztomo/beam/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/DeduplicateTest.java:21:
> error: cannot find symbol
> import static org.junit.Assert.assertThrows;
> ^
> symbol: static assertThrows
> location: class
> /Users/suztomo/beam/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/splittabledofn/WatermarkEstimatorsTest.java:21:
> error: cannot find symbol
> import static org.junit.Assert.assertThrows;
> ^
> symbol: static assertThrows
> location: class
> Note: Some input files use or override a deprecated API.
> Note: Recompile with -Xlint:deprecation for details.
> 2 errors
> <-> 36% EXECUTING [19m 37s]
> {noformat}
> Memo for my Mac:
> {noformat}
> suztomo-macbookpro44% ./gradlew -p sdks/java check -x
> extensions:sql:zetasql:check -x harness:test -x io:jdbc:test -x
> io:kafka:test -x io:solr:test -x core:test
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9444) Shall we use GCP Libraries BOM to specify Google-related library versions?
[
https://issues.apache.org/jira/browse/BEAM-9444?focusedWorklogId=405887=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-405887
]
ASF GitHub Bot logged work on BEAM-9444:
Author: ASF GitHub Bot
Created on: 19/Mar/20 01:07
Start Date: 19/Mar/20 01:07
Worklog Time Spent: 10m
Work Description: suztomo commented on issue #11156: [BEAM-9444] Use GCP
Libraries BOM for Google Cloud Dependencies
URL: https://github.com/apache/beam/pull/11156#issuecomment-600933706
@aaltay Thank you.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 405887)
Time Spent: 7h 10m (was: 7h)
> Shall we use GCP Libraries BOM to specify Google-related library versions?
> --
>
> Key: BEAM-9444
> URL: https://issues.apache.org/jira/browse/BEAM-9444
> Project: Beam
> Issue Type: Task
> Components: build-system
>Reporter: Tomo Suzuki
>Assignee: Tomo Suzuki
>Priority: Major
> Attachments: Screen Shot 2020-03-13 at 13.33.01.png, Screen Shot
> 2020-03-17 at 16.01.16.png
>
> Time Spent: 7h 10m
> Remaining Estimate: 0h
>
> Shall we use GCP Libraries BOM to specify Google-related library versions?
>
> I've been working on Beam's dependency upgrades in the past few months. I
> think it's time to consider a long-term solution to keep the libraries
> up-to-date with small maintenance effort. To achieve that, I propose Beam to
> use GCP Libraries BOM to set the Google-related library versions, rather than
> trying to make changes in each of ~30 Google libraries.
>
> h1. Background
> A BOM is pom.xml that provides dependencyManagement to importing projects.
>
> GCP Libraries BOM is a BOM that includes many Google Cloud related libraries
> + gRPC + protobuf. We (Google Cloud Java Diamond Dependency team) maintain
> the BOM so that the set of the libraries are compatible with each other.
>
> h1. Implementation
> Notes for obstacles.
> h2. BeamModulePlugin's "force" does not take BOM into account (thus fails)
> {{forcedModules}} via version resolution strategy is playing bad. This causes
> {noformat}
> A problem occurred evaluating project ':sdks:java:extensions:sql'.
> Could not resolve all dependencies for configuration
> ':sdks:java:extensions:sql:fmppTemplates'.
> Invalid format: 'com.google.cloud:google-cloud-core'. Group, name and version
> cannot be empty. Correct example: 'org.gradle:gradle-core:1.0'{noformat}
> !Screen Shot 2020-03-13 at 13.33.01.png|width=489,height=287!
>
> h2. :sdks:java:maven-archetypes:examples needs the version of
> google-http-client
> The task requires the version for the library:
> {code:java}
> 'google-http-client.version':
> dependencies.create(project.library.java.google_http_client).getVersion(),
> {code}
> This would generate NullPointerException. Running gradlew without the
> subproject:
>
> {code:java}
> ./gradlew -p sdks/java check -x :sdks:java:maven-archetypes:examples:check
> {code}
> h1. Problem in Gradle-generated pom files
> The generated Maven artifact POM has invalid data due to the BOM change. For
> example my locally installed
> {{~/.m2/repository/org/apache/beam/beam-sdks-java-io-google-cloud-platform/2.21.0-SNAPSHOT/beam-sdks-java-io-google-cloud-platform-2.21.0-SNAPSHOT.pom}}
> had the following problems.
> h2. The GCP Libraries BOM showing up in dependencies section:
> {noformat}
>
>
> com.google.cloud
> libraries-bom
> 4.2.0
> compile
>
>
> com.google.guava
> guava-jdk5
> ...
>
>
> {noformat}
> h2. The artifact that use the BOM in Gradle is missing version in the
> dependency.
> {noformat}
>
> com.google.api
> gax
>
> compile
> ...
>
> {noformat}
> h1. DependencyManagement section in generated pom.xml
> How can I check whether a entry in dependencies is "platform"?
> !Screen Shot 2020-03-17 at 16.01.16.png|width=504,height=344!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9542) Where the BeamModulePlugin's force is needed?
[
https://issues.apache.org/jira/browse/BEAM-9542?focusedWorklogId=405889=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-405889
]
ASF GitHub Bot logged work on BEAM-9542:
Author: ASF GitHub Bot
Created on: 19/Mar/20 01:08
Start Date: 19/Mar/20 01:08
Worklog Time Spent: 10m
Work Description: aaltay commented on issue #11168: [BEAM-9542] Limit and
clarify the effect of "force" in Java build
URL: https://github.com/apache/beam/pull/11168#issuecomment-600933744
Run BigQueryIO Streaming Performance Test Java
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 405889)
Time Spent: 40m (was: 0.5h)
> Where the BeamModulePlugin's force is needed?
> -
>
> Key: BEAM-9542
> URL: https://issues.apache.org/jira/browse/BEAM-9542
> Project: Beam
> Issue Type: Task
> Components: build-system
>Reporter: Tomo Suzuki
>Assignee: Tomo Suzuki
>Priority: Major
> Time Spent: 40m
> Remaining Estimate: 0h
>
> Followup of https://github.com/apache/beam/pull/11156#discussion_r394408735
> {noformat}
> > Task :sdks:java:core:compileTestJava FAILED
> /Users/suztomo/beam/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/DeduplicateTest.java:21:
> error: cannot find symbol
> import static org.junit.Assert.assertThrows;
> ^
> symbol: static assertThrows
> location: class
> /Users/suztomo/beam/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/splittabledofn/WatermarkEstimatorsTest.java:21:
> error: cannot find symbol
> import static org.junit.Assert.assertThrows;
> ^
> symbol: static assertThrows
> location: class
> Note: Some input files use or override a deprecated API.
> Note: Recompile with -Xlint:deprecation for details.
> 2 errors
> <-> 36% EXECUTING [19m 37s]
> {noformat}
> Memo for my Mac:
> {noformat}
> suztomo-macbookpro44% ./gradlew -p sdks/java check -x
> extensions:sql:zetasql:check -x harness:test -x io:jdbc:test -x
> io:kafka:test -x io:solr:test -x core:test
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9542) Where the BeamModulePlugin's force is needed?
[
https://issues.apache.org/jira/browse/BEAM-9542?focusedWorklogId=405892=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-405892
]
ASF GitHub Bot logged work on BEAM-9542:
Author: ASF GitHub Bot
Created on: 19/Mar/20 01:08
Start Date: 19/Mar/20 01:08
Worklog Time Spent: 10m
Work Description: aaltay commented on issue #11168: [BEAM-9542] Limit and
clarify the effect of "force" in Java build
URL: https://github.com/apache/beam/pull/11168#issuecomment-600933872
Run SQL Postcommit
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 405892)
Time Spent: 1h 10m (was: 1h)
> Where the BeamModulePlugin's force is needed?
> -
>
> Key: BEAM-9542
> URL: https://issues.apache.org/jira/browse/BEAM-9542
> Project: Beam
> Issue Type: Task
> Components: build-system
>Reporter: Tomo Suzuki
>Assignee: Tomo Suzuki
>Priority: Major
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> Followup of https://github.com/apache/beam/pull/11156#discussion_r394408735
> {noformat}
> > Task :sdks:java:core:compileTestJava FAILED
> /Users/suztomo/beam/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/DeduplicateTest.java:21:
> error: cannot find symbol
> import static org.junit.Assert.assertThrows;
> ^
> symbol: static assertThrows
> location: class
> /Users/suztomo/beam/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/splittabledofn/WatermarkEstimatorsTest.java:21:
> error: cannot find symbol
> import static org.junit.Assert.assertThrows;
> ^
> symbol: static assertThrows
> location: class
> Note: Some input files use or override a deprecated API.
> Note: Recompile with -Xlint:deprecation for details.
> 2 errors
> <-> 36% EXECUTING [19m 37s]
> {noformat}
> Memo for my Mac:
> {noformat}
> suztomo-macbookpro44% ./gradlew -p sdks/java check -x
> extensions:sql:zetasql:check -x harness:test -x io:jdbc:test -x
> io:kafka:test -x io:solr:test -x core:test
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9542) Where the BeamModulePlugin's force is needed?
[
https://issues.apache.org/jira/browse/BEAM-9542?focusedWorklogId=405890=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-405890
]
ASF GitHub Bot logged work on BEAM-9542:
Author: ASF GitHub Bot
Created on: 19/Mar/20 01:08
Start Date: 19/Mar/20 01:08
Worklog Time Spent: 10m
Work Description: aaltay commented on issue #11168: [BEAM-9542] Limit and
clarify the effect of "force" in Java build
URL: https://github.com/apache/beam/pull/11168#issuecomment-600933769
Run Dataflow ValidatesRunner
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 405890)
Time Spent: 50m (was: 40m)
> Where the BeamModulePlugin's force is needed?
> -
>
> Key: BEAM-9542
> URL: https://issues.apache.org/jira/browse/BEAM-9542
> Project: Beam
> Issue Type: Task
> Components: build-system
>Reporter: Tomo Suzuki
>Assignee: Tomo Suzuki
>Priority: Major
> Time Spent: 50m
> Remaining Estimate: 0h
>
> Followup of https://github.com/apache/beam/pull/11156#discussion_r394408735
> {noformat}
> > Task :sdks:java:core:compileTestJava FAILED
> /Users/suztomo/beam/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/DeduplicateTest.java:21:
> error: cannot find symbol
> import static org.junit.Assert.assertThrows;
> ^
> symbol: static assertThrows
> location: class
> /Users/suztomo/beam/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/splittabledofn/WatermarkEstimatorsTest.java:21:
> error: cannot find symbol
> import static org.junit.Assert.assertThrows;
> ^
> symbol: static assertThrows
> location: class
> Note: Some input files use or override a deprecated API.
> Note: Recompile with -Xlint:deprecation for details.
> 2 errors
> <-> 36% EXECUTING [19m 37s]
> {noformat}
> Memo for my Mac:
> {noformat}
> suztomo-macbookpro44% ./gradlew -p sdks/java check -x
> extensions:sql:zetasql:check -x harness:test -x io:jdbc:test -x
> io:kafka:test -x io:solr:test -x core:test
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9542) Where the BeamModulePlugin's force is needed?
[
https://issues.apache.org/jira/browse/BEAM-9542?focusedWorklogId=405891=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-405891
]
ASF GitHub Bot logged work on BEAM-9542:
Author: ASF GitHub Bot
Created on: 19/Mar/20 01:08
Start Date: 19/Mar/20 01:08
Worklog Time Spent: 10m
Work Description: aaltay commented on issue #11168: [BEAM-9542] Limit and
clarify the effect of "force" in Java build
URL: https://github.com/apache/beam/pull/11168#issuecomment-600933820
Run Spark ValidatesRunner
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 405891)
Time Spent: 1h (was: 50m)
> Where the BeamModulePlugin's force is needed?
> -
>
> Key: BEAM-9542
> URL: https://issues.apache.org/jira/browse/BEAM-9542
> Project: Beam
> Issue Type: Task
> Components: build-system
>Reporter: Tomo Suzuki
>Assignee: Tomo Suzuki
>Priority: Major
> Time Spent: 1h
> Remaining Estimate: 0h
>
> Followup of https://github.com/apache/beam/pull/11156#discussion_r394408735
> {noformat}
> > Task :sdks:java:core:compileTestJava FAILED
> /Users/suztomo/beam/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/DeduplicateTest.java:21:
> error: cannot find symbol
> import static org.junit.Assert.assertThrows;
> ^
> symbol: static assertThrows
> location: class
> /Users/suztomo/beam/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/splittabledofn/WatermarkEstimatorsTest.java:21:
> error: cannot find symbol
> import static org.junit.Assert.assertThrows;
> ^
> symbol: static assertThrows
> location: class
> Note: Some input files use or override a deprecated API.
> Note: Recompile with -Xlint:deprecation for details.
> 2 errors
> <-> 36% EXECUTING [19m 37s]
> {noformat}
> Memo for my Mac:
> {noformat}
> suztomo-macbookpro44% ./gradlew -p sdks/java check -x
> extensions:sql:zetasql:check -x harness:test -x io:jdbc:test -x
> io:kafka:test -x io:solr:test -x core:test
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9444) Shall we use GCP Libraries BOM to specify Google-related library versions?
[
https://issues.apache.org/jira/browse/BEAM-9444?focusedWorklogId=405885=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-405885
]
ASF GitHub Bot logged work on BEAM-9444:
Author: ASF GitHub Bot
Created on: 19/Mar/20 01:07
Start Date: 19/Mar/20 01:07
Worklog Time Spent: 10m
Work Description: aaltay commented on issue #11156: [BEAM-9444] Use GCP
Libraries BOM for Google Cloud Dependencies
URL: https://github.com/apache/beam/pull/11156#issuecomment-600933591
Run Python PreCommit
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 405885)
Time Spent: 7h (was: 6h 50m)
> Shall we use GCP Libraries BOM to specify Google-related library versions?
> --
>
> Key: BEAM-9444
> URL: https://issues.apache.org/jira/browse/BEAM-9444
> Project: Beam
> Issue Type: Task
> Components: build-system
>Reporter: Tomo Suzuki
>Assignee: Tomo Suzuki
>Priority: Major
> Attachments: Screen Shot 2020-03-13 at 13.33.01.png, Screen Shot
> 2020-03-17 at 16.01.16.png
>
> Time Spent: 7h
> Remaining Estimate: 0h
>
> Shall we use GCP Libraries BOM to specify Google-related library versions?
>
> I've been working on Beam's dependency upgrades in the past few months. I
> think it's time to consider a long-term solution to keep the libraries
> up-to-date with small maintenance effort. To achieve that, I propose Beam to
> use GCP Libraries BOM to set the Google-related library versions, rather than
> trying to make changes in each of ~30 Google libraries.
>
> h1. Background
> A BOM is pom.xml that provides dependencyManagement to importing projects.
>
> GCP Libraries BOM is a BOM that includes many Google Cloud related libraries
> + gRPC + protobuf. We (Google Cloud Java Diamond Dependency team) maintain
> the BOM so that the set of the libraries are compatible with each other.
>
> h1. Implementation
> Notes for obstacles.
> h2. BeamModulePlugin's "force" does not take BOM into account (thus fails)
> {{forcedModules}} via version resolution strategy is playing bad. This causes
> {noformat}
> A problem occurred evaluating project ':sdks:java:extensions:sql'.
> Could not resolve all dependencies for configuration
> ':sdks:java:extensions:sql:fmppTemplates'.
> Invalid format: 'com.google.cloud:google-cloud-core'. Group, name and version
> cannot be empty. Correct example: 'org.gradle:gradle-core:1.0'{noformat}
> !Screen Shot 2020-03-13 at 13.33.01.png|width=489,height=287!
>
> h2. :sdks:java:maven-archetypes:examples needs the version of
> google-http-client
> The task requires the version for the library:
> {code:java}
> 'google-http-client.version':
> dependencies.create(project.library.java.google_http_client).getVersion(),
> {code}
> This would generate NullPointerException. Running gradlew without the
> subproject:
>
> {code:java}
> ./gradlew -p sdks/java check -x :sdks:java:maven-archetypes:examples:check
> {code}
> h1. Problem in Gradle-generated pom files
> The generated Maven artifact POM has invalid data due to the BOM change. For
> example my locally installed
> {{~/.m2/repository/org/apache/beam/beam-sdks-java-io-google-cloud-platform/2.21.0-SNAPSHOT/beam-sdks-java-io-google-cloud-platform-2.21.0-SNAPSHOT.pom}}
> had the following problems.
> h2. The GCP Libraries BOM showing up in dependencies section:
> {noformat}
>
>
> com.google.cloud
> libraries-bom
> 4.2.0
> compile
>
>
> com.google.guava
> guava-jdk5
> ...
>
>
> {noformat}
> h2. The artifact that use the BOM in Gradle is missing version in the
> dependency.
> {noformat}
>
> com.google.api
> gax
>
> compile
> ...
>
> {noformat}
> h1. DependencyManagement section in generated pom.xml
> How can I check whether a entry in dependencies is "platform"?
> !Screen Shot 2020-03-17 at 16.01.16.png|width=504,height=344!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9542) Where the BeamModulePlugin's force is needed?
[
https://issues.apache.org/jira/browse/BEAM-9542?focusedWorklogId=405886=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-405886
]
ASF GitHub Bot logged work on BEAM-9542:
Author: ASF GitHub Bot
Created on: 19/Mar/20 01:07
Start Date: 19/Mar/20 01:07
Worklog Time Spent: 10m
Work Description: aaltay commented on issue #11168: [BEAM-9542] Limit and
clarify the effect of "force" in Java build
URL: https://github.com/apache/beam/pull/11168#issuecomment-600933671
Run Java PostCommit
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 405886)
Time Spent: 20m (was: 10m)
> Where the BeamModulePlugin's force is needed?
> -
>
> Key: BEAM-9542
> URL: https://issues.apache.org/jira/browse/BEAM-9542
> Project: Beam
> Issue Type: Task
> Components: build-system
>Reporter: Tomo Suzuki
>Assignee: Tomo Suzuki
>Priority: Major
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Followup of https://github.com/apache/beam/pull/11156#discussion_r394408735
> {noformat}
> > Task :sdks:java:core:compileTestJava FAILED
> /Users/suztomo/beam/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/DeduplicateTest.java:21:
> error: cannot find symbol
> import static org.junit.Assert.assertThrows;
> ^
> symbol: static assertThrows
> location: class
> /Users/suztomo/beam/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/splittabledofn/WatermarkEstimatorsTest.java:21:
> error: cannot find symbol
> import static org.junit.Assert.assertThrows;
> ^
> symbol: static assertThrows
> location: class
> Note: Some input files use or override a deprecated API.
> Note: Recompile with -Xlint:deprecation for details.
> 2 errors
> <-> 36% EXECUTING [19m 37s]
> {noformat}
> Memo for my Mac:
> {noformat}
> suztomo-macbookpro44% ./gradlew -p sdks/java check -x
> extensions:sql:zetasql:check -x harness:test -x io:jdbc:test -x
> io:kafka:test -x io:solr:test -x core:test
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9444) Shall we use GCP Libraries BOM to specify Google-related library versions?
[
https://issues.apache.org/jira/browse/BEAM-9444?focusedWorklogId=405884=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-405884
]
ASF GitHub Bot logged work on BEAM-9444:
Author: ASF GitHub Bot
Created on: 19/Mar/20 01:07
Start Date: 19/Mar/20 01:07
Worklog Time Spent: 10m
Work Description: aaltay commented on issue #11156: [BEAM-9444] Use GCP
Libraries BOM for Google Cloud Dependencies
URL: https://github.com/apache/beam/pull/11156#issuecomment-600933550
Run Java PreCommit
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 405884)
Time Spent: 6h 50m (was: 6h 40m)
> Shall we use GCP Libraries BOM to specify Google-related library versions?
> --
>
> Key: BEAM-9444
> URL: https://issues.apache.org/jira/browse/BEAM-9444
> Project: Beam
> Issue Type: Task
> Components: build-system
>Reporter: Tomo Suzuki
>Assignee: Tomo Suzuki
>Priority: Major
> Attachments: Screen Shot 2020-03-13 at 13.33.01.png, Screen Shot
> 2020-03-17 at 16.01.16.png
>
> Time Spent: 6h 50m
> Remaining Estimate: 0h
>
> Shall we use GCP Libraries BOM to specify Google-related library versions?
>
> I've been working on Beam's dependency upgrades in the past few months. I
> think it's time to consider a long-term solution to keep the libraries
> up-to-date with small maintenance effort. To achieve that, I propose Beam to
> use GCP Libraries BOM to set the Google-related library versions, rather than
> trying to make changes in each of ~30 Google libraries.
>
> h1. Background
> A BOM is pom.xml that provides dependencyManagement to importing projects.
>
> GCP Libraries BOM is a BOM that includes many Google Cloud related libraries
> + gRPC + protobuf. We (Google Cloud Java Diamond Dependency team) maintain
> the BOM so that the set of the libraries are compatible with each other.
>
> h1. Implementation
> Notes for obstacles.
> h2. BeamModulePlugin's "force" does not take BOM into account (thus fails)
> {{forcedModules}} via version resolution strategy is playing bad. This causes
> {noformat}
> A problem occurred evaluating project ':sdks:java:extensions:sql'.
> Could not resolve all dependencies for configuration
> ':sdks:java:extensions:sql:fmppTemplates'.
> Invalid format: 'com.google.cloud:google-cloud-core'. Group, name and version
> cannot be empty. Correct example: 'org.gradle:gradle-core:1.0'{noformat}
> !Screen Shot 2020-03-13 at 13.33.01.png|width=489,height=287!
>
> h2. :sdks:java:maven-archetypes:examples needs the version of
> google-http-client
> The task requires the version for the library:
> {code:java}
> 'google-http-client.version':
> dependencies.create(project.library.java.google_http_client).getVersion(),
> {code}
> This would generate NullPointerException. Running gradlew without the
> subproject:
>
> {code:java}
> ./gradlew -p sdks/java check -x :sdks:java:maven-archetypes:examples:check
> {code}
> h1. Problem in Gradle-generated pom files
> The generated Maven artifact POM has invalid data due to the BOM change. For
> example my locally installed
> {{~/.m2/repository/org/apache/beam/beam-sdks-java-io-google-cloud-platform/2.21.0-SNAPSHOT/beam-sdks-java-io-google-cloud-platform-2.21.0-SNAPSHOT.pom}}
> had the following problems.
> h2. The GCP Libraries BOM showing up in dependencies section:
> {noformat}
>
>
> com.google.cloud
> libraries-bom
> 4.2.0
> compile
>
>
> com.google.guava
> guava-jdk5
> ...
>
>
> {noformat}
> h2. The artifact that use the BOM in Gradle is missing version in the
> dependency.
> {noformat}
>
> com.google.api
> gax
>
> compile
> ...
>
> {noformat}
> h1. DependencyManagement section in generated pom.xml
> How can I check whether a entry in dependencies is "platform"?
> !Screen Shot 2020-03-17 at 16.01.16.png|width=504,height=344!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (BEAM-9553) portableWordCount test using incorrect job server
Kyle Weaver created BEAM-9553: - Summary: portableWordCount test using incorrect job server Key: BEAM-9553 URL: https://issues.apache.org/jira/browse/BEAM-9553 Project: Beam Issue Type: Bug Components: runner-flink Reporter: Kyle Weaver Assignee: Kyle Weaver portableWordCount test is building Flink job server 1.10 container [1], but is using 1.9 container in the test [2]. [1] https://github.com/lukecwik/incubator-beam/blob/9f4c5f66bb3979aeeebd648f76cbfaca301a1753/sdks/python/test-suites/portable/py2/build.gradle#L32 [2] https://github.com/lukecwik/incubator-beam/blob/9f4c5f66bb3979aeeebd648f76cbfaca301a1753/sdks/python/apache_beam/runners/portability/job_server.py#L185 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9553) portableWordCount test using incorrect job server
[ https://issues.apache.org/jira/browse/BEAM-9553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kyle Weaver updated BEAM-9553: -- Status: Open (was: Triage Needed) > portableWordCount test using incorrect job server > - > > Key: BEAM-9553 > URL: https://issues.apache.org/jira/browse/BEAM-9553 > Project: Beam > Issue Type: Bug > Components: runner-flink >Reporter: Kyle Weaver >Assignee: Kyle Weaver >Priority: Major > > portableWordCount test is building Flink job server 1.10 container [1], but > is using 1.9 container in the test [2]. > [1] > https://github.com/lukecwik/incubator-beam/blob/9f4c5f66bb3979aeeebd648f76cbfaca301a1753/sdks/python/test-suites/portable/py2/build.gradle#L32 > [2] > https://github.com/lukecwik/incubator-beam/blob/9f4c5f66bb3979aeeebd648f76cbfaca301a1753/sdks/python/apache_beam/runners/portability/job_server.py#L185 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-3301) Go SplittableDoFn support
[
https://issues.apache.org/jira/browse/BEAM-3301?focusedWorklogId=405882=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-405882
]
ASF GitHub Bot logged work on BEAM-3301:
Author: ASF GitHub Bot
Created on: 19/Mar/20 00:58
Start Date: 19/Mar/20 00:58
Worklog Time Spent: 10m
Work Description: lostluck commented on pull request #11144: [BEAM-3301]
Perform SDF validation (missing RestrictionTrackers).
URL: https://github.com/apache/beam/pull/11144#discussion_r394618692
##
File path: sdks/go/pkg/beam/core/graph/fn.go
##
@@ -196,13 +217,31 @@ func (f *DoFn) Name() string {
// IsSplittable returns whether the DoFn is a valid Splittable DoFn.
func (f *DoFn) IsSplittable() bool {
- return false // TODO(BEAM-3301): Implement this when we add SDFs.
+ isSdf, _ := validateSdfMethodsPresent((*Fn)(f))
Review comment:
A semantic note: the error shouldn't be ignored. Semantically if a function
returns an error, the non error return values are not guaranteed to be valid.
So conventionally, the way to handle IsSplittable here is to check for the
error, and return false if not nil, and otherwise return what the boolean is.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 405882)
Time Spent: 5h 50m (was: 5h 40m)
> Go SplittableDoFn support
> -
>
> Key: BEAM-3301
> URL: https://issues.apache.org/jira/browse/BEAM-3301
> Project: Beam
> Issue Type: Improvement
> Components: sdk-go
>Reporter: Henning Rohde
>Assignee: Daniel Oliveira
>Priority: Major
> Time Spent: 5h 50m
> Remaining Estimate: 0h
>
> SDFs will be the only way to add streaming and liquid sharded IO for Go.
> Design doc: https://s.apache.org/splittable-do-fn
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-3301) Go SplittableDoFn support
[
https://issues.apache.org/jira/browse/BEAM-3301?focusedWorklogId=405880=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-405880
]
ASF GitHub Bot logged work on BEAM-3301:
Author: ASF GitHub Bot
Created on: 19/Mar/20 00:58
Start Date: 19/Mar/20 00:58
Worklog Time Spent: 10m
Work Description: lostluck commented on pull request #11144: [BEAM-3301]
Perform SDF validation (missing RestrictionTrackers).
URL: https://github.com/apache/beam/pull/11144#discussion_r394722487
##
File path: sdks/go/pkg/beam/core/graph/fn.go
##
@@ -569,6 +622,188 @@ func validateSideInputsNumUnknown(processFnInputs
[]funcx.FnParam, method *funcx
return nil
}
+// validateSdfMethods validates that all SDF methods are either present or
+// missing in a Fn, and then returns true if they're present and false
+// otherwise. If some are present and some are missing, it returns an error.
+func validateSdfMethodsPresent(fn *Fn) (bool, error) {
+ // Check if first sdf method is present or not, and compare all
subsequent
+ // methods to that result. If there's a mismatch, then we only fail
after
+ // finishing the loop so we can output all the missing methods.
+ missing := make([]string, 0)
Review comment:
Prefer declaring empty slices using the var syntax.
var missing []string
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 405880)
Time Spent: 5h 40m (was: 5.5h)
> Go SplittableDoFn support
> -
>
> Key: BEAM-3301
> URL: https://issues.apache.org/jira/browse/BEAM-3301
> Project: Beam
> Issue Type: Improvement
> Components: sdk-go
>Reporter: Henning Rohde
>Assignee: Daniel Oliveira
>Priority: Major
> Time Spent: 5h 40m
> Remaining Estimate: 0h
>
> SDFs will be the only way to add streaming and liquid sharded IO for Go.
> Design doc: https://s.apache.org/splittable-do-fn
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-3301) Go SplittableDoFn support
[
https://issues.apache.org/jira/browse/BEAM-3301?focusedWorklogId=405881=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-405881
]
ASF GitHub Bot logged work on BEAM-3301:
Author: ASF GitHub Bot
Created on: 19/Mar/20 00:58
Start Date: 19/Mar/20 00:58
Worklog Time Spent: 10m
Work Description: lostluck commented on pull request #11144: [BEAM-3301]
Perform SDF validation (missing RestrictionTrackers).
URL: https://github.com/apache/beam/pull/11144#discussion_r394725254
##
File path: sdks/go/pkg/beam/core/graph/fn.go
##
@@ -569,6 +622,188 @@ func validateSideInputsNumUnknown(processFnInputs
[]funcx.FnParam, method *funcx
return nil
}
+// validateSdfMethods validates that all SDF methods are either present or
+// missing in a Fn, and then returns true if they're present and false
+// otherwise. If some are present and some are missing, it returns an error.
+func validateSdfMethodsPresent(fn *Fn) (bool, error) {
+ // Check if first sdf method is present or not, and compare all
subsequent
+ // methods to that result. If there's a mismatch, then we only fail
after
+ // finishing the loop so we can output all the missing methods.
+ missing := make([]string, 0)
+ var first, fail bool
+
+ for i, name := range sdfNames {
+ _, ok := fn.methods[name]
+ if !ok {
+ missing = append(missing, name)
+ }
+
+ if i == 0 {
+ first = ok
+ } else if ok != first {
+ fail = true
+ }
+ }
+
+ if fail {
+ err := errors.Errorf("not all SplittableDoFn methods are
present. Missing methods: %v", missing)
+ return false, err
+ }
+
+ return first, nil
Review comment:
I see that this function is trying to distinguish between partial coverage
or complete coverage, but I think it could be simpler.
Consider that the booleans could be removed by comparing whether
len(missing) == len(sdfNames) to check if it's simply not an SDF at all (and
thus, no error should be returned).
```
switch len(missing) {
case 0:
return true, nil
case len(sdfNames):
return false, nil
default:
err := errors.Errorf("not all SplittableDoFn methods are present. Missing
methods: %v", missing)
return false, err
}
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 405881)
Time Spent: 5h 40m (was: 5.5h)
> Go SplittableDoFn support
> -
>
> Key: BEAM-3301
> URL: https://issues.apache.org/jira/browse/BEAM-3301
> Project: Beam
> Issue Type: Improvement
> Components: sdk-go
>Reporter: Henning Rohde
>Assignee: Daniel Oliveira
>Priority: Major
> Time Spent: 5h 40m
> Remaining Estimate: 0h
>
> SDFs will be the only way to add streaming and liquid sharded IO for Go.
> Design doc: https://s.apache.org/splittable-do-fn
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-3301) Go SplittableDoFn support
[
https://issues.apache.org/jira/browse/BEAM-3301?focusedWorklogId=405879=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-405879
]
ASF GitHub Bot logged work on BEAM-3301:
Author: ASF GitHub Bot
Created on: 19/Mar/20 00:58
Start Date: 19/Mar/20 00:58
Worklog Time Spent: 10m
Work Description: lostluck commented on pull request #11144: [BEAM-3301]
Perform SDF validation (missing RestrictionTrackers).
URL: https://github.com/apache/beam/pull/11144#discussion_r394728377
##
File path: sdks/go/pkg/beam/core/graph/fn_test.go
##
@@ -470,6 +542,169 @@ func (fn *BadDoFnAmbiguousSideInput) StartBundle(bool) {
func (fn *BadDoFnAmbiguousSideInput) FinishBundle(bool) {
}
+// Examples of correct SplittableDoFn signatures
+
+type RestT struct{}
+
+type GoodSdf struct {
Review comment:
Shouldn't the ProcessElement method be adjusted here for the Restriction
tracker parameter for a "GoodSdf" ?
I guess technically if we don't care about dynamic splitting at all, it
should be allowed. It's not unreasonable to prevent it until we have the right
idea how to do that, but if so, lets put a TODO in here somewhere to make the
intent explicit.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 405879)
Time Spent: 5h 40m (was: 5.5h)
> Go SplittableDoFn support
> -
>
> Key: BEAM-3301
> URL: https://issues.apache.org/jira/browse/BEAM-3301
> Project: Beam
> Issue Type: Improvement
> Components: sdk-go
>Reporter: Henning Rohde
>Assignee: Daniel Oliveira
>Priority: Major
> Time Spent: 5h 40m
> Remaining Estimate: 0h
>
> SDFs will be the only way to add streaming and liquid sharded IO for Go.
> Design doc: https://s.apache.org/splittable-do-fn
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9444) Shall we use GCP Libraries BOM to specify Google-related library versions?
[
https://issues.apache.org/jira/browse/BEAM-9444?focusedWorklogId=405878=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-405878
]
ASF GitHub Bot logged work on BEAM-9444:
Author: ASF GitHub Bot
Created on: 19/Mar/20 00:53
Start Date: 19/Mar/20 00:53
Worklog Time Spent: 10m
Work Description: suztomo commented on pull request #11156: [BEAM-9444]
Use GCP Libraries BOM for Google Cloud Dependencies
URL: https://github.com/apache/beam/pull/11156#discussion_r394727429
##
File path:
buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
##
@@ -1274,7 +1296,9 @@ class BeamModulePlugin implements Plugin {
// has different dependencies than our project.
if (config.getName() != "errorprone" && !inDependencyUpdates) {
config.resolutionStrategy {
-force project.library.java.values()
+// Filtering versionless coordinates that depend on BOM
+def librariesWithVersion = project.library.java.values().findAll {
it.split(':').size() > 2 }
+force librariesWithVersion
Review comment:
Created PR for my suggestion https://github.com/apache/beam/pull/11168
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 405878)
Time Spent: 6h 40m (was: 6.5h)
> Shall we use GCP Libraries BOM to specify Google-related library versions?
> --
>
> Key: BEAM-9444
> URL: https://issues.apache.org/jira/browse/BEAM-9444
> Project: Beam
> Issue Type: Task
> Components: build-system
>Reporter: Tomo Suzuki
>Assignee: Tomo Suzuki
>Priority: Major
> Attachments: Screen Shot 2020-03-13 at 13.33.01.png, Screen Shot
> 2020-03-17 at 16.01.16.png
>
> Time Spent: 6h 40m
> Remaining Estimate: 0h
>
> Shall we use GCP Libraries BOM to specify Google-related library versions?
>
> I've been working on Beam's dependency upgrades in the past few months. I
> think it's time to consider a long-term solution to keep the libraries
> up-to-date with small maintenance effort. To achieve that, I propose Beam to
> use GCP Libraries BOM to set the Google-related library versions, rather than
> trying to make changes in each of ~30 Google libraries.
>
> h1. Background
> A BOM is pom.xml that provides dependencyManagement to importing projects.
>
> GCP Libraries BOM is a BOM that includes many Google Cloud related libraries
> + gRPC + protobuf. We (Google Cloud Java Diamond Dependency team) maintain
> the BOM so that the set of the libraries are compatible with each other.
>
> h1. Implementation
> Notes for obstacles.
> h2. BeamModulePlugin's "force" does not take BOM into account (thus fails)
> {{forcedModules}} via version resolution strategy is playing bad. This causes
> {noformat}
> A problem occurred evaluating project ':sdks:java:extensions:sql'.
> Could not resolve all dependencies for configuration
> ':sdks:java:extensions:sql:fmppTemplates'.
> Invalid format: 'com.google.cloud:google-cloud-core'. Group, name and version
> cannot be empty. Correct example: 'org.gradle:gradle-core:1.0'{noformat}
> !Screen Shot 2020-03-13 at 13.33.01.png|width=489,height=287!
>
> h2. :sdks:java:maven-archetypes:examples needs the version of
> google-http-client
> The task requires the version for the library:
> {code:java}
> 'google-http-client.version':
> dependencies.create(project.library.java.google_http_client).getVersion(),
> {code}
> This would generate NullPointerException. Running gradlew without the
> subproject:
>
> {code:java}
> ./gradlew -p sdks/java check -x :sdks:java:maven-archetypes:examples:check
> {code}
> h1. Problem in Gradle-generated pom files
> The generated Maven artifact POM has invalid data due to the BOM change. For
> example my locally installed
> {{~/.m2/repository/org/apache/beam/beam-sdks-java-io-google-cloud-platform/2.21.0-SNAPSHOT/beam-sdks-java-io-google-cloud-platform-2.21.0-SNAPSHOT.pom}}
> had the following problems.
> h2. The GCP Libraries BOM showing up in dependencies section:
> {noformat}
>
>
> com.google.cloud
> libraries-bom
> 4.2.0
> compile
>
>
> com.google.guava
> guava-jdk5
> ...
>
>
> {noformat}
> h2. The artifact that use the BOM in Gradle is missing version in the
> dependency.
> {noformat}
>
> com.google.api
> gax
>
[jira] [Work logged] (BEAM-8078) streaming_wordcount_debugging.py is missing a test
[ https://issues.apache.org/jira/browse/BEAM-8078?focusedWorklogId=405875=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-405875 ] ASF GitHub Bot logged work on BEAM-8078: Author: ASF GitHub Bot Created on: 19/Mar/20 00:47 Start Date: 19/Mar/20 00:47 Worklog Time Spent: 10m Work Description: aaltay commented on issue #10914: [BEAM-8078] streaming_wordcount_debugging.py is missing a test URL: https://github.com/apache/beam/pull/10914#issuecomment-600928606 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 405875) Time Spent: 2h 50m (was: 2h 40m) > streaming_wordcount_debugging.py is missing a test > -- > > Key: BEAM-8078 > URL: https://issues.apache.org/jira/browse/BEAM-8078 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Reporter: Udi Meiri >Assignee: Aleksey Vysotin >Priority: Minor > Labels: beginner, easy, newbie, starter > Time Spent: 2h 50m > Remaining Estimate: 0h > > It's example code and should have a basic_test (like the other wordcount > variants in [1]) to at least verify that it runs in the latest Beam release. > [1] > https://github.com/apache/beam/tree/master/sdks/python/apache_beam/examples -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9552) TestPubsub ACK deadline is too short
[ https://issues.apache.org/jira/browse/BEAM-9552?focusedWorklogId=405870=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-405870 ] ASF GitHub Bot logged work on BEAM-9552: Author: ASF GitHub Bot Created on: 19/Mar/20 00:37 Start Date: 19/Mar/20 00:37 Worklog Time Spent: 10m Work Description: TheNeuralBit commented on pull request #11169: [BEAM-9552] Bump TestPubsub subscription creation ACK deadline to 60s URL: https://github.com/apache/beam/pull/11169 Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/) | --- | [![Build
[jira] [Work logged] (BEAM-9383) Staging Dataflow artifacts from environment
[
https://issues.apache.org/jira/browse/BEAM-9383?focusedWorklogId=405868=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-405868
]
ASF GitHub Bot logged work on BEAM-9383:
Author: ASF GitHub Bot
Created on: 19/Mar/20 00:35
Start Date: 19/Mar/20 00:35
Worklog Time Spent: 10m
Work Description: ihji commented on pull request #11039: [BEAM-9383]
Staging Dataflow artifacts from environment
URL: https://github.com/apache/beam/pull/11039#discussion_r394722832
##
File path:
runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
##
@@ -752,6 +759,27 @@ private Debuggee registerDebuggee(CloudDebugger
debuggerClient, String uniquifie
}
}
+ private List stageArtifacts(RunnerApi.Pipeline pipeline) {
+ImmutableList.Builder filesToStageBuilder =
ImmutableList.builder();
+for (Map.Entry entry :
+pipeline.getComponents().getEnvironmentsMap().entrySet()) {
+ for (RunnerApi.ArtifactInformation info :
entry.getValue().getDependenciesList()) {
+if
(!BeamUrns.getUrn(RunnerApi.StandardArtifacts.Types.FILE).equals(info.getTypeUrn()))
{
Review comment:
Currently, DataflowRunner auto-generates staging names based on source file
names. So ROLE (STAGING_TO) is ignored anyway. We could change this behavior
later in BEAM-9455.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 405868)
Time Spent: 1h 40m (was: 1.5h)
> Staging Dataflow artifacts from environment
> ---
>
> Key: BEAM-9383
> URL: https://issues.apache.org/jira/browse/BEAM-9383
> Project: Beam
> Issue Type: Sub-task
> Components: java-fn-execution
>Reporter: Heejong Lee
>Assignee: Heejong Lee
>Priority: Major
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> Staging Dataflow artifacts from environment
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9383) Staging Dataflow artifacts from environment
[ https://issues.apache.org/jira/browse/BEAM-9383?focusedWorklogId=405869=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-405869 ] ASF GitHub Bot logged work on BEAM-9383: Author: ASF GitHub Bot Created on: 19/Mar/20 00:35 Start Date: 19/Mar/20 00:35 Worklog Time Spent: 10m Work Description: ihji commented on issue #11039: [BEAM-9383] Staging Dataflow artifacts from environment URL: https://github.com/apache/beam/pull/11039#issuecomment-600925930 @chamikaramj PTAL and rerun the tests. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 405869) Time Spent: 1h 50m (was: 1h 40m) > Staging Dataflow artifacts from environment > --- > > Key: BEAM-9383 > URL: https://issues.apache.org/jira/browse/BEAM-9383 > Project: Beam > Issue Type: Sub-task > Components: java-fn-execution >Reporter: Heejong Lee >Assignee: Heejong Lee >Priority: Major > Time Spent: 1h 50m > Remaining Estimate: 0h > > Staging Dataflow artifacts from environment -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (BEAM-9552) TestPubsub ACK deadline is too short
Brian Hulette created BEAM-9552: --- Summary: TestPubsub ACK deadline is too short Key: BEAM-9552 URL: https://issues.apache.org/jira/browse/BEAM-9552 Project: Beam Issue Type: Improvement Components: sdk-java-core, testing Reporter: Brian Hulette Assignee: Brian Hulette Increase to 60 to match PubsubUnboundedSource -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9383) Staging Dataflow artifacts from environment
[
https://issues.apache.org/jira/browse/BEAM-9383?focusedWorklogId=405867=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-405867
]
ASF GitHub Bot logged work on BEAM-9383:
Author: ASF GitHub Bot
Created on: 19/Mar/20 00:31
Start Date: 19/Mar/20 00:31
Worklog Time Spent: 10m
Work Description: ihji commented on pull request #11039: [BEAM-9383]
Staging Dataflow artifacts from environment
URL: https://github.com/apache/beam/pull/11039#discussion_r394721868
##
File path:
runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
##
@@ -752,6 +759,27 @@ private Debuggee registerDebuggee(CloudDebugger
debuggerClient, String uniquifie
}
}
+ private List stageArtifacts(RunnerApi.Pipeline pipeline) {
Review comment:
It's private method so we can't directly test it. Existing test (such as
https://github.com/apache/beam/blob/master/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/DataflowRunnerTest.java#L758)
should cover this method too.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 405867)
Time Spent: 1.5h (was: 1h 20m)
> Staging Dataflow artifacts from environment
> ---
>
> Key: BEAM-9383
> URL: https://issues.apache.org/jira/browse/BEAM-9383
> Project: Beam
> Issue Type: Sub-task
> Components: java-fn-execution
>Reporter: Heejong Lee
>Assignee: Heejong Lee
>Priority: Major
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> Staging Dataflow artifacts from environment
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9551) Pass around Environment PB as pointer not value
[ https://issues.apache.org/jira/browse/BEAM-9551?focusedWorklogId=405866=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-405866 ] ASF GitHub Bot logged work on BEAM-9551: Author: ASF GitHub Bot Created on: 19/Mar/20 00:30 Start Date: 19/Mar/20 00:30 Worklog Time Spent: 10m Work Description: lostluck commented on issue #11164: [BEAM-9551] Environment PB Pointer cleanup URL: https://github.com/apache/beam/pull/11164#issuecomment-600924609 R: @youngoli This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 405866) Time Spent: 0.5h (was: 20m) > Pass around Environment PB as pointer not value > --- > > Key: BEAM-9551 > URL: https://issues.apache.org/jira/browse/BEAM-9551 > Project: Beam > Issue Type: Bug > Components: sdk-go >Reporter: Robert Burke >Assignee: Robert Burke >Priority: Minor > Time Spent: 0.5h > Remaining Estimate: 0h > > Go Protocol buffers prefer being passed around by Pointer than by value. > Caught by a linter, and should be fixed for good practice. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9542) Where the BeamModulePlugin's force is needed?
[ https://issues.apache.org/jira/browse/BEAM-9542?focusedWorklogId=405865=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-405865 ] ASF GitHub Bot logged work on BEAM-9542: Author: ASF GitHub Bot Created on: 19/Mar/20 00:28 Start Date: 19/Mar/20 00:28 Worklog Time Spent: 10m Work Description: suztomo commented on pull request #11168: [BEAM-9542] Limit and clarify the effect of "force" in Java build URL: https://github.com/apache/beam/pull/11168 Before: forcing all dependencies for all class paths After: forcing only 4 dependencies needed for tests. Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] Update `CHANGES.md` with noteworthy changes. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier). Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[![Build
[jira] [Work logged] (BEAM-9339) Declare capabilities in SDK environments
[
https://issues.apache.org/jira/browse/BEAM-9339?focusedWorklogId=405864=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-405864
]
ASF GitHub Bot logged work on BEAM-9339:
Author: ASF GitHub Bot
Created on: 19/Mar/20 00:26
Start Date: 19/Mar/20 00:26
Worklog Time Spent: 10m
Work Description: lukecwik commented on pull request #11160: [BEAM-9339]
Declare capabilities in the Java SDK.
URL: https://github.com/apache/beam/pull/11160#discussion_r394720559
##
File path:
runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/Environments.java
##
@@ -228,6 +232,10 @@ public static Environment createProcessEnvironment(
return filesToStage.build();
}
+ public static Set getJavaCapabilities() {
+return ModelCoders.urns();
Review comment:
Yes but only some metrics are being reported.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 405864)
Time Spent: 5h 10m (was: 5h)
> Declare capabilities in SDK environments
>
>
> Key: BEAM-9339
> URL: https://issues.apache.org/jira/browse/BEAM-9339
> Project: Beam
> Issue Type: New Feature
> Components: sdk-go, sdk-java-harness, sdk-py-harness
>Reporter: Robert Bradshaw
>Assignee: Robert Bradshaw
>Priority: Major
> Time Spent: 5h 10m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9540) Rename beam:source:runner:0.1/beam:sink:runner:0.1 to beam:runner:source:v1/beam:runner:sink:v1
[ https://issues.apache.org/jira/browse/BEAM-9540?focusedWorklogId=405858=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-405858 ] ASF GitHub Bot logged work on BEAM-9540: Author: ASF GitHub Bot Created on: 19/Mar/20 00:20 Start Date: 19/Mar/20 00:20 Worklog Time Spent: 10m Work Description: lukecwik commented on issue #11157: [BEAM-9540] Rename beam:source:runner:0.1/beam:sink:runner:0.1 to beam:runner:source:v1/beam:runner:sink:v1 URL: https://github.com/apache/beam/pull/11157#issuecomment-600921437 Run Portable_Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 405858) Time Spent: 2.5h (was: 2h 20m) > Rename beam:source:runner:0.1/beam:sink:runner:0.1 to > beam:runner:source:v1/beam:runner:sink:v1 > --- > > Key: BEAM-9540 > URL: https://issues.apache.org/jira/browse/BEAM-9540 > Project: Beam > Issue Type: Bug > Components: sdk-java-harness, sdk-py-harness >Reporter: Luke Cwik >Assignee: Luke Cwik >Priority: Minor > Time Spent: 2.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-4374) Update existing metrics in the FN API to use new Metric Schema
[ https://issues.apache.org/jira/browse/BEAM-4374?focusedWorklogId=405855=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-405855 ] ASF GitHub Bot logged work on BEAM-4374: Author: ASF GitHub Bot Created on: 19/Mar/20 00:12 Start Date: 19/Mar/20 00:12 Worklog Time Spent: 10m Work Description: lukecwik commented on issue #11167: [BEAM-4374] Define the protos for a "short" id mechanism for metrics URL: https://github.com/apache/beam/pull/11167#issuecomment-600919341 R: @robertwb This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 405855) Time Spent: 17h 10m (was: 17h) > Update existing metrics in the FN API to use new Metric Schema > -- > > Key: BEAM-4374 > URL: https://issues.apache.org/jira/browse/BEAM-4374 > Project: Beam > Issue Type: New Feature > Components: beam-model >Reporter: Alex Amato >Priority: Major > Time Spent: 17h 10m > Remaining Estimate: 0h > > Update existing metrics to use the new proto and cataloging schema defined in: > [_https://s.apache.org/beam-fn-api-metrics_] > * Check in new protos > * Define catalog file for metrics > * Port existing metrics to use this new format, based on catalog > names+metadata -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-4374) Update existing metrics in the FN API to use new Metric Schema
[ https://issues.apache.org/jira/browse/BEAM-4374?focusedWorklogId=405854=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-405854 ] ASF GitHub Bot logged work on BEAM-4374: Author: ASF GitHub Bot Created on: 19/Mar/20 00:08 Start Date: 19/Mar/20 00:08 Worklog Time Spent: 10m Work Description: lukecwik commented on pull request #11167: [BEAM-4374] Define the protos for a "short" id mechanism for metrics URL: https://github.com/apache/beam/pull/11167 Typical flow: Runner requests metrics and SDK responds with metrics data and ids, SDK stores id -> full MonitoringInfo mapping. if Runner is missing ids: Runner requests full data given for missing ids and SDK provides full data Runner "remembers" id -> full mapping for future process bundle responses Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] Update `CHANGES.md` with noteworthy changes. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier). Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[![Build
[jira] [Work logged] (BEAM-9339) Declare capabilities in SDK environments
[ https://issues.apache.org/jira/browse/BEAM-9339?focusedWorklogId=405852=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-405852 ] ASF GitHub Bot logged work on BEAM-9339: Author: ASF GitHub Bot Created on: 19/Mar/20 00:04 Start Date: 19/Mar/20 00:04 Worklog Time Spent: 10m Work Description: lukecwik commented on issue #11162: [BEAM-9339, BEAM-2939] Drop splittable field from ParDoPayload, add splittable dofn requirement to Python SDK. URL: https://github.com/apache/beam/pull/11162#issuecomment-600917167 Run Portable_Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 405852) Time Spent: 5h (was: 4h 50m) > Declare capabilities in SDK environments > > > Key: BEAM-9339 > URL: https://issues.apache.org/jira/browse/BEAM-9339 > Project: Beam > Issue Type: New Feature > Components: sdk-go, sdk-java-harness, sdk-py-harness >Reporter: Robert Bradshaw >Assignee: Robert Bradshaw >Priority: Major > Time Spent: 5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-7923) Interactive Beam
[ https://issues.apache.org/jira/browse/BEAM-7923?focusedWorklogId=405836=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-405836 ] ASF GitHub Bot logged work on BEAM-7923: Author: ASF GitHub Bot Created on: 18/Mar/20 23:52 Start Date: 18/Mar/20 23:52 Worklog Time Spent: 10m Work Description: KevinGG commented on issue #11166: [BEAM-7923] Emit info when capture control options are configured. URL: https://github.com/apache/beam/pull/11166#issuecomment-600914076 Per discussion offline, we've decided to emit some logging for options APIs. yapf formatted. lint passed locally. R: @pabloem R: @davidyan74 R: @rohdesamuel This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 405836) Time Spent: 6h 10m (was: 6h) > Interactive Beam > > > Key: BEAM-7923 > URL: https://issues.apache.org/jira/browse/BEAM-7923 > Project: Beam > Issue Type: New Feature > Components: runner-py-interactive >Reporter: Ning Kang >Assignee: Ning Kang >Priority: Major > Time Spent: 6h 10m > Remaining Estimate: 0h > > This is the top level ticket for all efforts leveraging [interactive > Beam|[https://github.com/apache/beam/tree/master/sdks/python/apache_beam/runners/interactive]] > As the development goes, blocking tickets will be added to this one. > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-7923) Interactive Beam
[ https://issues.apache.org/jira/browse/BEAM-7923?focusedWorklogId=405837=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-405837 ] ASF GitHub Bot logged work on BEAM-7923: Author: ASF GitHub Bot Created on: 18/Mar/20 23:52 Start Date: 18/Mar/20 23:52 Worklog Time Spent: 10m Work Description: KevinGG commented on issue #11166: [BEAM-7923] Emit info when capture control options are configured. URL: https://github.com/apache/beam/pull/11166#issuecomment-600914076 Per discussion offline, we've decided to emit some logging for options APIs. yapf formatted. lint passed locally. R: @pabloem R: @davidyan74 R: @rohdesamuel PTAL, thx! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 405837) Time Spent: 6h 20m (was: 6h 10m) > Interactive Beam > > > Key: BEAM-7923 > URL: https://issues.apache.org/jira/browse/BEAM-7923 > Project: Beam > Issue Type: New Feature > Components: runner-py-interactive >Reporter: Ning Kang >Assignee: Ning Kang >Priority: Major > Time Spent: 6h 20m > Remaining Estimate: 0h > > This is the top level ticket for all efforts leveraging [interactive > Beam|[https://github.com/apache/beam/tree/master/sdks/python/apache_beam/runners/interactive]] > As the development goes, blocking tickets will be added to this one. > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-7923) Interactive Beam
[ https://issues.apache.org/jira/browse/BEAM-7923?focusedWorklogId=405833=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-405833 ] ASF GitHub Bot logged work on BEAM-7923: Author: ASF GitHub Bot Created on: 18/Mar/20 23:50 Start Date: 18/Mar/20 23:50 Worklog Time Spent: 10m Work Description: KevinGG commented on pull request #11166: [BEAM-7923] Emit info when capture control options are configured. URL: https://github.com/apache/beam/pull/11166 1. Info logging to notify users about invoking evict_captured_data on changing capture_duration, capture_size_limit, and capturable_sources options. The users have to explicitly evict captured data because the process is destructive and the usage should be the same as evicting captured data even when they don't modify options at all. 2. Info logging about behavior difference between enabling capture replay and disabling it. This functions as a switch enabling/disabling the replayability, thus works without the need of evicting captured data. 3. Changed the default capture_duration from 5s to 60s. **Please** add a meaningful description for your change here Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] Update `CHANGES.md` with noteworthy changes. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier). Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [![Build
[jira] [Updated] (BEAM-9551) Pass around Environment PB as pointer not value
[ https://issues.apache.org/jira/browse/BEAM-9551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Burke updated BEAM-9551: --- Status: Open (was: Triage Needed) > Pass around Environment PB as pointer not value > --- > > Key: BEAM-9551 > URL: https://issues.apache.org/jira/browse/BEAM-9551 > Project: Beam > Issue Type: Bug > Components: sdk-go >Reporter: Robert Burke >Assignee: Robert Burke >Priority: Minor > Time Spent: 20m > Remaining Estimate: 0h > > Go Protocol buffers prefer being passed around by Pointer than by value. > Caught by a linter, and should be fixed for good practice. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9339) Declare capabilities in SDK environments
[ https://issues.apache.org/jira/browse/BEAM-9339?focusedWorklogId=405828=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-405828 ] ASF GitHub Bot logged work on BEAM-9339: Author: ASF GitHub Bot Created on: 18/Mar/20 23:38 Start Date: 18/Mar/20 23:38 Worklog Time Spent: 10m Work Description: lukecwik commented on issue #11160: [BEAM-9339] Declare capabilities in the Java SDK. URL: https://github.com/apache/beam/pull/11160#issuecomment-600910480 Run Java PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 405828) Time Spent: 4h 50m (was: 4h 40m) > Declare capabilities in SDK environments > > > Key: BEAM-9339 > URL: https://issues.apache.org/jira/browse/BEAM-9339 > Project: Beam > Issue Type: New Feature > Components: sdk-go, sdk-java-harness, sdk-py-harness >Reporter: Robert Bradshaw >Assignee: Robert Bradshaw >Priority: Major > Time Spent: 4h 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9339) Declare capabilities in SDK environments
[ https://issues.apache.org/jira/browse/BEAM-9339?focusedWorklogId=405827=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-405827 ] ASF GitHub Bot logged work on BEAM-9339: Author: ASF GitHub Bot Created on: 18/Mar/20 23:37 Start Date: 18/Mar/20 23:37 Worklog Time Spent: 10m Work Description: lukecwik commented on issue #11162: [BEAM-9339, BEAM-2939] Drop splittable field from ParDoPayload, add splittable dofn requirement to Python SDK. URL: https://github.com/apache/beam/pull/11162#issuecomment-600910385 Run Portable_Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 405827) Time Spent: 4h 40m (was: 4.5h) > Declare capabilities in SDK environments > > > Key: BEAM-9339 > URL: https://issues.apache.org/jira/browse/BEAM-9339 > Project: Beam > Issue Type: New Feature > Components: sdk-go, sdk-java-harness, sdk-py-harness >Reporter: Robert Bradshaw >Assignee: Robert Bradshaw >Priority: Major > Time Spent: 4h 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9551) Pass around Environment PB as pointer not value
[ https://issues.apache.org/jira/browse/BEAM-9551?focusedWorklogId=405825=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-405825 ] ASF GitHub Bot logged work on BEAM-9551: Author: ASF GitHub Bot Created on: 18/Mar/20 23:33 Start Date: 18/Mar/20 23:33 Worklog Time Spent: 10m Work Description: lostluck commented on issue #11164: [BEAM-9551] Environment PB Pointer cleanup URL: https://github.com/apache/beam/pull/11164#issuecomment-600909382 Run Go Postcommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 405825) Time Spent: 20m (was: 10m) > Pass around Environment PB as pointer not value > --- > > Key: BEAM-9551 > URL: https://issues.apache.org/jira/browse/BEAM-9551 > Project: Beam > Issue Type: Bug > Components: sdk-go >Reporter: Robert Burke >Assignee: Robert Burke >Priority: Minor > Time Spent: 20m > Remaining Estimate: 0h > > Go Protocol buffers prefer being passed around by Pointer than by value. > Caught by a linter, and should be fixed for good practice. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9340) Properly populate pipeline proto requirements.
[ https://issues.apache.org/jira/browse/BEAM-9340?focusedWorklogId=405820=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-405820 ] ASF GitHub Bot logged work on BEAM-9340: Author: ASF GitHub Bot Created on: 18/Mar/20 23:22 Start Date: 18/Mar/20 23:22 Worklog Time Spent: 10m Work Description: robertwb commented on pull request #11165: [BEAM-9340] Populate requirements for Java. URL: https://github.com/apache/beam/pull/11165 Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] Update `CHANGES.md` with noteworthy changes. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier). Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) Python | [![Build
[jira] [Work logged] (BEAM-9511) ArrayScanToUncollectConverter: ResolvedParameter cannot be cast to ResolvedLiteral
[
https://issues.apache.org/jira/browse/BEAM-9511?focusedWorklogId=405815=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-405815
]
ASF GitHub Bot logged work on BEAM-9511:
Author: ASF GitHub Bot
Created on: 18/Mar/20 23:06
Start Date: 18/Mar/20 23:06
Worklog Time Spent: 10m
Work Description: apilloud commented on pull request #11133:
[BEAM-9511][BEAM-9515][BEAM-9516] Uncollect takes arbitrary expressions
URL: https://github.com/apache/beam/pull/11133
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 405815)
Time Spent: 0.5h (was: 20m)
> ArrayScanToUncollectConverter: ResolvedParameter cannot be cast to
> ResolvedLiteral
> --
>
> Key: BEAM-9511
> URL: https://issues.apache.org/jira/browse/BEAM-9511
> Project: Beam
> Issue Type: Bug
> Components: dsl-sql-zetasql
>Reporter: Andrew Pilloud
>Assignee: Andrew Pilloud
>Priority: Major
> Labels: zetasql-compliance
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
>
> {code:java}
> Mar 16, 2020 12:57:42 PM
> cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
> executeQuery
> INFO: Processing Sql statement: SELECT elem FROM UNNEST(@string_arr) AS elem
> Mar 16, 2020 12:57:42 PM
> com.google.zetasql.io.grpc.internal.SerializingExecutor run
> SEVERE: Exception while executing runnable
> com.google.zetasql.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed@2f5b49be
> java.lang.ClassCastException:
> com.google.zetasql.resolvedast.ResolvedNodes$ResolvedParameter cannot be cast
> to com.google.zetasql.resolvedast.ResolvedNodes$ResolvedLiteral
> at
> org.apache.beam.sdk.extensions.sql.zetasql.translation.ArrayScanToUncollectConverter.convert(ArrayScanToUncollectConverter.java:45)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.translation.ArrayScanToUncollectConverter.convert(ArrayScanToUncollectConverter.java:31)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.translation.QueryStatementConverter.convertNode(QueryStatementConverter.java:97)
> at
> java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
> at java.util.Collections$2.tryAdvance(Collections.java:4717)
> at java.util.Collections$2.forEachRemaining(Collections.java:4725)
> at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
> at
> java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
> at
> java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
> at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
> at
> java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.translation.QueryStatementConverter.convertNode(QueryStatementConverter.java:96)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.translation.QueryStatementConverter.convert(QueryStatementConverter.java:84)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.translation.QueryStatementConverter.convertRootQuery(QueryStatementConverter.java:51)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.ZetaSQLPlannerImpl.rel(ZetaSQLPlannerImpl.java:160)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.ZetaSQLQueryPlanner.convertToBeamRelInternal(ZetaSQLQueryPlanner.java:131)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.ZetaSQLQueryPlanner.convertToBeamRel(ZetaSQLQueryPlanner.java:115)
> at
> cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl.executeQuery(ExecuteQueryServiceServer.java:242)
> at
> com.google.zetasql.testing.SqlComplianceServiceGrpc$MethodHandlers.invoke(SqlComplianceServiceGrpc.java:423)
> at
> com.google.zetasql.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171)
> at
> com.google.zetasql.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283)
> at
> com.google.zetasql.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:711)
> at
> com.google.zetasql.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
> at
> com.google.zetasql.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
> at
>
[jira] [Work logged] (BEAM-9339) Declare capabilities in SDK environments
[ https://issues.apache.org/jira/browse/BEAM-9339?focusedWorklogId=405814=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-405814 ] ASF GitHub Bot logged work on BEAM-9339: Author: ASF GitHub Bot Created on: 18/Mar/20 23:03 Start Date: 18/Mar/20 23:03 Worklog Time Spent: 10m Work Description: lukecwik commented on issue #11162: [BEAM-9339, BEAM-2939] Drop splittable field from ParDoPayload, add splittable dofn requirement to Python SDK. URL: https://github.com/apache/beam/pull/11162#issuecomment-600901398 Run Portable_Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 405814) Time Spent: 4.5h (was: 4h 20m) > Declare capabilities in SDK environments > > > Key: BEAM-9339 > URL: https://issues.apache.org/jira/browse/BEAM-9339 > Project: Beam > Issue Type: New Feature > Components: sdk-go, sdk-java-harness, sdk-py-harness >Reporter: Robert Bradshaw >Assignee: Robert Bradshaw >Priority: Major > Time Spent: 4.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9540) Rename beam:source:runner:0.1/beam:sink:runner:0.1 to beam:runner:source:v1/beam:runner:sink:v1
[ https://issues.apache.org/jira/browse/BEAM-9540?focusedWorklogId=405813=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-405813 ] ASF GitHub Bot logged work on BEAM-9540: Author: ASF GitHub Bot Created on: 18/Mar/20 23:03 Start Date: 18/Mar/20 23:03 Worklog Time Spent: 10m Work Description: lukecwik commented on issue #11157: [BEAM-9540] Rename beam:source:runner:0.1/beam:sink:runner:0.1 to beam:runner:source:v1/beam:runner:sink:v1 URL: https://github.com/apache/beam/pull/11157#issuecomment-600901181 Run Portable_Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 405813) Time Spent: 2h 20m (was: 2h 10m) > Rename beam:source:runner:0.1/beam:sink:runner:0.1 to > beam:runner:source:v1/beam:runner:sink:v1 > --- > > Key: BEAM-9540 > URL: https://issues.apache.org/jira/browse/BEAM-9540 > Project: Beam > Issue Type: Bug > Components: sdk-java-harness, sdk-py-harness >Reporter: Luke Cwik >Assignee: Luke Cwik >Priority: Minor > Time Spent: 2h 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9540) Rename beam:source:runner:0.1/beam:sink:runner:0.1 to beam:runner:source:v1/beam:runner:sink:v1
[ https://issues.apache.org/jira/browse/BEAM-9540?focusedWorklogId=405812=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-405812 ] ASF GitHub Bot logged work on BEAM-9540: Author: ASF GitHub Bot Created on: 18/Mar/20 23:03 Start Date: 18/Mar/20 23:03 Worklog Time Spent: 10m Work Description: lukecwik commented on issue #11157: [BEAM-9540] Rename beam:source:runner:0.1/beam:sink:runner:0.1 to beam:runner:source:v1/beam:runner:sink:v1 URL: https://github.com/apache/beam/pull/11157#issuecomment-600901119 Run Java PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 405812) Time Spent: 2h 10m (was: 2h) > Rename beam:source:runner:0.1/beam:sink:runner:0.1 to > beam:runner:source:v1/beam:runner:sink:v1 > --- > > Key: BEAM-9540 > URL: https://issues.apache.org/jira/browse/BEAM-9540 > Project: Beam > Issue Type: Bug > Components: sdk-java-harness, sdk-py-harness >Reporter: Luke Cwik >Assignee: Luke Cwik >Priority: Minor > Time Spent: 2h 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9551) Pass around Environment PB as pointer not value
[ https://issues.apache.org/jira/browse/BEAM-9551?focusedWorklogId=405811=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-405811 ] ASF GitHub Bot logged work on BEAM-9551: Author: ASF GitHub Bot Created on: 18/Mar/20 22:57 Start Date: 18/Mar/20 22:57 Worklog Time Spent: 10m Work Description: lostluck commented on pull request #11164: [BEAM-9551] Environment PB Pointer cleanup URL: https://github.com/apache/beam/pull/11164 Go Protocol buffers strongly prefer to be pointers, and could cause issues otherwise WRT copying the proto's non-exposed state incorrectly. Likely hasn't caused any bugs, but it's a good practice to clean it up ASAP. Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] Update `CHANGES.md` with noteworthy changes. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier). Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[![Build
[jira] [Created] (BEAM-9551) Pass around Environment PB as pointer not value
Robert Burke created BEAM-9551: -- Summary: Pass around Environment PB as pointer not value Key: BEAM-9551 URL: https://issues.apache.org/jira/browse/BEAM-9551 Project: Beam Issue Type: Bug Components: sdk-go Reporter: Robert Burke Assignee: Robert Burke Go Protocol buffers prefer being passed around by Pointer than by value. Caught by a linter, and should be fixed for good practice. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9468) Add Google Cloud Healthcare API IO Connectors
[
https://issues.apache.org/jira/browse/BEAM-9468?focusedWorklogId=405808=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-405808
]
ASF GitHub Bot logged work on BEAM-9468:
Author: ASF GitHub Bot
Created on: 18/Mar/20 22:52
Start Date: 18/Mar/20 22:52
Worklog Time Spent: 10m
Work Description: pabloem commented on pull request #11151: [BEAM-9468]
Hl7v2 io
URL: https://github.com/apache/beam/pull/11151#discussion_r394668514
##
File path:
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/healthcare/HL7v2IO.java
##
@@ -0,0 +1,620 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.gcp.healthcare;
+
+import com.google.api.services.healthcare.v1alpha2.model.IngestMessageResponse;
+import com.google.api.services.healthcare.v1alpha2.model.Message;
+import com.google.auto.value.AutoValue;
+import java.io.IOException;
+import java.text.ParseException;
+import java.util.Collections;
+import java.util.List;
+import org.apache.beam.sdk.io.gcp.datastore.AdaptiveThrottler;
+import org.apache.beam.sdk.io.gcp.pubsub.PubsubIO;
+import org.apache.beam.sdk.metrics.Counter;
+import org.apache.beam.sdk.metrics.Metrics;
+import org.apache.beam.sdk.transforms.Create;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.util.Sleeper;
+import org.apache.beam.sdk.values.PBegin;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.PCollectionTuple;
+import org.apache.beam.sdk.values.PDone;
+import org.apache.beam.sdk.values.TupleTag;
+import org.apache.beam.sdk.values.TupleTagList;
+import
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Throwables;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * {@link HL7v2IO} provides an API for reading from and writing to https://cloud.google.com/healthcare/docs/concepts/hl7v2;>Google Cloud
Healthcare HL7v2 API.
+ *
+ *
+ * Read HL7v2 Messages are fetched from the HL7v2 store based on the {@link
PCollection} of of
+ * message IDs {@link String}s produced by the {@link
AutoValue_HL7v2IO_Read#getMessageIDTransform}
+ * as {@link PCollectionTuple}*** containing an {@link HL7v2IO.Read#OUT} tag
for successfully
+ * fetched messages and a {@link HL7v2IO.Read#DEAD_LETTER} tag for message IDs
that could not be
+ * fetched.
+ *
+ * HL7v2 stores can be read in several ways: - Unbounded: based on the
Pub/Sub Notification
+ * Channel {@link HL7v2IO#readNotificationSubscription(String)} - Bounded:
based on reading an
+ * entire HL7v2 store (or stores) {@link HL7v2IO#readHL7v2Store(String)} -
Bounded: based on reading
+ * an HL7v2 store with a filter
+ *
+ * Note, due to the flexibility of this Read transform, this must output a
dead letter queue.
+ * This handles the scenario where the the PTransform that populates a
PCollection of message IDs
+ * contains message IDs that do not exist in the HL7v2 stores.
+ *
+ * Example:
+ *
+ * {@code
+ * PipelineOptions options = ...;
+ * Pipeline pipeline = Pipeline.create(options)
+ *
+ *
+ * PCollectionTuple messages = pipeline.apply(
+ * new HLv2IO.readNotifications(options.getNotificationSubscription())
+ *
+ * // Write errors to your favorite dead letter queue (e.g. Pub/Sub, GCS,
BigQuery)
+ * messages.get(PubsubNotificationToHL7v2Message.DEAD_LETTER)
+ *.apply("WriteToDeadLetterQueue", ...);
+ *
+ * PCollection fetchedMessages =
fetchResults.get(PubsubNotificationToHL7v2Message.OUT)
+ *.apply("ExtractFetchedMessage",
+ *MapElements
+ *.into(TypeDescriptor.of(Message.class))
+ *.via(FailsafeElement::getPayload));
+ *
+ * // Go about your happy path transformations.
+ * PCollection out = fetchedMessages.apply("ProcessFetchedMessages",
...);
+ *
+ * // Write using the Message.Ingest method of the HL7v2 REST API.
+ * out.apply(HL7v2IO.ingestMessages(options.getOutputHL7v2Store()));
+ *
+ * pipeline.run();
+ *
+ * }***
+ *
+ */
+public class HL7v2IO {
+ // TODO add metrics
[jira] [Work logged] (BEAM-9468) Add Google Cloud Healthcare API IO Connectors
[
https://issues.apache.org/jira/browse/BEAM-9468?focusedWorklogId=405810=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-405810
]
ASF GitHub Bot logged work on BEAM-9468:
Author: ASF GitHub Bot
Created on: 18/Mar/20 22:52
Start Date: 18/Mar/20 22:52
Worklog Time Spent: 10m
Work Description: pabloem commented on pull request #11151: [BEAM-9468]
Hl7v2 io
URL: https://github.com/apache/beam/pull/11151#discussion_r394632407
##
File path:
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/healthcare/HL7v2IO.java
##
@@ -0,0 +1,620 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.gcp.healthcare;
+
+import com.google.api.services.healthcare.v1alpha2.model.IngestMessageResponse;
+import com.google.api.services.healthcare.v1alpha2.model.Message;
+import com.google.auto.value.AutoValue;
+import java.io.IOException;
+import java.text.ParseException;
+import java.util.Collections;
+import java.util.List;
+import org.apache.beam.sdk.io.gcp.datastore.AdaptiveThrottler;
+import org.apache.beam.sdk.io.gcp.pubsub.PubsubIO;
+import org.apache.beam.sdk.metrics.Counter;
+import org.apache.beam.sdk.metrics.Metrics;
+import org.apache.beam.sdk.transforms.Create;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.util.Sleeper;
+import org.apache.beam.sdk.values.PBegin;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.PCollectionTuple;
+import org.apache.beam.sdk.values.PDone;
+import org.apache.beam.sdk.values.TupleTag;
+import org.apache.beam.sdk.values.TupleTagList;
+import
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Throwables;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * {@link HL7v2IO} provides an API for reading from and writing to https://cloud.google.com/healthcare/docs/concepts/hl7v2;>Google Cloud
Healthcare HL7v2 API.
+ *
+ *
+ * Read HL7v2 Messages are fetched from the HL7v2 store based on the {@link
PCollection} of of
+ * message IDs {@link String}s produced by the {@link
AutoValue_HL7v2IO_Read#getMessageIDTransform}
+ * as {@link PCollectionTuple}*** containing an {@link HL7v2IO.Read#OUT} tag
for successfully
+ * fetched messages and a {@link HL7v2IO.Read#DEAD_LETTER} tag for message IDs
that could not be
+ * fetched.
+ *
+ * HL7v2 stores can be read in several ways: - Unbounded: based on the
Pub/Sub Notification
+ * Channel {@link HL7v2IO#readNotificationSubscription(String)} - Bounded:
based on reading an
+ * entire HL7v2 store (or stores) {@link HL7v2IO#readHL7v2Store(String)} -
Bounded: based on reading
+ * an HL7v2 store with a filter
+ *
+ * Note, due to the flexibility of this Read transform, this must output a
dead letter queue.
+ * This handles the scenario where the the PTransform that populates a
PCollection of message IDs
+ * contains message IDs that do not exist in the HL7v2 stores.
+ *
+ * Example:
+ *
+ * {@code
+ * PipelineOptions options = ...;
+ * Pipeline pipeline = Pipeline.create(options)
+ *
+ *
+ * PCollectionTuple messages = pipeline.apply(
+ * new HLv2IO.readNotifications(options.getNotificationSubscription())
Review comment:
I believe you don't need the `new` here.
```suggestion
* HLv2IO.readNotifications(options.getNotificationSubscription())
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 405810)
Time Spent: 2h 40m (was: 2.5h)
> Add Google Cloud Healthcare API IO Connectors
> -
>
> Key: BEAM-9468
> URL: https://issues.apache.org/jira/browse/BEAM-9468
>
[jira] [Work logged] (BEAM-9468) Add Google Cloud Healthcare API IO Connectors
[
https://issues.apache.org/jira/browse/BEAM-9468?focusedWorklogId=405809=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-405809
]
ASF GitHub Bot logged work on BEAM-9468:
Author: ASF GitHub Bot
Created on: 18/Mar/20 22:52
Start Date: 18/Mar/20 22:52
Worklog Time Spent: 10m
Work Description: pabloem commented on pull request #11151: [BEAM-9468]
Hl7v2 io
URL: https://github.com/apache/beam/pull/11151#discussion_r394631597
##
File path:
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/healthcare/HL7v2IO.java
##
@@ -0,0 +1,620 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.gcp.healthcare;
+
+import com.google.api.services.healthcare.v1alpha2.model.IngestMessageResponse;
+import com.google.api.services.healthcare.v1alpha2.model.Message;
+import com.google.auto.value.AutoValue;
+import java.io.IOException;
+import java.text.ParseException;
+import java.util.Collections;
+import java.util.List;
+import org.apache.beam.sdk.io.gcp.datastore.AdaptiveThrottler;
+import org.apache.beam.sdk.io.gcp.pubsub.PubsubIO;
+import org.apache.beam.sdk.metrics.Counter;
+import org.apache.beam.sdk.metrics.Metrics;
+import org.apache.beam.sdk.transforms.Create;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.util.Sleeper;
+import org.apache.beam.sdk.values.PBegin;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.PCollectionTuple;
+import org.apache.beam.sdk.values.PDone;
+import org.apache.beam.sdk.values.TupleTag;
+import org.apache.beam.sdk.values.TupleTagList;
+import
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Throwables;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * {@link HL7v2IO} provides an API for reading from and writing to https://cloud.google.com/healthcare/docs/concepts/hl7v2;>Google Cloud
Healthcare HL7v2 API.
+ *
+ *
+ * Read HL7v2 Messages are fetched from the HL7v2 store based on the {@link
PCollection} of of
+ * message IDs {@link String}s produced by the {@link
AutoValue_HL7v2IO_Read#getMessageIDTransform}
+ * as {@link PCollectionTuple}*** containing an {@link HL7v2IO.Read#OUT} tag
for successfully
Review comment:
I would recommend that the output of the transform be something like
[WriteResult](https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/WriteResult.java#L34)
returned by
[BigQueryIO.Write](https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java#L1720).
That should give you a more digestable output type for new users, with
things like `getFailedElements`, etc. Thoughts?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 405809)
Time Spent: 2h 40m (was: 2.5h)
> Add Google Cloud Healthcare API IO Connectors
> -
>
> Key: BEAM-9468
> URL: https://issues.apache.org/jira/browse/BEAM-9468
> Project: Beam
> Issue Type: New Feature
> Components: io-java-gcp
>Reporter: Jacob Ferriero
>Assignee: Jacob Ferriero
>Priority: Minor
> Time Spent: 2h 40m
> Remaining Estimate: 0h
>
> Add IO Transforms for the HL7v2, FHIR and DICOM stores in the [Google Cloud
> Healthcare API|https://cloud.google.com/healthcare/docs/]
> HL7v2IO
> FHIRIO
> DICOM
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9535) Cleanup portability protos.
[ https://issues.apache.org/jira/browse/BEAM-9535?focusedWorklogId=405802=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-405802 ] ASF GitHub Bot logged work on BEAM-9535: Author: ASF GitHub Bot Created on: 18/Mar/20 22:41 Start Date: 18/Mar/20 22:41 Worklog Time Spent: 10m Work Description: robertwb commented on issue #11150: [BEAM-9535] Remove unused ParDoPayload.Parameters. URL: https://github.com/apache/beam/pull/11150#issuecomment-600895150 Run Portable_Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 405802) Time Spent: 50m (was: 40m) > Cleanup portability protos. > --- > > Key: BEAM-9535 > URL: https://issues.apache.org/jira/browse/BEAM-9535 > Project: Beam > Issue Type: Improvement > Components: beam-model >Reporter: Robert Bradshaw >Priority: Major > Fix For: 2.21.0 > > Time Spent: 50m > Remaining Estimate: 0h > > This is to provide a stable version of the FnAPI protos going forward. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9339) Declare capabilities in SDK environments
[
https://issues.apache.org/jira/browse/BEAM-9339?focusedWorklogId=405798=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-405798
]
ASF GitHub Bot logged work on BEAM-9339:
Author: ASF GitHub Bot
Created on: 18/Mar/20 22:39
Start Date: 18/Mar/20 22:39
Worklog Time Spent: 10m
Work Description: robertwb commented on pull request #11160: [BEAM-9339]
Declare capabilities in the Java SDK.
URL: https://github.com/apache/beam/pull/11160#discussion_r394679769
##
File path:
runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/Environments.java
##
@@ -228,6 +232,10 @@ public static Environment createProcessEnvironment(
return filesToStage.build();
}
+ public static Set getJavaCapabilities() {
+return ModelCoders.urns();
Review comment:
(I suppose that's just the new metrics PCollection element counts, right?)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 405798)
Time Spent: 4h 20m (was: 4h 10m)
> Declare capabilities in SDK environments
>
>
> Key: BEAM-9339
> URL: https://issues.apache.org/jira/browse/BEAM-9339
> Project: Beam
> Issue Type: New Feature
> Components: sdk-go, sdk-java-harness, sdk-py-harness
>Reporter: Robert Bradshaw
>Assignee: Robert Bradshaw
>Priority: Major
> Time Spent: 4h 20m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9339) Declare capabilities in SDK environments
[
https://issues.apache.org/jira/browse/BEAM-9339?focusedWorklogId=405797=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-405797
]
ASF GitHub Bot logged work on BEAM-9339:
Author: ASF GitHub Bot
Created on: 18/Mar/20 22:38
Start Date: 18/Mar/20 22:38
Worklog Time Spent: 10m
Work Description: robertwb commented on pull request #11160: [BEAM-9339]
Declare capabilities in the Java SDK.
URL: https://github.com/apache/beam/pull/11160#discussion_r394679580
##
File path:
runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/Environments.java
##
@@ -228,6 +232,10 @@ public static Environment createProcessEnvironment(
return filesToStage.build();
}
+ public static Set getJavaCapabilities() {
+return ModelCoders.urns();
Review comment:
Good call on multi-core. Is progress reporting actually done now?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 405797)
Time Spent: 4h 10m (was: 4h)
> Declare capabilities in SDK environments
>
>
> Key: BEAM-9339
> URL: https://issues.apache.org/jira/browse/BEAM-9339
> Project: Beam
> Issue Type: New Feature
> Components: sdk-go, sdk-java-harness, sdk-py-harness
>Reporter: Robert Bradshaw
>Assignee: Robert Bradshaw
>Priority: Major
> Time Spent: 4h 10m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (BEAM-9542) Where the BeamModulePlugin's force is needed?
[
https://issues.apache.org/jira/browse/BEAM-9542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17062097#comment-17062097
]
Tomo Suzuki commented on BEAM-9542:
---
Why test classes are in main???
{noformat}
> Task :sdks:java:io:google-cloud-platform:compileJava
/Users/suztomo/beam/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/TestPubsub.java:23:
error: cannot find symbol
import static org.hamcrest.Matchers.containsInAnyOrder;
^
symbol: class Matchers
location: package org.hamcrest
/Users/suztomo/beam/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/TestPubsub.java:23:
error: static import only from classes and interfaces
import static org.hamcrest.Matchers.containsInAnyOrder;
^
Note: Some input files use or override a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
2 errors
> Task :sdks:java:io:google-cloud-platform:compileJava FAILED
{noformat}
> Where the BeamModulePlugin's force is needed?
> -
>
> Key: BEAM-9542
> URL: https://issues.apache.org/jira/browse/BEAM-9542
> Project: Beam
> Issue Type: Task
> Components: build-system
>Reporter: Tomo Suzuki
>Assignee: Tomo Suzuki
>Priority: Major
>
> Followup of https://github.com/apache/beam/pull/11156#discussion_r394408735
> {noformat}
> > Task :sdks:java:core:compileTestJava FAILED
> /Users/suztomo/beam/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/DeduplicateTest.java:21:
> error: cannot find symbol
> import static org.junit.Assert.assertThrows;
> ^
> symbol: static assertThrows
> location: class
> /Users/suztomo/beam/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/splittabledofn/WatermarkEstimatorsTest.java:21:
> error: cannot find symbol
> import static org.junit.Assert.assertThrows;
> ^
> symbol: static assertThrows
> location: class
> Note: Some input files use or override a deprecated API.
> Note: Recompile with -Xlint:deprecation for details.
> 2 errors
> <-> 36% EXECUTING [19m 37s]
> {noformat}
> Memo for my Mac:
> {noformat}
> suztomo-macbookpro44% ./gradlew -p sdks/java check -x
> extensions:sql:zetasql:check -x harness:test -x io:jdbc:test -x
> io:kafka:test -x io:solr:test -x core:test
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (BEAM-9513) NullPointerException in convertRexNodeFromResolvedExprWithRefScan
[
https://issues.apache.org/jira/browse/BEAM-9513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Pilloud resolved BEAM-9513.
--
Fix Version/s: 2.25.0
Resolution: Fixed
> NullPointerException in convertRexNodeFromResolvedExprWithRefScan
> -
>
> Key: BEAM-9513
> URL: https://issues.apache.org/jira/browse/BEAM-9513
> Project: Beam
> Issue Type: Bug
> Components: dsl-sql-zetasql
>Reporter: Andrew Pilloud
>Assignee: Andrew Pilloud
>Priority: Major
> Labels: zetasql-compliance
> Fix For: 2.25.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> {code:java}
> Mar 16, 2020 12:58:26 PM
> cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
> executeQuery
> INFO: Processing Sql statement: WITH
> R AS (SELECT 2 a),
> S AS (SELECT a-1 x, a+1 y FROM R)
> SELECT a, x, y FROM R, S
> Mar 16, 2020 12:58:26 PM
> com.google.zetasql.io.grpc.internal.SerializingExecutor run
> SEVERE: Exception while executing runnable
> com.google.zetasql.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed@5341cc1c
> java.lang.NullPointerException
> at
> org.apache.beam.sdk.extensions.sql.zetasql.translation.ExpressionConverter.convertRexNodeFromResolvedExprWithRefScan(ExpressionConverter.java:374)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.translation.JoinScanWithRefConverter.convert(JoinScanWithRefConverter.java:63)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.translation.JoinScanWithRefConverter.convert(JoinScanWithRefConverter.java:35)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.translation.QueryStatementConverter.convertNode(QueryStatementConverter.java:97)
> at
> java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
> at java.util.Collections$2.tryAdvance(Collections.java:4717)
> at java.util.Collections$2.forEachRemaining(Collections.java:4725)
> at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
> at
> java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
> at
> java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
> at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
> at
> java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.translation.QueryStatementConverter.convertNode(QueryStatementConverter.java:96)
> at
> java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
> at java.util.Collections$2.tryAdvance(Collections.java:4717)
> at java.util.Collections$2.forEachRemaining(Collections.java:4725)
> at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
> at
> java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
> at
> java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
> at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
> at
> java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.translation.QueryStatementConverter.convertNode(QueryStatementConverter.java:96)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.translation.QueryStatementConverter.convert(QueryStatementConverter.java:84)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.translation.QueryStatementConverter.convertRootQuery(QueryStatementConverter.java:51)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.ZetaSQLPlannerImpl.rel(ZetaSQLPlannerImpl.java:160)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.ZetaSQLQueryPlanner.convertToBeamRelInternal(ZetaSQLQueryPlanner.java:131)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.ZetaSQLQueryPlanner.convertToBeamRel(ZetaSQLQueryPlanner.java:115)
> at
> cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl.executeQuery(ExecuteQueryServiceServer.java:242)
> at
> com.google.zetasql.testing.SqlComplianceServiceGrpc$MethodHandlers.invoke(SqlComplianceServiceGrpc.java:423)
> at
> com.google.zetasql.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171)
> at
> com.google.zetasql.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283)
> at
> com.google.zetasql.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:711)
> at
> com.google.zetasql.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
> at
>
[jira] [Resolved] (BEAM-9521) NullPointerException in convertRexNodeFromResolvedExpr
[
https://issues.apache.org/jira/browse/BEAM-9521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Pilloud resolved BEAM-9521.
--
Fix Version/s: 2.25.0
Resolution: Fixed
> NullPointerException in convertRexNodeFromResolvedExpr
> --
>
> Key: BEAM-9521
> URL: https://issues.apache.org/jira/browse/BEAM-9521
> Project: Beam
> Issue Type: Bug
> Components: dsl-sql-zetasql
>Reporter: Andrew Pilloud
>Assignee: Andrew Pilloud
>Priority: Major
> Labels: zetasql-compliance
> Fix For: 2.25.0
>
>
> {code:java}
> Mar 16, 2020 12:57:10 PM
> cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
> executeQuery
> INFO: Processing Sql statement: SELECT (SELECT AS STRUCT
> (SELECT AS STRUCT l.float_val lf, r.float_val rf),
> l.float_val = r.float_val),
>(SELECT AS STRUCT
> (SELECT AS STRUCT l.double_val ld, r.double_val rd),
> l.double_val = r.double_val)
> FROM TestTableZero l, TestTableZero r
> WHERE l.row_id > r.row_id
> Mar 16, 2020 12:57:10 PM
> com.google.zetasql.io.grpc.internal.SerializingExecutor run
> SEVERE: Exception while executing runnable
> com.google.zetasql.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed@710f8c0d
> java.lang.NullPointerException
> at
> org.apache.beam.sdk.extensions.sql.zetasql.translation.ExpressionConverter.convertRexNodeFromResolvedExpr(ExpressionConverter.java:301)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.translation.JoinScanConverter.convert(JoinScanConverter.java:83)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.translation.JoinScanConverter.convert(JoinScanConverter.java:36)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.translation.QueryStatementConverter.convertNode(QueryStatementConverter.java:97)
> at
> java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
> at java.util.Collections$2.tryAdvance(Collections.java:4717)
> at java.util.Collections$2.forEachRemaining(Collections.java:4725)
> at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
> at
> java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
> at
> java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
> at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
> at
> java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.translation.QueryStatementConverter.convertNode(QueryStatementConverter.java:96)
> at
> java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
> at java.util.Collections$2.tryAdvance(Collections.java:4717)
> at java.util.Collections$2.forEachRemaining(Collections.java:4725)
> at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
> at
> java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
> at
> java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
> at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
> at
> java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.translation.QueryStatementConverter.convertNode(QueryStatementConverter.java:96)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.translation.QueryStatementConverter.convert(QueryStatementConverter.java:84)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.translation.QueryStatementConverter.convertRootQuery(QueryStatementConverter.java:51)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.ZetaSQLPlannerImpl.rel(ZetaSQLPlannerImpl.java:160)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.ZetaSQLQueryPlanner.convertToBeamRelInternal(ZetaSQLQueryPlanner.java:131)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.ZetaSQLQueryPlanner.convertToBeamRel(ZetaSQLQueryPlanner.java:115)
> at
> cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl.executeQuery(ExecuteQueryServiceServer.java:242)
> at
> com.google.zetasql.testing.SqlComplianceServiceGrpc$MethodHandlers.invoke(SqlComplianceServiceGrpc.java:423)
> at
> com.google.zetasql.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171)
> at
> com.google.zetasql.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283)
> at
> com.google.zetasql.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:711)
> at
>
[jira] [Work logged] (BEAM-9513) NullPointerException in convertRexNodeFromResolvedExprWithRefScan
[
https://issues.apache.org/jira/browse/BEAM-9513?focusedWorklogId=405789=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-405789
]
ASF GitHub Bot logged work on BEAM-9513:
Author: ASF GitHub Bot
Created on: 18/Mar/20 22:29
Start Date: 18/Mar/20 22:29
Worklog Time Spent: 10m
Work Description: apilloud commented on pull request #11132:
[BEAM-9513][BEAM-9521] Translate ZetaSQL joins without condition
URL: https://github.com/apache/beam/pull/11132
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 405789)
Remaining Estimate: 0h
Time Spent: 10m
> NullPointerException in convertRexNodeFromResolvedExprWithRefScan
> -
>
> Key: BEAM-9513
> URL: https://issues.apache.org/jira/browse/BEAM-9513
> Project: Beam
> Issue Type: Bug
> Components: dsl-sql-zetasql
>Reporter: Andrew Pilloud
>Assignee: Andrew Pilloud
>Priority: Major
> Labels: zetasql-compliance
> Fix For: 2.25.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> {code:java}
> Mar 16, 2020 12:58:26 PM
> cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
> executeQuery
> INFO: Processing Sql statement: WITH
> R AS (SELECT 2 a),
> S AS (SELECT a-1 x, a+1 y FROM R)
> SELECT a, x, y FROM R, S
> Mar 16, 2020 12:58:26 PM
> com.google.zetasql.io.grpc.internal.SerializingExecutor run
> SEVERE: Exception while executing runnable
> com.google.zetasql.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed@5341cc1c
> java.lang.NullPointerException
> at
> org.apache.beam.sdk.extensions.sql.zetasql.translation.ExpressionConverter.convertRexNodeFromResolvedExprWithRefScan(ExpressionConverter.java:374)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.translation.JoinScanWithRefConverter.convert(JoinScanWithRefConverter.java:63)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.translation.JoinScanWithRefConverter.convert(JoinScanWithRefConverter.java:35)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.translation.QueryStatementConverter.convertNode(QueryStatementConverter.java:97)
> at
> java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
> at java.util.Collections$2.tryAdvance(Collections.java:4717)
> at java.util.Collections$2.forEachRemaining(Collections.java:4725)
> at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
> at
> java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
> at
> java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
> at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
> at
> java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.translation.QueryStatementConverter.convertNode(QueryStatementConverter.java:96)
> at
> java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
> at java.util.Collections$2.tryAdvance(Collections.java:4717)
> at java.util.Collections$2.forEachRemaining(Collections.java:4725)
> at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
> at
> java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
> at
> java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
> at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
> at
> java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.translation.QueryStatementConverter.convertNode(QueryStatementConverter.java:96)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.translation.QueryStatementConverter.convert(QueryStatementConverter.java:84)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.translation.QueryStatementConverter.convertRootQuery(QueryStatementConverter.java:51)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.ZetaSQLPlannerImpl.rel(ZetaSQLPlannerImpl.java:160)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.ZetaSQLQueryPlanner.convertToBeamRelInternal(ZetaSQLQueryPlanner.java:131)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.ZetaSQLQueryPlanner.convertToBeamRel(ZetaSQLQueryPlanner.java:115)
> at
>
[jira] [Work logged] (BEAM-9468) Add Google Cloud Healthcare API IO Connectors
[ https://issues.apache.org/jira/browse/BEAM-9468?focusedWorklogId=405787=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-405787 ] ASF GitHub Bot logged work on BEAM-9468: Author: ASF GitHub Bot Created on: 18/Mar/20 22:28 Start Date: 18/Mar/20 22:28 Worklog Time Spent: 10m Work Description: jaketf commented on issue #11151: [BEAM-9468] Hl7v2 io URL: https://github.com/apache/beam/pull/11151#issuecomment-600890728 @lastomato Thoughts on how we could rework the HCAPI client to get around this failing CI Check? https://builds.apache.org/job/beam_PreCommit_Java_Commit/10418/testReport/org.apache.beam.sdk.io.gcp/GcpApiSurfaceTest/testGcpApiSurface/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 405787) Time Spent: 2.5h (was: 2h 20m) > Add Google Cloud Healthcare API IO Connectors > - > > Key: BEAM-9468 > URL: https://issues.apache.org/jira/browse/BEAM-9468 > Project: Beam > Issue Type: New Feature > Components: io-java-gcp >Reporter: Jacob Ferriero >Assignee: Jacob Ferriero >Priority: Minor > Time Spent: 2.5h > Remaining Estimate: 0h > > Add IO Transforms for the HL7v2, FHIR and DICOM stores in the [Google Cloud > Healthcare API|https://cloud.google.com/healthcare/docs/] > HL7v2IO > FHIRIO > DICOM -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9533) Replace *-gcp/*-aws tox suites with *-cloud suites to run unit tests for both
[ https://issues.apache.org/jira/browse/BEAM-9533?focusedWorklogId=405785=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-405785 ] ASF GitHub Bot logged work on BEAM-9533: Author: ASF GitHub Bot Created on: 18/Mar/20 22:23 Start Date: 18/Mar/20 22:23 Worklog Time Spent: 10m Work Description: pabloem commented on issue #11158: [BEAM-9533] Fixing tox.ini variants URL: https://github.com/apache/beam/pull/11158#issuecomment-600888942 Run Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 405785) Time Spent: 1h (was: 50m) > Replace *-gcp/*-aws tox suites with *-cloud suites to run unit tests for both > - > > Key: BEAM-9533 > URL: https://issues.apache.org/jira/browse/BEAM-9533 > Project: Beam > Issue Type: Bug > Components: testing >Reporter: Pablo Estrada >Assignee: Pablo Estrada >Priority: Minor > Time Spent: 1h > Remaining Estimate: 0h > > Currently there are `py37-gcp`, py37-aws test suites. Let's consolidate all > of them into py37-cloud, along with other py35-gcp, py27-gcp, etc. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9550) beam_PostCommit_Python_Chicago_Taxi_Flink OOM
[ https://issues.apache.org/jira/browse/BEAM-9550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kyle Weaver updated BEAM-9550: -- Description: https://builds.apache.org/job/beam_PostCommit_Python_Chicago_Taxi_Flink/ The following error has been occurring consistently for several days: 07:57:26 ERROR:root:java.lang.OutOfMemoryError: Metaspace 07:57:27 Traceback (most recent call last): 07:57:27 File "tfdv_analyze_and_validate.py", line 227, in 07:57:27 main() 07:57:27 File "tfdv_analyze_and_validate.py", line 212, in main 07:57:27 project=known_args.metric_reporting_project) 07:57:27 File "tfdv_analyze_and_validate.py", line 132, in compute_stats 07:57:27 result.wait_until_finish() 07:57:27 File "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Chicago_Taxi_Flink/src/build/gradleenv/1866363813/local/lib/python2.7/site-packages/apache_beam/runners/portability/portable_runner.py", line 545, in wait_until_finish 07:57:27 (self._job_id, self._state, self._last_error_message())) 07:57:27 RuntimeError: Pipeline chicago-taxi-tfdv-20200317-144954-eval_9742ac2b-26bf-4d1d-835e-572d4efacfcb failed in state FAILED: java.lang.OutOfMemoryError: Metaspace was: The following error has been occurring consistently for several days: 07:57:26 ERROR:root:java.lang.OutOfMemoryError: Metaspace 07:57:27 Traceback (most recent call last): 07:57:27 File "tfdv_analyze_and_validate.py", line 227, in 07:57:27 main() 07:57:27 File "tfdv_analyze_and_validate.py", line 212, in main 07:57:27 project=known_args.metric_reporting_project) 07:57:27 File "tfdv_analyze_and_validate.py", line 132, in compute_stats 07:57:27 result.wait_until_finish() 07:57:27 File "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Chicago_Taxi_Flink/src/build/gradleenv/1866363813/local/lib/python2.7/site-packages/apache_beam/runners/portability/portable_runner.py", line 545, in wait_until_finish 07:57:27 (self._job_id, self._state, self._last_error_message())) 07:57:27 RuntimeError: Pipeline chicago-taxi-tfdv-20200317-144954-eval_9742ac2b-26bf-4d1d-835e-572d4efacfcb failed in state FAILED: java.lang.OutOfMemoryError: Metaspace > beam_PostCommit_Python_Chicago_Taxi_Flink OOM > - > > Key: BEAM-9550 > URL: https://issues.apache.org/jira/browse/BEAM-9550 > Project: Beam > Issue Type: Bug > Components: runner-flink, test-failures >Reporter: Kyle Weaver >Assignee: Kamil Wasilewski >Priority: Major > Labels: currently-failing > > https://builds.apache.org/job/beam_PostCommit_Python_Chicago_Taxi_Flink/ > The following error has been occurring consistently for several days: > 07:57:26 ERROR:root:java.lang.OutOfMemoryError: Metaspace > 07:57:27 Traceback (most recent call last): > 07:57:27 File "tfdv_analyze_and_validate.py", line 227, in > 07:57:27 main() > 07:57:27 File "tfdv_analyze_and_validate.py", line 212, in main > 07:57:27 project=known_args.metric_reporting_project) > 07:57:27 File "tfdv_analyze_and_validate.py", line 132, in compute_stats > 07:57:27 result.wait_until_finish() > 07:57:27 File > "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Chicago_Taxi_Flink/src/build/gradleenv/1866363813/local/lib/python2.7/site-packages/apache_beam/runners/portability/portable_runner.py", > line 545, in wait_until_finish > 07:57:27 (self._job_id, self._state, self._last_error_message())) > 07:57:27 RuntimeError: Pipeline > chicago-taxi-tfdv-20200317-144954-eval_9742ac2b-26bf-4d1d-835e-572d4efacfcb > failed in state FAILED: java.lang.OutOfMemoryError: Metaspace -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9511) ArrayScanToUncollectConverter: ResolvedParameter cannot be cast to ResolvedLiteral
[
https://issues.apache.org/jira/browse/BEAM-9511?focusedWorklogId=405783=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-405783
]
ASF GitHub Bot logged work on BEAM-9511:
Author: ASF GitHub Bot
Created on: 18/Mar/20 22:20
Start Date: 18/Mar/20 22:20
Worklog Time Spent: 10m
Work Description: apilloud commented on issue #11133:
[BEAM-9511][BEAM-9515][BEAM-9516] Uncollect takes arbitrary expressions
URL: https://github.com/apache/beam/pull/11133#issuecomment-600887799
Added tests.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 405783)
Time Spent: 20m (was: 10m)
> ArrayScanToUncollectConverter: ResolvedParameter cannot be cast to
> ResolvedLiteral
> --
>
> Key: BEAM-9511
> URL: https://issues.apache.org/jira/browse/BEAM-9511
> Project: Beam
> Issue Type: Bug
> Components: dsl-sql-zetasql
>Reporter: Andrew Pilloud
>Assignee: Andrew Pilloud
>Priority: Major
> Labels: zetasql-compliance
> Time Spent: 20m
> Remaining Estimate: 0h
>
>
> {code:java}
> Mar 16, 2020 12:57:42 PM
> cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
> executeQuery
> INFO: Processing Sql statement: SELECT elem FROM UNNEST(@string_arr) AS elem
> Mar 16, 2020 12:57:42 PM
> com.google.zetasql.io.grpc.internal.SerializingExecutor run
> SEVERE: Exception while executing runnable
> com.google.zetasql.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed@2f5b49be
> java.lang.ClassCastException:
> com.google.zetasql.resolvedast.ResolvedNodes$ResolvedParameter cannot be cast
> to com.google.zetasql.resolvedast.ResolvedNodes$ResolvedLiteral
> at
> org.apache.beam.sdk.extensions.sql.zetasql.translation.ArrayScanToUncollectConverter.convert(ArrayScanToUncollectConverter.java:45)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.translation.ArrayScanToUncollectConverter.convert(ArrayScanToUncollectConverter.java:31)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.translation.QueryStatementConverter.convertNode(QueryStatementConverter.java:97)
> at
> java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
> at java.util.Collections$2.tryAdvance(Collections.java:4717)
> at java.util.Collections$2.forEachRemaining(Collections.java:4725)
> at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
> at
> java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
> at
> java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
> at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
> at
> java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.translation.QueryStatementConverter.convertNode(QueryStatementConverter.java:96)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.translation.QueryStatementConverter.convert(QueryStatementConverter.java:84)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.translation.QueryStatementConverter.convertRootQuery(QueryStatementConverter.java:51)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.ZetaSQLPlannerImpl.rel(ZetaSQLPlannerImpl.java:160)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.ZetaSQLQueryPlanner.convertToBeamRelInternal(ZetaSQLQueryPlanner.java:131)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.ZetaSQLQueryPlanner.convertToBeamRel(ZetaSQLQueryPlanner.java:115)
> at
> cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl.executeQuery(ExecuteQueryServiceServer.java:242)
> at
> com.google.zetasql.testing.SqlComplianceServiceGrpc$MethodHandlers.invoke(SqlComplianceServiceGrpc.java:423)
> at
> com.google.zetasql.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171)
> at
> com.google.zetasql.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283)
> at
> com.google.zetasql.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:711)
> at
> com.google.zetasql.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
> at
>
[jira] [Updated] (BEAM-9515) ArrayScanToUncollectConverter Unnest does not support sub-queries
[
https://issues.apache.org/jira/browse/BEAM-9515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Pilloud updated BEAM-9515:
-
Summary: ArrayScanToUncollectConverter Unnest does not support sub-queries
(was: Unnest does not support sub-queries)
> ArrayScanToUncollectConverter Unnest does not support sub-queries
> -
>
> Key: BEAM-9515
> URL: https://issues.apache.org/jira/browse/BEAM-9515
> Project: Beam
> Issue Type: New Feature
> Components: dsl-sql-zetasql
>Reporter: Andrew Pilloud
>Assignee: Andrew Pilloud
>Priority: Major
> Labels: zetasql-compliance
>
> {code:java}
> Mar 16, 2020 1:00:02 PM
> cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
> executeQuery
> INFO: Processing Sql statement: SELECT * FROM UNNEST(ARRAY(
> SELECT bool_val FROM AllTypesTable t
> ORDER BY bool_val ASC
> )) x WITH OFFSET POS
> Mar 16, 2020 1:00:02 PM
> com.google.zetasql.io.grpc.internal.SerializingExecutor run
> SEVERE: Exception while executing runnable
> com.google.zetasql.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed@7b42f9e7
> java.lang.ClassCastException:
> com.google.zetasql.resolvedast.ResolvedNodes$ResolvedSubqueryExpr cannot be
> cast to com.google.zetasql.resolvedast.ResolvedNodes$ResolvedLiteral
> at
> org.apache.beam.sdk.extensions.sql.zetasql.translation.ArrayScanToUncollectConverter.convert(ArrayScanToUncollectConverter.java:45)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.translation.ArrayScanToUncollectConverter.convert(ArrayScanToUncollectConverter.java:31)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.translation.QueryStatementConverter.convertNode(QueryStatementConverter.java:97)
> at
> java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
> at java.util.Collections$2.tryAdvance(Collections.java:4717)
> at java.util.Collections$2.forEachRemaining(Collections.java:4725)
> at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
> at
> java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
> at
> java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
> at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
> at
> java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.translation.QueryStatementConverter.convertNode(QueryStatementConverter.java:96)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.translation.QueryStatementConverter.convert(QueryStatementConverter.java:84)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.translation.QueryStatementConverter.convertRootQuery(QueryStatementConverter.java:51)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.ZetaSQLPlannerImpl.rel(ZetaSQLPlannerImpl.java:160)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.ZetaSQLQueryPlanner.convertToBeamRelInternal(ZetaSQLQueryPlanner.java:131)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.ZetaSQLQueryPlanner.convertToBeamRel(ZetaSQLQueryPlanner.java:115)
> at
> cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl.executeQuery(ExecuteQueryServiceServer.java:242)
> at
> com.google.zetasql.testing.SqlComplianceServiceGrpc$MethodHandlers.invoke(SqlComplianceServiceGrpc.java:423)
> at
> com.google.zetasql.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171)
> at
> com.google.zetasql.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283)
> at
> com.google.zetasql.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:711)
> at
> com.google.zetasql.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
> at
> com.google.zetasql.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (BEAM-9515) ResolvedSubqueryExpr can not be cast to ResolvedLiteral
[
https://issues.apache.org/jira/browse/BEAM-9515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Pilloud updated BEAM-9515:
-
Issue Type: New Feature (was: Bug)
> ResolvedSubqueryExpr can not be cast to ResolvedLiteral
> ---
>
> Key: BEAM-9515
> URL: https://issues.apache.org/jira/browse/BEAM-9515
> Project: Beam
> Issue Type: New Feature
> Components: dsl-sql-zetasql
>Reporter: Andrew Pilloud
>Assignee: Andrew Pilloud
>Priority: Major
> Labels: zetasql-compliance
>
> {code:java}
> Mar 16, 2020 1:00:02 PM
> cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
> executeQuery
> INFO: Processing Sql statement: SELECT * FROM UNNEST(ARRAY(
> SELECT bool_val FROM AllTypesTable t
> ORDER BY bool_val ASC
> )) x WITH OFFSET POS
> Mar 16, 2020 1:00:02 PM
> com.google.zetasql.io.grpc.internal.SerializingExecutor run
> SEVERE: Exception while executing runnable
> com.google.zetasql.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed@7b42f9e7
> java.lang.ClassCastException:
> com.google.zetasql.resolvedast.ResolvedNodes$ResolvedSubqueryExpr cannot be
> cast to com.google.zetasql.resolvedast.ResolvedNodes$ResolvedLiteral
> at
> org.apache.beam.sdk.extensions.sql.zetasql.translation.ArrayScanToUncollectConverter.convert(ArrayScanToUncollectConverter.java:45)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.translation.ArrayScanToUncollectConverter.convert(ArrayScanToUncollectConverter.java:31)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.translation.QueryStatementConverter.convertNode(QueryStatementConverter.java:97)
> at
> java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
> at java.util.Collections$2.tryAdvance(Collections.java:4717)
> at java.util.Collections$2.forEachRemaining(Collections.java:4725)
> at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
> at
> java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
> at
> java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
> at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
> at
> java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.translation.QueryStatementConverter.convertNode(QueryStatementConverter.java:96)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.translation.QueryStatementConverter.convert(QueryStatementConverter.java:84)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.translation.QueryStatementConverter.convertRootQuery(QueryStatementConverter.java:51)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.ZetaSQLPlannerImpl.rel(ZetaSQLPlannerImpl.java:160)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.ZetaSQLQueryPlanner.convertToBeamRelInternal(ZetaSQLQueryPlanner.java:131)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.ZetaSQLQueryPlanner.convertToBeamRel(ZetaSQLQueryPlanner.java:115)
> at
> cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl.executeQuery(ExecuteQueryServiceServer.java:242)
> at
> com.google.zetasql.testing.SqlComplianceServiceGrpc$MethodHandlers.invoke(SqlComplianceServiceGrpc.java:423)
> at
> com.google.zetasql.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171)
> at
> com.google.zetasql.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283)
> at
> com.google.zetasql.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:711)
> at
> com.google.zetasql.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
> at
> com.google.zetasql.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (BEAM-9515) Unnest does not support sub-queries
[
https://issues.apache.org/jira/browse/BEAM-9515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Pilloud updated BEAM-9515:
-
Summary: Unnest does not support sub-queries (was: ResolvedSubqueryExpr
can not be cast to ResolvedLiteral)
> Unnest does not support sub-queries
> ---
>
> Key: BEAM-9515
> URL: https://issues.apache.org/jira/browse/BEAM-9515
> Project: Beam
> Issue Type: New Feature
> Components: dsl-sql-zetasql
>Reporter: Andrew Pilloud
>Assignee: Andrew Pilloud
>Priority: Major
> Labels: zetasql-compliance
>
> {code:java}
> Mar 16, 2020 1:00:02 PM
> cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
> executeQuery
> INFO: Processing Sql statement: SELECT * FROM UNNEST(ARRAY(
> SELECT bool_val FROM AllTypesTable t
> ORDER BY bool_val ASC
> )) x WITH OFFSET POS
> Mar 16, 2020 1:00:02 PM
> com.google.zetasql.io.grpc.internal.SerializingExecutor run
> SEVERE: Exception while executing runnable
> com.google.zetasql.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed@7b42f9e7
> java.lang.ClassCastException:
> com.google.zetasql.resolvedast.ResolvedNodes$ResolvedSubqueryExpr cannot be
> cast to com.google.zetasql.resolvedast.ResolvedNodes$ResolvedLiteral
> at
> org.apache.beam.sdk.extensions.sql.zetasql.translation.ArrayScanToUncollectConverter.convert(ArrayScanToUncollectConverter.java:45)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.translation.ArrayScanToUncollectConverter.convert(ArrayScanToUncollectConverter.java:31)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.translation.QueryStatementConverter.convertNode(QueryStatementConverter.java:97)
> at
> java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
> at java.util.Collections$2.tryAdvance(Collections.java:4717)
> at java.util.Collections$2.forEachRemaining(Collections.java:4725)
> at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
> at
> java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
> at
> java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
> at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
> at
> java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.translation.QueryStatementConverter.convertNode(QueryStatementConverter.java:96)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.translation.QueryStatementConverter.convert(QueryStatementConverter.java:84)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.translation.QueryStatementConverter.convertRootQuery(QueryStatementConverter.java:51)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.ZetaSQLPlannerImpl.rel(ZetaSQLPlannerImpl.java:160)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.ZetaSQLQueryPlanner.convertToBeamRelInternal(ZetaSQLQueryPlanner.java:131)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.ZetaSQLQueryPlanner.convertToBeamRel(ZetaSQLQueryPlanner.java:115)
> at
> cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl.executeQuery(ExecuteQueryServiceServer.java:242)
> at
> com.google.zetasql.testing.SqlComplianceServiceGrpc$MethodHandlers.invoke(SqlComplianceServiceGrpc.java:423)
> at
> com.google.zetasql.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171)
> at
> com.google.zetasql.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283)
> at
> com.google.zetasql.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:711)
> at
> com.google.zetasql.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
> at
> com.google.zetasql.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (BEAM-9515) ResolvedSubqueryExpr can not be cast to ResolvedLiteral
[
https://issues.apache.org/jira/browse/BEAM-9515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Pilloud updated BEAM-9515:
-
Priority: Major (was: Trivial)
> ResolvedSubqueryExpr can not be cast to ResolvedLiteral
> ---
>
> Key: BEAM-9515
> URL: https://issues.apache.org/jira/browse/BEAM-9515
> Project: Beam
> Issue Type: Bug
> Components: dsl-sql-zetasql
>Reporter: Andrew Pilloud
>Assignee: Andrew Pilloud
>Priority: Major
> Labels: zetasql-compliance
>
> {code:java}
> Mar 16, 2020 1:00:02 PM
> cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
> executeQuery
> INFO: Processing Sql statement: SELECT * FROM UNNEST(ARRAY(
> SELECT bool_val FROM AllTypesTable t
> ORDER BY bool_val ASC
> )) x WITH OFFSET POS
> Mar 16, 2020 1:00:02 PM
> com.google.zetasql.io.grpc.internal.SerializingExecutor run
> SEVERE: Exception while executing runnable
> com.google.zetasql.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed@7b42f9e7
> java.lang.ClassCastException:
> com.google.zetasql.resolvedast.ResolvedNodes$ResolvedSubqueryExpr cannot be
> cast to com.google.zetasql.resolvedast.ResolvedNodes$ResolvedLiteral
> at
> org.apache.beam.sdk.extensions.sql.zetasql.translation.ArrayScanToUncollectConverter.convert(ArrayScanToUncollectConverter.java:45)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.translation.ArrayScanToUncollectConverter.convert(ArrayScanToUncollectConverter.java:31)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.translation.QueryStatementConverter.convertNode(QueryStatementConverter.java:97)
> at
> java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
> at java.util.Collections$2.tryAdvance(Collections.java:4717)
> at java.util.Collections$2.forEachRemaining(Collections.java:4725)
> at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
> at
> java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
> at
> java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
> at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
> at
> java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.translation.QueryStatementConverter.convertNode(QueryStatementConverter.java:96)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.translation.QueryStatementConverter.convert(QueryStatementConverter.java:84)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.translation.QueryStatementConverter.convertRootQuery(QueryStatementConverter.java:51)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.ZetaSQLPlannerImpl.rel(ZetaSQLPlannerImpl.java:160)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.ZetaSQLQueryPlanner.convertToBeamRelInternal(ZetaSQLQueryPlanner.java:131)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.ZetaSQLQueryPlanner.convertToBeamRel(ZetaSQLQueryPlanner.java:115)
> at
> cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl.executeQuery(ExecuteQueryServiceServer.java:242)
> at
> com.google.zetasql.testing.SqlComplianceServiceGrpc$MethodHandlers.invoke(SqlComplianceServiceGrpc.java:423)
> at
> com.google.zetasql.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171)
> at
> com.google.zetasql.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283)
> at
> com.google.zetasql.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:711)
> at
> com.google.zetasql.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
> at
> com.google.zetasql.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (BEAM-9515) ResolvedSubqueryExpr can not be cast to ResolvedLiteral
[
https://issues.apache.org/jira/browse/BEAM-9515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Pilloud updated BEAM-9515:
-
Priority: Trivial (was: Major)
> ResolvedSubqueryExpr can not be cast to ResolvedLiteral
> ---
>
> Key: BEAM-9515
> URL: https://issues.apache.org/jira/browse/BEAM-9515
> Project: Beam
> Issue Type: Bug
> Components: dsl-sql-zetasql
>Reporter: Andrew Pilloud
>Assignee: Andrew Pilloud
>Priority: Trivial
> Labels: zetasql-compliance
>
> {code:java}
> Mar 16, 2020 1:00:02 PM
> cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
> executeQuery
> INFO: Processing Sql statement: SELECT * FROM UNNEST(ARRAY(
> SELECT bool_val FROM AllTypesTable t
> ORDER BY bool_val ASC
> )) x WITH OFFSET POS
> Mar 16, 2020 1:00:02 PM
> com.google.zetasql.io.grpc.internal.SerializingExecutor run
> SEVERE: Exception while executing runnable
> com.google.zetasql.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed@7b42f9e7
> java.lang.ClassCastException:
> com.google.zetasql.resolvedast.ResolvedNodes$ResolvedSubqueryExpr cannot be
> cast to com.google.zetasql.resolvedast.ResolvedNodes$ResolvedLiteral
> at
> org.apache.beam.sdk.extensions.sql.zetasql.translation.ArrayScanToUncollectConverter.convert(ArrayScanToUncollectConverter.java:45)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.translation.ArrayScanToUncollectConverter.convert(ArrayScanToUncollectConverter.java:31)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.translation.QueryStatementConverter.convertNode(QueryStatementConverter.java:97)
> at
> java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
> at java.util.Collections$2.tryAdvance(Collections.java:4717)
> at java.util.Collections$2.forEachRemaining(Collections.java:4725)
> at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
> at
> java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
> at
> java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
> at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
> at
> java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.translation.QueryStatementConverter.convertNode(QueryStatementConverter.java:96)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.translation.QueryStatementConverter.convert(QueryStatementConverter.java:84)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.translation.QueryStatementConverter.convertRootQuery(QueryStatementConverter.java:51)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.ZetaSQLPlannerImpl.rel(ZetaSQLPlannerImpl.java:160)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.ZetaSQLQueryPlanner.convertToBeamRelInternal(ZetaSQLQueryPlanner.java:131)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.ZetaSQLQueryPlanner.convertToBeamRel(ZetaSQLQueryPlanner.java:115)
> at
> cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl.executeQuery(ExecuteQueryServiceServer.java:242)
> at
> com.google.zetasql.testing.SqlComplianceServiceGrpc$MethodHandlers.invoke(SqlComplianceServiceGrpc.java:423)
> at
> com.google.zetasql.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171)
> at
> com.google.zetasql.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283)
> at
> com.google.zetasql.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:711)
> at
> com.google.zetasql.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
> at
> com.google.zetasql.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (BEAM-9516) ArrayScanToUncollectConverter: ResolvedSubqueryExpr cannot be cast to ResolvedLiteral
[
https://issues.apache.org/jira/browse/BEAM-9516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Pilloud resolved BEAM-9516.
--
Fix Version/s: Not applicable
Resolution: Duplicate
> ArrayScanToUncollectConverter: ResolvedSubqueryExpr cannot be cast to
> ResolvedLiteral
> -
>
> Key: BEAM-9516
> URL: https://issues.apache.org/jira/browse/BEAM-9516
> Project: Beam
> Issue Type: Bug
> Components: dsl-sql-zetasql
>Reporter: Andrew Pilloud
>Assignee: Andrew Pilloud
>Priority: Major
> Labels: zetasql-compliance
> Fix For: Not applicable
>
>
> {code:java}
> Mar 16, 2020 12:58:49 PM
> com.google.zetasql.io.grpc.internal.SerializingExecutor run
> SEVERE: Exception while executing runnable
> com.google.zetasql.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed@7c1d4ea6
> java.lang.ClassCastException:
> com.google.zetasql.resolvedast.ResolvedNodes$ResolvedSubqueryExpr cannot be
> cast to com.google.zetasql.resolvedast.ResolvedNodes$ResolvedLiteral
> at
> org.apache.beam.sdk.extensions.sql.zetasql.translation.ArrayScanToUncollectConverter.convert(ArrayScanToUncollectConverter.java:45)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.translation.ArrayScanToUncollectConverter.convert(ArrayScanToUncollectConverter.java:31)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.translation.QueryStatementConverter.convertNode(QueryStatementConverter.java:97)
> at
> java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
> at java.util.Collections$2.tryAdvance(Collections.java:4717)
> at java.util.Collections$2.forEachRemaining(Collections.java:4725)
> at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
> at
> java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
> at
> java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
> at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
> at
> java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.translation.QueryStatementConverter.convertNode(QueryStatementConverter.java:96)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.translation.QueryStatementConverter.convert(QueryStatementConverter.java:84)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.translation.QueryStatementConverter.convertRootQuery(QueryStatementConverter.java:51)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.ZetaSQLPlannerImpl.rel(ZetaSQLPlannerImpl.java:160)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.ZetaSQLQueryPlanner.convertToBeamRelInternal(ZetaSQLQueryPlanner.java:131)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.ZetaSQLQueryPlanner.convertToBeamRel(ZetaSQLQueryPlanner.java:115)
> at
> cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl.executeQuery(ExecuteQueryServiceServer.java:242)
> at
> com.google.zetasql.testing.SqlComplianceServiceGrpc$MethodHandlers.invoke(SqlComplianceServiceGrpc.java:423)
> at
> com.google.zetasql.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171)
> at
> com.google.zetasql.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283)
> at
> com.google.zetasql.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:711)
> at
> com.google.zetasql.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
> at
> com.google.zetasql.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (BEAM-9550) beam_PostCommit_Python_Chicago_Taxi_Flink OOM
Kyle Weaver created BEAM-9550: - Summary: beam_PostCommit_Python_Chicago_Taxi_Flink OOM Key: BEAM-9550 URL: https://issues.apache.org/jira/browse/BEAM-9550 Project: Beam Issue Type: Bug Components: runner-flink, test-failures Reporter: Kyle Weaver Assignee: Kamil Wasilewski The following error has been occurring consistently for several days: 07:57:26 ERROR:root:java.lang.OutOfMemoryError: Metaspace 07:57:27 Traceback (most recent call last): 07:57:27 File "tfdv_analyze_and_validate.py", line 227, in 07:57:27 main() 07:57:27 File "tfdv_analyze_and_validate.py", line 212, in main 07:57:27 project=known_args.metric_reporting_project) 07:57:27 File "tfdv_analyze_and_validate.py", line 132, in compute_stats 07:57:27 result.wait_until_finish() 07:57:27 File "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Chicago_Taxi_Flink/src/build/gradleenv/1866363813/local/lib/python2.7/site-packages/apache_beam/runners/portability/portable_runner.py", line 545, in wait_until_finish 07:57:27 (self._job_id, self._state, self._last_error_message())) 07:57:27 RuntimeError: Pipeline chicago-taxi-tfdv-20200317-144954-eval_9742ac2b-26bf-4d1d-835e-572d4efacfcb failed in state FAILED: java.lang.OutOfMemoryError: Metaspace -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-9515) ResolvedSubqueryExpr can not be cast to ResolvedLiteral
[
https://issues.apache.org/jira/browse/BEAM-9515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17062091#comment-17062091
]
Andrew Pilloud commented on BEAM-9515:
--
After [https://github.com/apache/beam/pull/11133] this is a
"java.lang.UnsupportedOperationException: Does not support sub-queries "
> ResolvedSubqueryExpr can not be cast to ResolvedLiteral
> ---
>
> Key: BEAM-9515
> URL: https://issues.apache.org/jira/browse/BEAM-9515
> Project: Beam
> Issue Type: Bug
> Components: dsl-sql-zetasql
>Reporter: Andrew Pilloud
>Assignee: Andrew Pilloud
>Priority: Major
> Labels: zetasql-compliance
>
> {code:java}
> Mar 16, 2020 1:00:02 PM
> cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl
> executeQuery
> INFO: Processing Sql statement: SELECT * FROM UNNEST(ARRAY(
> SELECT bool_val FROM AllTypesTable t
> ORDER BY bool_val ASC
> )) x WITH OFFSET POS
> Mar 16, 2020 1:00:02 PM
> com.google.zetasql.io.grpc.internal.SerializingExecutor run
> SEVERE: Exception while executing runnable
> com.google.zetasql.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed@7b42f9e7
> java.lang.ClassCastException:
> com.google.zetasql.resolvedast.ResolvedNodes$ResolvedSubqueryExpr cannot be
> cast to com.google.zetasql.resolvedast.ResolvedNodes$ResolvedLiteral
> at
> org.apache.beam.sdk.extensions.sql.zetasql.translation.ArrayScanToUncollectConverter.convert(ArrayScanToUncollectConverter.java:45)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.translation.ArrayScanToUncollectConverter.convert(ArrayScanToUncollectConverter.java:31)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.translation.QueryStatementConverter.convertNode(QueryStatementConverter.java:97)
> at
> java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
> at java.util.Collections$2.tryAdvance(Collections.java:4717)
> at java.util.Collections$2.forEachRemaining(Collections.java:4725)
> at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
> at
> java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
> at
> java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
> at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
> at
> java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.translation.QueryStatementConverter.convertNode(QueryStatementConverter.java:96)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.translation.QueryStatementConverter.convert(QueryStatementConverter.java:84)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.translation.QueryStatementConverter.convertRootQuery(QueryStatementConverter.java:51)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.ZetaSQLPlannerImpl.rel(ZetaSQLPlannerImpl.java:160)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.ZetaSQLQueryPlanner.convertToBeamRelInternal(ZetaSQLQueryPlanner.java:131)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.ZetaSQLQueryPlanner.convertToBeamRel(ZetaSQLQueryPlanner.java:115)
> at
> cloud.dataflow.sql.ExecuteQueryServiceServer$SqlComplianceServiceImpl.executeQuery(ExecuteQueryServiceServer.java:242)
> at
> com.google.zetasql.testing.SqlComplianceServiceGrpc$MethodHandlers.invoke(SqlComplianceServiceGrpc.java:423)
> at
> com.google.zetasql.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171)
> at
> com.google.zetasql.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283)
> at
> com.google.zetasql.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:711)
> at
> com.google.zetasql.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
> at
> com.google.zetasql.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9535) Cleanup portability protos.
[ https://issues.apache.org/jira/browse/BEAM-9535?focusedWorklogId=405771=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-405771 ] ASF GitHub Bot logged work on BEAM-9535: Author: ASF GitHub Bot Created on: 18/Mar/20 21:58 Start Date: 18/Mar/20 21:58 Worklog Time Spent: 10m Work Description: lukecwik commented on issue #11150: [BEAM-9535] Remove unused ParDoPayload.Parameters. URL: https://github.com/apache/beam/pull/11150#issuecomment-600879517 Run Portable_Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 405771) Time Spent: 40m (was: 0.5h) > Cleanup portability protos. > --- > > Key: BEAM-9535 > URL: https://issues.apache.org/jira/browse/BEAM-9535 > Project: Beam > Issue Type: Improvement > Components: beam-model >Reporter: Robert Bradshaw >Priority: Major > Fix For: 2.21.0 > > Time Spent: 40m > Remaining Estimate: 0h > > This is to provide a stable version of the FnAPI protos going forward. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9540) Rename beam:source:runner:0.1/beam:sink:runner:0.1 to beam:runner:source:v1/beam:runner:sink:v1
[ https://issues.apache.org/jira/browse/BEAM-9540?focusedWorklogId=405774=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-405774 ] ASF GitHub Bot logged work on BEAM-9540: Author: ASF GitHub Bot Created on: 18/Mar/20 21:58 Start Date: 18/Mar/20 21:58 Worklog Time Spent: 10m Work Description: lukecwik commented on issue #11157: [BEAM-9540] Rename beam:source:runner:0.1/beam:sink:runner:0.1 to beam:runner:source:v1/beam:runner:sink:v1 URL: https://github.com/apache/beam/pull/11157#issuecomment-600879773 Run Java PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 405774) Time Spent: 1h 50m (was: 1h 40m) > Rename beam:source:runner:0.1/beam:sink:runner:0.1 to > beam:runner:source:v1/beam:runner:sink:v1 > --- > > Key: BEAM-9540 > URL: https://issues.apache.org/jira/browse/BEAM-9540 > Project: Beam > Issue Type: Bug > Components: sdk-java-harness, sdk-py-harness >Reporter: Luke Cwik >Assignee: Luke Cwik >Priority: Minor > Time Spent: 1h 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9540) Rename beam:source:runner:0.1/beam:sink:runner:0.1 to beam:runner:source:v1/beam:runner:sink:v1
[ https://issues.apache.org/jira/browse/BEAM-9540?focusedWorklogId=405775=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-405775 ] ASF GitHub Bot logged work on BEAM-9540: Author: ASF GitHub Bot Created on: 18/Mar/20 21:58 Start Date: 18/Mar/20 21:58 Worklog Time Spent: 10m Work Description: lukecwik commented on issue #11157: [BEAM-9540] Rename beam:source:runner:0.1/beam:sink:runner:0.1 to beam:runner:source:v1/beam:runner:sink:v1 URL: https://github.com/apache/beam/pull/11157#issuecomment-600879819 Run Portable_Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 405775) Time Spent: 2h (was: 1h 50m) > Rename beam:source:runner:0.1/beam:sink:runner:0.1 to > beam:runner:source:v1/beam:runner:sink:v1 > --- > > Key: BEAM-9540 > URL: https://issues.apache.org/jira/browse/BEAM-9540 > Project: Beam > Issue Type: Bug > Components: sdk-java-harness, sdk-py-harness >Reporter: Luke Cwik >Assignee: Luke Cwik >Priority: Minor > Time Spent: 2h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9524) ib.show() spins forever when cells are re-executed
[
https://issues.apache.org/jira/browse/BEAM-9524?focusedWorklogId=405769=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-405769
]
ASF GitHub Bot logged work on BEAM-9524:
Author: ASF GitHub Bot
Created on: 18/Mar/20 21:57
Start Date: 18/Mar/20 21:57
Worklog Time Spent: 10m
Work Description: davidyan74 commented on pull request #11128:
[BEAM-9524] Fix for ib.show() executing indefinitely
URL: https://github.com/apache/beam/pull/11128#discussion_r394662812
##
File path:
sdks/python/apache_beam/runners/interactive/caching/streaming_cache.py
##
@@ -153,7 +153,8 @@ def __init__(
self._coder = coder
self._labels = labels
self._is_cache_complete = (
-is_cache_complete if is_cache_complete else lambda: True)
+is_cache_complete if is_cache_complete else lambda _: True)
+self._pipeline_id = labels[-1].split('_')[-1]
Review comment:
This looks kinda like a hack, and if we change the format of the label, this
will break.
Can we at least put the code that encodes and decodes labels in one place,
perhaps in a util module, and have this line call the decode function?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 405769)
Time Spent: 1h (was: 50m)
> ib.show() spins forever when cells are re-executed
> --
>
> Key: BEAM-9524
> URL: https://issues.apache.org/jira/browse/BEAM-9524
> Project: Beam
> Issue Type: Bug
> Components: runner-py-interactive
>Reporter: Sam Rohde
>Priority: Major
> Time Spent: 1h
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9535) Cleanup portability protos.
[ https://issues.apache.org/jira/browse/BEAM-9535?focusedWorklogId=405770=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-405770 ] ASF GitHub Bot logged work on BEAM-9535: Author: ASF GitHub Bot Created on: 18/Mar/20 21:57 Start Date: 18/Mar/20 21:57 Worklog Time Spent: 10m Work Description: lukecwik commented on issue #11150: [BEAM-9535] Remove unused ParDoPayload.Parameters. URL: https://github.com/apache/beam/pull/11150#issuecomment-600879449 Run Python2_PVR_Flink PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 405770) Time Spent: 0.5h (was: 20m) > Cleanup portability protos. > --- > > Key: BEAM-9535 > URL: https://issues.apache.org/jira/browse/BEAM-9535 > Project: Beam > Issue Type: Improvement > Components: beam-model >Reporter: Robert Bradshaw >Priority: Major > Fix For: 2.21.0 > > Time Spent: 0.5h > Remaining Estimate: 0h > > This is to provide a stable version of the FnAPI protos going forward. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9536) Return type of window start/end functions is incorrectly inferred to be INT64
[ https://issues.apache.org/jira/browse/BEAM-9536?focusedWorklogId=405766=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-405766 ] ASF GitHub Bot logged work on BEAM-9536: Author: ASF GitHub Bot Created on: 18/Mar/20 21:54 Start Date: 18/Mar/20 21:54 Worklog Time Spent: 10m Work Description: amaliujia commented on pull request #11152: [BEAM-9536] Specify return types of window start/end functions explicitly URL: https://github.com/apache/beam/pull/11152 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 405766) Time Spent: 40m (was: 0.5h) > Return type of window start/end functions is incorrectly inferred to be INT64 > - > > Key: BEAM-9536 > URL: https://issues.apache.org/jira/browse/BEAM-9536 > Project: Beam > Issue Type: Improvement > Components: dsl-sql-zetasql >Reporter: Yueyang Qiu >Assignee: Yueyang Qiu >Priority: Major > Time Spent: 40m > Remaining Estimate: 0h > > Our current implementation does not have the correct return type information > of the window start/end functions (e.g. TUMBLE_START). > LOC that cause the problem: > [https://github.com/apache/beam/blob/646f596988be9d6a739090f48d2fed07c8dfc17c/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/SqlAnalyzer.java#L62] > We should specify the return type explicitly here in the CREATE statements. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9549) Flaky portableWordCountBatch and portableWordCountStreaming tests
[
https://issues.apache.org/jira/browse/BEAM-9549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ning Kang updated BEAM-9549:
Description:
The tests :sdks:python:test-suites:portable:py2:portableWordCountBatch and
:sdks:python:test-suites:portable:py2:portableWordCountStreaming are flaky,
sometimes throws grpc errrors.
Stacktrace
!Sr5cNnx8sAW.png|width=2049,height=1001!
In text:
{code:java}
INFO:root:Using Python SDK docker image: apache/beam_python2.7_sdk:2.21.0.dev.
If the image is not available at local, we will try to pull from
hub.docker.comINFO:apache_beam.runners.portability.fn_api_runner_transforms:
INFO:apache_beam.utils.subprocess_server:Starting service
with ['docker' 'run' '-v' '/usr/bin/docker:/bin/docker' '-v'
'/var/run/docker.sock:/var/run/docker.sock' '--network=host'
'apache/beam_flink1.9_job_server:latest' '--job-host' 'localhost' '--job-port'
'58753' '--artifact-port' '60175' '--expansion-port'
'33067']INFO:apache_beam.utils.subprocess_server:[main] INFO
org.apache.beam.runners.fnexecution.jobsubmission.JobServerDriver -
ArtifactStagingService started on
localhost:60175INFO:apache_beam.utils.subprocess_server:[main] INFO
org.apache.beam.runners.fnexecution.jobsubmission.JobServerDriver - Java
ExpansionService started on
localhost:33067INFO:apache_beam.utils.subprocess_server:[main] INFO
org.apache.beam.runners.fnexecution.jobsubmission.JobServerDriver - JobService
started on localhost:58753ERROR:grpc._common:Exception deserializing
message!Traceback (most recent call last): File
"/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Portable_Python_Commit/src/build/gradleenv/1866363813/local/lib/python2.7/site-packages/grpc/_common.py",
line 84, in _transformreturn transformer(message)DecodeError: Error
parsing messageTraceback (most recent call last): File
"/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main"__main__",
fname, loader, pkg_name) File "/usr/lib/python2.7/runpy.py", line 72, in
_run_codeexec code in run_globals File
"/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Portable_Python_Commit/src/build/gradleenv/1866363813/lib/python2.7/site-packages/apache_beam/examples/wordcount.py",
line 142, in run() File
"/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Portable_Python_Commit/src/build/gradleenv/1866363813/lib/python2.7/site-packages/apache_beam/examples/wordcount.py",
line 121, in runresult = p.run() File
"/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Portable_Python_Commit/src/build/gradleenv/1866363813/local/lib/python2.7/site-packages/apache_beam/pipeline.py",
line 495, in runself._options).run(False) File
"/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Portable_Python_Commit/src/build/gradleenv/1866363813/local/lib/python2.7/site-packages/apache_beam/pipeline.py",
line 508, in runreturn self.runner.run_pipeline(self, self._options) File
"/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Portable_Python_Commit/src/build/gradleenv/1866363813/local/lib/python2.7/site-packages/apache_beam/runners/portability/portable_runner.py",
line 401, in run_pipelinejob_service_handle.submit(proto_pipeline) File
"/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Portable_Python_Commit/src/build/gradleenv/1866363813/local/lib/python2.7/site-packages/apache_beam/runners/portability/portable_runner.py",
line 102, in submitprepare_response = self.prepare(proto_pipeline) File
"/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Portable_Python_Commit/src/build/gradleenv/1866363813/local/lib/python2.7/site-packages/apache_beam/runners/portability/portable_runner.py",
line 179, in preparetimeout=self.timeout) File
"/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Portable_Python_Commit/src/build/gradleenv/1866363813/local/lib/python2.7/site-packages/grpc/_channel.py",
line 826, in __call__return _end_unary_response_blocking(state, call,
False, None) File
"/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Portable_Python_Commit/src/build/gradleenv/1866363813/local/lib/python2.7/site-packages/grpc/_channel.py",
line 729, in _end_unary_response_blockingraise
_InactiveRpcError(state)grpc._channel._InactiveRpcError: <_InactiveRpcError of
RPC that terminated with: status = StatusCode.INTERNALdetails = "Exception
deserializing response!" debug_error_string = "None">
{code}
A gradle scan [example|https://scans.gradle.com/s/yv63cefske4cy].
was:
The tests :sdks:python:test-suites:portable:py2:portableWordCountBatch and
:sdks:python:test-suites:portable:py2:portableWordCountStreaming are flaky,
sometimes throws grpc errrors.
Stacktrace
{code:java}
INFO:root:Using Python SDK docker image: apache/beam_python2.7_sdk:2.21.0.dev.
If the image is not available at local, we will try to pull from
[jira] [Work logged] (BEAM-7923) Interactive Beam
[ https://issues.apache.org/jira/browse/BEAM-7923?focusedWorklogId=405765=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-405765 ] ASF GitHub Bot logged work on BEAM-7923: Author: ASF GitHub Bot Created on: 18/Mar/20 21:51 Start Date: 18/Mar/20 21:51 Worklog Time Spent: 10m Work Description: aaltay commented on pull request #11161: [BEAM-7923] Change Transform Label Prefix Syntax URL: https://github.com/apache/beam/pull/11161 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 405765) Time Spent: 5h 50m (was: 5h 40m) > Interactive Beam > > > Key: BEAM-7923 > URL: https://issues.apache.org/jira/browse/BEAM-7923 > Project: Beam > Issue Type: New Feature > Components: runner-py-interactive >Reporter: Ning Kang >Assignee: Ning Kang >Priority: Major > Time Spent: 5h 50m > Remaining Estimate: 0h > > This is the top level ticket for all efforts leveraging [interactive > Beam|[https://github.com/apache/beam/tree/master/sdks/python/apache_beam/runners/interactive]] > As the development goes, blocking tickets will be added to this one. > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (BEAM-9549) Flaky portableWordCountBatch and portableWordCountStreaming tests
[
https://issues.apache.org/jira/browse/BEAM-9549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ahmet Altay reassigned BEAM-9549:
-
Assignee: Ankur Goenka
> Flaky portableWordCountBatch and portableWordCountStreaming tests
> -
>
> Key: BEAM-9549
> URL: https://issues.apache.org/jira/browse/BEAM-9549
> Project: Beam
> Issue Type: Test
> Components: test-failures
>Reporter: Ning Kang
>Assignee: Ankur Goenka
>Priority: Major
> Attachments: Sr5cNnx8sAW.png
>
>
> The tests :sdks:python:test-suites:portable:py2:portableWordCountBatch and
> :sdks:python:test-suites:portable:py2:portableWordCountStreaming are flaky,
> sometimes throws grpc errrors.
> Stacktrace
> {code:java}
> INFO:root:Using Python SDK docker image:
> apache/beam_python2.7_sdk:2.21.0.dev. If the image is not available at local,
> we will try to pull from
> hub.docker.comINFO:apache_beam.runners.portability.fn_api_runner_transforms:
>
> INFO:apache_beam.utils.subprocess_server:Starting service
> with ['docker' 'run' '-v' '/usr/bin/docker:/bin/docker' '-v'
> '/var/run/docker.sock:/var/run/docker.sock' '--network=host'
> 'apache/beam_flink1.9_job_server:latest' '--job-host' 'localhost'
> '--job-port' '58753' '--artifact-port' '60175' '--expansion-port'
> '33067']INFO:apache_beam.utils.subprocess_server:[main] INFO
> org.apache.beam.runners.fnexecution.jobsubmission.JobServerDriver -
> ArtifactStagingService started on
> localhost:60175INFO:apache_beam.utils.subprocess_server:[main] INFO
> org.apache.beam.runners.fnexecution.jobsubmission.JobServerDriver - Java
> ExpansionService started on
> localhost:33067INFO:apache_beam.utils.subprocess_server:[main] INFO
> org.apache.beam.runners.fnexecution.jobsubmission.JobServerDriver -
> JobService started on localhost:58753ERROR:grpc._common:Exception
> deserializing message!Traceback (most recent call last): File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Portable_Python_Commit/src/build/gradleenv/1866363813/local/lib/python2.7/site-packages/grpc/_common.py",
> line 84, in _transformreturn transformer(message)DecodeError: Error
> parsing messageTraceback (most recent call last): File
> "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main
> "__main__", fname, loader, pkg_name) File "/usr/lib/python2.7/runpy.py",
> line 72, in _run_codeexec code in run_globals File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Portable_Python_Commit/src/build/gradleenv/1866363813/lib/python2.7/site-packages/apache_beam/examples/wordcount.py",
> line 142, in run() File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Portable_Python_Commit/src/build/gradleenv/1866363813/lib/python2.7/site-packages/apache_beam/examples/wordcount.py",
> line 121, in runresult = p.run() File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Portable_Python_Commit/src/build/gradleenv/1866363813/local/lib/python2.7/site-packages/apache_beam/pipeline.py",
> line 495, in runself._options).run(False) File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Portable_Python_Commit/src/build/gradleenv/1866363813/local/lib/python2.7/site-packages/apache_beam/pipeline.py",
> line 508, in runreturn self.runner.run_pipeline(self, self._options)
> File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Portable_Python_Commit/src/build/gradleenv/1866363813/local/lib/python2.7/site-packages/apache_beam/runners/portability/portable_runner.py",
> line 401, in run_pipelinejob_service_handle.submit(proto_pipeline) File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Portable_Python_Commit/src/build/gradleenv/1866363813/local/lib/python2.7/site-packages/apache_beam/runners/portability/portable_runner.py",
> line 102, in submitprepare_response = self.prepare(proto_pipeline) File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Portable_Python_Commit/src/build/gradleenv/1866363813/local/lib/python2.7/site-packages/apache_beam/runners/portability/portable_runner.py",
> line 179, in preparetimeout=self.timeout) File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Portable_Python_Commit/src/build/gradleenv/1866363813/local/lib/python2.7/site-packages/grpc/_channel.py",
> line 826, in __call__return _end_unary_response_blocking(state, call,
> False, None) File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Portable_Python_Commit/src/build/gradleenv/1866363813/local/lib/python2.7/site-packages/grpc/_channel.py",
> line 729, in _end_unary_response_blockingraise
> _InactiveRpcError(state)grpc._channel._InactiveRpcError: <_InactiveRpcError
> of RPC that terminated with: status = StatusCode.INTERNAL
[jira] [Updated] (BEAM-9549) Flaky portableWordCountBatch and portableWordCountStreaming tests
[
https://issues.apache.org/jira/browse/BEAM-9549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ning Kang updated BEAM-9549:
Attachment: Sr5cNnx8sAW.png
> Flaky portableWordCountBatch and portableWordCountStreaming tests
> -
>
> Key: BEAM-9549
> URL: https://issues.apache.org/jira/browse/BEAM-9549
> Project: Beam
> Issue Type: Test
> Components: test-failures
>Reporter: Ning Kang
>Assignee: Ankur Goenka
>Priority: Major
> Attachments: Sr5cNnx8sAW.png
>
>
> The tests :sdks:python:test-suites:portable:py2:portableWordCountBatch and
> :sdks:python:test-suites:portable:py2:portableWordCountStreaming are flaky,
> sometimes throws grpc errrors.
> Stacktrace
> {code:java}
> INFO:root:Using Python SDK docker image:
> apache/beam_python2.7_sdk:2.21.0.dev. If the image is not available at local,
> we will try to pull from
> hub.docker.comINFO:apache_beam.runners.portability.fn_api_runner_transforms:
>
> INFO:apache_beam.utils.subprocess_server:Starting service
> with ['docker' 'run' '-v' '/usr/bin/docker:/bin/docker' '-v'
> '/var/run/docker.sock:/var/run/docker.sock' '--network=host'
> 'apache/beam_flink1.9_job_server:latest' '--job-host' 'localhost'
> '--job-port' '58753' '--artifact-port' '60175' '--expansion-port'
> '33067']INFO:apache_beam.utils.subprocess_server:[main] INFO
> org.apache.beam.runners.fnexecution.jobsubmission.JobServerDriver -
> ArtifactStagingService started on
> localhost:60175INFO:apache_beam.utils.subprocess_server:[main] INFO
> org.apache.beam.runners.fnexecution.jobsubmission.JobServerDriver - Java
> ExpansionService started on
> localhost:33067INFO:apache_beam.utils.subprocess_server:[main] INFO
> org.apache.beam.runners.fnexecution.jobsubmission.JobServerDriver -
> JobService started on localhost:58753ERROR:grpc._common:Exception
> deserializing message!Traceback (most recent call last): File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Portable_Python_Commit/src/build/gradleenv/1866363813/local/lib/python2.7/site-packages/grpc/_common.py",
> line 84, in _transformreturn transformer(message)DecodeError: Error
> parsing messageTraceback (most recent call last): File
> "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main
> "__main__", fname, loader, pkg_name) File "/usr/lib/python2.7/runpy.py",
> line 72, in _run_codeexec code in run_globals File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Portable_Python_Commit/src/build/gradleenv/1866363813/lib/python2.7/site-packages/apache_beam/examples/wordcount.py",
> line 142, in run() File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Portable_Python_Commit/src/build/gradleenv/1866363813/lib/python2.7/site-packages/apache_beam/examples/wordcount.py",
> line 121, in runresult = p.run() File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Portable_Python_Commit/src/build/gradleenv/1866363813/local/lib/python2.7/site-packages/apache_beam/pipeline.py",
> line 495, in runself._options).run(False) File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Portable_Python_Commit/src/build/gradleenv/1866363813/local/lib/python2.7/site-packages/apache_beam/pipeline.py",
> line 508, in runreturn self.runner.run_pipeline(self, self._options)
> File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Portable_Python_Commit/src/build/gradleenv/1866363813/local/lib/python2.7/site-packages/apache_beam/runners/portability/portable_runner.py",
> line 401, in run_pipelinejob_service_handle.submit(proto_pipeline) File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Portable_Python_Commit/src/build/gradleenv/1866363813/local/lib/python2.7/site-packages/apache_beam/runners/portability/portable_runner.py",
> line 102, in submitprepare_response = self.prepare(proto_pipeline) File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Portable_Python_Commit/src/build/gradleenv/1866363813/local/lib/python2.7/site-packages/apache_beam/runners/portability/portable_runner.py",
> line 179, in preparetimeout=self.timeout) File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Portable_Python_Commit/src/build/gradleenv/1866363813/local/lib/python2.7/site-packages/grpc/_channel.py",
> line 826, in __call__return _end_unary_response_blocking(state, call,
> False, None) File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Portable_Python_Commit/src/build/gradleenv/1866363813/local/lib/python2.7/site-packages/grpc/_channel.py",
> line 729, in _end_unary_response_blockingraise
> _InactiveRpcError(state)grpc._channel._InactiveRpcError: <_InactiveRpcError
> of RPC that terminated with: status = StatusCode.INTERNAL
[jira] [Resolved] (BEAM-8070) Support empty array literal
[ https://issues.apache.org/jira/browse/BEAM-8070?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Pilloud resolved BEAM-8070. -- Fix Version/s: 2.25.0 Resolution: Fixed > Support empty array literal > --- > > Key: BEAM-8070 > URL: https://issues.apache.org/jira/browse/BEAM-8070 > Project: Beam > Issue Type: Sub-task > Components: dsl-sql-zetasql >Reporter: Rui Wang >Assignee: Andrew Pilloud >Priority: Major > Labels: zetasql-compliance > Fix For: 2.25.0 > > Time Spent: 1h 20m > Remaining Estimate: 0h > > Currently BeamSQL throws an IndexOutOfBoundsException when given a query with > an empty array literal. This happens because Calcite attempts to infer the > element types [1,2] from an empty element list. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (BEAM-9549) Flaky portableWordCountBatch and portableWordCountStreaming tests
Ning Kang created BEAM-9549:
---
Summary: Flaky portableWordCountBatch and
portableWordCountStreaming tests
Key: BEAM-9549
URL: https://issues.apache.org/jira/browse/BEAM-9549
Project: Beam
Issue Type: Test
Components: test-failures
Reporter: Ning Kang
The tests :sdks:python:test-suites:portable:py2:portableWordCountBatch and
:sdks:python:test-suites:portable:py2:portableWordCountStreaming are flaky,
sometimes throws grpc errrors.
Stacktrace
{code:java}
INFO:root:Using Python SDK docker image: apache/beam_python2.7_sdk:2.21.0.dev.
If the image is not available at local, we will try to pull from
hub.docker.comINFO:apache_beam.runners.portability.fn_api_runner_transforms:
INFO:apache_beam.utils.subprocess_server:Starting service
with ['docker' 'run' '-v' '/usr/bin/docker:/bin/docker' '-v'
'/var/run/docker.sock:/var/run/docker.sock' '--network=host'
'apache/beam_flink1.9_job_server:latest' '--job-host' 'localhost' '--job-port'
'58753' '--artifact-port' '60175' '--expansion-port'
'33067']INFO:apache_beam.utils.subprocess_server:[main] INFO
org.apache.beam.runners.fnexecution.jobsubmission.JobServerDriver -
ArtifactStagingService started on
localhost:60175INFO:apache_beam.utils.subprocess_server:[main] INFO
org.apache.beam.runners.fnexecution.jobsubmission.JobServerDriver - Java
ExpansionService started on
localhost:33067INFO:apache_beam.utils.subprocess_server:[main] INFO
org.apache.beam.runners.fnexecution.jobsubmission.JobServerDriver - JobService
started on localhost:58753ERROR:grpc._common:Exception deserializing
message!Traceback (most recent call last): File
"/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Portable_Python_Commit/src/build/gradleenv/1866363813/local/lib/python2.7/site-packages/grpc/_common.py",
line 84, in _transformreturn transformer(message)DecodeError: Error
parsing messageTraceback (most recent call last): File
"/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main"__main__",
fname, loader, pkg_name) File "/usr/lib/python2.7/runpy.py", line 72, in
_run_codeexec code in run_globals File
"/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Portable_Python_Commit/src/build/gradleenv/1866363813/lib/python2.7/site-packages/apache_beam/examples/wordcount.py",
line 142, in run() File
"/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Portable_Python_Commit/src/build/gradleenv/1866363813/lib/python2.7/site-packages/apache_beam/examples/wordcount.py",
line 121, in runresult = p.run() File
"/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Portable_Python_Commit/src/build/gradleenv/1866363813/local/lib/python2.7/site-packages/apache_beam/pipeline.py",
line 495, in runself._options).run(False) File
"/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Portable_Python_Commit/src/build/gradleenv/1866363813/local/lib/python2.7/site-packages/apache_beam/pipeline.py",
line 508, in runreturn self.runner.run_pipeline(self, self._options) File
"/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Portable_Python_Commit/src/build/gradleenv/1866363813/local/lib/python2.7/site-packages/apache_beam/runners/portability/portable_runner.py",
line 401, in run_pipelinejob_service_handle.submit(proto_pipeline) File
"/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Portable_Python_Commit/src/build/gradleenv/1866363813/local/lib/python2.7/site-packages/apache_beam/runners/portability/portable_runner.py",
line 102, in submitprepare_response = self.prepare(proto_pipeline) File
"/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Portable_Python_Commit/src/build/gradleenv/1866363813/local/lib/python2.7/site-packages/apache_beam/runners/portability/portable_runner.py",
line 179, in preparetimeout=self.timeout) File
"/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Portable_Python_Commit/src/build/gradleenv/1866363813/local/lib/python2.7/site-packages/grpc/_channel.py",
line 826, in __call__return _end_unary_response_blocking(state, call,
False, None) File
"/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Portable_Python_Commit/src/build/gradleenv/1866363813/local/lib/python2.7/site-packages/grpc/_channel.py",
line 729, in _end_unary_response_blockingraise
_InactiveRpcError(state)grpc._channel._InactiveRpcError: <_InactiveRpcError of
RPC that terminated with: status = StatusCode.INTERNALdetails = "Exception
deserializing response!" debug_error_string = "None">
{code}
A gradle scan [example|https://scans.gradle.com/s/yv63cefske4cy].
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9524) ib.show() spins forever when cells are re-executed
[
https://issues.apache.org/jira/browse/BEAM-9524?focusedWorklogId=405757=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-405757
]
ASF GitHub Bot logged work on BEAM-9524:
Author: ASF GitHub Bot
Created on: 18/Mar/20 21:40
Start Date: 18/Mar/20 21:40
Worklog Time Spent: 10m
Work Description: rohdesamuel commented on pull request #11128:
[BEAM-9524] Fix for ib.show() executing indefinitely
URL: https://github.com/apache/beam/pull/11128#discussion_r394655524
##
File path:
sdks/python/apache_beam/runners/interactive/caching/streaming_cache.py
##
@@ -153,7 +153,8 @@ def __init__(
self._coder = coder
self._labels = labels
self._is_cache_complete = (
-is_cache_complete if is_cache_complete else lambda: True)
+is_cache_complete if is_cache_complete else lambda _: True)
+self._pipeline_id = labels[-1].split('_')[-1]
Review comment:
Yeah the pipeline id is just a pointer to the pipeline, which is guaranteed
to be unique and constant.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 405757)
Time Spent: 50m (was: 40m)
> ib.show() spins forever when cells are re-executed
> --
>
> Key: BEAM-9524
> URL: https://issues.apache.org/jira/browse/BEAM-9524
> Project: Beam
> Issue Type: Bug
> Components: runner-py-interactive
>Reporter: Sam Rohde
>Priority: Major
> Time Spent: 50m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9533) Replace *-gcp/*-aws tox suites with *-cloud suites to run unit tests for both
[ https://issues.apache.org/jira/browse/BEAM-9533?focusedWorklogId=405755=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-405755 ] ASF GitHub Bot logged work on BEAM-9533: Author: ASF GitHub Bot Created on: 18/Mar/20 21:39 Start Date: 18/Mar/20 21:39 Worklog Time Spent: 10m Work Description: pabloem commented on issue #11158: [BEAM-9533] Fixing tox.ini variants URL: https://github.com/apache/beam/pull/11158#issuecomment-600872144 Run Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 405755) Time Spent: 50m (was: 40m) > Replace *-gcp/*-aws tox suites with *-cloud suites to run unit tests for both > - > > Key: BEAM-9533 > URL: https://issues.apache.org/jira/browse/BEAM-9533 > Project: Beam > Issue Type: Bug > Components: testing >Reporter: Pablo Estrada >Assignee: Pablo Estrada >Priority: Minor > Time Spent: 50m > Remaining Estimate: 0h > > Currently there are `py37-gcp`, py37-aws test suites. Let's consolidate all > of them into py37-cloud, along with other py35-gcp, py27-gcp, etc. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9444) Shall we use GCP Libraries BOM to specify Google-related library versions?
[
https://issues.apache.org/jira/browse/BEAM-9444?focusedWorklogId=405754=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-405754
]
ASF GitHub Bot logged work on BEAM-9444:
Author: ASF GitHub Bot
Created on: 18/Mar/20 21:31
Start Date: 18/Mar/20 21:31
Worklog Time Spent: 10m
Work Description: suztomo commented on issue #11156: [BEAM-9444] Use GCP
Libraries BOM for Google Cloud Dependencies
URL: https://github.com/apache/beam/pull/11156#issuecomment-600868999
## Java Precommit failed
`org.apache.beam.runners.flink.translation.wrappers.streaming.io.UnboundedSourceWrapperTest`
is BEAM-9470.
Samza error:
https://builds.apache.org/job/beam_PreCommit_Java_Commit/10414/testReport/junit/org.apache.beam.runners.samza.runtime/SamzaTimerInternalsFactoryTest/testRestore/
```
Caused by: org.rocksdb.RocksDBException: Can't access /000370.sst: IO error:
while stat a file for size: /tmp/store2/000370.sst: No such file or directory
```
## Python PreCommit failed
https://builds.apache.org/job/beam_PreCommit_Python_Commit/11782/testReport/junit/apache_beam.runners.portability.fn_api_runner_test/FnApiRunnerTestWithGrpc/test_large_elements/
complains `IndexError: index out of range`.
I find another irrelevant build
([11780](https://builds.apache.org/job/beam_PreCommit_Python_Commit/11780/testReport/junit/apache_beam.runners.portability.fn_api_runner_test/FnApiRunnerTestWithGrpc/test_large_elements/))
also show this error.
They all seem irrelevant to this PR.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 405754)
Time Spent: 6.5h (was: 6h 20m)
> Shall we use GCP Libraries BOM to specify Google-related library versions?
> --
>
> Key: BEAM-9444
> URL: https://issues.apache.org/jira/browse/BEAM-9444
> Project: Beam
> Issue Type: Task
> Components: build-system
>Reporter: Tomo Suzuki
>Assignee: Tomo Suzuki
>Priority: Major
> Attachments: Screen Shot 2020-03-13 at 13.33.01.png, Screen Shot
> 2020-03-17 at 16.01.16.png
>
> Time Spent: 6.5h
> Remaining Estimate: 0h
>
> Shall we use GCP Libraries BOM to specify Google-related library versions?
>
> I've been working on Beam's dependency upgrades in the past few months. I
> think it's time to consider a long-term solution to keep the libraries
> up-to-date with small maintenance effort. To achieve that, I propose Beam to
> use GCP Libraries BOM to set the Google-related library versions, rather than
> trying to make changes in each of ~30 Google libraries.
>
> h1. Background
> A BOM is pom.xml that provides dependencyManagement to importing projects.
>
> GCP Libraries BOM is a BOM that includes many Google Cloud related libraries
> + gRPC + protobuf. We (Google Cloud Java Diamond Dependency team) maintain
> the BOM so that the set of the libraries are compatible with each other.
>
> h1. Implementation
> Notes for obstacles.
> h2. BeamModulePlugin's "force" does not take BOM into account (thus fails)
> {{forcedModules}} via version resolution strategy is playing bad. This causes
> {noformat}
> A problem occurred evaluating project ':sdks:java:extensions:sql'.
> Could not resolve all dependencies for configuration
> ':sdks:java:extensions:sql:fmppTemplates'.
> Invalid format: 'com.google.cloud:google-cloud-core'. Group, name and version
> cannot be empty. Correct example: 'org.gradle:gradle-core:1.0'{noformat}
> !Screen Shot 2020-03-13 at 13.33.01.png|width=489,height=287!
>
> h2. :sdks:java:maven-archetypes:examples needs the version of
> google-http-client
> The task requires the version for the library:
> {code:java}
> 'google-http-client.version':
> dependencies.create(project.library.java.google_http_client).getVersion(),
> {code}
> This would generate NullPointerException. Running gradlew without the
> subproject:
>
> {code:java}
> ./gradlew -p sdks/java check -x :sdks:java:maven-archetypes:examples:check
> {code}
> h1. Problem in Gradle-generated pom files
> The generated Maven artifact POM has invalid data due to the BOM change. For
> example my locally installed
> {{~/.m2/repository/org/apache/beam/beam-sdks-java-io-google-cloud-platform/2.21.0-SNAPSHOT/beam-sdks-java-io-google-cloud-platform-2.21.0-SNAPSHOT.pom}}
> had the following problems.
> h2. The GCP Libraries BOM showing up in dependencies section:
> {noformat}
[jira] [Work logged] (BEAM-9524) ib.show() spins forever when cells are re-executed
[
https://issues.apache.org/jira/browse/BEAM-9524?focusedWorklogId=405753=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-405753
]
ASF GitHub Bot logged work on BEAM-9524:
Author: ASF GitHub Bot
Created on: 18/Mar/20 21:25
Start Date: 18/Mar/20 21:25
Worklog Time Spent: 10m
Work Description: pabloem commented on pull request #11128: [BEAM-9524]
Fix for ib.show() executing indefinitely
URL: https://github.com/apache/beam/pull/11128#discussion_r394648905
##
File path:
sdks/python/apache_beam/runners/interactive/caching/streaming_cache.py
##
@@ -153,7 +153,8 @@ def __init__(
self._coder = coder
self._labels = labels
self._is_cache_complete = (
-is_cache_complete if is_cache_complete else lambda: True)
+is_cache_complete if is_cache_complete else lambda _: True)
+self._pipeline_id = labels[-1].split('_')[-1]
Review comment:
Will it come in as a label?
the ID will be unique for the pipeline object itself. I guess we will not be
reusing it anywhere?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 405753)
Time Spent: 40m (was: 0.5h)
> ib.show() spins forever when cells are re-executed
> --
>
> Key: BEAM-9524
> URL: https://issues.apache.org/jira/browse/BEAM-9524
> Project: Beam
> Issue Type: Bug
> Components: runner-py-interactive
>Reporter: Sam Rohde
>Priority: Major
> Time Spent: 40m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9537) Refactor FnApiRunner into its own package
[ https://issues.apache.org/jira/browse/BEAM-9537?focusedWorklogId=405751=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-405751 ] ASF GitHub Bot logged work on BEAM-9537: Author: ASF GitHub Bot Created on: 18/Mar/20 21:21 Start Date: 18/Mar/20 21:21 Worklog Time Spent: 10m Work Description: pabloem commented on issue #11153: [BEAM-9537] Adding a new module for FnApiRunner URL: https://github.com/apache/beam/pull/11153#issuecomment-600865229 I've started rebasing https://github.com/apache/beam/pull/10291 to keep it up to date. Since the module refactoring is likely to make lots of trouble merging, I'm getting it out of the way early. r: @robertwb This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 405751) Time Spent: 1h 10m (was: 1h) > Refactor FnApiRunner into its own package > - > > Key: BEAM-9537 > URL: https://issues.apache.org/jira/browse/BEAM-9537 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Pablo Estrada >Assignee: Pablo Estrada >Priority: Minor > Time Spent: 1h 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9035) BIP-1: Typed options for Row Schema and Fields
[ https://issues.apache.org/jira/browse/BEAM-9035?focusedWorklogId=405750=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-405750 ] ASF GitHub Bot logged work on BEAM-9035: Author: ASF GitHub Bot Created on: 18/Mar/20 21:20 Start Date: 18/Mar/20 21:20 Worklog Time Spent: 10m Work Description: reuvenlax commented on issue #10413: [BEAM-9035] Typed options for Row Schema and Field URL: https://github.com/apache/beam/pull/10413#issuecomment-600864690 @alexvanboxel I'm still not 100% sure that I agree that options are not schemas, and I don't love making SchemaVerification public. However I think it will be easier to iterate on those issues in isolation once the basic support is in, so I'm approving for now. LGTM This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 405750) Time Spent: 8h 20m (was: 8h 10m) > BIP-1: Typed options for Row Schema and Fields > -- > > Key: BEAM-9035 > URL: https://issues.apache.org/jira/browse/BEAM-9035 > Project: Beam > Issue Type: Sub-task > Components: sdk-java-core >Reporter: Alex Van Boxel >Assignee: Alex Van Boxel >Priority: Major > Fix For: 2.19.0 > > Time Spent: 8h 20m > Remaining Estimate: 0h > > This is the first issue of a multipart commit: this ticket implements the > basic infrastructure of options on row and field. > Full explanation: > Introduce the concept of Options in Beam Schema’s to add extra context to > fields and schema. In contracts to metadata, options would be added to > fields, logical types and rows. In the options schema convertors can add > options/annotations/decorators that were in the original schema, this context > can be used in the rest of the pipeline for specific transformations or > augment the end schema in the target output. > Examples of options are: > * informational: like the source of the data, ... > * drive decisions further in the pipeline: flatten a row into another, > rename a field, ... > * influence something in the output: like cluster index, primary key, ... > * logical type information > And option is a key/typed value combination. The advantages of having the > value types is: > * Having strongly typed options would give a *portable way of Logical Types* > to have structured information that could be shared over different languages. > * This could keep the type intact when mapping from a formats that have > strongly typed options (example: Protobuf). > This is part of a multi ticket implementation. The following tickets are > related: > # Typed options for Row Schema and Fields > # Convert Proto Options to Beam Schema options > # Convert Avro extra information for Beam string options > # Replace meta data with Logical Type options > # Extract meta data in Calcite SQL to Beam options > # Extract meta data in Zeta SQL to Beam options > # Add java example of using option in a transform > This feature is discussed with Reuven Lax, Brian Hulette -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9540) Rename beam:source:runner:0.1/beam:sink:runner:0.1 to beam:runner:source:v1/beam:runner:sink:v1
[ https://issues.apache.org/jira/browse/BEAM-9540?focusedWorklogId=405748=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-405748 ] ASF GitHub Bot logged work on BEAM-9540: Author: ASF GitHub Bot Created on: 18/Mar/20 21:19 Start Date: 18/Mar/20 21:19 Worklog Time Spent: 10m Work Description: amaliujia commented on issue #11157: [BEAM-9540] Rename beam:source:runner:0.1/beam:sink:runner:0.1 to beam:runner:source:v1/beam:runner:sink:v1 URL: https://github.com/apache/beam/pull/11157#issuecomment-600864297 LGTM This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 405748) Time Spent: 1h 40m (was: 1.5h) > Rename beam:source:runner:0.1/beam:sink:runner:0.1 to > beam:runner:source:v1/beam:runner:sink:v1 > --- > > Key: BEAM-9540 > URL: https://issues.apache.org/jira/browse/BEAM-9540 > Project: Beam > Issue Type: Bug > Components: sdk-java-harness, sdk-py-harness >Reporter: Luke Cwik >Assignee: Luke Cwik >Priority: Minor > Time Spent: 1h 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (BEAM-9548) Bad error handling with errors from TestStreamService when using Interactive Beam
Sam Rohde created BEAM-9548: --- Summary: Bad error handling with errors from TestStreamService when using Interactive Beam Key: BEAM-9548 URL: https://issues.apache.org/jira/browse/BEAM-9548 Project: Beam Issue Type: Bug Components: runner-py-interactive Reporter: Sam Rohde Assignee: Sam Rohde The error handling when an error is generated on the GRPC server side is very verbose and hides the problem. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9524) ib.show() spins forever when cells are re-executed
[
https://issues.apache.org/jira/browse/BEAM-9524?focusedWorklogId=405746=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-405746
]
ASF GitHub Bot logged work on BEAM-9524:
Author: ASF GitHub Bot
Created on: 18/Mar/20 21:14
Start Date: 18/Mar/20 21:14
Worklog Time Spent: 10m
Work Description: pabloem commented on pull request #11128: [BEAM-9524]
Fix for ib.show() executing indefinitely
URL: https://github.com/apache/beam/pull/11128#discussion_r394495708
##
File path:
sdks/python/apache_beam/runners/interactive/caching/streaming_cache.py
##
@@ -153,7 +153,8 @@ def __init__(
self._coder = coder
self._labels = labels
self._is_cache_complete = (
-is_cache_complete if is_cache_complete else lambda: True)
+is_cache_complete if is_cache_complete else lambda _: True)
+self._pipeline_id = labels[-1].split('_')[-1]
Review comment:
what is a pipeline ID? Are you sure it's unique? I see elsewhere you're
using the `id(..)` of the objects?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 405746)
Time Spent: 0.5h (was: 20m)
> ib.show() spins forever when cells are re-executed
> --
>
> Key: BEAM-9524
> URL: https://issues.apache.org/jira/browse/BEAM-9524
> Project: Beam
> Issue Type: Bug
> Components: runner-py-interactive
>Reporter: Sam Rohde
>Priority: Major
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-7923) Interactive Beam
[ https://issues.apache.org/jira/browse/BEAM-7923?focusedWorklogId=405744=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-405744 ] ASF GitHub Bot logged work on BEAM-7923: Author: ASF GitHub Bot Created on: 18/Mar/20 21:00 Start Date: 18/Mar/20 21:00 Worklog Time Spent: 10m Work Description: aaltay commented on issue #11161: [BEAM-7923] Change Transform Label Prefix Syntax URL: https://github.com/apache/beam/pull/11161#issuecomment-600855753 Run Portable_Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 405744) Time Spent: 5h 40m (was: 5.5h) > Interactive Beam > > > Key: BEAM-7923 > URL: https://issues.apache.org/jira/browse/BEAM-7923 > Project: Beam > Issue Type: New Feature > Components: runner-py-interactive >Reporter: Ning Kang >Assignee: Ning Kang >Priority: Major > Time Spent: 5h 40m > Remaining Estimate: 0h > > This is the top level ticket for all efforts leveraging [interactive > Beam|[https://github.com/apache/beam/tree/master/sdks/python/apache_beam/runners/interactive]] > As the development goes, blocking tickets will be added to this one. > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9545) MVP: DataframeTransform
[
https://issues.apache.org/jira/browse/BEAM-9545?focusedWorklogId=405734=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-405734
]
ASF GitHub Bot logged work on BEAM-9545:
Author: ASF GitHub Bot
Created on: 18/Mar/20 20:49
Start Date: 18/Mar/20 20:49
Worklog Time Spent: 10m
Work Description: TheNeuralBit commented on pull request #10760:
[BEAM-9545] Dataframe transforms
URL: https://github.com/apache/beam/pull/10760#discussion_r394598085
##
File path: sdks/python/apache_beam/dataframe/transforms_test.py
##
@@ -0,0 +1,87 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from __future__ import absolute_import
+
+import unittest
+
+import pandas as pd
+
+import apache_beam as beam
+from apache_beam.dataframe import expressions
+from apache_beam.dataframe import frame_base
+from apache_beam.dataframe import transforms
+from apache_beam.testing.util import assert_that
+
+
+class TransformTest(unittest.TestCase):
+ def run_test(self, input, func):
+expected = func(input)
+
+empty = input[0:0]
+input_placeholder = expressions.PlaceholderExpression(empty)
+input_deferred = frame_base.DeferredFrame.wrap(input_placeholder)
+actual_deferred = func(input_deferred)._expr.evaluate_at(
+expressions.Session({input_placeholder: input}))
+
+def check_correct(actual):
+ if actual is None:
+raise AssertionError('Empty frame but expected: \n\n%s' % (expected))
+ sorted_actual = actual.sort_index()
+ sorted_expected = expected.sort_index()
+ if not sorted_actual.equals(sorted_expected):
+raise AssertionError(
+'Dataframes not equal: \n\n%s\n\n%s' %
+(sorted_actual, sorted_expected))
+
+check_correct(actual_deferred)
+
+with beam.Pipeline() as p:
+ input_pcoll = p | beam.Create([input[::2], input[1::2]])
+ output_pcoll = input_pcoll | transforms.DataframeTransform(
+ func, proxy=empty)
+ assert_that(
+ output_pcoll,
+ lambda actual: check_correct(pd.concat(actual) if actual else None))
+
+ def test_identity(self):
+df = pd.DataFrame({
+'Animal': ['Falcon', 'Falcon', 'Parrot', 'Parrot'],
+'Speed': [380., 370., 24., 26.]
+})
+self.run_test(df, lambda x: x)
+
+ def test_sum_mean(self):
+df = pd.DataFrame({
+'Animal': ['Falcon', 'Falcon', 'Parrot', 'Parrot'],
+'Speed': [380., 370., 24., 26.]
+})
+self.run_test(df, lambda df: df.groupby('Animal').sum())
+self.run_test(df, lambda df: df.groupby('Animal').mean())
+
+ def test_filter(self):
+df = pd.DataFrame({
+'Animal': ['Aardvark', 'Ant', 'Elephant', 'Zebra'],
+'Speed': [5, 2, 35, 40]
+})
+self.run_test(df, lambda df: df.filter(items=['Animal']))
+self.run_test(df, lambda df: df.filter(regex='A.*'))
+self.run_test(
+df, lambda df: df.set_index('Animal').filter(regex='F.*',
axis='index'))
+
+
+if __name__ == '__main__':
+ unittest.main()
Review comment:
Is there a way to leverage pandas test suites for more complete testing?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 405734)
Remaining Estimate: 0h
Time Spent: 10m
> MVP: DataframeTransform
> ---
>
> Key: BEAM-9545
> URL: https://issues.apache.org/jira/browse/BEAM-9545
> Project: Beam
> Issue Type: Improvement
> Components: sdk-py-core
>Reporter: Brian Hulette
>Assignee: Robert Bradshaw
>Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
