[jira] [Work logged] (BEAM-5852) Function extension in BeamSQL
[ https://issues.apache.org/jira/browse/BEAM-5852?focusedWorklogId=161017&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-161017 ] ASF GitHub Bot logged work on BEAM-5852: Author: ASF GitHub Bot Created on: 31/Oct/18 07:15 Start Date: 31/Oct/18 07:15 Worklog Time Spent: 10m Work Description: amaliujia opened a new pull request #6898: [BEAM-5852] BeamSQL functions URL: https://github.com/apache/beam/pull/6898 1. Add annotation to allow define functions. 2. Add some functions as an extension to BeamSQL, which allow BeamSQL provides more rich function set. Follow this checklist to help us incorporate your contribution quickly and easily: - [x] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). It will help us expedite review of your Pull Request if you tag someone (e.g. `@username`) to look at it. Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/) | --- | --- | --- | --- | --- | --- Java | [](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/) [](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/) [](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Python_VR_Flink/lastCompletedBuild/) | --- | --- | --- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 161017) Time Spent: 10m Remaining Estimate: 0h > Function extension in BeamSQL > -
[jira] [Work logged] (BEAM-5852) Function extension in BeamSQL
[ https://issues.apache.org/jira/browse/BEAM-5852?focusedWorklogId=161019&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-161019 ] ASF GitHub Bot logged work on BEAM-5852: Author: ASF GitHub Bot Created on: 31/Oct/18 07:20 Start Date: 31/Oct/18 07:20 Worklog Time Spent: 10m Work Description: amaliujia commented on issue #6898: [BEAM-5852] BeamSQL functions URL: https://github.com/apache/beam/pull/6898#issuecomment-434585949 @kennknowles @apilloud @akedin This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 161019) Time Spent: 20m (was: 10m) > Function extension in BeamSQL > - > > Key: BEAM-5852 > URL: https://issues.apache.org/jira/browse/BEAM-5852 > Project: Beam > Issue Type: New Feature > Components: dsl-sql >Reporter: Rui Wang >Assignee: Rui Wang >Priority: Major > Time Spent: 20m > Remaining Estimate: 0h > > We could add more functions to BeamSQL (as UDFs) to provide rich > functionalities than standard/Calcite functions. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5299) Define max global window as a shared value in protos like URN enums.
[ https://issues.apache.org/jira/browse/BEAM-5299?focusedWorklogId=161062&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-161062 ] ASF GitHub Bot logged work on BEAM-5299: Author: ASF GitHub Bot Created on: 31/Oct/18 09:23 Start Date: 31/Oct/18 09:23 Worklog Time Spent: 10m Work Description: mxm commented on a change in pull request #6381: [BEAM-5299] Define max timestamp for global window in proto URL: https://github.com/apache/beam/pull/6381#discussion_r229615676 ## File path: sdks/python/apache_beam/transforms/window.py ## @@ -295,19 +294,14 @@ def __lt__(self, other): class GlobalWindow(BoundedWindow): """The default window into which all data is placed (via GlobalWindows).""" _instance = None - # The maximum timestamp for global windows is MAX_TIMESTAMP - 1 day. - # This is due to timers triggering when the watermark passes the trigger - # time, which is only possible for timestamps < MAX_TIMESTAMP. - # See also GlobalWindow in the Java SDK. - _END_OF_GLOBAL_WINDOW = MAX_TIMESTAMP - (24 * 60 * 60) def __new__(cls): if cls._instance is None: cls._instance = super(GlobalWindow, cls).__new__(cls) return cls._instance def __init__(self): -super(GlobalWindow, self).__init__(GlobalWindow._END_OF_GLOBAL_WINDOW) +super(GlobalWindow, self).__init__(GlobalWindow._getTimestampFromProto()) Review comment: Yep, makes sense. I'll CC you in a follow-up PR. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 161062) Time Spent: 6h 10m (was: 6h) > Define max global window as a shared value in protos like URN enums. > > > Key: BEAM-5299 > URL: https://issues.apache.org/jira/browse/BEAM-5299 > Project: Beam > Issue Type: Improvement > Components: beam-model, sdk-go, sdk-java-core, sdk-py-core >Reporter: Luke Cwik >Assignee: Maximilian Michels >Priority: Minor > Labels: portability > Time Spent: 6h 10m > Remaining Estimate: 0h > > Instead of having each language define a max timestamp themselves, define the > max timestamps within proto to be shared across different languages. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5299) Define max global window as a shared value in protos like URN enums.
[ https://issues.apache.org/jira/browse/BEAM-5299?focusedWorklogId=161063&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-161063 ] ASF GitHub Bot logged work on BEAM-5299: Author: ASF GitHub Bot Created on: 31/Oct/18 09:25 Start Date: 31/Oct/18 09:25 Worklog Time Spent: 10m Work Description: mxm commented on issue #6381: [BEAM-5299] Define max timestamp for global window in proto URL: https://github.com/apache/beam/pull/6381#issuecomment-434617221 Nice that we got this merged! For the Gophers out there, the Go SDK still needs to be changed to use the values from the Proto. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 161063) Time Spent: 6h 20m (was: 6h 10m) > Define max global window as a shared value in protos like URN enums. > > > Key: BEAM-5299 > URL: https://issues.apache.org/jira/browse/BEAM-5299 > Project: Beam > Issue Type: Improvement > Components: beam-model, sdk-go, sdk-java-core, sdk-py-core >Reporter: Luke Cwik >Assignee: Maximilian Michels >Priority: Minor > Labels: portability > Time Spent: 6h 20m > Remaining Estimate: 0h > > Instead of having each language define a max timestamp themselves, define the > max timestamps within proto to be shared across different languages. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5299) Define max global window as a shared value in protos like URN enums.
[ https://issues.apache.org/jira/browse/BEAM-5299?focusedWorklogId=161069&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-161069 ] ASF GitHub Bot logged work on BEAM-5299: Author: ASF GitHub Bot Created on: 31/Oct/18 09:36 Start Date: 31/Oct/18 09:36 Worklog Time Spent: 10m Work Description: mxm commented on a change in pull request #6381: [BEAM-5299] Define max timestamp for global window in proto URL: https://github.com/apache/beam/pull/6381#discussion_r229619777 ## File path: sdks/python/apache_beam/transforms/window.py ## @@ -295,19 +294,14 @@ def __lt__(self, other): class GlobalWindow(BoundedWindow): """The default window into which all data is placed (via GlobalWindows).""" _instance = None - # The maximum timestamp for global windows is MAX_TIMESTAMP - 1 day. - # This is due to timers triggering when the watermark passes the trigger - # time, which is only possible for timestamps < MAX_TIMESTAMP. - # See also GlobalWindow in the Java SDK. - _END_OF_GLOBAL_WINDOW = MAX_TIMESTAMP - (24 * 60 * 60) def __new__(cls): if cls._instance is None: cls._instance = super(GlobalWindow, cls).__new__(cls) return cls._instance def __init__(self): -super(GlobalWindow, self).__init__(GlobalWindow._END_OF_GLOBAL_WINDOW) +super(GlobalWindow, self).__init__(GlobalWindow._getTimestampFromProto()) Review comment: As far as I understand `GlobalWindow` is already a singleton and `__init__` will only be called once by the `__new__` method to initialize the `_instance` class member. Correct me if I'm wrong. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 161069) Time Spent: 6.5h (was: 6h 20m) > Define max global window as a shared value in protos like URN enums. > > > Key: BEAM-5299 > URL: https://issues.apache.org/jira/browse/BEAM-5299 > Project: Beam > Issue Type: Improvement > Components: beam-model, sdk-go, sdk-java-core, sdk-py-core >Reporter: Luke Cwik >Assignee: Maximilian Michels >Priority: Minor > Labels: portability > Time Spent: 6.5h > Remaining Estimate: 0h > > Instead of having each language define a max timestamp themselves, define the > max timestamps within proto to be shared across different languages. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5464) Portable beam hangs while running TFX preprocessing step on a distributed cluster
[
https://issues.apache.org/jira/browse/BEAM-5464?focusedWorklogId=161075&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-161075
]
ASF GitHub Bot logged work on BEAM-5464:
Author: ASF GitHub Bot
Created on: 31/Oct/18 09:45
Start Date: 31/Oct/18 09:45
Worklog Time Spent: 10m
Work Description: mxm commented on a change in pull request #6897:
[BEAM-5464] Use BATCH_FORCED as the default ExecutionMode for batch pipeline
URL: https://github.com/apache/beam/pull/6897#discussion_r229622422
##
File path:
runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkPipelineOptions.java
##
@@ -187,4 +188,12 @@
Long getLatencyTrackingInterval();
void setLatencyTrackingInterval(Long interval);
+
+ @Description(
+ "Flink mode for data exchange for batch pipeline. "
+ + "Reference {@link org.apache.flink.api.common.ExecutionMode}")
+ @Default.Enum("BATCH_FORCED")
Review comment:
I'm a bit hesitant to change the default value here. This is also used by
the non-portable FlinkRunner and the default is `PIPELINED`. We haven't heard
from anyone having issues with the batch execution. I'd leave this at the Flink
default until we have found out the exact issue.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 161075)
Time Spent: 0.5h (was: 20m)
> Portable beam hangs while running TFX preprocessing step on a distributed
> cluster
> -
>
> Key: BEAM-5464
> URL: https://issues.apache.org/jira/browse/BEAM-5464
> Project: Beam
> Issue Type: Bug
> Components: java-fn-execution
>Reporter: Axel Magnuson
>Assignee: Ankur Goenka
>Priority: Major
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Recently I went through the exercise of running the TFX taxi example on a
> dataproc cluster. However it would always hang indefinitely. The flink UI
> indicated that the job was halfway done. However I could not see any clear
> errors in the job driver logs, the job service logs, or the Flink logs. The
> root cause is still a mystery to me.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-5464) Portable beam hangs while running TFX preprocessing step on a distributed cluster
[
https://issues.apache.org/jira/browse/BEAM-5464?focusedWorklogId=161076&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-161076
]
ASF GitHub Bot logged work on BEAM-5464:
Author: ASF GitHub Bot
Created on: 31/Oct/18 09:46
Start Date: 31/Oct/18 09:46
Worklog Time Spent: 10m
Work Description: mxm commented on a change in pull request #6897:
[BEAM-5464] Use BATCH_FORCED as the default ExecutionMode for batch pipeline
URL: https://github.com/apache/beam/pull/6897#discussion_r229623346
##
File path:
runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkPipelineOptions.java
##
@@ -187,4 +188,12 @@
Long getLatencyTrackingInterval();
void setLatencyTrackingInterval(Long interval);
+
+ @Description(
+ "Flink mode for data exchange for batch pipeline. "
+ + "Reference {@link org.apache.flink.api.common.ExecutionMode}")
+ @Default.Enum("BATCH_FORCED")
+ ExecutionMode getExecutionModeForBatch();
Review comment:
Can we add a test in `PipelineOptionsTest`?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 161076)
Time Spent: 40m (was: 0.5h)
> Portable beam hangs while running TFX preprocessing step on a distributed
> cluster
> -
>
> Key: BEAM-5464
> URL: https://issues.apache.org/jira/browse/BEAM-5464
> Project: Beam
> Issue Type: Bug
> Components: java-fn-execution
>Reporter: Axel Magnuson
>Assignee: Ankur Goenka
>Priority: Major
> Time Spent: 40m
> Remaining Estimate: 0h
>
> Recently I went through the exercise of running the TFX taxi example on a
> dataproc cluster. However it would always hang indefinitely. The flink UI
> indicated that the job was halfway done. However I could not see any clear
> errors in the job driver logs, the job service logs, or the Flink logs. The
> root cause is still a mystery to me.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-5917) Update Flink Runner to 1.5.5
[ https://issues.apache.org/jira/browse/BEAM-5917?focusedWorklogId=161083&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-161083 ] ASF GitHub Bot logged work on BEAM-5917: Author: ASF GitHub Bot Created on: 31/Oct/18 10:00 Start Date: 31/Oct/18 10:00 Worklog Time Spent: 10m Work Description: mxm commented on issue #6885: [BEAM-5917] Update Flink to 1.5.5 URL: https://github.com/apache/beam/pull/6885#issuecomment-434628297 IMHO updating to the latest bugfix release will give us only benefits. For minor releases I agree that it makes sense to wait a bit until it stabilizes. The Flink issue is reported here: https://issues.apache.org/jira/browse/FLINK-10732 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 161083) Time Spent: 20m (was: 10m) > Update Flink Runner to 1.5.5 > > > Key: BEAM-5917 > URL: https://issues.apache.org/jira/browse/BEAM-5917 > Project: Beam > Issue Type: Improvement > Components: runner-flink >Reporter: Maximilian Michels >Assignee: Maximilian Michels >Priority: Minor > Fix For: 2.9.0 > > Time Spent: 20m > Remaining Estimate: 0h > > We should be on the latest hotfix version to ensure stability for users: > https://flink.apache.org/news/2018/10/29/release-1.5.5.html -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5919) Build breaks when including Multi-release Jars
[
https://issues.apache.org/jira/browse/BEAM-5919?focusedWorklogId=161092&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-161092
]
ASF GitHub Bot logged work on BEAM-5919:
Author: ASF GitHub Bot
Created on: 31/Oct/18 10:24
Start Date: 31/Oct/18 10:24
Worklog Time Spent: 10m
Work Description: iemejia commented on a change in pull request #6889:
[BEAM-5919] Exclude META-INF from validation of shaded jar files
URL: https://github.com/apache/beam/pull/6889#discussion_r229635884
##
File path:
buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
##
@@ -802,6 +802,7 @@ class BeamModulePlugin implements Plugin {
project.configurations.shadow.artifacts.files.each {
FileTree exposedClasses = project.zipTree(it).matching {
include "**/*.class"
+exclude "META-INF/**" // BEAM-5919: Exclude module-info.class
for Java 9 build support
Review comment:
Wondering if to be future proof we do better add here
`META-INF/versions/**/module-info.class` and
`META-INF/versions/**/org/apache/beam/`
I am assuming this after reading:
https://blog.codefx.org/tools/multi-release-jars-multiple-java-versions/
WDYT?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 161092)
Time Spent: 50m (was: 40m)
> Build breaks when including Multi-release Jars
> --
>
> Key: BEAM-5919
> URL: https://issues.apache.org/jira/browse/BEAM-5919
> Project: Beam
> Issue Type: Bug
> Components: build-system
>Affects Versions: 2.8.0
>Reporter: Ismaël Mejía
>Assignee: Luke Cwik
>Priority: Minor
> Attachments: logjava9jarissue.txt
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> As part of BEAM-5891 there is a reported problem in the build system with
> jars that contain multiple versions of the classes (this is a feature
> introduced in Java 9). This should not be a problem when the main classes are
> still Java 8 compatible as it is the case for byte buddy.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-5919) Build breaks when including Multi-release Jars
[
https://issues.apache.org/jira/browse/BEAM-5919?focusedWorklogId=161091&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-161091
]
ASF GitHub Bot logged work on BEAM-5919:
Author: ASF GitHub Bot
Created on: 31/Oct/18 10:24
Start Date: 31/Oct/18 10:24
Worklog Time Spent: 10m
Work Description: iemejia commented on a change in pull request #6889:
[BEAM-5919] Exclude META-INF from validation of shaded jar files
URL: https://github.com/apache/beam/pull/6889#discussion_r229635884
##
File path:
buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
##
@@ -802,6 +802,7 @@ class BeamModulePlugin implements Plugin {
project.configurations.shadow.artifacts.files.each {
FileTree exposedClasses = project.zipTree(it).matching {
include "**/*.class"
+exclude "META-INF/**" // BEAM-5919: Exclude module-info.class
for Java 9 build support
Review comment:
Wondering if to be future proof we do better add here
`META-INF/versions/module-info.class` and
`META-INF/versions/**/org/apache/beam/`
I am assuming this after reading:
https://blog.codefx.org/tools/multi-release-jars-multiple-java-versions/
WDYT?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 161091)
Time Spent: 40m (was: 0.5h)
> Build breaks when including Multi-release Jars
> --
>
> Key: BEAM-5919
> URL: https://issues.apache.org/jira/browse/BEAM-5919
> Project: Beam
> Issue Type: Bug
> Components: build-system
>Affects Versions: 2.8.0
>Reporter: Ismaël Mejía
>Assignee: Luke Cwik
>Priority: Minor
> Attachments: logjava9jarissue.txt
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> As part of BEAM-5891 there is a reported problem in the build system with
> jars that contain multiple versions of the classes (this is a feature
> introduced in Java 9). This should not be a problem when the main classes are
> still Java 8 compatible as it is the case for byte buddy.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-5919) Build breaks when including Multi-release Jars
[
https://issues.apache.org/jira/browse/BEAM-5919?focusedWorklogId=161093&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-161093
]
ASF GitHub Bot logged work on BEAM-5919:
Author: ASF GitHub Bot
Created on: 31/Oct/18 10:25
Start Date: 31/Oct/18 10:25
Worklog Time Spent: 10m
Work Description: iemejia commented on a change in pull request #6889:
[BEAM-5919] Exclude META-INF from validation of shaded jar files
URL: https://github.com/apache/beam/pull/6889#discussion_r229635884
##
File path:
buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
##
@@ -802,6 +802,7 @@ class BeamModulePlugin implements Plugin {
project.configurations.shadow.artifacts.files.each {
FileTree exposedClasses = project.zipTree(it).matching {
include "**/*.class"
+exclude "META-INF/**" // BEAM-5919: Exclude module-info.class
for Java 9 build support
Review comment:
Wondering if to be future proof we do better add here
`META-INF/versions/**/module-info.class` and
`META-INF/versions/**/org/apache/beam/**`
I am assuming this after reading:
https://blog.codefx.org/tools/multi-release-jars-multiple-java-versions/
WDYT?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 161093)
Time Spent: 1h (was: 50m)
> Build breaks when including Multi-release Jars
> --
>
> Key: BEAM-5919
> URL: https://issues.apache.org/jira/browse/BEAM-5919
> Project: Beam
> Issue Type: Bug
> Components: build-system
>Affects Versions: 2.8.0
>Reporter: Ismaël Mejía
>Assignee: Luke Cwik
>Priority: Minor
> Attachments: logjava9jarissue.txt
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> As part of BEAM-5891 there is a reported problem in the build system with
> jars that contain multiple versions of the classes (this is a feature
> introduced in Java 9). This should not be a problem when the main classes are
> still Java 8 compatible as it is the case for byte buddy.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-5875) Nexmark perf tests fail due to NoClassDefFoundError for Iterables.
[
https://issues.apache.org/jira/browse/BEAM-5875?focusedWorklogId=16&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-16
]
ASF GitHub Bot logged work on BEAM-5875:
Author: ASF GitHub Bot
Created on: 31/Oct/18 11:46
Start Date: 31/Oct/18 11:46
Worklog Time Spent: 10m
Work Description: lgajowy commented on a change in pull request #6894:
[BEAM-5875] Fix scope of dependency ensuring that it is packaged within the jar.
URL: https://github.com/apache/beam/pull/6894#discussion_r229659917
##
File path: sdks/java/testing/test-utils/build.gradle
##
@@ -23,7 +23,7 @@ description = "Apache Beam :: SDKs :: Java :: Test Utils"
dependencies {
shadow project(path: ":beam-sdks-java-core", configuration: "shadow")
- shadow library.java.guava
+ compile library.java.guava
Review comment:
Could you explain what is the difference between "compile" and "shadow"? I
thought I understand this but as I read through "Gradle Primier" doc and
"BeamModulePlugin.java" file, I start to have doubts. Some questions:
- both shadow and compile configurations "shadow" guava, because it is
listed in the `DEFAULT_SHADOW_CLOSURE` that is used there, right? So whichever
configuration I use it will still get shadowed the same way. What if Guava
wasn't listed in the `DEFAULT_SHADOW_CLOSURE`? Would It get shadowed if I use
"compile"?
- you said that using "compile" packages the depencency "within the jar".
So what happens with the guava dependency when I use "shadow"?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 16)
Time Spent: 3h (was: 2h 50m)
> Nexmark perf tests fail due to NoClassDefFoundError for Iterables.
> --
>
> Key: BEAM-5875
> URL: https://issues.apache.org/jira/browse/BEAM-5875
> Project: Beam
> Issue Type: Bug
> Components: test-failures
>Reporter: Henning Rohde
>Assignee: Luke Cwik
>Priority: Critical
> Labels: currently-failing
> Time Spent: 3h
> Remaining Estimate: 0h
>
> https://scans.gradle.com/s/vjkiys2xc3age/console-log?task=:beam-sdks-java-nexmark:run
> I see:
> Caused by: java.lang.NoClassDefFoundError:
> org/apache/beam/repackaged/beam_sdks_java_test_utils/com/google/common/collect/Iterables
> at
> org.apache.beam.sdk.testutils.metrics.MetricsReader.checkIfMetricResultIsUnique(MetricsReader.java:128)
> at
> org.apache.beam.sdk.testutils.metrics.MetricsReader.getCounterMetric(MetricsReader.java:65)
> at
> org.apache.beam.sdk.nexmark.NexmarkLauncher.currentPerf(NexmarkLauncher.java:250)
> at
> org.apache.beam.sdk.nexmark.NexmarkLauncher.monitor(NexmarkLauncher.java:435)
> at
> org.apache.beam.sdk.nexmark.NexmarkLauncher.run(NexmarkLauncher.java:1156)
> at org.apache.beam.sdk.nexmark.Main$Run.call(Main.java:108)
> at org.apache.beam.sdk.nexmark.Main$Run.call(Main.java:96)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.ClassNotFoundException:
> org.apache.beam.repackaged.beam_sdks_java_test_utils.com.google.common.collect.Iterables
> at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> ... 13 more
> PRs for the first red run:
> [BEAM-5716] Move nexmark to "testing" directory in java sdk (commit: 0074138)
> (detail / githubweb)
> [BEAM-5716] Move load-tests code to "testing" directory in java sdk (commit:
> 6674c9d) (detail / githubweb)
> [BEAM-5716] Create module for testing utils (commit: 0628951) (detail /
> githubweb)
> [BEAM-5716] Extract MetricReader class, test it and use in Nexmark code
> (commit: 69730fc) (detail / githubweb)
> [BEAM-5355] Use MetricsReader in GroupByKeyLoadTest (commit: 7374eb6) (detail
> / githubweb)
> Łukasz -- would you mind taking a look? Looks like a shading issue.
--
Th
[jira] [Work logged] (BEAM-5866) RowCoder doesn't implement structuralValue
[
https://issues.apache.org/jira/browse/BEAM-5866?focusedWorklogId=161120&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-161120
]
ASF GitHub Bot logged work on BEAM-5866:
Author: ASF GitHub Bot
Created on: 31/Oct/18 12:21
Start Date: 31/Oct/18 12:21
Worklog Time Spent: 10m
Work Description: kanterov commented on issue #6845: [BEAM-5866] Fix
`Row#equals`
URL: https://github.com/apache/beam/pull/6845#issuecomment-434666535
@reuvenlax yes, I did it in
https://github.com/kanterov/beam/commit/08f300e800ef056238360dcea33a47f95ddb5d3d,
quite amazing stuff in byte buddy btw :).
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 161120)
Time Spent: 6h 10m (was: 6h)
> RowCoder doesn't implement structuralValue
> --
>
> Key: BEAM-5866
> URL: https://issues.apache.org/jira/browse/BEAM-5866
> Project: Beam
> Issue Type: Bug
> Components: sdk-java-core
>Reporter: Gleb Kanterov
>Assignee: Gleb Kanterov
>Priority: Major
> Time Spent: 6h 10m
> Remaining Estimate: 0h
>
> These two properties fail for RowCoder with `BYTES` field, or `Map`
> field.
> {code}
> public static void testConsistentWithEquals(Coder coder, T example) {
> assumeTrue(coder.consistentWithEquals());
> byte[] bytes = encodeBytes(coder, example);
> // even if the coder is non-deterministic, if the encoded bytes match,
> // coder is consistent with equals, decoded values must be equal
> T out0 = decodeBytes(coder, bytes);
> T out1 = decodeBytes(coder, bytes);
> assertEquals("If the encoded bytes match, decoded values must be equal",
> out0, out1);
> assertEquals(
> "If two values are equal, their hash codes must be equal",
> out0.hashCode(),
> out1.hashCode());
> }
> public static void testStructuralValueConsistentWithEquals(Coder
> coder, T example) {
> byte[] bytes = encodeBytes(coder, example);
> // even if coder is non-deterministic, if the encoded bytes match,
> // structural values must be equal
> Object out0 = coder.structuralValue(decodeBytes(coder, bytes));
> Object out1 = coder.structuralValue(decodeBytes(coder, bytes));
> assertEquals("If the encoded bytes match, structural values must be
> equal", out0, out1);
> assertEquals(
> "If two values are equal, their hash codes must be equal",
> out0.hashCode(),
> out1.hashCode());
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (BEAM-5725) ElasticsearchIO RetryConfiguration response parse failure
[ https://issues.apache.org/jira/browse/BEAM-5725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16670108#comment-16670108 ] Wout Scheepers edited comment on BEAM-5725 at 10/31/18 1:30 PM: I've been digging into it and came up with the following so far: As Tim already pointed out, the problem here is that parseResponse(response) is indeed not repeatable, the content can only be consumed once as the Elastic Response object encapsulates a HttpEntity object. I added a reproducible unit test[1], which tries to insert a valid document with the retryConfiguration set. It becomes clear that handleRetry() does not have to be called for the bug to appear. My first thought on a solution was to use the reset() method on the InputStream of the Entity content, however, this is not supported for the InputStream used in HttpEntity. A possible solution would be the following: Encapulate the Response object in a wrapper object, making sure the content of the HttpEntity object can be parsed repeatable. I think the best way to do this is by implementing a BufferedHttpEntity[2] in the wrapper. This can be done either only in the case a retryConfiguration is set, but I guess it's probably better to create a wrapper for the response objects everywhere in the ElasticSearchIO class? Is there a better or more elegant solution for this? I've also found a way of getting control on how to read bytes from the buffer in elastic docs[3] but I'm not sure it can be of any help (it's for async calls, not sure they are used in ElasticSearchIO): "... As for reading the response body, the HttpEntity#getContent method comes handy which returns an InputStream reading from the previously buffered response body. As an alternative, it is possible to provide a custom org.apache.http.nio.protocol.HttpAsyncResponseConsumer that controls how bytes are read and buffered. ..." I'm happy to get some thoughts on a good solution. Thanks, Wout [1] [https://github.com/wscheep/beam/commit/8f2093066a2908f0472983cfc640bc7644b728d9] [2] [https://hc.apache.org/httpcomponents-core-ga/httpcore/apidocs/org/apache/http/entity/BufferedHttpEntity.html] [3] [https://www.elastic.co/guide/en/elasticsearch/client/java-rest/master/java-rest-low-usage-responses.html] was (Author: wouts): I've been digging into it and came up with the following so far: As Tim already pointed out, the problem here is that parseResponse(response) is indeed not repeatable, the content can only be consumed once as the Elastic Reponse object encapsulates a HttpEntity object. I added a reproducible unit test[1], which tries to insert a valid document with the retryConfiguration set. It becomes clear that handleRetry() does not have to be called for the bug to appear. My first thought on a solution was to use the reset() method on the InputStream of the Entity content, however, this is not supported for the InputStream used in HttpEntity. A possible solution would be the following: Encapulate the Response object in a wrapper object, making sure the content of the HttpEntity object can be parsed repeatable. I think the best way to do this is by implementing a BufferedHttpEntity[2] in the wrapper. This can be done either only in the case a retryConfiguration is set, but I guess it's probably better to create a wrapper for the response objects everywhere in the ElasticSearchIO class? Is there a better or more elegant solution for this? I've also found a way of getting control on how to read bytes from the buffer in elastic docs[3] but I'm not sure it can be of any help (it's for async calls, not sure they are used in ElasticSearchIO): "... As for reading the response body, the HttpEntity#getContent method comes handy which returns an InputStream reading from the previously buffered response body. As an alternative, it is possible to provide a custom org.apache.http.nio.protocol.HttpAsyncResponseConsumer that controls how bytes are read and buffered. ..." I'm happy to get some thoughts on a good solution. Thanks, Wout [1] https://github.com/wscheep/beam/commit/8f2093066a2908f0472983cfc640bc7644b728d9 [2] https://hc.apache.org/httpcomponents-core-ga/httpcore/apidocs/org/apache/http/entity/BufferedHttpEntity.html [3] https://www.elastic.co/guide/en/elasticsearch/client/java-rest/master/java-rest-low-usage-responses.html > ElasticsearchIO RetryConfiguration response parse failure > - > > Key: BEAM-5725 > URL: https://issues.apache.org/jira/browse/BEAM-5725 > Project: Beam > Issue Type: Bug > Components: io-java-elasticsearch >Reporter: Wout Scheepers >Assignee: Wout Scheepers >Priority: Major > > When using .withRetryConfiguration() for ElasticsearchIO, I get the following
[jira] [Commented] (BEAM-5725) ElasticsearchIO RetryConfiguration response parse failure
[
https://issues.apache.org/jira/browse/BEAM-5725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16670108#comment-16670108
]
Wout Scheepers commented on BEAM-5725:
--
I've been digging into it and came up with the following so far:
As Tim already pointed out, the problem here is that parseResponse(response) is
indeed not repeatable, the content can only be consumed once as the Elastic
Reponse object encapsulates a HttpEntity object.
I added a reproducible unit test[1], which tries to insert a valid document
with the retryConfiguration set.
It becomes clear that handleRetry() does not have to be called for the bug to
appear.
My first thought on a solution was to use the reset() method on the InputStream
of the Entity content, however, this is not supported for the InputStream used
in HttpEntity.
A possible solution would be the following:
Encapulate the Response object in a wrapper object, making sure the content of
the HttpEntity object can be parsed repeatable.
I think the best way to do this is by implementing a BufferedHttpEntity[2] in
the wrapper.
This can be done either only in the case a retryConfiguration is set, but I
guess it's probably better to create a wrapper for the response objects
everywhere in the ElasticSearchIO class?
Is there a better or more elegant solution for this?
I've also found a way of getting control on how to read bytes from the buffer
in elastic docs[3] but I'm not sure it can be of any help (it's for async
calls, not sure they are used in ElasticSearchIO):
"... As for reading the response body, the HttpEntity#getContent method comes
handy which returns an InputStream reading from the previously buffered
response body. As an alternative, it is possible to provide a custom
org.apache.http.nio.protocol.HttpAsyncResponseConsumer that controls how bytes
are read and buffered. ..."
I'm happy to get some thoughts on a good solution.
Thanks,
Wout
[1]
https://github.com/wscheep/beam/commit/8f2093066a2908f0472983cfc640bc7644b728d9
[2]
https://hc.apache.org/httpcomponents-core-ga/httpcore/apidocs/org/apache/http/entity/BufferedHttpEntity.html
[3]
https://www.elastic.co/guide/en/elasticsearch/client/java-rest/master/java-rest-low-usage-responses.html
> ElasticsearchIO RetryConfiguration response parse failure
> -
>
> Key: BEAM-5725
> URL: https://issues.apache.org/jira/browse/BEAM-5725
> Project: Beam
> Issue Type: Bug
> Components: io-java-elasticsearch
>Reporter: Wout Scheepers
>Assignee: Wout Scheepers
>Priority: Major
>
> When using .withRetryConfiguration() for ElasticsearchIO, I get the following
> stacktrace:
>
>
> {code:java}
> Caused by: com.fasterxml.jackson.databind.exc.MismatchedInputException: No
> content to map due to end-of-input
> at [Source: (org.apache.http.nio.entity.ContentInputStream); line: 1, column:
> 0]
> at
> com.fasterxml.jackson.databind.exc.MismatchedInputException.from(MismatchedInputException.java:59)
> at
> com.fasterxml.jackson.databind.ObjectMapper._initForReading(ObjectMapper.java:4133)
> at
> com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:3988)
> at
> com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3058)
> at
> org.apache.beam.sdk.io.elasticsearch.ElasticsearchIO.parseResponse(ElasticsearchIO.java:173)
> at
> org.apache.beam.sdk.io.elasticsearch.ElasticsearchIO.checkForErrors(ElasticsearchIO.java:177)
> at
> org.apache.beam.sdk.io.elasticsearch.ElasticsearchIO$Write$WriteFn.flushBatch(ElasticsearchIO.java:1204)
> at
> org.apache.beam.sdk.io.elasticsearch.ElasticsearchIO$Write$WriteFn.finishBundle(ElasticsearchIO.java:1175)
> {code}
>
>
> Probably the elastic response object's content stream is consumed twice,
> resulting in a MismatchedInputException.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (BEAM-5925) Test flake in ElasticsearchIOTest.testWriteFullAddressing
[
https://issues.apache.org/jira/browse/BEAM-5925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16670115#comment-16670115
]
Etienne Chauchot commented on BEAM-5925:
Thanks Kenn for pointing out. It has been a lot of flakiness lately with ESIO
v6. I'll take a look at them
> Test flake in ElasticsearchIOTest.testWriteFullAddressing
> -
>
> Key: BEAM-5925
> URL: https://issues.apache.org/jira/browse/BEAM-5925
> Project: Beam
> Issue Type: Bug
> Components: io-java-elasticsearch
>Reporter: Kenneth Knowles
>Assignee: Etienne Chauchot
>Priority: Critical
>
> https://builds.apache.org/view/A-D/view/Beam/job/beam_PostCommit_Java_GradleBuild/1789/
> https://scans.gradle.com/s/j42mwdsn5svcs
> {code}
> org.apache.beam.sdk.Pipeline$PipelineExecutionException: java.io.IOException:
> listener timeout after waiting for [3] ms
> {code}
> Log looks like this:
> {code}
> [2018-10-31T04:06:07,571][INFO ][o.a.b.s.i.e.ElasticsearchIOTest]
> [testWriteFullAddressing]: before test
> [2018-10-31T04:06:07,572][INFO ][o.a.b.s.i.e.ElasticsearchIOTest]
> [ElasticsearchIOTest#testWriteFullAddressing]: setting up test
> [2018-10-31T04:06:07,589][INFO ][o.e.c.m.MetaDataIndexTemplateService]
> [node_s0] adding template [random_index_template] for index patterns [*]
> [2018-10-31T04:06:07,645][INFO ][o.a.b.s.i.e.ElasticsearchIOTest]
> [ElasticsearchIOTest#testWriteFullAddressing]: all set up test
> [2018-10-31T04:06:10,536][INFO ][o.e.c.m.MetaDataCreateIndexService]
> [node_s0] [galilei] creating index, cause [auto(bulk api)], templates
> [random_index_template], shards [6]/[0], mappings []
> [2018-10-31T04:06:33,963][INFO ][o.e.c.m.MetaDataCreateIndexService]
> [node_s0] [curie] creating index, cause [auto(bulk api)], templates
> [random_index_template], shards [6]/[0], mappings []
> [2018-10-31T04:06:34,034][INFO ][o.e.c.m.MetaDataCreateIndexService]
> [node_s0] [darwin] creating index, cause [auto(bulk api)], templates
> [random_index_template], shards [6]/[0], mappings []
> [2018-10-31T04:06:34,050][INFO ][o.e.c.m.MetaDataCreateIndexService]
> [node_s0] [copernicus] creating index, cause [auto(bulk api)], templates
> [random_index_template], shards [6]/[0], mappings []
> [2018-10-31T04:06:34,075][INFO ][o.e.c.m.MetaDataCreateIndexService]
> [node_s0] [faraday] creating index, cause [auto(bulk api)], templates
> [random_index_template], shards [6]/[0], mappings []
> [2018-10-31T04:06:34,095][INFO ][o.e.c.m.MetaDataCreateIndexService]
> [node_s0] [bohr] creating index, cause [auto(bulk api)], templates
> [random_index_template], shards [6]/[0], mappings []
> [2018-10-31T04:06:34,113][INFO ][o.e.c.m.MetaDataCreateIndexService]
> [node_s0] [pasteur] creating index, cause [auto(bulk api)], templates
> [random_index_template], shards [6]/[0], mappings []
> [2018-10-31T04:06:34,142][INFO ][o.e.c.m.MetaDataCreateIndexService]
> [node_s0] [einstein] creating index, cause [auto(bulk api)], templates
> [random_index_template], shards [6]/[0], mappings []
> [2018-10-31T04:06:34,205][INFO ][o.e.c.m.MetaDataCreateIndexService]
> [node_s0] [maxwell] creating index, cause [auto(bulk api)], templates
> [random_index_template], shards [6]/[0], mappings []
> [2018-10-31T04:06:34,226][INFO ][o.e.c.m.MetaDataCreateIndexService]
> [node_s0] [newton] creating index, cause [auto(bulk api)], templates
> [random_index_template], shards [6]/[0], mappings []
> [2018-10-31T04:06:36,914][INFO ][o.e.c.r.a.AllocationService] [node_s0]
> Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards
> started [[galilei][4], [galilei][5]] ...]).
> [2018-10-31T04:06:36,970][INFO ][o.e.c.m.MetaDataMappingService] [node_s0]
> [galilei/Vn1b8XXVSAmrTb5BVe2IJQ] create_mapping [TYPE_1]
> [2018-10-31T04:06:37,137][INFO ][o.e.c.m.MetaDataMappingService] [node_s0]
> [newton/bjnImLt_QguBGEFH9lBJ6Q] create_mapping [TYPE_-1]
> [2018-10-31T04:06:37,385][INFO ][o.e.c.m.MetaDataMappingService] [node_s0]
> [maxwell/-RZ32NbRRZWaGaVfaptFIA] create_mapping [TYPE_0]
> [2018-10-31T04:06:37,636][INFO ][o.e.c.m.MetaDataMappingService] [node_s0]
> [einstein/2lgF5Vj6Ti2KTS-pYSzv3Q] create_mapping [TYPE_1]
> [2018-10-31T04:06:37,806][INFO ][o.e.c.m.MetaDataMappingService] [node_s0]
> [pasteur/832OwzleRSOHsWx85vOH-w] create_mapping [TYPE_0]
> [2018-10-31T04:06:38,103][INFO ][o.e.c.m.MetaDataMappingService] [node_s0]
> [bohr/9YTwB1yvTYKf9YjYCmHjwg] create_mapping [TYPE_1]
> [2018-10-31T04:06:38,229][INFO ][o.e.c.m.MetaDataMappingService] [node_s0]
> [faraday/vIMYG8vpTQKqNkyajcFOxw] create_mapping [TYPE_0]
> [2018-10-31T04:06:38,576][INFO ][o.e.c.m.MetaDataMappingService] [node_s0]
> [copernicus/NzCZssInSiOdZKTmLCoXRw] create_mapping [TYPE_1]
> [2018-10-31T04:06:38,890][INFO ][o.e.c.m.MetaDataMappingServic
[jira] [Comment Edited] (BEAM-5725) ElasticsearchIO RetryConfiguration response parse failure
[ https://issues.apache.org/jira/browse/BEAM-5725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16670108#comment-16670108 ] Wout Scheepers edited comment on BEAM-5725 at 10/31/18 1:48 PM: I've been digging into it and came up with the following so far: As Tim already pointed out, the problem here is that parseResponse(response) is indeed not repeatable, the content can only be consumed once as the Elastic Response object encapsulates a HttpEntity object. I added a reproducible unit test[1], which tries to insert a valid document with the retryConfiguration set. It becomes clear that handleRetry() does not have to be called for the bug to appear. My first thought on a solution was to use the reset() method on the InputStream of the Entity content, however, this is not supported for the InputStream used in HttpEntity. A possible solution would be the following: Encapsulate the Response object in a wrapper object, making sure the content of the HttpEntity object can be parsed repeatable. I think the best way to do this is by implementing a BufferedHttpEntity[2] in the wrapper. This can be done either only in the case a retryConfiguration is set, but I guess it's probably better to create a wrapper for the response objects everywhere in the ElasticSearchIO class? Is there a better or more elegant solution for this? I've also found a way of getting control on how to read bytes from the buffer in elastic docs[3] but I'm not sure it can be of any help (it's for async calls, not sure they are used in ElasticSearchIO): "... As for reading the response body, the HttpEntity#getContent method comes handy which returns an InputStream reading from the previously buffered response body. As an alternative, it is possible to provide a custom org.apache.http.nio.protocol.HttpAsyncResponseConsumer that controls how bytes are read and buffered. ..." I'm happy to get some thoughts on a good solution. Thanks, Wout [1] [https://github.com/wscheep/beam/commit/8f2093066a2908f0472983cfc640bc7644b728d9] [2] [https://hc.apache.org/httpcomponents-core-ga/httpcore/apidocs/org/apache/http/entity/BufferedHttpEntity.html] [3] [https://www.elastic.co/guide/en/elasticsearch/client/java-rest/master/java-rest-low-usage-responses.html] was (Author: wouts): I've been digging into it and came up with the following so far: As Tim already pointed out, the problem here is that parseResponse(response) is indeed not repeatable, the content can only be consumed once as the Elastic Response object encapsulates a HttpEntity object. I added a reproducible unit test[1], which tries to insert a valid document with the retryConfiguration set. It becomes clear that handleRetry() does not have to be called for the bug to appear. My first thought on a solution was to use the reset() method on the InputStream of the Entity content, however, this is not supported for the InputStream used in HttpEntity. A possible solution would be the following: Encapulate the Response object in a wrapper object, making sure the content of the HttpEntity object can be parsed repeatable. I think the best way to do this is by implementing a BufferedHttpEntity[2] in the wrapper. This can be done either only in the case a retryConfiguration is set, but I guess it's probably better to create a wrapper for the response objects everywhere in the ElasticSearchIO class? Is there a better or more elegant solution for this? I've also found a way of getting control on how to read bytes from the buffer in elastic docs[3] but I'm not sure it can be of any help (it's for async calls, not sure they are used in ElasticSearchIO): "... As for reading the response body, the HttpEntity#getContent method comes handy which returns an InputStream reading from the previously buffered response body. As an alternative, it is possible to provide a custom org.apache.http.nio.protocol.HttpAsyncResponseConsumer that controls how bytes are read and buffered. ..." I'm happy to get some thoughts on a good solution. Thanks, Wout [1] [https://github.com/wscheep/beam/commit/8f2093066a2908f0472983cfc640bc7644b728d9] [2] [https://hc.apache.org/httpcomponents-core-ga/httpcore/apidocs/org/apache/http/entity/BufferedHttpEntity.html] [3] [https://www.elastic.co/guide/en/elasticsearch/client/java-rest/master/java-rest-low-usage-responses.html] > ElasticsearchIO RetryConfiguration response parse failure > - > > Key: BEAM-5725 > URL: https://issues.apache.org/jira/browse/BEAM-5725 > Project: Beam > Issue Type: Bug > Components: io-java-elasticsearch >Reporter: Wout Scheepers >Assignee: Wout Scheepers >Priority: Major > > When using .withRetryConfiguration() for ElasticsearchIO, I ge
[jira] [Created] (BEAM-5928) ConcurrentModificationException from RowCoderGenerator lazy caching
Benson Tucker created BEAM-5928:
---
Summary: ConcurrentModificationException from RowCoderGenerator
lazy caching
Key: BEAM-5928
URL: https://issues.apache.org/jira/browse/BEAM-5928
Project: Beam
Issue Type: Bug
Components: sdk-java-core
Reporter: Benson Tucker
Assignee: Kenneth Knowles
h3. Summary:
RowCoderGenerator caches a delegate Coder once encode or decode is
exercised, but there's not an API for caching this delegate eagerly.
h3. Use Case:
When creating several PCollections to perform distinct reads with the same
schema, you might create one RowCoder.of(schema) before creating the list of
PCollections / PCollectionsList. However, once the pipeline begins and rows
arrive for encoding, these pipelines will simultaneously try to cache a
delegate coder for the row's schema.
h3. Workaround:
You can force the eager caching of the code by exercising encode in the main
application before creating PCollections using the RowCoder:
{code:java}
try {
myRowCoder.encode(null, null);
} catch (IOException | NullPointerException e) {
// do nothing
}
{code}
h3. Context:
I've only encountered this during development with the direct runner.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-5299) Define max global window as a shared value in protos like URN enums.
[ https://issues.apache.org/jira/browse/BEAM-5299?focusedWorklogId=161168&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-161168 ] ASF GitHub Bot logged work on BEAM-5299: Author: ASF GitHub Bot Created on: 31/Oct/18 15:03 Start Date: 31/Oct/18 15:03 Worklog Time Spent: 10m Work Description: lgajowy opened a new pull request #6899: Revert "[BEAM-5299] Define max timestamp for global window in proto (… URL: https://github.com/apache/beam/pull/6899 …#6381)" This reverts commit f20e90a65874945dbbeb33270cab2a91d619af03. **Please** add a meaningful description for your change here Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). It will help us expedite review of your Pull Request if you tag someone (e.g. `@username`) to look at it. Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/) | --- | --- | --- | --- | --- | --- Java | [](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/) [](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/) [](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Python_VR_Flink/lastCompletedBuild/) | --- | --- | --- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 161168) Time Spent: 6h 40m (was: 6.5h) > Define max global window as a shared value in proto
[jira] [Commented] (BEAM-5605) Support Portable SplittableDoFn for batch
[ https://issues.apache.org/jira/browse/BEAM-5605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16670211#comment-16670211 ] Scott Wegner commented on BEAM-5605: Yup, thanks. > Support Portable SplittableDoFn for batch > - > > Key: BEAM-5605 > URL: https://issues.apache.org/jira/browse/BEAM-5605 > Project: Beam > Issue Type: New Feature > Components: sdk-java-core >Reporter: Scott Wegner >Assignee: Scott Wegner >Priority: Major > Labels: portability > > Roll-up item tracking work towards supporting portable SplittableDoFn for > batch -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5299) Define max global window as a shared value in protos like URN enums.
[ https://issues.apache.org/jira/browse/BEAM-5299?focusedWorklogId=161172&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-161172 ] ASF GitHub Bot logged work on BEAM-5299: Author: ASF GitHub Bot Created on: 31/Oct/18 15:15 Start Date: 31/Oct/18 15:15 Worklog Time Spent: 10m Work Description: lgajowy commented on issue #6381: [BEAM-5299] Define max timestamp for global window in proto URL: https://github.com/apache/beam/pull/6381#issuecomment-434726637 It seems that merging this broke Performance Tests (example logs): https://builds.apache.org/view/A-D/view/Beam/job/beam_PerformanceTests_TextIOIT/1197/console I proposed a revert here: https://github.com/apache/beam/pull/6899 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 161172) Time Spent: 6h 50m (was: 6h 40m) > Define max global window as a shared value in protos like URN enums. > > > Key: BEAM-5299 > URL: https://issues.apache.org/jira/browse/BEAM-5299 > Project: Beam > Issue Type: Improvement > Components: beam-model, sdk-go, sdk-java-core, sdk-py-core >Reporter: Luke Cwik >Assignee: Maximilian Michels >Priority: Minor > Labels: portability > Time Spent: 6h 50m > Remaining Estimate: 0h > > Instead of having each language define a max timestamp themselves, define the > max timestamps within proto to be shared across different languages. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5299) Define max global window as a shared value in protos like URN enums.
[ https://issues.apache.org/jira/browse/BEAM-5299?focusedWorklogId=161173&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-161173 ] ASF GitHub Bot logged work on BEAM-5299: Author: ASF GitHub Bot Created on: 31/Oct/18 15:15 Start Date: 31/Oct/18 15:15 Worklog Time Spent: 10m Work Description: lgajowy edited a comment on issue #6381: [BEAM-5299] Define max timestamp for global window in proto URL: https://github.com/apache/beam/pull/6381#issuecomment-434726637 It seems that merging this broke Performance Tests. Example logs: https://builds.apache.org/view/A-D/view/Beam/job/beam_PerformanceTests_TextIOIT/1197/console I proposed a revert here: https://github.com/apache/beam/pull/6899 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 161173) Time Spent: 7h (was: 6h 50m) > Define max global window as a shared value in protos like URN enums. > > > Key: BEAM-5299 > URL: https://issues.apache.org/jira/browse/BEAM-5299 > Project: Beam > Issue Type: Improvement > Components: beam-model, sdk-go, sdk-java-core, sdk-py-core >Reporter: Luke Cwik >Assignee: Maximilian Michels >Priority: Minor > Labels: portability > Time Spent: 7h > Remaining Estimate: 0h > > Instead of having each language define a max timestamp themselves, define the > max timestamps within proto to be shared across different languages. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-5910) FileSystems should retrieve lastModified time
[ https://issues.apache.org/jira/browse/BEAM-5910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16670230#comment-16670230 ] Ismaël Mejía commented on BEAM-5910: Big +1 on this, makes total sense. > FileSystems should retrieve lastModified time > - > > Key: BEAM-5910 > URL: https://issues.apache.org/jira/browse/BEAM-5910 > Project: Beam > Issue Type: Improvement > Components: sdk-java-core >Reporter: Jeff Klukas >Assignee: Jeff Klukas >Priority: Minor > > In the Java SDK, the Filesystems.match facilities are aimed at listing file > names and collect very limited additional metadata from the filesystem > (sizeBytes and isReadSeekEfficient). I propose adding a new field for > lastModified time to MatchResult.Metadata that each FileSystem would populate > when listing files. > This would be a basis for a future improvement to > FileIO.match(...).continuously(...) where we could let the user opt to poll > not just for new file names, but also for existing file names if their > content has been updated. > In the near term, the addition of lastModified to Metadata would allow users > to implement their own polling logic on top of Filesystems.match to detect > and download new files from any of the supported filesystems. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5299) Define max global window as a shared value in protos like URN enums.
[ https://issues.apache.org/jira/browse/BEAM-5299?focusedWorklogId=161176&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-161176 ] ASF GitHub Bot logged work on BEAM-5299: Author: ASF GitHub Bot Created on: 31/Oct/18 15:18 Start Date: 31/Oct/18 15:18 Worklog Time Spent: 10m Work Description: lgajowy edited a comment on issue #6381: [BEAM-5299] Define max timestamp for global window in proto URL: https://github.com/apache/beam/pull/6381#issuecomment-434726637 It seems that merging this broke Performance Tests. Example logs: https://builds.apache.org/view/A-D/view/Beam/job/beam_PerformanceTests_TextIOIT/1197/console I proposed a revert here: https://github.com/apache/beam/pull/6899 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 161176) Time Spent: 7h 20m (was: 7h 10m) > Define max global window as a shared value in protos like URN enums. > > > Key: BEAM-5299 > URL: https://issues.apache.org/jira/browse/BEAM-5299 > Project: Beam > Issue Type: Improvement > Components: beam-model, sdk-go, sdk-java-core, sdk-py-core >Reporter: Luke Cwik >Assignee: Maximilian Michels >Priority: Minor > Labels: portability > Time Spent: 7h 20m > Remaining Estimate: 0h > > Instead of having each language define a max timestamp themselves, define the > max timestamps within proto to be shared across different languages. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5299) Define max global window as a shared value in protos like URN enums.
[ https://issues.apache.org/jira/browse/BEAM-5299?focusedWorklogId=161174&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-161174 ] ASF GitHub Bot logged work on BEAM-5299: Author: ASF GitHub Bot Created on: 31/Oct/18 15:18 Start Date: 31/Oct/18 15:18 Worklog Time Spent: 10m Work Description: lgajowy edited a comment on issue #6381: [BEAM-5299] Define max timestamp for global window in proto URL: https://github.com/apache/beam/pull/6381#issuecomment-434726637 It seems that merging this broke Performance Tests and Nexmark_Dataflow job. Example logs: https://builds.apache.org/view/A-D/view/Beam/job/beam_PerformanceTests_TextIOIT/1197/console I proposed a revert here: https://github.com/apache/beam/pull/6899 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 161174) Time Spent: 7h 10m (was: 7h) > Define max global window as a shared value in protos like URN enums. > > > Key: BEAM-5299 > URL: https://issues.apache.org/jira/browse/BEAM-5299 > Project: Beam > Issue Type: Improvement > Components: beam-model, sdk-go, sdk-java-core, sdk-py-core >Reporter: Luke Cwik >Assignee: Maximilian Michels >Priority: Minor > Labels: portability > Time Spent: 7h 10m > Remaining Estimate: 0h > > Instead of having each language define a max timestamp themselves, define the > max timestamps within proto to be shared across different languages. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (BEAM-5929) BigQueryIO DynamicDestination should support JSONSchema
Benson Tucker created BEAM-5929: --- Summary: BigQueryIO DynamicDestination should support JSONSchema Key: BEAM-5929 URL: https://issues.apache.org/jira/browse/BEAM-5929 Project: Beam Issue Type: Improvement Components: io-java-gcp Reporter: Benson Tucker Assignee: Chamikara Jayalath JSON-formatted String schemas for BQ tables should be allowed in the getSchema method of a DynamicDestination allowing dynamically-targeted writes to take advantage of a data model like that expected by `withSchemaFromView`. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5299) Define max global window as a shared value in protos like URN enums.
[
https://issues.apache.org/jira/browse/BEAM-5299?focusedWorklogId=161180&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-161180
]
ASF GitHub Bot logged work on BEAM-5299:
Author: ASF GitHub Bot
Created on: 31/Oct/18 15:27
Start Date: 31/Oct/18 15:27
Worklog Time Spent: 10m
Work Description: mxm commented on a change in pull request #6899: Revert
"[BEAM-5299] Define max timestamp for global window in proto (…
URL: https://github.com/apache/beam/pull/6899#discussion_r229743814
##
File path: sdks/java/core/build.gradle
##
@@ -51,8 +51,6 @@ test {
}
dependencies {
- // Required to load constants from the model, e.g. max timestamp for global
window
- shadow project(path: ":beam-model-pipeline", configuration: "shadow")
Review comment:
Pretty sure this is the culprit.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 161180)
Time Spent: 7.5h (was: 7h 20m)
> Define max global window as a shared value in protos like URN enums.
>
>
> Key: BEAM-5299
> URL: https://issues.apache.org/jira/browse/BEAM-5299
> Project: Beam
> Issue Type: Improvement
> Components: beam-model, sdk-go, sdk-java-core, sdk-py-core
>Reporter: Luke Cwik
>Assignee: Maximilian Michels
>Priority: Minor
> Labels: portability
> Time Spent: 7.5h
> Remaining Estimate: 0h
>
> Instead of having each language define a max timestamp themselves, define the
> max timestamps within proto to be shared across different languages.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-5299) Define max global window as a shared value in protos like URN enums.
[ https://issues.apache.org/jira/browse/BEAM-5299?focusedWorklogId=161185&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-161185 ] ASF GitHub Bot logged work on BEAM-5299: Author: ASF GitHub Bot Created on: 31/Oct/18 15:35 Start Date: 31/Oct/18 15:35 Worklog Time Spent: 10m Work Description: mxm commented on issue #6899: Revert "[BEAM-5299] Define max timestamp for global window in proto (… URL: https://github.com/apache/beam/pull/6899#issuecomment-434734614 @lgajowy and me are trying to find a fix but will revert otherwise. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 161185) Time Spent: 7h 40m (was: 7.5h) > Define max global window as a shared value in protos like URN enums. > > > Key: BEAM-5299 > URL: https://issues.apache.org/jira/browse/BEAM-5299 > Project: Beam > Issue Type: Improvement > Components: beam-model, sdk-go, sdk-java-core, sdk-py-core >Reporter: Luke Cwik >Assignee: Maximilian Michels >Priority: Minor > Labels: portability > Time Spent: 7h 40m > Remaining Estimate: 0h > > Instead of having each language define a max timestamp themselves, define the > max timestamps within proto to be shared across different languages. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5299) Define max global window as a shared value in protos like URN enums.
[
https://issues.apache.org/jira/browse/BEAM-5299?focusedWorklogId=161191&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-161191
]
ASF GitHub Bot logged work on BEAM-5299:
Author: ASF GitHub Bot
Created on: 31/Oct/18 15:48
Start Date: 31/Oct/18 15:48
Worklog Time Spent: 10m
Work Description: lgajowy commented on a change in pull request #6899:
Revert "[BEAM-5299] Define max timestamp for global window in proto (…
URL: https://github.com/apache/beam/pull/6899#discussion_r229753966
##
File path: sdks/java/core/build.gradle
##
@@ -51,8 +51,6 @@ test {
}
dependencies {
- // Required to load constants from the model, e.g. max timestamp for global
window
- shadow project(path: ":beam-model-pipeline", configuration: "shadow")
Review comment:
changed to compile, run the TextIOIT on Dataflow. Didn't help.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 161191)
Time Spent: 7h 50m (was: 7h 40m)
> Define max global window as a shared value in protos like URN enums.
>
>
> Key: BEAM-5299
> URL: https://issues.apache.org/jira/browse/BEAM-5299
> Project: Beam
> Issue Type: Improvement
> Components: beam-model, sdk-go, sdk-java-core, sdk-py-core
>Reporter: Luke Cwik
>Assignee: Maximilian Michels
>Priority: Minor
> Labels: portability
> Time Spent: 7h 50m
> Remaining Estimate: 0h
>
> Instead of having each language define a max timestamp themselves, define the
> max timestamps within proto to be shared across different languages.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (BEAM-5928) ConcurrentModificationException from RowCoderGenerator lazy caching
[
https://issues.apache.org/jira/browse/BEAM-5928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kenneth Knowles reassigned BEAM-5928:
-
Assignee: Reuven Lax (was: Kenneth Knowles)
> ConcurrentModificationException from RowCoderGenerator lazy caching
> ---
>
> Key: BEAM-5928
> URL: https://issues.apache.org/jira/browse/BEAM-5928
> Project: Beam
> Issue Type: Bug
> Components: sdk-java-core
>Reporter: Benson Tucker
>Assignee: Reuven Lax
>Priority: Major
>
> h3. Summary:
> RowCoderGenerator caches a delegate Coder once encode or decode is
> exercised, but there's not an API for caching this delegate eagerly.
> h3. Use Case:
> When creating several PCollections to perform distinct reads with the same
> schema, you might create one RowCoder.of(schema) before creating the list of
> PCollections / PCollectionsList. However, once the pipeline begins and rows
> arrive for encoding, these pipelines will simultaneously try to cache a
> delegate coder for the row's schema.
> h3. Workaround:
> You can force the eager caching of the code by exercising encode in the main
> application before creating PCollections using the RowCoder:
> {code:java}
> try {
> myRowCoder.encode(null, null);
> } catch (IOException | NullPointerException e) {
> // do nothing
> }
> {code}
> h3. Context:
> I've only encountered this during development with the direct runner.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-5299) Define max global window as a shared value in protos like URN enums.
[ https://issues.apache.org/jira/browse/BEAM-5299?focusedWorklogId=161193&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-161193 ] ASF GitHub Bot logged work on BEAM-5299: Author: ASF GitHub Bot Created on: 31/Oct/18 15:54 Start Date: 31/Oct/18 15:54 Worklog Time Spent: 10m Work Description: lgajowy commented on issue #6899: Revert "[BEAM-5299] Define max timestamp for global window in proto (… URL: https://github.com/apache/beam/pull/6899#issuecomment-434742048 Run Java PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 161193) Time Spent: 8h (was: 7h 50m) > Define max global window as a shared value in protos like URN enums. > > > Key: BEAM-5299 > URL: https://issues.apache.org/jira/browse/BEAM-5299 > Project: Beam > Issue Type: Improvement > Components: beam-model, sdk-go, sdk-java-core, sdk-py-core >Reporter: Luke Cwik >Assignee: Maximilian Michels >Priority: Minor > Labels: portability > Time Spent: 8h > Remaining Estimate: 0h > > Instead of having each language define a max timestamp themselves, define the > max timestamps within proto to be shared across different languages. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5875) Nexmark perf tests fail due to NoClassDefFoundError for Iterables.
[
https://issues.apache.org/jira/browse/BEAM-5875?focusedWorklogId=161197&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-161197
]
ASF GitHub Bot logged work on BEAM-5875:
Author: ASF GitHub Bot
Created on: 31/Oct/18 15:58
Start Date: 31/Oct/18 15:58
Worklog Time Spent: 10m
Work Description: kennknowles commented on a change in pull request
#6894: [BEAM-5875] Fix scope of dependency ensuring that it is packaged within
the jar.
URL: https://github.com/apache/beam/pull/6894#discussion_r229758168
##
File path: sdks/java/testing/test-utils/build.gradle
##
@@ -23,7 +23,7 @@ description = "Apache Beam :: SDKs :: Java :: Test Utils"
dependencies {
shadow project(path: ":beam-sdks-java-core", configuration: "shadow")
- shadow library.java.guava
+ compile library.java.guava
Review comment:
There are two independent parts to what we call "shading": relocating
(moving classes into different namespace) and bundling (putting classes into
jars). You can do one and not the other. The difference between `compile` and
`shadow` is bundling. If you have a `compile` dependency, then it will be
bundled into the `shadowJar`. If you have a `shadow` dependency, then it will
not be bundled into the `shadowJar`.
The `DEFAULT_SHADOW_CLOSURE` configures relocations and TBH not sure what
`include(dependency(...))` does. It is not the same as putting it in the scope
to be bundled.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 161197)
Time Spent: 3h 10m (was: 3h)
> Nexmark perf tests fail due to NoClassDefFoundError for Iterables.
> --
>
> Key: BEAM-5875
> URL: https://issues.apache.org/jira/browse/BEAM-5875
> Project: Beam
> Issue Type: Bug
> Components: test-failures
>Reporter: Henning Rohde
>Assignee: Luke Cwik
>Priority: Critical
> Labels: currently-failing
> Time Spent: 3h 10m
> Remaining Estimate: 0h
>
> https://scans.gradle.com/s/vjkiys2xc3age/console-log?task=:beam-sdks-java-nexmark:run
> I see:
> Caused by: java.lang.NoClassDefFoundError:
> org/apache/beam/repackaged/beam_sdks_java_test_utils/com/google/common/collect/Iterables
> at
> org.apache.beam.sdk.testutils.metrics.MetricsReader.checkIfMetricResultIsUnique(MetricsReader.java:128)
> at
> org.apache.beam.sdk.testutils.metrics.MetricsReader.getCounterMetric(MetricsReader.java:65)
> at
> org.apache.beam.sdk.nexmark.NexmarkLauncher.currentPerf(NexmarkLauncher.java:250)
> at
> org.apache.beam.sdk.nexmark.NexmarkLauncher.monitor(NexmarkLauncher.java:435)
> at
> org.apache.beam.sdk.nexmark.NexmarkLauncher.run(NexmarkLauncher.java:1156)
> at org.apache.beam.sdk.nexmark.Main$Run.call(Main.java:108)
> at org.apache.beam.sdk.nexmark.Main$Run.call(Main.java:96)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.ClassNotFoundException:
> org.apache.beam.repackaged.beam_sdks_java_test_utils.com.google.common.collect.Iterables
> at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> ... 13 more
> PRs for the first red run:
> [BEAM-5716] Move nexmark to "testing" directory in java sdk (commit: 0074138)
> (detail / githubweb)
> [BEAM-5716] Move load-tests code to "testing" directory in java sdk (commit:
> 6674c9d) (detail / githubweb)
> [BEAM-5716] Create module for testing utils (commit: 0628951) (detail /
> githubweb)
> [BEAM-5716] Extract MetricReader class, test it and use in Nexmark code
> (commit: 69730fc) (detail / githubweb)
> [BEAM-5355] Use MetricsReader in GroupByKeyLoadTest (commit: 7374eb6) (detail
> / githubweb)
> Łukasz -- would you mind taking a look? Looks like a shading issue.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-5299) Define max global window as a shared value in protos like URN enums.
[ https://issues.apache.org/jira/browse/BEAM-5299?focusedWorklogId=161198&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-161198 ] ASF GitHub Bot logged work on BEAM-5299: Author: ASF GitHub Bot Created on: 31/Oct/18 16:08 Start Date: 31/Oct/18 16:08 Worklog Time Spent: 10m Work Description: lukecwik commented on a change in pull request #6381: [BEAM-5299] Define max timestamp for global window in proto URL: https://github.com/apache/beam/pull/6381#discussion_r229762765 ## File path: sdks/python/apache_beam/transforms/window.py ## @@ -295,19 +294,14 @@ def __lt__(self, other): class GlobalWindow(BoundedWindow): """The default window into which all data is placed (via GlobalWindows).""" _instance = None - # The maximum timestamp for global windows is MAX_TIMESTAMP - 1 day. - # This is due to timers triggering when the watermark passes the trigger - # time, which is only possible for timestamps < MAX_TIMESTAMP. - # See also GlobalWindow in the Java SDK. - _END_OF_GLOBAL_WINDOW = MAX_TIMESTAMP - (24 * 60 * 60) def __new__(cls): if cls._instance is None: cls._instance = super(GlobalWindow, cls).__new__(cls) return cls._instance def __init__(self): -super(GlobalWindow, self).__init__(GlobalWindow._END_OF_GLOBAL_WINDOW) +super(GlobalWindow, self).__init__(GlobalWindow._getTimestampFromProto()) Review comment: Yes your right. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 161198) Time Spent: 8h 10m (was: 8h) > Define max global window as a shared value in protos like URN enums. > > > Key: BEAM-5299 > URL: https://issues.apache.org/jira/browse/BEAM-5299 > Project: Beam > Issue Type: Improvement > Components: beam-model, sdk-go, sdk-java-core, sdk-py-core >Reporter: Luke Cwik >Assignee: Maximilian Michels >Priority: Minor > Labels: portability > Time Spent: 8h 10m > Remaining Estimate: 0h > > Instead of having each language define a max timestamp themselves, define the > max timestamps within proto to be shared across different languages. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5299) Define max global window as a shared value in protos like URN enums.
[ https://issues.apache.org/jira/browse/BEAM-5299?focusedWorklogId=161200&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-161200 ] ASF GitHub Bot logged work on BEAM-5299: Author: ASF GitHub Bot Created on: 31/Oct/18 16:14 Start Date: 31/Oct/18 16:14 Worklog Time Spent: 10m Work Description: lukecwik commented on issue #6899: Revert "[BEAM-5299] Define max timestamp for global window in proto (… URL: https://github.com/apache/beam/pull/6899#issuecomment-434749771 I don't think there is a good way to get this change in without breaking Dataflow temporarily. I believe the Dataflow container image needs to be rebuilt to include this change. Note that this is slowly changing, more and more tests are migrating to use artifacts built from here instead of inside Google. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 161200) Time Spent: 8h 20m (was: 8h 10m) > Define max global window as a shared value in protos like URN enums. > > > Key: BEAM-5299 > URL: https://issues.apache.org/jira/browse/BEAM-5299 > Project: Beam > Issue Type: Improvement > Components: beam-model, sdk-go, sdk-java-core, sdk-py-core >Reporter: Luke Cwik >Assignee: Maximilian Michels >Priority: Minor > Labels: portability > Time Spent: 8h 20m > Remaining Estimate: 0h > > Instead of having each language define a max timestamp themselves, define the > max timestamps within proto to be shared across different languages. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5299) Define max global window as a shared value in protos like URN enums.
[
https://issues.apache.org/jira/browse/BEAM-5299?focusedWorklogId=161201&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-161201
]
ASF GitHub Bot logged work on BEAM-5299:
Author: ASF GitHub Bot
Created on: 31/Oct/18 16:15
Start Date: 31/Oct/18 16:15
Worklog Time Spent: 10m
Work Description: lukecwik commented on a change in pull request #6899:
Revert "[BEAM-5299] Define max timestamp for global window in proto (…
URL: https://github.com/apache/beam/pull/6899#discussion_r229765589
##
File path: sdks/java/core/build.gradle
##
@@ -51,8 +51,6 @@ test {
}
dependencies {
- // Required to load constants from the model, e.g. max timestamp for global
window
- shadow project(path: ":beam-model-pipeline", configuration: "shadow")
Review comment:
I don't believe this to be the culprit.
The issue is that Dataflow has an internal container image that contains
classes that were built from a previous source version. These classes become
incompatible with versions submitted from "HEAD".
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 161201)
Time Spent: 8.5h (was: 8h 20m)
> Define max global window as a shared value in protos like URN enums.
>
>
> Key: BEAM-5299
> URL: https://issues.apache.org/jira/browse/BEAM-5299
> Project: Beam
> Issue Type: Improvement
> Components: beam-model, sdk-go, sdk-java-core, sdk-py-core
>Reporter: Luke Cwik
>Assignee: Maximilian Michels
>Priority: Minor
> Labels: portability
> Time Spent: 8.5h
> Remaining Estimate: 0h
>
> Instead of having each language define a max timestamp themselves, define the
> max timestamps within proto to be shared across different languages.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (BEAM-4124) Support elements larger than 4 MB
[ https://issues.apache.org/jira/browse/BEAM-4124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16670326#comment-16670326 ] Robert Burke commented on BEAM-4124: [~dfbrowne] I was chatting with [~wcn3] and he expressed interest in tackling this. Otherwise I'll get this done next week. > Support elements larger than 4 MB > - > > Key: BEAM-4124 > URL: https://issues.apache.org/jira/browse/BEAM-4124 > Project: Beam > Issue Type: Bug > Components: sdk-go >Reporter: Cody Schroeder >Priority: Major > > The Go SDK harness is limited by a gRPC message size limit of 4 MB. > https://github.com/apache/beam/blob/4a32353/sdks/go/pkg/beam/core/runtime/harness/datamgr.go#L31 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5299) Define max global window as a shared value in protos like URN enums.
[ https://issues.apache.org/jira/browse/BEAM-5299?focusedWorklogId=161203&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-161203 ] ASF GitHub Bot logged work on BEAM-5299: Author: ASF GitHub Bot Created on: 31/Oct/18 16:16 Start Date: 31/Oct/18 16:16 Worklog Time Spent: 10m Work Description: lukecwik edited a comment on issue #6899: Revert "[BEAM-5299] Define max timestamp for global window in proto (… URL: https://github.com/apache/beam/pull/6899#issuecomment-434749771 I believe the Dataflow container image needs to be rebuilt to include this change. We can revert this change for now until I rebuild a new container image. Note that this is slowly changing, more and more tests are migrating to use artifacts built from here instead of inside Google. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 161203) Time Spent: 8h 40m (was: 8.5h) > Define max global window as a shared value in protos like URN enums. > > > Key: BEAM-5299 > URL: https://issues.apache.org/jira/browse/BEAM-5299 > Project: Beam > Issue Type: Improvement > Components: beam-model, sdk-go, sdk-java-core, sdk-py-core >Reporter: Luke Cwik >Assignee: Maximilian Michels >Priority: Minor > Labels: portability > Time Spent: 8h 40m > Remaining Estimate: 0h > > Instead of having each language define a max timestamp themselves, define the > max timestamps within proto to be shared across different languages. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5912) The python dependency check report shows same release dates for different versions of libraries.
[ https://issues.apache.org/jira/browse/BEAM-5912?focusedWorklogId=161202&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-161202 ] ASF GitHub Bot logged work on BEAM-5912: Author: ASF GitHub Bot Created on: 31/Oct/18 16:16 Start Date: 31/Oct/18 16:16 Worklog Time Spent: 10m Work Description: yifanzou commented on issue #6879: [BEAM-5912] fix the python dependency check has the same relase date … URL: https://github.com/apache/beam/pull/6879#issuecomment-434750622 Thanks, Cham. Please merge. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 161202) Time Spent: 2h 20m (was: 2h 10m) > The python dependency check report shows same release dates for different > versions of libraries. > > > Key: BEAM-5912 > URL: https://issues.apache.org/jira/browse/BEAM-5912 > Project: Beam > Issue Type: Bug > Components: dependencies >Reporter: yifan zou >Assignee: yifan zou >Priority: Major > Time Spent: 2h 20m > Remaining Estimate: 0h > > example: > [https://builds.apache.org/job/beam_Dependency_Check/160/artifact/src/build/dependencyUpdates/beam-dependency-check-report.html] > > |*Dependency Name*|*Current Version*|*Latest Version*|*Release Date Of the > Current Used Version*|*Release Date Of The Latest Release*|*JIRA Issue*| > |[google-cloud-pubsub|https://pypi.org/project/google-cloud-pubsub]|0.35.4|0.38.0|2018-06-06|2018-06-06|BEAM-5539| > |[pytz|https://pypi.org/project/pytz]|2018.4|2018.7|2018-04-10|2018-04-10|BEAM-5893| -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5299) Define max global window as a shared value in protos like URN enums.
[ https://issues.apache.org/jira/browse/BEAM-5299?focusedWorklogId=161207&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-161207 ] ASF GitHub Bot logged work on BEAM-5299: Author: ASF GitHub Bot Created on: 31/Oct/18 16:24 Start Date: 31/Oct/18 16:24 Worklog Time Spent: 10m Work Description: lukecwik commented on issue #6899: Revert "[BEAM-5299] Define max timestamp for global window in proto (… URL: https://github.com/apache/beam/pull/6899#issuecomment-434753435 We could migrate the IOITs to build/run tests using Dataflow built from this github repo. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 161207) Time Spent: 8h 50m (was: 8h 40m) > Define max global window as a shared value in protos like URN enums. > > > Key: BEAM-5299 > URL: https://issues.apache.org/jira/browse/BEAM-5299 > Project: Beam > Issue Type: Improvement > Components: beam-model, sdk-go, sdk-java-core, sdk-py-core >Reporter: Luke Cwik >Assignee: Maximilian Michels >Priority: Minor > Labels: portability > Time Spent: 8h 50m > Remaining Estimate: 0h > > Instead of having each language define a max timestamp themselves, define the > max timestamps within proto to be shared across different languages. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5917) Update Flink Runner to 1.5.5
[ https://issues.apache.org/jira/browse/BEAM-5917?focusedWorklogId=161208&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-161208 ] ASF GitHub Bot logged work on BEAM-5917: Author: ASF GitHub Bot Created on: 31/Oct/18 16:25 Start Date: 31/Oct/18 16:25 Worklog Time Spent: 10m Work Description: tweise commented on issue #6885: [BEAM-5917] Update Flink to 1.5.5 URL: https://github.com/apache/beam/pull/6885#issuecomment-434753818 We can safely defer this change until after the Flink release is fixed, at which time it will be just a version number change with no other workarounds. It would also be nice to run something on a Flink 1.5.5 cluster, if you had not already done that. That can be done without any change in Beam (we are using 1.5.4 at Lyft currently). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 161208) Time Spent: 0.5h (was: 20m) > Update Flink Runner to 1.5.5 > > > Key: BEAM-5917 > URL: https://issues.apache.org/jira/browse/BEAM-5917 > Project: Beam > Issue Type: Improvement > Components: runner-flink >Reporter: Maximilian Michels >Assignee: Maximilian Michels >Priority: Minor > Fix For: 2.9.0 > > Time Spent: 0.5h > Remaining Estimate: 0h > > We should be on the latest hotfix version to ensure stability for users: > https://flink.apache.org/news/2018/10/29/release-1.5.5.html -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5299) Define max global window as a shared value in protos like URN enums.
[
https://issues.apache.org/jira/browse/BEAM-5299?focusedWorklogId=161209&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-161209
]
ASF GitHub Bot logged work on BEAM-5299:
Author: ASF GitHub Bot
Created on: 31/Oct/18 16:28
Start Date: 31/Oct/18 16:28
Worklog Time Spent: 10m
Work Description: mxm commented on a change in pull request #6899: Revert
"[BEAM-5299] Define max timestamp for global window in proto (…
URL: https://github.com/apache/beam/pull/6899#discussion_r229770745
##
File path: sdks/java/core/build.gradle
##
@@ -51,8 +51,6 @@ test {
}
dependencies {
- // Required to load constants from the model, e.g. max timestamp for global
window
- shadow project(path: ":beam-model-pipeline", configuration: "shadow")
Review comment:
Makes sense, the model project doesn't contain any of the conflicting
classes.
This should break quite often then, whenever classes are changed in
`sdsks/java/core`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 161209)
Time Spent: 9h (was: 8h 50m)
> Define max global window as a shared value in protos like URN enums.
>
>
> Key: BEAM-5299
> URL: https://issues.apache.org/jira/browse/BEAM-5299
> Project: Beam
> Issue Type: Improvement
> Components: beam-model, sdk-go, sdk-java-core, sdk-py-core
>Reporter: Luke Cwik
>Assignee: Maximilian Michels
>Priority: Minor
> Labels: portability
> Time Spent: 9h
> Remaining Estimate: 0h
>
> Instead of having each language define a max timestamp themselves, define the
> max timestamps within proto to be shared across different languages.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-5875) Nexmark perf tests fail due to NoClassDefFoundError for Iterables.
[
https://issues.apache.org/jira/browse/BEAM-5875?focusedWorklogId=161211&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-161211
]
ASF GitHub Bot logged work on BEAM-5875:
Author: ASF GitHub Bot
Created on: 31/Oct/18 16:35
Start Date: 31/Oct/18 16:35
Worklog Time Spent: 10m
Work Description: lukecwik closed pull request #6894: [BEAM-5875] Fix
scope of dependency ensuring that it is packaged within the jar.
URL: https://github.com/apache/beam/pull/6894
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/sdks/java/testing/test-utils/build.gradle
b/sdks/java/testing/test-utils/build.gradle
index 1d0040073699..4672093f39ac 100644
--- a/sdks/java/testing/test-utils/build.gradle
+++ b/sdks/java/testing/test-utils/build.gradle
@@ -23,7 +23,7 @@ description = "Apache Beam :: SDKs :: Java :: Test Utils"
dependencies {
shadow project(path: ":beam-sdks-java-core", configuration: "shadow")
- shadow library.java.guava
+ compile library.java.guava
shadowTest library.java.junit
shadowTest library.java.mockito_core
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 161211)
Time Spent: 3.5h (was: 3h 20m)
> Nexmark perf tests fail due to NoClassDefFoundError for Iterables.
> --
>
> Key: BEAM-5875
> URL: https://issues.apache.org/jira/browse/BEAM-5875
> Project: Beam
> Issue Type: Bug
> Components: test-failures
>Reporter: Henning Rohde
>Assignee: Luke Cwik
>Priority: Critical
> Labels: currently-failing
> Time Spent: 3.5h
> Remaining Estimate: 0h
>
> https://scans.gradle.com/s/vjkiys2xc3age/console-log?task=:beam-sdks-java-nexmark:run
> I see:
> Caused by: java.lang.NoClassDefFoundError:
> org/apache/beam/repackaged/beam_sdks_java_test_utils/com/google/common/collect/Iterables
> at
> org.apache.beam.sdk.testutils.metrics.MetricsReader.checkIfMetricResultIsUnique(MetricsReader.java:128)
> at
> org.apache.beam.sdk.testutils.metrics.MetricsReader.getCounterMetric(MetricsReader.java:65)
> at
> org.apache.beam.sdk.nexmark.NexmarkLauncher.currentPerf(NexmarkLauncher.java:250)
> at
> org.apache.beam.sdk.nexmark.NexmarkLauncher.monitor(NexmarkLauncher.java:435)
> at
> org.apache.beam.sdk.nexmark.NexmarkLauncher.run(NexmarkLauncher.java:1156)
> at org.apache.beam.sdk.nexmark.Main$Run.call(Main.java:108)
> at org.apache.beam.sdk.nexmark.Main$Run.call(Main.java:96)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.ClassNotFoundException:
> org.apache.beam.repackaged.beam_sdks_java_test_utils.com.google.common.collect.Iterables
> at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> ... 13 more
> PRs for the first red run:
> [BEAM-5716] Move nexmark to "testing" directory in java sdk (commit: 0074138)
> (detail / githubweb)
> [BEAM-5716] Move load-tests code to "testing" directory in java sdk (commit:
> 6674c9d) (detail / githubweb)
> [BEAM-5716] Create module for testing utils (commit: 0628951) (detail /
> githubweb)
> [BEAM-5716] Extract MetricReader class, test it and use in Nexmark code
> (commit: 69730fc) (detail / githubweb)
> [BEAM-5355] Use MetricsReader in GroupByKeyLoadTest (commit: 7374eb6) (detail
> / githubweb)
> Łukasz -- would you mind taking a look? Looks like a shading issue.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-5875) Nexmark perf tests fail due to NoClassDefFoundError for Iterables.
[
https://issues.apache.org/jira/browse/BEAM-5875?focusedWorklogId=161210&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-161210
]
ASF GitHub Bot logged work on BEAM-5875:
Author: ASF GitHub Bot
Created on: 31/Oct/18 16:35
Start Date: 31/Oct/18 16:35
Worklog Time Spent: 10m
Work Description: lukecwik commented on a change in pull request #6894:
[BEAM-5875] Fix scope of dependency ensuring that it is packaged within the jar.
URL: https://github.com/apache/beam/pull/6894#discussion_r229773307
##
File path: sdks/java/testing/test-utils/build.gradle
##
@@ -23,7 +23,7 @@ description = "Apache Beam :: SDKs :: Java :: Test Utils"
dependencies {
shadow project(path: ":beam-sdks-java-core", configuration: "shadow")
- shadow library.java.guava
+ compile library.java.guava
Review comment:
Another part which is important for shading between `compile` and `shadow`
is whether the dependency is listed in the `pom.xml` that is uploaded to Maven
central for the artifact.
`compile`: Not part of `pom.xml`
`shadow`: Part of `pom.xml`
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 161210)
Time Spent: 3h 20m (was: 3h 10m)
> Nexmark perf tests fail due to NoClassDefFoundError for Iterables.
> --
>
> Key: BEAM-5875
> URL: https://issues.apache.org/jira/browse/BEAM-5875
> Project: Beam
> Issue Type: Bug
> Components: test-failures
>Reporter: Henning Rohde
>Assignee: Luke Cwik
>Priority: Critical
> Labels: currently-failing
> Time Spent: 3h 20m
> Remaining Estimate: 0h
>
> https://scans.gradle.com/s/vjkiys2xc3age/console-log?task=:beam-sdks-java-nexmark:run
> I see:
> Caused by: java.lang.NoClassDefFoundError:
> org/apache/beam/repackaged/beam_sdks_java_test_utils/com/google/common/collect/Iterables
> at
> org.apache.beam.sdk.testutils.metrics.MetricsReader.checkIfMetricResultIsUnique(MetricsReader.java:128)
> at
> org.apache.beam.sdk.testutils.metrics.MetricsReader.getCounterMetric(MetricsReader.java:65)
> at
> org.apache.beam.sdk.nexmark.NexmarkLauncher.currentPerf(NexmarkLauncher.java:250)
> at
> org.apache.beam.sdk.nexmark.NexmarkLauncher.monitor(NexmarkLauncher.java:435)
> at
> org.apache.beam.sdk.nexmark.NexmarkLauncher.run(NexmarkLauncher.java:1156)
> at org.apache.beam.sdk.nexmark.Main$Run.call(Main.java:108)
> at org.apache.beam.sdk.nexmark.Main$Run.call(Main.java:96)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.ClassNotFoundException:
> org.apache.beam.repackaged.beam_sdks_java_test_utils.com.google.common.collect.Iterables
> at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> ... 13 more
> PRs for the first red run:
> [BEAM-5716] Move nexmark to "testing" directory in java sdk (commit: 0074138)
> (detail / githubweb)
> [BEAM-5716] Move load-tests code to "testing" directory in java sdk (commit:
> 6674c9d) (detail / githubweb)
> [BEAM-5716] Create module for testing utils (commit: 0628951) (detail /
> githubweb)
> [BEAM-5716] Extract MetricReader class, test it and use in Nexmark code
> (commit: 69730fc) (detail / githubweb)
> [BEAM-5355] Use MetricsReader in GroupByKeyLoadTest (commit: 7374eb6) (detail
> / githubweb)
> Łukasz -- would you mind taking a look? Looks like a shading issue.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-5299) Define max global window as a shared value in protos like URN enums.
[ https://issues.apache.org/jira/browse/BEAM-5299?focusedWorklogId=161212&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-161212 ] ASF GitHub Bot logged work on BEAM-5299: Author: ASF GitHub Bot Created on: 31/Oct/18 16:41 Start Date: 31/Oct/18 16:41 Worklog Time Spent: 10m Work Description: lukecwik commented on issue #6899: Revert "[BEAM-5299] Define max timestamp for global window in proto (… URL: https://github.com/apache/beam/pull/6899#issuecomment-43475 Lets revert for now, we have two paths forward: 1) (BEST) Migrate Dataflow ITs to use Dataflow jar built at head 2) Update Dataflow worker image This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 161212) Time Spent: 9h 10m (was: 9h) > Define max global window as a shared value in protos like URN enums. > > > Key: BEAM-5299 > URL: https://issues.apache.org/jira/browse/BEAM-5299 > Project: Beam > Issue Type: Improvement > Components: beam-model, sdk-go, sdk-java-core, sdk-py-core >Reporter: Luke Cwik >Assignee: Maximilian Michels >Priority: Minor > Labels: portability > Time Spent: 9h 10m > Remaining Estimate: 0h > > Instead of having each language define a max timestamp themselves, define the > max timestamps within proto to be shared across different languages. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5299) Define max global window as a shared value in protos like URN enums.
[ https://issues.apache.org/jira/browse/BEAM-5299?focusedWorklogId=161213&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-161213 ] ASF GitHub Bot logged work on BEAM-5299: Author: ASF GitHub Bot Created on: 31/Oct/18 16:42 Start Date: 31/Oct/18 16:42 Worklog Time Spent: 10m Work Description: lgajowy commented on issue #6899: Revert "[BEAM-5299] Define max timestamp for global window in proto (… URL: https://github.com/apache/beam/pull/6899#issuecomment-434760352 I'm not sure 100% but it seems that nexmark_Dataflow jobs are affected too: https://builds.apache.org/view/A-D/view/Beam/job/beam_PostCommit_Java_Nexmark_Dataflow/873/consoleText So: to not require rebuilding the dataflow container image we need to migrate both nexmark and IOITs. I don't have bandwidth to do this this week (taking 4 days off) so if no one else can do this - I suggest reverting. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 161213) Time Spent: 9h 20m (was: 9h 10m) > Define max global window as a shared value in protos like URN enums. > > > Key: BEAM-5299 > URL: https://issues.apache.org/jira/browse/BEAM-5299 > Project: Beam > Issue Type: Improvement > Components: beam-model, sdk-go, sdk-java-core, sdk-py-core >Reporter: Luke Cwik >Assignee: Maximilian Michels >Priority: Minor > Labels: portability > Time Spent: 9h 20m > Remaining Estimate: 0h > > Instead of having each language define a max timestamp themselves, define the > max timestamps within proto to be shared across different languages. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5299) Define max global window as a shared value in protos like URN enums.
[ https://issues.apache.org/jira/browse/BEAM-5299?focusedWorklogId=161215&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-161215 ] ASF GitHub Bot logged work on BEAM-5299: Author: ASF GitHub Bot Created on: 31/Oct/18 16:43 Start Date: 31/Oct/18 16:43 Worklog Time Spent: 10m Work Description: lgajowy edited a comment on issue #6899: Revert "[BEAM-5299] Define max timestamp for global window in proto (… URL: https://github.com/apache/beam/pull/6899#issuecomment-434760352 I'm not sure 100% but it seems that nexmark_Dataflow jobs are affected too: https://builds.apache.org/view/A-D/view/Beam/job/beam_PostCommit_Java_Nexmark_Dataflow/873/consoleText So to not require rebuilding the dataflow container image we need to migrate both nexmark and IOITs. I don't have bandwidth to do this this week (taking 4 days off) so if no one else can do this - I suggest reverting. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 161215) Time Spent: 9.5h (was: 9h 20m) > Define max global window as a shared value in protos like URN enums. > > > Key: BEAM-5299 > URL: https://issues.apache.org/jira/browse/BEAM-5299 > Project: Beam > Issue Type: Improvement > Components: beam-model, sdk-go, sdk-java-core, sdk-py-core >Reporter: Luke Cwik >Assignee: Maximilian Michels >Priority: Minor > Labels: portability > Time Spent: 9.5h > Remaining Estimate: 0h > > Instead of having each language define a max timestamp themselves, define the > max timestamps within proto to be shared across different languages. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5875) Nexmark perf tests fail due to NoClassDefFoundError for Iterables.
[
https://issues.apache.org/jira/browse/BEAM-5875?focusedWorklogId=161216&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-161216
]
ASF GitHub Bot logged work on BEAM-5875:
Author: ASF GitHub Bot
Created on: 31/Oct/18 16:45
Start Date: 31/Oct/18 16:45
Worklog Time Spent: 10m
Work Description: lgajowy commented on a change in pull request #6894:
[BEAM-5875] Fix scope of dependency ensuring that it is packaged within the jar.
URL: https://github.com/apache/beam/pull/6894#discussion_r229777512
##
File path: sdks/java/testing/test-utils/build.gradle
##
@@ -23,7 +23,7 @@ description = "Apache Beam :: SDKs :: Java :: Test Utils"
dependencies {
shadow project(path: ":beam-sdks-java-core", configuration: "shadow")
- shadow library.java.guava
+ compile library.java.guava
Review comment:
Thanks for the clarification.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 161216)
Time Spent: 3h 40m (was: 3.5h)
> Nexmark perf tests fail due to NoClassDefFoundError for Iterables.
> --
>
> Key: BEAM-5875
> URL: https://issues.apache.org/jira/browse/BEAM-5875
> Project: Beam
> Issue Type: Bug
> Components: test-failures
>Reporter: Henning Rohde
>Assignee: Luke Cwik
>Priority: Critical
> Labels: currently-failing
> Time Spent: 3h 40m
> Remaining Estimate: 0h
>
> https://scans.gradle.com/s/vjkiys2xc3age/console-log?task=:beam-sdks-java-nexmark:run
> I see:
> Caused by: java.lang.NoClassDefFoundError:
> org/apache/beam/repackaged/beam_sdks_java_test_utils/com/google/common/collect/Iterables
> at
> org.apache.beam.sdk.testutils.metrics.MetricsReader.checkIfMetricResultIsUnique(MetricsReader.java:128)
> at
> org.apache.beam.sdk.testutils.metrics.MetricsReader.getCounterMetric(MetricsReader.java:65)
> at
> org.apache.beam.sdk.nexmark.NexmarkLauncher.currentPerf(NexmarkLauncher.java:250)
> at
> org.apache.beam.sdk.nexmark.NexmarkLauncher.monitor(NexmarkLauncher.java:435)
> at
> org.apache.beam.sdk.nexmark.NexmarkLauncher.run(NexmarkLauncher.java:1156)
> at org.apache.beam.sdk.nexmark.Main$Run.call(Main.java:108)
> at org.apache.beam.sdk.nexmark.Main$Run.call(Main.java:96)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.ClassNotFoundException:
> org.apache.beam.repackaged.beam_sdks_java_test_utils.com.google.common.collect.Iterables
> at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> ... 13 more
> PRs for the first red run:
> [BEAM-5716] Move nexmark to "testing" directory in java sdk (commit: 0074138)
> (detail / githubweb)
> [BEAM-5716] Move load-tests code to "testing" directory in java sdk (commit:
> 6674c9d) (detail / githubweb)
> [BEAM-5716] Create module for testing utils (commit: 0628951) (detail /
> githubweb)
> [BEAM-5716] Extract MetricReader class, test it and use in Nexmark code
> (commit: 69730fc) (detail / githubweb)
> [BEAM-5355] Use MetricsReader in GroupByKeyLoadTest (commit: 7374eb6) (detail
> / githubweb)
> Łukasz -- would you mind taking a look? Looks like a shading issue.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-5299) Define max global window as a shared value in protos like URN enums.
[ https://issues.apache.org/jira/browse/BEAM-5299?focusedWorklogId=161217&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-161217 ] ASF GitHub Bot logged work on BEAM-5299: Author: ASF GitHub Bot Created on: 31/Oct/18 16:46 Start Date: 31/Oct/18 16:46 Worklog Time Spent: 10m Work Description: lukecwik commented on issue #6899: Revert "[BEAM-5299] Define max timestamp for global window in proto (… URL: https://github.com/apache/beam/pull/6899#issuecomment-434761658 @lgajowy Yes, I agree. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 161217) Time Spent: 9h 40m (was: 9.5h) > Define max global window as a shared value in protos like URN enums. > > > Key: BEAM-5299 > URL: https://issues.apache.org/jira/browse/BEAM-5299 > Project: Beam > Issue Type: Improvement > Components: beam-model, sdk-go, sdk-java-core, sdk-py-core >Reporter: Luke Cwik >Assignee: Maximilian Michels >Priority: Minor > Labels: portability > Time Spent: 9h 40m > Remaining Estimate: 0h > > Instead of having each language define a max timestamp themselves, define the > max timestamps within proto to be shared across different languages. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5299) Define max global window as a shared value in protos like URN enums.
[
https://issues.apache.org/jira/browse/BEAM-5299?focusedWorklogId=161220&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-161220
]
ASF GitHub Bot logged work on BEAM-5299:
Author: ASF GitHub Bot
Created on: 31/Oct/18 16:50
Start Date: 31/Oct/18 16:50
Worklog Time Spent: 10m
Work Description: lukecwik closed pull request #6899: Revert "[BEAM-5299]
Define max timestamp for global window in proto (…
URL: https://github.com/apache/beam/pull/6899
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/model/pipeline/src/main/proto/beam_runner_api.proto
b/model/pipeline/src/main/proto/beam_runner_api.proto
index 4b537b0c4f6b..194797f166c0 100644
--- a/model/pipeline/src/main/proto/beam_runner_api.proto
+++ b/model/pipeline/src/main/proto/beam_runner_api.proto
@@ -32,22 +32,6 @@ option java_outer_classname = "RunnerApi";
import "google/protobuf/any.proto";
import "google/protobuf/descriptor.proto";
-message BeamConstants {
- enum Constants {
-// All timestamps in milliseconds since Jan 1, 1970.
-MIN_TIMESTAMP_MILLIS = 0 [(beam_constant) = "-9223372036854775"];
-MAX_TIMESTAMP_MILLIS = 1 [(beam_constant) = "9223372036854775"];
-// The maximum timestamp for the global window.
-// Triggers use maxTimestamp to set timers' timestamp. Timers fires when
-// the watermark passes their timestamps. So, the timestamp needs to be
-// smaller than the MAX_TIMESTAMP_MILLIS.
-// One standard day is subtracted from MAX_TIMESTAMP_MILLIS to make sure
-// the maxTimestamp is smaller than MAX_TIMESTAMP_MILLIS even after
rounding up
-// to seconds or minutes. See also GlobalWindow in the Java SDK.
-GLOBAL_WINDOW_MAX_TIMESTAMP_MILLIS = 2 [(beam_constant) =
"9223371950454775"];
- }
-}
-
// A set of mappings from id to message. This is included as an optional field
// on any proto message that may contain references needing resolution.
message Components {
@@ -1081,8 +1065,6 @@ extend google.protobuf.EnumValueOptions {
// }
// }
string beam_urn = 185324356;
- // A value to store other constants
- string beam_constant = 185324357;
}
// A URN along with a parameter object whose schema is determined by the
diff --git a/sdks/go/pkg/beam/core/graph/mtime/time.go
b/sdks/go/pkg/beam/core/graph/mtime/time.go
index 7ef82fa9b30a..627deb9c0756 100644
--- a/sdks/go/pkg/beam/core/graph/mtime/time.go
+++ b/sdks/go/pkg/beam/core/graph/mtime/time.go
@@ -37,7 +37,6 @@ const (
// EndOfGlobalWindowTime is the timestamp at the end of the global
window. It
// is a day before the max timestamp.
- // TODO Use GLOBAL_WINDOW_MAX_TIMESTAMP_MILLIS from the Runner API
constants
EndOfGlobalWindowTime = MaxTimestamp - 24*60*60*1000
// ZeroTimestamp is the default zero value time. It corresponds to the
unix epoch.
diff --git a/sdks/java/core/build.gradle b/sdks/java/core/build.gradle
index 61c4d66956bc..2b976a285efc 100644
--- a/sdks/java/core/build.gradle
+++ b/sdks/java/core/build.gradle
@@ -51,8 +51,6 @@ test {
}
dependencies {
- // Required to load constants from the model, e.g. max timestamp for global
window
- shadow project(path: ":beam-model-pipeline", configuration: "shadow")
compile library.java.guava
compile library.java.protobuf_java
compile library.java.byte_buddy
diff --git
a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/BoundedWindow.java
b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/BoundedWindow.java
index 1d6b13c49dda..eb9ae7fee29d 100644
---
a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/BoundedWindow.java
+++
b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/BoundedWindow.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.sdk.transforms.windowing;
-import org.apache.beam.model.pipeline.v1.RunnerApi;
+import java.util.concurrent.TimeUnit;
import org.joda.time.Instant;
/**
@@ -46,7 +46,7 @@
* microseconds-since-epoch can be safely represented with a {@code long}.
*/
public static final Instant TIMESTAMP_MIN_VALUE =
-
extractTimestampFromProto(RunnerApi.BeamConstants.Constants.MIN_TIMESTAMP_MILLIS);
+ new Instant(TimeUnit.MICROSECONDS.toMillis(Long.MIN_VALUE));
/**
* The maximum value for any Beam timestamp. Often referred to as
"+infinity".
@@ -55,7 +55,7 @@
* microseconds-since-epoch can be safely represented with a {@code long}.
*/
public static final Instant TIMESTAMP_MAX_VALUE =
-
extractTimestampFromProto(RunnerApi.BeamConstants.Constants.MAX_TIMESTAMP_MILLIS);
+ new Instant(TimeUnit.MICROS
[jira] [Work logged] (BEAM-5299) Define max global window as a shared value in protos like URN enums.
[ https://issues.apache.org/jira/browse/BEAM-5299?focusedWorklogId=161222&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-161222 ] ASF GitHub Bot logged work on BEAM-5299: Author: ASF GitHub Bot Created on: 31/Oct/18 16:52 Start Date: 31/Oct/18 16:52 Worklog Time Spent: 10m Work Description: mxm commented on issue #6899: Revert "[BEAM-5299] Define max timestamp for global window in proto (… URL: https://github.com/apache/beam/pull/6899#issuecomment-434763911 Let's trigger rebuilding of the image if it is not much work. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 161222) Time Spent: 10h (was: 9h 50m) > Define max global window as a shared value in protos like URN enums. > > > Key: BEAM-5299 > URL: https://issues.apache.org/jira/browse/BEAM-5299 > Project: Beam > Issue Type: Improvement > Components: beam-model, sdk-go, sdk-java-core, sdk-py-core >Reporter: Luke Cwik >Assignee: Maximilian Michels >Priority: Minor > Labels: portability > Time Spent: 10h > Remaining Estimate: 0h > > Instead of having each language define a max timestamp themselves, define the > max timestamps within proto to be shared across different languages. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5299) Define max global window as a shared value in protos like URN enums.
[ https://issues.apache.org/jira/browse/BEAM-5299?focusedWorklogId=161227&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-161227 ] ASF GitHub Bot logged work on BEAM-5299: Author: ASF GitHub Bot Created on: 31/Oct/18 17:01 Start Date: 31/Oct/18 17:01 Worklog Time Spent: 10m Work Description: lukecwik commented on issue #6899: Revert "[BEAM-5299] Define max timestamp for global window in proto (… URL: https://github.com/apache/beam/pull/6899#issuecomment-434767498 Its a good chunk of work to rebuild the image. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 161227) Time Spent: 10h 10m (was: 10h) > Define max global window as a shared value in protos like URN enums. > > > Key: BEAM-5299 > URL: https://issues.apache.org/jira/browse/BEAM-5299 > Project: Beam > Issue Type: Improvement > Components: beam-model, sdk-go, sdk-java-core, sdk-py-core >Reporter: Luke Cwik >Assignee: Maximilian Michels >Priority: Minor > Labels: portability > Time Spent: 10h 10m > Remaining Estimate: 0h > > Instead of having each language define a max timestamp themselves, define the > max timestamps within proto to be shared across different languages. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work started] (BEAM-5875) Nexmark perf tests fail due to NoClassDefFoundError for Iterables.
[ https://issues.apache.org/jira/browse/BEAM-5875?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on BEAM-5875 started by Luke Cwik. --- > Nexmark perf tests fail due to NoClassDefFoundError for Iterables. > -- > > Key: BEAM-5875 > URL: https://issues.apache.org/jira/browse/BEAM-5875 > Project: Beam > Issue Type: Bug > Components: test-failures >Reporter: Henning Rohde >Assignee: Luke Cwik >Priority: Critical > Labels: currently-failing > Fix For: Not applicable > > Time Spent: 3h 40m > Remaining Estimate: 0h > > https://scans.gradle.com/s/vjkiys2xc3age/console-log?task=:beam-sdks-java-nexmark:run > I see: > Caused by: java.lang.NoClassDefFoundError: > org/apache/beam/repackaged/beam_sdks_java_test_utils/com/google/common/collect/Iterables > at > org.apache.beam.sdk.testutils.metrics.MetricsReader.checkIfMetricResultIsUnique(MetricsReader.java:128) > at > org.apache.beam.sdk.testutils.metrics.MetricsReader.getCounterMetric(MetricsReader.java:65) > at > org.apache.beam.sdk.nexmark.NexmarkLauncher.currentPerf(NexmarkLauncher.java:250) > at > org.apache.beam.sdk.nexmark.NexmarkLauncher.monitor(NexmarkLauncher.java:435) > at > org.apache.beam.sdk.nexmark.NexmarkLauncher.run(NexmarkLauncher.java:1156) > at org.apache.beam.sdk.nexmark.Main$Run.call(Main.java:108) > at org.apache.beam.sdk.nexmark.Main$Run.call(Main.java:96) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > Caused by: java.lang.ClassNotFoundException: > org.apache.beam.repackaged.beam_sdks_java_test_utils.com.google.common.collect.Iterables > at java.net.URLClassLoader.findClass(URLClassLoader.java:381) > at java.lang.ClassLoader.loadClass(ClassLoader.java:424) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349) > at java.lang.ClassLoader.loadClass(ClassLoader.java:357) > ... 13 more > PRs for the first red run: > [BEAM-5716] Move nexmark to "testing" directory in java sdk (commit: 0074138) > (detail / githubweb) > [BEAM-5716] Move load-tests code to "testing" directory in java sdk (commit: > 6674c9d) (detail / githubweb) > [BEAM-5716] Create module for testing utils (commit: 0628951) (detail / > githubweb) > [BEAM-5716] Extract MetricReader class, test it and use in Nexmark code > (commit: 69730fc) (detail / githubweb) > [BEAM-5355] Use MetricsReader in GroupByKeyLoadTest (commit: 7374eb6) (detail > / githubweb) > Łukasz -- would you mind taking a look? Looks like a shading issue. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (BEAM-5875) Nexmark perf tests fail due to NoClassDefFoundError for Iterables.
[ https://issues.apache.org/jira/browse/BEAM-5875?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luke Cwik resolved BEAM-5875. - Resolution: Fixed Fix Version/s: Not applicable Ran smoke test using direct runner and it didn't crash: ./gradlew :beam-sdks-java-nexmark:run -Pnexmark.runner=":beam-runners-direct-java" -Pnexmark.args=" --runner=DirectRunner --streaming=false --suite=SMOKE --manageResources=false --monitorJobs=true --enforceEncodability=true --enforceImmutability=true" > Nexmark perf tests fail due to NoClassDefFoundError for Iterables. > -- > > Key: BEAM-5875 > URL: https://issues.apache.org/jira/browse/BEAM-5875 > Project: Beam > Issue Type: Bug > Components: test-failures >Reporter: Henning Rohde >Assignee: Luke Cwik >Priority: Critical > Labels: currently-failing > Fix For: Not applicable > > Time Spent: 3h 40m > Remaining Estimate: 0h > > https://scans.gradle.com/s/vjkiys2xc3age/console-log?task=:beam-sdks-java-nexmark:run > I see: > Caused by: java.lang.NoClassDefFoundError: > org/apache/beam/repackaged/beam_sdks_java_test_utils/com/google/common/collect/Iterables > at > org.apache.beam.sdk.testutils.metrics.MetricsReader.checkIfMetricResultIsUnique(MetricsReader.java:128) > at > org.apache.beam.sdk.testutils.metrics.MetricsReader.getCounterMetric(MetricsReader.java:65) > at > org.apache.beam.sdk.nexmark.NexmarkLauncher.currentPerf(NexmarkLauncher.java:250) > at > org.apache.beam.sdk.nexmark.NexmarkLauncher.monitor(NexmarkLauncher.java:435) > at > org.apache.beam.sdk.nexmark.NexmarkLauncher.run(NexmarkLauncher.java:1156) > at org.apache.beam.sdk.nexmark.Main$Run.call(Main.java:108) > at org.apache.beam.sdk.nexmark.Main$Run.call(Main.java:96) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > Caused by: java.lang.ClassNotFoundException: > org.apache.beam.repackaged.beam_sdks_java_test_utils.com.google.common.collect.Iterables > at java.net.URLClassLoader.findClass(URLClassLoader.java:381) > at java.lang.ClassLoader.loadClass(ClassLoader.java:424) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349) > at java.lang.ClassLoader.loadClass(ClassLoader.java:357) > ... 13 more > PRs for the first red run: > [BEAM-5716] Move nexmark to "testing" directory in java sdk (commit: 0074138) > (detail / githubweb) > [BEAM-5716] Move load-tests code to "testing" directory in java sdk (commit: > 6674c9d) (detail / githubweb) > [BEAM-5716] Create module for testing utils (commit: 0628951) (detail / > githubweb) > [BEAM-5716] Extract MetricReader class, test it and use in Nexmark code > (commit: 69730fc) (detail / githubweb) > [BEAM-5355] Use MetricsReader in GroupByKeyLoadTest (commit: 7374eb6) (detail > / githubweb) > Łukasz -- would you mind taking a look? Looks like a shading issue. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5464) Portable beam hangs while running TFX preprocessing step on a distributed cluster
[
https://issues.apache.org/jira/browse/BEAM-5464?focusedWorklogId=161228&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-161228
]
ASF GitHub Bot logged work on BEAM-5464:
Author: ASF GitHub Bot
Created on: 31/Oct/18 17:08
Start Date: 31/Oct/18 17:08
Worklog Time Spent: 10m
Work Description: mxm commented on a change in pull request #6897:
[BEAM-5464] Use BATCH_FORCED as the default ExecutionMode for batch pipeline
URL: https://github.com/apache/beam/pull/6897#discussion_r229786520
##
File path:
runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkPipelineOptions.java
##
@@ -187,4 +188,12 @@
Long getLatencyTrackingInterval();
void setLatencyTrackingInterval(Long interval);
+
+ @Description(
+ "Flink mode for data exchange for batch pipeline. "
+ + "Reference {@link org.apache.flink.api.common.ExecutionMode}")
+ @Default.Enum("BATCH_FORCED")
Review comment:
Also ok for me to add a link to a JIRA issue to further investigate this
issue.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 161228)
Time Spent: 50m (was: 40m)
> Portable beam hangs while running TFX preprocessing step on a distributed
> cluster
> -
>
> Key: BEAM-5464
> URL: https://issues.apache.org/jira/browse/BEAM-5464
> Project: Beam
> Issue Type: Bug
> Components: java-fn-execution
>Reporter: Axel Magnuson
>Assignee: Ankur Goenka
>Priority: Major
> Time Spent: 50m
> Remaining Estimate: 0h
>
> Recently I went through the exercise of running the TFX taxi example on a
> dataproc cluster. However it would always hang indefinitely. The flink UI
> indicated that the job was halfway done. However I could not see any clear
> errors in the job driver logs, the job service logs, or the Flink logs. The
> root cause is still a mystery to me.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (BEAM-5928) ConcurrentModificationException from RowCoderGenerator lazy caching
[
https://issues.apache.org/jira/browse/BEAM-5928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16670401#comment-16670401
]
Reuven Lax commented on BEAM-5928:
--
I believe the correct fix here is probably to change to ConcurrentHashMap. So
replace
private static Map> generatedCoders = Maps.newHashMap();
with
private static Map> generatedCoders = Maps.newConcurrentMap();
> ConcurrentModificationException from RowCoderGenerator lazy caching
> ---
>
> Key: BEAM-5928
> URL: https://issues.apache.org/jira/browse/BEAM-5928
> Project: Beam
> Issue Type: Bug
> Components: sdk-java-core
>Reporter: Benson Tucker
>Assignee: Reuven Lax
>Priority: Major
>
> h3. Summary:
> RowCoderGenerator caches a delegate Coder once encode or decode is
> exercised, but there's not an API for caching this delegate eagerly.
> h3. Use Case:
> When creating several PCollections to perform distinct reads with the same
> schema, you might create one RowCoder.of(schema) before creating the list of
> PCollections / PCollectionsList. However, once the pipeline begins and rows
> arrive for encoding, these pipelines will simultaneously try to cache a
> delegate coder for the row's schema.
> h3. Workaround:
> You can force the eager caching of the code by exercising encode in the main
> application before creating PCollections using the RowCoder:
> {code:java}
> try {
> myRowCoder.encode(null, null);
> } catch (IOException | NullPointerException e) {
> // do nothing
> }
> {code}
> h3. Context:
> I've only encountered this during development with the direct runner.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-5852) Function extension in BeamSQL
[ https://issues.apache.org/jira/browse/BEAM-5852?focusedWorklogId=161230&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-161230 ] ASF GitHub Bot logged work on BEAM-5852: Author: ASF GitHub Bot Created on: 31/Oct/18 17:15 Start Date: 31/Oct/18 17:15 Worklog Time Spent: 10m Work Description: amaliujia commented on issue #6898: [BEAM-5852] BeamSQL functions URL: https://github.com/apache/beam/pull/6898#issuecomment-434772304 run java precommit This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 161230) Time Spent: 0.5h (was: 20m) > Function extension in BeamSQL > - > > Key: BEAM-5852 > URL: https://issues.apache.org/jira/browse/BEAM-5852 > Project: Beam > Issue Type: New Feature > Components: dsl-sql >Reporter: Rui Wang >Assignee: Rui Wang >Priority: Major > Time Spent: 0.5h > Remaining Estimate: 0h > > We could add more functions to BeamSQL (as UDFs) to provide rich > functionalities than standard/Calcite functions. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5464) Portable beam hangs while running TFX preprocessing step on a distributed cluster
[
https://issues.apache.org/jira/browse/BEAM-5464?focusedWorklogId=161235&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-161235
]
ASF GitHub Bot logged work on BEAM-5464:
Author: ASF GitHub Bot
Created on: 31/Oct/18 17:20
Start Date: 31/Oct/18 17:20
Worklog Time Spent: 10m
Work Description: angoenka commented on a change in pull request #6897:
[BEAM-5464] Use BATCH_FORCED as the default ExecutionMode for batch pipeline
URL: https://github.com/apache/beam/pull/6897#discussion_r229790932
##
File path:
runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkPipelineOptions.java
##
@@ -187,4 +188,12 @@
Long getLatencyTrackingInterval();
void setLatencyTrackingInterval(Long interval);
+
+ @Description(
+ "Flink mode for data exchange for batch pipeline. "
+ + "Reference {@link org.apache.flink.api.common.ExecutionMode}")
+ @Default.Enum("BATCH_FORCED")
+ ExecutionMode getExecutionModeForBatch();
Review comment:
Sure
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 161235)
Time Spent: 1h 10m (was: 1h)
> Portable beam hangs while running TFX preprocessing step on a distributed
> cluster
> -
>
> Key: BEAM-5464
> URL: https://issues.apache.org/jira/browse/BEAM-5464
> Project: Beam
> Issue Type: Bug
> Components: java-fn-execution
>Reporter: Axel Magnuson
>Assignee: Ankur Goenka
>Priority: Major
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> Recently I went through the exercise of running the TFX taxi example on a
> dataproc cluster. However it would always hang indefinitely. The flink UI
> indicated that the job was halfway done. However I could not see any clear
> errors in the job driver logs, the job service logs, or the Flink logs. The
> root cause is still a mystery to me.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-5464) Portable beam hangs while running TFX preprocessing step on a distributed cluster
[
https://issues.apache.org/jira/browse/BEAM-5464?focusedWorklogId=161234&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-161234
]
ASF GitHub Bot logged work on BEAM-5464:
Author: ASF GitHub Bot
Created on: 31/Oct/18 17:20
Start Date: 31/Oct/18 17:20
Worklog Time Spent: 10m
Work Description: angoenka commented on a change in pull request #6897:
[BEAM-5464] Use BATCH_FORCED as the default ExecutionMode for batch pipeline
URL: https://github.com/apache/beam/pull/6897#discussion_r229790869
##
File path:
runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkPipelineOptions.java
##
@@ -187,4 +188,12 @@
Long getLatencyTrackingInterval();
void setLatencyTrackingInterval(Long interval);
+
+ @Description(
+ "Flink mode for data exchange for batch pipeline. "
+ + "Reference {@link org.apache.flink.api.common.ExecutionMode}")
+ @Default.Enum("BATCH_FORCED")
Review comment:
Sounds good, I will update the default.
Flink Jira for reference https://issues.apache.org/jira/browse/FLINK-10672
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 161234)
Time Spent: 1h (was: 50m)
> Portable beam hangs while running TFX preprocessing step on a distributed
> cluster
> -
>
> Key: BEAM-5464
> URL: https://issues.apache.org/jira/browse/BEAM-5464
> Project: Beam
> Issue Type: Bug
> Components: java-fn-execution
>Reporter: Axel Magnuson
>Assignee: Ankur Goenka
>Priority: Major
> Time Spent: 1h
> Remaining Estimate: 0h
>
> Recently I went through the exercise of running the TFX taxi example on a
> dataproc cluster. However it would always hang indefinitely. The flink UI
> indicated that the job was halfway done. However I could not see any clear
> errors in the job driver logs, the job service logs, or the Flink logs. The
> root cause is still a mystery to me.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-5058) Python precommits should run E2E tests
[
https://issues.apache.org/jira/browse/BEAM-5058?focusedWorklogId=161240&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-161240
]
ASF GitHub Bot logged work on BEAM-5058:
Author: ASF GitHub Bot
Created on: 31/Oct/18 17:26
Start Date: 31/Oct/18 17:26
Worklog Time Spent: 10m
Work Description: markflyhigh commented on issue #6707: [BEAM-5058]
Parallelize Python Precommit and add basic ITs
URL: https://github.com/apache/beam/pull/6707#issuecomment-434776546
One thing to notice is that with above change, calling `setupVirtualenv` in
different sub-projects will install virtualenv in their own build directory.
For example, tasks in `:beam-sdks-python-precommit-dataflow` will setup
virtualenv in `sdks/python/precommit/dataflow/build/`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 161240)
Time Spent: 2h 40m (was: 2.5h)
> Python precommits should run E2E tests
> --
>
> Key: BEAM-5058
> URL: https://issues.apache.org/jira/browse/BEAM-5058
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core, testing
>Reporter: Udi Meiri
>Assignee: Mark Liu
>Priority: Major
> Time Spent: 2h 40m
> Remaining Estimate: 0h
>
> According to [https://beam.apache.org/contribute/testing/] (which I'm working
> on), end-to-end tests should be run in precommit on each combination of
> \{batch, streaming}x\{SDK language}x\{supported runner}.
> At least 2 tests need to be added to Python's precommit: wordcount and
> wordcount_streaming on Dataflow, and possibly on other supported runners
> (direct runner and new runners plz).
> These tests should be configured to run from a Gradle sub-project, so that
> they're run in parallel to the unit tests.
> Example that parallelizes Java precommit integration tests:
> [https://github.com/apache/beam/pull/5731]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (BEAM-5724) Beam creates too many sdk_worker processes with --sdk-worker-parallelism=stage
[ https://issues.apache.org/jira/browse/BEAM-5724?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Micah Wylde resolved BEAM-5724. --- Resolution: Fixed Fix Version/s: 2.9.0 > Beam creates too many sdk_worker processes with --sdk-worker-parallelism=stage > -- > > Key: BEAM-5724 > URL: https://issues.apache.org/jira/browse/BEAM-5724 > Project: Beam > Issue Type: Improvement > Components: runner-flink >Reporter: Micah Wylde >Assignee: Micah Wylde >Priority: Major > Labels: portability-flink > Fix For: 2.9.0 > > Time Spent: 4h 20m > Remaining Estimate: 0h > > In the flink portable runner, we currently support two options for sdk worker > parallelism (how many python worker processes we run). The default is one per > taskmanager, and with --sdk-worker-parallelism=stage you get one per stage. > However, for complex pipelines with many beam operators that get fused into a > single flink task this can produce hundreds of worker processes per TM. > Flink uses the notion of task slots to limit resource utilization on a box; I > think that beam should try to respect those limits as well. I think ideally > we'd produce a single python worker per task slot/flink operator chain. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5922) Common base class for FlatMapElements, MapElements, and Filter
[
https://issues.apache.org/jira/browse/BEAM-5922?focusedWorklogId=161241&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-161241
]
ASF GitHub Bot logged work on BEAM-5922:
Author: ASF GitHub Bot
Created on: 31/Oct/18 17:31
Start Date: 31/Oct/18 17:31
Worklog Time Spent: 10m
Work Description: jklukas commented on a change in pull request #6892:
[BEAM-5922] Refactor Map, FlatMap, and Filter on MapperBase
URL: https://github.com/apache/beam/pull/6892#discussion_r229795011
##
File path:
runners/spark/src/test/java/org/apache/beam/runners/spark/SparkRunnerDebuggerTest.java
##
@@ -81,14 +81,14 @@ public void debugBatchPipeline() {
"sparkContext.parallelize(Arrays.asList(...))\n"
+ "_.mapPartitions("
+ "new
org.apache.beam.runners.spark.examples.WordCount$ExtractWordsFn())\n"
-+ "_.mapPartitions(new
org.apache.beam.sdk.transforms.Contextful())\n"
++ "_.mapPartitions(new
org.apache.beam.sdk.transforms.MapperBase$1())\n"
Review comment:
It's not clear to me why this PR makes this show up as `MapperBase$1` rather
than `Contextful`. Is this potentially pointing to a scope leak causing more to
get serialized than necessary?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 161241)
Time Spent: 1h (was: 50m)
> Common base class for FlatMapElements, MapElements, and Filter
> --
>
> Key: BEAM-5922
> URL: https://issues.apache.org/jira/browse/BEAM-5922
> Project: Beam
> Issue Type: Improvement
> Components: sdk-java-core
>Reporter: Jeff Klukas
>Assignee: Jeff Klukas
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 1h
> Remaining Estimate: 0h
>
> By having FlatMapElements, MapElements, and Filter all inherit from a common
> MapperBase, we can reduce code duplication between these three classes for
> better consistency in future changes. There should be no change in
> functionality or public interface with this change.
> In particular, this change will simplify the implementation of failure
> handling
> across these transforms in https://issues.apache.org/jira/browse/BEAM-5638
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-5462) get rid of .options deprecation warnings in tests
[
https://issues.apache.org/jira/browse/BEAM-5462?focusedWorklogId=161243&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-161243
]
ASF GitHub Bot logged work on BEAM-5462:
Author: ASF GitHub Bot
Created on: 31/Oct/18 17:37
Start Date: 31/Oct/18 17:37
Worklog Time Spent: 10m
Work Description: markflyhigh commented on issue #6834: [BEAM-5462] Fix
.options in game example since it's deprecated
URL: https://github.com/apache/beam/pull/6834#issuecomment-434780383
PTAL @aaltay @udim
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 161243)
Time Spent: 1.5h (was: 1h 20m)
> get rid of .options deprecation warnings in tests
> ---
>
> Key: BEAM-5462
> URL: https://issues.apache.org/jira/browse/BEAM-5462
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core
>Reporter: Udi Meiri
>Priority: Minor
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> Messages look like:
> {{/usr/local/google/home/ehudm/src/beam/sdks/python/apache_beam/runners/direct/direct_runner.py:360:
> DeprecationWarning: options is deprecated since First stable release.
> References to .options will not be supported}}
> {{pipeline.replace_all(_get_transform_overrides(pipeline.options))}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-5464) Portable beam hangs while running TFX preprocessing step on a distributed cluster
[
https://issues.apache.org/jira/browse/BEAM-5464?focusedWorklogId=161245&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-161245
]
ASF GitHub Bot logged work on BEAM-5464:
Author: ASF GitHub Bot
Created on: 31/Oct/18 17:41
Start Date: 31/Oct/18 17:41
Worklog Time Spent: 10m
Work Description: angoenka commented on a change in pull request #6897:
[BEAM-5464] Use BATCH_FORCED as the default ExecutionMode for batch pipeline
URL: https://github.com/apache/beam/pull/6897#discussion_r229799016
##
File path:
runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkPipelineOptions.java
##
@@ -187,4 +188,12 @@
Long getLatencyTrackingInterval();
void setLatencyTrackingInterval(Long interval);
+
+ @Description(
+ "Flink mode for data exchange for batch pipeline. "
+ + "Reference {@link org.apache.flink.api.common.ExecutionMode}")
+ @Default.Enum("BATCH_FORCED")
+ ExecutionMode getExecutionModeForBatch();
Review comment:
TheDefault Enum test is done in ProxyInvocationHandlerTest.java
Please let me know if you are referring to some other test.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 161245)
Time Spent: 1h 20m (was: 1h 10m)
> Portable beam hangs while running TFX preprocessing step on a distributed
> cluster
> -
>
> Key: BEAM-5464
> URL: https://issues.apache.org/jira/browse/BEAM-5464
> Project: Beam
> Issue Type: Bug
> Components: java-fn-execution
>Reporter: Axel Magnuson
>Assignee: Ankur Goenka
>Priority: Major
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> Recently I went through the exercise of running the TFX taxi example on a
> dataproc cluster. However it would always hang indefinitely. The flink UI
> indicated that the job was halfway done. However I could not see any clear
> errors in the job driver logs, the job service logs, or the Flink logs. The
> root cause is still a mystery to me.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-5304) Support running user pipelines with the Universal Local Runner in Java.
[
https://issues.apache.org/jira/browse/BEAM-5304?focusedWorklogId=161251&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-161251
]
ASF GitHub Bot logged work on BEAM-5304:
Author: ASF GitHub Bot
Created on: 31/Oct/18 18:04
Start Date: 31/Oct/18 18:04
Worklog Time Spent: 10m
Work Description: lukecwik closed pull request #6797: [BEAM-5304] Small
improvements to ULR Job Server run command.
URL: https://github.com/apache/beam/pull/6797
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/runners/reference/job-server/build.gradle
b/runners/reference/job-server/build.gradle
index 1c81fba2dd5..919573d7df0 100644
--- a/runners/reference/job-server/build.gradle
+++ b/runners/reference/job-server/build.gradle
@@ -45,4 +45,9 @@ run {
jvmArgs = ["-Xdebug",
"-Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=5005"]
if (project.hasProperty("logLevel"))
jvmArgs +=
["-Dorg.slf4j.simpleLogger.defaultLogLevel=${project.property('logLevel')}"]
+ if (project.hasProperty("vendorLogLevel")) {
+jvmArgs +=
["-Dorg.slf4j.simpleLogger.log.org.apache.beam.vendor=${project.property('vendorLogLevel')}"]
+ } else {
+jvmArgs += ["-Dorg.slf4j.simpleLogger.log.org.apache.beam.vendor=info"]
+ }
}
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 161251)
Time Spent: 2h 40m (was: 2.5h)
> Support running user pipelines with the Universal Local Runner in Java.
> ---
>
> Key: BEAM-5304
> URL: https://issues.apache.org/jira/browse/BEAM-5304
> Project: Beam
> Issue Type: Sub-task
> Components: runner-direct
>Reporter: Daniel Oliveira
>Assignee: Daniel Oliveira
>Priority: Major
> Time Spent: 2h 40m
> Remaining Estimate: 0h
>
> In order to aid testing, devs should be able to write pipelines and then
> easily run them with the ULR. The Flink runner seems to have this
> functionality, so the implementation could be based on that.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (BEAM-5930) Java SDK harness fails to access state during timer callbacks
Luke Cwik created BEAM-5930:
---
Summary: Java SDK harness fails to access state during timer
callbacks
Key: BEAM-5930
URL: https://issues.apache.org/jira/browse/BEAM-5930
Project: Beam
Issue Type: Bug
Components: sdk-java-harness
Reporter: Luke Cwik
Assignee: Luke Cwik
The FnApiStateAccessor is given a callback to get the current element and it is
not handling the case where the current element is a timer.
callback:
https://github.com/apache/beam/blob/29c443162a2fe4c89d26336b30aa6e3a3bfbade8/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/FnApiDoFnRunner.java#L212
where the current "element" gets set:
https://github.com/apache/beam/blob/29c443162a2fe4c89d26336b30aa6e3a3bfbade8/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/FnApiDoFnRunner.java#L220
where the current "timer" gets set:
https://github.com/apache/beam/blob/29c443162a2fe4c89d26336b30aa6e3a3bfbade8/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/FnApiDoFnRunner.java#L237
The easiest fix would be to have the callback return the first non null from
currentElement/currentTimer but longer term I think we'll want a different
solution. Alternatively, we could collapse currentElement and currentTimer to
be currentElementOrTimer which would solve the accessor issue.
{code:java}
Caused by: java.util.concurrent.ExecutionException: java.lang.RuntimeException:
Error received from SDK harness for instruction 72:
java.util.concurrent.ExecutionException: java.lang.NullPointerException
at
java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
at
java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895)
at
org.apache.beam.sdk.fn.data.CompletableFutureInboundDataClient.awaitCompletion(CompletableFutureInboundDataClient.java:49)
at
org.apache.beam.sdk.fn.data.BeamFnDataInboundObserver.awaitCompletion(BeamFnDataInboundObserver.java:90)
at
org.apache.beam.fn.harness.BeamFnDataReadRunner.blockTillReadFinishes(BeamFnDataReadRunner.java:185)
at
org.apache.beam.fn.harness.control.ProcessBundleHandler.processBundle(ProcessBundleHandler.java:292)
at
org.apache.beam.fn.harness.control.BeamFnControlClient.delegateOnInstructionRequestType(BeamFnControlClient.java:161)
at
org.apache.beam.fn.harness.control.BeamFnControlClient.lambda$processInstructionRequests$0(BeamFnControlClient.java:145)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NullPointerException
at
org.apache.beam.model.fnexecution.v1.BeamFnApi$StateKey$BagUserState$Builder.setKey(BeamFnApi.java:49694)
at
org.apache.beam.fn.harness.state.FnApiStateAccessor.createBagUserStateKey(FnApiStateAccessor.java:451)
at
org.apache.beam.fn.harness.state.FnApiStateAccessor.bindBag(FnApiStateAccessor.java:244)
at
org.apache.beam.sdk.state.StateSpecs$BagStateSpec.bind(StateSpecs.java:487)
at
org.apache.beam.sdk.state.StateSpecs$BagStateSpec.bind(StateSpecs.java:477)
at
org.apache.beam.fn.harness.FnApiDoFnRunner$OnTimerContext.state(FnApiDoFnRunner.java:671)
at StateTest$5$OnTimerInvoker$expiry$ZXhwaXJ5.invokeOnTimer(Unknown
Source)
at
org.apache.beam.sdk.transforms.reflect.ByteBuddyDoFnInvokerFactory$DoFnInvokerBase.invokeOnTimer(ByteBuddyDoFnInvokerFactory.java:187)
at
org.apache.beam.fn.harness.FnApiDoFnRunner.processTimer(FnApiDoFnRunner.java:244)
at
org.apache.beam.fn.harness.DoFnPTransformRunnerFactory.lambda$createRunnerForPTransform$0(DoFnPTransformRunnerFactory.java:134)
at
org.apache.beam.sdk.fn.data.BeamFnDataInboundObserver.accept(BeamFnDataInboundObserver.java:81)
at
org.apache.beam.sdk.fn.data.BeamFnDataInboundObserver.accept(BeamFnDataInboundObserver.java:32)
at
org.apache.beam.sdk.fn.data.BeamFnDataGrpcMultiplexer$InboundObserver.onNext(BeamFnDataGrpcMultiplexer.java:139)
at
org.apache.beam.sdk.fn.data.BeamFnDataGrpcMultiplexer$InboundObserver.onNext(BeamFnDataGrpcMultiplexer.java:125)
at
org.apache.beam.sdk.fn.stream.ForwardingClientResponseObserver.onNext(ForwardingClientResponseObserver.java:50)
at
org.apache.beam.vendor.grpc.v1.io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onMessage(ClientCalls.java:407)
at
org.apache.beam.vendor.grpc.v1.io.grpc.ForwardingClientCallListener.onMessage(ForwardingClientCallListener.java:33)
at
org.apache.beam.vendor.grpc.v1.io.grpc.ForwardingClientCallListener.onMessage(ForwardingClientCallListener.java:33)
at
org.apache.beam.vendor.grpc.v1.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1MessagesAv
[jira] [Assigned] (BEAM-5930) Java SDK harness fails to access state during timer callbacks
[
https://issues.apache.org/jira/browse/BEAM-5930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Luke Cwik reassigned BEAM-5930:
---
Assignee: (was: Luke Cwik)
> Java SDK harness fails to access state during timer callbacks
> -
>
> Key: BEAM-5930
> URL: https://issues.apache.org/jira/browse/BEAM-5930
> Project: Beam
> Issue Type: Bug
> Components: sdk-java-harness
>Reporter: Luke Cwik
>Priority: Major
> Labels: portability
>
> The FnApiStateAccessor is given a callback to get the current element and it
> is not handling the case where the current element is a timer.
> callback:
> https://github.com/apache/beam/blob/29c443162a2fe4c89d26336b30aa6e3a3bfbade8/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/FnApiDoFnRunner.java#L212
> where the current "element" gets set:
> https://github.com/apache/beam/blob/29c443162a2fe4c89d26336b30aa6e3a3bfbade8/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/FnApiDoFnRunner.java#L220
> where the current "timer" gets set:
> https://github.com/apache/beam/blob/29c443162a2fe4c89d26336b30aa6e3a3bfbade8/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/FnApiDoFnRunner.java#L237
> The easiest fix would be to have the callback return the first non null from
> currentElement/currentTimer but longer term I think we'll want a different
> solution. Alternatively, we could collapse currentElement and currentTimer to
> be currentElementOrTimer which would solve the accessor issue.
> {code:java}
> Caused by: java.util.concurrent.ExecutionException:
> java.lang.RuntimeException: Error received from SDK harness for instruction
> 72: java.util.concurrent.ExecutionException: java.lang.NullPointerException
> at
> java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
> at
> java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895)
> at
> org.apache.beam.sdk.fn.data.CompletableFutureInboundDataClient.awaitCompletion(CompletableFutureInboundDataClient.java:49)
> at
> org.apache.beam.sdk.fn.data.BeamFnDataInboundObserver.awaitCompletion(BeamFnDataInboundObserver.java:90)
> at
> org.apache.beam.fn.harness.BeamFnDataReadRunner.blockTillReadFinishes(BeamFnDataReadRunner.java:185)
> at
> org.apache.beam.fn.harness.control.ProcessBundleHandler.processBundle(ProcessBundleHandler.java:292)
> at
> org.apache.beam.fn.harness.control.BeamFnControlClient.delegateOnInstructionRequestType(BeamFnControlClient.java:161)
> at
> org.apache.beam.fn.harness.control.BeamFnControlClient.lambda$processInstructionRequests$0(BeamFnControlClient.java:145)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.NullPointerException
> at
> org.apache.beam.model.fnexecution.v1.BeamFnApi$StateKey$BagUserState$Builder.setKey(BeamFnApi.java:49694)
> at
> org.apache.beam.fn.harness.state.FnApiStateAccessor.createBagUserStateKey(FnApiStateAccessor.java:451)
> at
> org.apache.beam.fn.harness.state.FnApiStateAccessor.bindBag(FnApiStateAccessor.java:244)
> at
> org.apache.beam.sdk.state.StateSpecs$BagStateSpec.bind(StateSpecs.java:487)
> at
> org.apache.beam.sdk.state.StateSpecs$BagStateSpec.bind(StateSpecs.java:477)
> at
> org.apache.beam.fn.harness.FnApiDoFnRunner$OnTimerContext.state(FnApiDoFnRunner.java:671)
> at StateTest$5$OnTimerInvoker$expiry$ZXhwaXJ5.invokeOnTimer(Unknown
> Source)
> at
> org.apache.beam.sdk.transforms.reflect.ByteBuddyDoFnInvokerFactory$DoFnInvokerBase.invokeOnTimer(ByteBuddyDoFnInvokerFactory.java:187)
> at
> org.apache.beam.fn.harness.FnApiDoFnRunner.processTimer(FnApiDoFnRunner.java:244)
> at
> org.apache.beam.fn.harness.DoFnPTransformRunnerFactory.lambda$createRunnerForPTransform$0(DoFnPTransformRunnerFactory.java:134)
> at
> org.apache.beam.sdk.fn.data.BeamFnDataInboundObserver.accept(BeamFnDataInboundObserver.java:81)
> at
> org.apache.beam.sdk.fn.data.BeamFnDataInboundObserver.accept(BeamFnDataInboundObserver.java:32)
> at
> org.apache.beam.sdk.fn.data.BeamFnDataGrpcMultiplexer$InboundObserver.onNext(BeamFnDataGrpcMultiplexer.java:139)
> at
> org.apache.beam.sdk.fn.data.BeamFnDataGrpcMultiplexer$InboundObserver.onNext(BeamFnDataGrpcMultiplexer.java:125)
> at
> org.apache.beam.sdk.fn.stream.ForwardingClientResponseObserver.onNext(ForwardingClientResponseObserver.java:50)
> at
> org.apache.beam.vendor.grpc.v1.io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onMessage(Cli
[jira] [Updated] (BEAM-5930) Java SDK harness fails to access state during timer callbacks
[
https://issues.apache.org/jira/browse/BEAM-5930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Luke Cwik updated BEAM-5930:
Labels: portability (was: )
> Java SDK harness fails to access state during timer callbacks
> -
>
> Key: BEAM-5930
> URL: https://issues.apache.org/jira/browse/BEAM-5930
> Project: Beam
> Issue Type: Bug
> Components: sdk-java-harness
>Reporter: Luke Cwik
>Priority: Major
> Labels: portability
>
> The FnApiStateAccessor is given a callback to get the current element and it
> is not handling the case where the current element is a timer.
> callback:
> https://github.com/apache/beam/blob/29c443162a2fe4c89d26336b30aa6e3a3bfbade8/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/FnApiDoFnRunner.java#L212
> where the current "element" gets set:
> https://github.com/apache/beam/blob/29c443162a2fe4c89d26336b30aa6e3a3bfbade8/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/FnApiDoFnRunner.java#L220
> where the current "timer" gets set:
> https://github.com/apache/beam/blob/29c443162a2fe4c89d26336b30aa6e3a3bfbade8/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/FnApiDoFnRunner.java#L237
> The easiest fix would be to have the callback return the first non null from
> currentElement/currentTimer but longer term I think we'll want a different
> solution. Alternatively, we could collapse currentElement and currentTimer to
> be currentElementOrTimer which would solve the accessor issue.
> {code:java}
> Caused by: java.util.concurrent.ExecutionException:
> java.lang.RuntimeException: Error received from SDK harness for instruction
> 72: java.util.concurrent.ExecutionException: java.lang.NullPointerException
> at
> java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
> at
> java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895)
> at
> org.apache.beam.sdk.fn.data.CompletableFutureInboundDataClient.awaitCompletion(CompletableFutureInboundDataClient.java:49)
> at
> org.apache.beam.sdk.fn.data.BeamFnDataInboundObserver.awaitCompletion(BeamFnDataInboundObserver.java:90)
> at
> org.apache.beam.fn.harness.BeamFnDataReadRunner.blockTillReadFinishes(BeamFnDataReadRunner.java:185)
> at
> org.apache.beam.fn.harness.control.ProcessBundleHandler.processBundle(ProcessBundleHandler.java:292)
> at
> org.apache.beam.fn.harness.control.BeamFnControlClient.delegateOnInstructionRequestType(BeamFnControlClient.java:161)
> at
> org.apache.beam.fn.harness.control.BeamFnControlClient.lambda$processInstructionRequests$0(BeamFnControlClient.java:145)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.NullPointerException
> at
> org.apache.beam.model.fnexecution.v1.BeamFnApi$StateKey$BagUserState$Builder.setKey(BeamFnApi.java:49694)
> at
> org.apache.beam.fn.harness.state.FnApiStateAccessor.createBagUserStateKey(FnApiStateAccessor.java:451)
> at
> org.apache.beam.fn.harness.state.FnApiStateAccessor.bindBag(FnApiStateAccessor.java:244)
> at
> org.apache.beam.sdk.state.StateSpecs$BagStateSpec.bind(StateSpecs.java:487)
> at
> org.apache.beam.sdk.state.StateSpecs$BagStateSpec.bind(StateSpecs.java:477)
> at
> org.apache.beam.fn.harness.FnApiDoFnRunner$OnTimerContext.state(FnApiDoFnRunner.java:671)
> at StateTest$5$OnTimerInvoker$expiry$ZXhwaXJ5.invokeOnTimer(Unknown
> Source)
> at
> org.apache.beam.sdk.transforms.reflect.ByteBuddyDoFnInvokerFactory$DoFnInvokerBase.invokeOnTimer(ByteBuddyDoFnInvokerFactory.java:187)
> at
> org.apache.beam.fn.harness.FnApiDoFnRunner.processTimer(FnApiDoFnRunner.java:244)
> at
> org.apache.beam.fn.harness.DoFnPTransformRunnerFactory.lambda$createRunnerForPTransform$0(DoFnPTransformRunnerFactory.java:134)
> at
> org.apache.beam.sdk.fn.data.BeamFnDataInboundObserver.accept(BeamFnDataInboundObserver.java:81)
> at
> org.apache.beam.sdk.fn.data.BeamFnDataInboundObserver.accept(BeamFnDataInboundObserver.java:32)
> at
> org.apache.beam.sdk.fn.data.BeamFnDataGrpcMultiplexer$InboundObserver.onNext(BeamFnDataGrpcMultiplexer.java:139)
> at
> org.apache.beam.sdk.fn.data.BeamFnDataGrpcMultiplexer$InboundObserver.onNext(BeamFnDataGrpcMultiplexer.java:125)
> at
> org.apache.beam.sdk.fn.stream.ForwardingClientResponseObserver.onNext(ForwardingClientResponseObserver.java:50)
> at
> org.apache.beam.vendor.grpc.v1.io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onMessage(ClientCalls.
[jira] [Work logged] (BEAM-5878) Support DoFns with Keyword-only arguments in Python 3.
[ https://issues.apache.org/jira/browse/BEAM-5878?focusedWorklogId=161262&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-161262 ] ASF GitHub Bot logged work on BEAM-5878: Author: ASF GitHub Bot Created on: 31/Oct/18 18:15 Start Date: 31/Oct/18 18:15 Worklog Time Spent: 10m Work Description: tvalentyn commented on issue #6781: [BEAM-5878] [BEAM-5490] Add partial support for functions with keyword-only arguments. URL: https://github.com/apache/beam/pull/6781#issuecomment-434793488 @aaltay Thanks for the review! PTAL. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 161262) Time Spent: 1.5h (was: 1h 20m) > Support DoFns with Keyword-only arguments in Python 3. > -- > > Key: BEAM-5878 > URL: https://issues.apache.org/jira/browse/BEAM-5878 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Valentyn Tymofieiev >Priority: Minor > Time Spent: 1.5h > Remaining Estimate: 0h > > Python 3.0 [adds a possibility|https://www.python.org/dev/peps/pep-3102/] to > define functions with keyword-only arguments. > Currently Beam does not handle them correctly. [~ruoyu] pointed out [one > place|https://github.com/apache/beam/blob/a56ce43109c97c739fa08adca45528c41e3c925c/sdks/python/apache_beam/typehints/decorators.py#L118] > in our codebase that we should fix: in Python in 3.0 inspect.getargspec() > will fail on functions with keyword-only arguments, but a new method > [inspect.getfullargspec()|https://docs.python.org/3/library/inspect.html#inspect.getfullargspec] > supports them. > There may be implications for our (best-effort) type-hints machinery. > We should also add a Py3-only unit tests that covers DoFn's with keyword-only > arguments once Beam Python 3 tests are in a good shape. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (BEAM-5930) Java SDK harness fails to access state during timer callbacks
[
https://issues.apache.org/jira/browse/BEAM-5930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Luke Cwik reassigned BEAM-5930:
---
Assignee: Luke Cwik
> Java SDK harness fails to access state during timer callbacks
> -
>
> Key: BEAM-5930
> URL: https://issues.apache.org/jira/browse/BEAM-5930
> Project: Beam
> Issue Type: Bug
> Components: sdk-java-harness
>Reporter: Luke Cwik
>Assignee: Luke Cwik
>Priority: Major
> Labels: portability
>
> The FnApiStateAccessor is given a callback to get the current element and it
> is not handling the case where the current element is a timer.
> callback:
> https://github.com/apache/beam/blob/29c443162a2fe4c89d26336b30aa6e3a3bfbade8/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/FnApiDoFnRunner.java#L212
> where the current "element" gets set:
> https://github.com/apache/beam/blob/29c443162a2fe4c89d26336b30aa6e3a3bfbade8/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/FnApiDoFnRunner.java#L220
> where the current "timer" gets set:
> https://github.com/apache/beam/blob/29c443162a2fe4c89d26336b30aa6e3a3bfbade8/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/FnApiDoFnRunner.java#L237
> The easiest fix would be to have the callback return the first non null from
> currentElement/currentTimer but longer term I think we'll want a different
> solution. Alternatively, we could collapse currentElement and currentTimer to
> be currentElementOrTimer which would solve the accessor issue.
> {code:java}
> Caused by: java.util.concurrent.ExecutionException:
> java.lang.RuntimeException: Error received from SDK harness for instruction
> 72: java.util.concurrent.ExecutionException: java.lang.NullPointerException
> at
> java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
> at
> java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895)
> at
> org.apache.beam.sdk.fn.data.CompletableFutureInboundDataClient.awaitCompletion(CompletableFutureInboundDataClient.java:49)
> at
> org.apache.beam.sdk.fn.data.BeamFnDataInboundObserver.awaitCompletion(BeamFnDataInboundObserver.java:90)
> at
> org.apache.beam.fn.harness.BeamFnDataReadRunner.blockTillReadFinishes(BeamFnDataReadRunner.java:185)
> at
> org.apache.beam.fn.harness.control.ProcessBundleHandler.processBundle(ProcessBundleHandler.java:292)
> at
> org.apache.beam.fn.harness.control.BeamFnControlClient.delegateOnInstructionRequestType(BeamFnControlClient.java:161)
> at
> org.apache.beam.fn.harness.control.BeamFnControlClient.lambda$processInstructionRequests$0(BeamFnControlClient.java:145)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.NullPointerException
> at
> org.apache.beam.model.fnexecution.v1.BeamFnApi$StateKey$BagUserState$Builder.setKey(BeamFnApi.java:49694)
> at
> org.apache.beam.fn.harness.state.FnApiStateAccessor.createBagUserStateKey(FnApiStateAccessor.java:451)
> at
> org.apache.beam.fn.harness.state.FnApiStateAccessor.bindBag(FnApiStateAccessor.java:244)
> at
> org.apache.beam.sdk.state.StateSpecs$BagStateSpec.bind(StateSpecs.java:487)
> at
> org.apache.beam.sdk.state.StateSpecs$BagStateSpec.bind(StateSpecs.java:477)
> at
> org.apache.beam.fn.harness.FnApiDoFnRunner$OnTimerContext.state(FnApiDoFnRunner.java:671)
> at StateTest$5$OnTimerInvoker$expiry$ZXhwaXJ5.invokeOnTimer(Unknown
> Source)
> at
> org.apache.beam.sdk.transforms.reflect.ByteBuddyDoFnInvokerFactory$DoFnInvokerBase.invokeOnTimer(ByteBuddyDoFnInvokerFactory.java:187)
> at
> org.apache.beam.fn.harness.FnApiDoFnRunner.processTimer(FnApiDoFnRunner.java:244)
> at
> org.apache.beam.fn.harness.DoFnPTransformRunnerFactory.lambda$createRunnerForPTransform$0(DoFnPTransformRunnerFactory.java:134)
> at
> org.apache.beam.sdk.fn.data.BeamFnDataInboundObserver.accept(BeamFnDataInboundObserver.java:81)
> at
> org.apache.beam.sdk.fn.data.BeamFnDataInboundObserver.accept(BeamFnDataInboundObserver.java:32)
> at
> org.apache.beam.sdk.fn.data.BeamFnDataGrpcMultiplexer$InboundObserver.onNext(BeamFnDataGrpcMultiplexer.java:139)
> at
> org.apache.beam.sdk.fn.data.BeamFnDataGrpcMultiplexer$InboundObserver.onNext(BeamFnDataGrpcMultiplexer.java:125)
> at
> org.apache.beam.sdk.fn.stream.ForwardingClientResponseObserver.onNext(ForwardingClientResponseObserver.java:50)
> at
> org.apache.beam.vendor.grpc.v1.io.grpc.stub.ClientCalls$StreamObserverToCallListene
[jira] [Work logged] (BEAM-5919) Build breaks when including Multi-release Jars
[
https://issues.apache.org/jira/browse/BEAM-5919?focusedWorklogId=161264&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-161264
]
ASF GitHub Bot logged work on BEAM-5919:
Author: ASF GitHub Bot
Created on: 31/Oct/18 18:23
Start Date: 31/Oct/18 18:23
Worklog Time Spent: 10m
Work Description: lukecwik commented on a change in pull request #6889:
[BEAM-5919] Exclude META-INF from validation of shaded jar files
URL: https://github.com/apache/beam/pull/6889#discussion_r229815832
##
File path:
buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
##
@@ -802,6 +802,7 @@ class BeamModulePlugin implements Plugin {
project.configurations.shadow.artifacts.files.each {
FileTree exposedClasses = project.zipTree(it).matching {
include "**/*.class"
+exclude "META-INF/**" // BEAM-5919: Exclude module-info.class
for Java 9 build support
Review comment:
We currently are excluding everything under META-INF/
Are you suggesting that I make the exclude more restrictive by changing it
to be:
```
exclude "META-INF/versions/**/module-info.class"
exclude "META-INF/versions/**/org/apache/beam/**"
```
or to just update the comment?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 161264)
Time Spent: 1h 10m (was: 1h)
> Build breaks when including Multi-release Jars
> --
>
> Key: BEAM-5919
> URL: https://issues.apache.org/jira/browse/BEAM-5919
> Project: Beam
> Issue Type: Bug
> Components: build-system
>Affects Versions: 2.8.0
>Reporter: Ismaël Mejía
>Assignee: Luke Cwik
>Priority: Minor
> Attachments: logjava9jarissue.txt
>
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> As part of BEAM-5891 there is a reported problem in the build system with
> jars that contain multiple versions of the classes (this is a feature
> introduced in Java 9). This should not be a problem when the main classes are
> still Java 8 compatible as it is the case for byte buddy.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-5919) Build breaks when including Multi-release Jars
[
https://issues.apache.org/jira/browse/BEAM-5919?focusedWorklogId=161265&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-161265
]
ASF GitHub Bot logged work on BEAM-5919:
Author: ASF GitHub Bot
Created on: 31/Oct/18 18:26
Start Date: 31/Oct/18 18:26
Worklog Time Spent: 10m
Work Description: lukecwik commented on a change in pull request #6889:
[BEAM-5919] Exclude META-INF from validation of shaded jar files
URL: https://github.com/apache/beam/pull/6889#discussion_r229815832
##
File path:
buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
##
@@ -802,6 +802,7 @@ class BeamModulePlugin implements Plugin {
project.configurations.shadow.artifacts.files.each {
FileTree exposedClasses = project.zipTree(it).matching {
include "**/*.class"
+exclude "META-INF/**" // BEAM-5919: Exclude module-info.class
for Java 9 build support
Review comment:
I see what you mean. Thanks for the pointer and valuable new knowledge about
how Java 9 is going to support multiple code versions.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 161265)
Time Spent: 1h 20m (was: 1h 10m)
> Build breaks when including Multi-release Jars
> --
>
> Key: BEAM-5919
> URL: https://issues.apache.org/jira/browse/BEAM-5919
> Project: Beam
> Issue Type: Bug
> Components: build-system
>Affects Versions: 2.8.0
>Reporter: Ismaël Mejía
>Assignee: Luke Cwik
>Priority: Minor
> Attachments: logjava9jarissue.txt
>
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> As part of BEAM-5891 there is a reported problem in the build system with
> jars that contain multiple versions of the classes (this is a feature
> introduced in Java 9). This should not be a problem when the main classes are
> still Java 8 compatible as it is the case for byte buddy.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-5919) Build breaks when including Multi-release Jars
[
https://issues.apache.org/jira/browse/BEAM-5919?focusedWorklogId=161266&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-161266
]
ASF GitHub Bot logged work on BEAM-5919:
Author: ASF GitHub Bot
Created on: 31/Oct/18 18:30
Start Date: 31/Oct/18 18:30
Worklog Time Spent: 10m
Work Description: lukecwik commented on a change in pull request #6889:
[BEAM-5919] Exclude META-INF from validation of shaded jar files
URL: https://github.com/apache/beam/pull/6889#discussion_r229818021
##
File path:
buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
##
@@ -802,6 +802,7 @@ class BeamModulePlugin implements Plugin {
project.configurations.shadow.artifacts.files.each {
FileTree exposedClasses = project.zipTree(it).matching {
include "**/*.class"
+exclude "META-INF/**" // BEAM-5919: Exclude module-info.class
for Java 9 build support
Review comment:
I feel like it will be a long while before "shading" tools support
multi-release jars and can see the amount of confusion/bugs people attempting
to shade these will cause.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 161266)
Time Spent: 1.5h (was: 1h 20m)
> Build breaks when including Multi-release Jars
> --
>
> Key: BEAM-5919
> URL: https://issues.apache.org/jira/browse/BEAM-5919
> Project: Beam
> Issue Type: Bug
> Components: build-system
>Affects Versions: 2.8.0
>Reporter: Ismaël Mejía
>Assignee: Luke Cwik
>Priority: Minor
> Attachments: logjava9jarissue.txt
>
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> As part of BEAM-5891 there is a reported problem in the build system with
> jars that contain multiple versions of the classes (this is a feature
> introduced in Java 9). This should not be a problem when the main classes are
> still Java 8 compatible as it is the case for byte buddy.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-5919) Build breaks when including Multi-release Jars
[
https://issues.apache.org/jira/browse/BEAM-5919?focusedWorklogId=161267&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-161267
]
ASF GitHub Bot logged work on BEAM-5919:
Author: ASF GitHub Bot
Created on: 31/Oct/18 18:32
Start Date: 31/Oct/18 18:32
Worklog Time Spent: 10m
Work Description: lukecwik commented on a change in pull request #6889:
[BEAM-5919] Exclude META-INF from validation of shaded jar files
URL: https://github.com/apache/beam/pull/6889#discussion_r229818830
##
File path:
buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
##
@@ -802,6 +802,7 @@ class BeamModulePlugin implements Plugin {
project.configurations.shadow.artifacts.files.each {
FileTree exposedClasses = project.zipTree(it).matching {
include "**/*.class"
+exclude "META-INF/**" // BEAM-5919: Exclude module-info.class
for Java 9 build support
Review comment:
I decided to use a single `*` for the version directory instead of `**`.
Since `**` matches any number of directories as per:
https://docs.gradle.org/current/javadoc/org/gradle/api/tasks/util/PatternFilterable.html
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 161267)
Time Spent: 1h 40m (was: 1.5h)
> Build breaks when including Multi-release Jars
> --
>
> Key: BEAM-5919
> URL: https://issues.apache.org/jira/browse/BEAM-5919
> Project: Beam
> Issue Type: Bug
> Components: build-system
>Affects Versions: 2.8.0
>Reporter: Ismaël Mejía
>Assignee: Luke Cwik
>Priority: Minor
> Attachments: logjava9jarissue.txt
>
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> As part of BEAM-5891 there is a reported problem in the build system with
> jars that contain multiple versions of the classes (this is a feature
> introduced in Java 9). This should not be a problem when the main classes are
> still Java 8 compatible as it is the case for byte buddy.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (BEAM-5919) Build breaks when including Multi-release Jars
[ https://issues.apache.org/jira/browse/BEAM-5919?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16670519#comment-16670519 ] Luke Cwik commented on BEAM-5919: - Note that https://blog.codefx.org/tools/multi-release-jars-multiple-java-versions/ describes the structure of Java 9+ multi-release jars: > Build breaks when including Multi-release Jars > -- > > Key: BEAM-5919 > URL: https://issues.apache.org/jira/browse/BEAM-5919 > Project: Beam > Issue Type: Bug > Components: build-system >Affects Versions: 2.8.0 >Reporter: Ismaël Mejía >Assignee: Luke Cwik >Priority: Minor > Attachments: logjava9jarissue.txt > > Time Spent: 1h 40m > Remaining Estimate: 0h > > As part of BEAM-5891 there is a reported problem in the build system with > jars that contain multiple versions of the classes (this is a feature > introduced in Java 9). This should not be a problem when the main classes are > still Java 8 compatible as it is the case for byte buddy. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5912) The python dependency check report shows same release dates for different versions of libraries.
[ https://issues.apache.org/jira/browse/BEAM-5912?focusedWorklogId=161273&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-161273 ] ASF GitHub Bot logged work on BEAM-5912: Author: ASF GitHub Bot Created on: 31/Oct/18 18:41 Start Date: 31/Oct/18 18:41 Worklog Time Spent: 10m Work Description: chamikaramj closed pull request #6879: [BEAM-5912] fix the python dependency check has the same relase date … URL: https://github.com/apache/beam/pull/6879 This is a PR merged from a forked repository. As GitHub hides the original diff on merge, it is displayed below for the sake of provenance: As this is a foreign pull request (from a fork), the diff is supplied below (as it won't show otherwise due to GitHub magic): diff --git a/.test-infra/jenkins/dependency_check/dependency_check_report_generator.py b/.test-infra/jenkins/dependency_check/dependency_check_report_generator.py index a1fe2ee0e2d..b4dfd35c4f6 100644 --- a/.test-infra/jenkins/dependency_check/dependency_check_report_generator.py +++ b/.test-infra/jenkins/dependency_check/dependency_check_report_generator.py @@ -118,7 +118,7 @@ def prioritize_dependencies(deps, sdk_type): else: dep_details_url = ReportGeneratorConfig.PYPI_URL + dep_name curr_release_date = find_release_time_from_python_compatibility_checking_service(dep_name, curr_ver) -latest_release_date = find_release_time_from_python_compatibility_checking_service(dep_name, curr_ver) +latest_release_date = find_release_time_from_python_compatibility_checking_service(dep_name, latest_ver) if not curr_release_date or not latest_release_date: curr_release_date, latest_release_date = query_dependency_release_dates_from_bigquery(bigquery_client, This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 161273) Time Spent: 2.5h (was: 2h 20m) > The python dependency check report shows same release dates for different > versions of libraries. > > > Key: BEAM-5912 > URL: https://issues.apache.org/jira/browse/BEAM-5912 > Project: Beam > Issue Type: Bug > Components: dependencies >Reporter: yifan zou >Assignee: yifan zou >Priority: Major > Time Spent: 2.5h > Remaining Estimate: 0h > > example: > [https://builds.apache.org/job/beam_Dependency_Check/160/artifact/src/build/dependencyUpdates/beam-dependency-check-report.html] > > |*Dependency Name*|*Current Version*|*Latest Version*|*Release Date Of the > Current Used Version*|*Release Date Of The Latest Release*|*JIRA Issue*| > |[google-cloud-pubsub|https://pypi.org/project/google-cloud-pubsub]|0.35.4|0.38.0|2018-06-06|2018-06-06|BEAM-5539| > |[pytz|https://pypi.org/project/pytz]|2018.4|2018.7|2018-04-10|2018-04-10|BEAM-5893| -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5878) Support DoFns with Keyword-only arguments in Python 3.
[ https://issues.apache.org/jira/browse/BEAM-5878?focusedWorklogId=161280&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-161280 ] ASF GitHub Bot logged work on BEAM-5878: Author: ASF GitHub Bot Created on: 31/Oct/18 18:49 Start Date: 31/Oct/18 18:49 Worklog Time Spent: 10m Work Description: aaltay closed pull request #6781: [BEAM-5878] [BEAM-5490] Add partial support for functions with keyword-only arguments. URL: https://github.com/apache/beam/pull/6781 This is a PR merged from a forked repository. As GitHub hides the original diff on merge, it is displayed below for the sake of provenance: As this is a foreign pull request (from a fork), the diff is supplied below (as it won't show otherwise due to GitHub magic): diff --git a/sdks/python/apache_beam/runners/common.py b/sdks/python/apache_beam/runners/common.py index 47222a18a78..033650048fd 100644 --- a/sdks/python/apache_beam/runners/common.py +++ b/sdks/python/apache_beam/runners/common.py @@ -150,8 +150,13 @@ def __init__(self, obj_to_invoke, method_name): 'a \'RestrictionProvider\'. Received %r instead.' % obj_to_invoke) -args, _, _, defaults = core.get_function_arguments( +fullargspec = core.get_function_arguments( obj_to_invoke, method_name) + +# TODO(BEAM-5878) support kwonlyargs on Python 3. +args = fullargspec[0] +defaults = fullargspec[3] + defaults = defaults if defaults else [] method_value = getattr(obj_to_invoke, method_name) self.method_value = method_value diff --git a/sdks/python/apache_beam/transforms/core.py b/sdks/python/apache_beam/transforms/core.py index 5505de55522..9027b97ac50 100644 --- a/sdks/python/apache_beam/transforms/core.py +++ b/sdks/python/apache_beam/transforms/core.py @@ -20,7 +20,6 @@ from __future__ import absolute_import import copy -import inspect import logging import random import re @@ -64,6 +63,7 @@ from apache_beam.typehints.decorators import TypeCheckError from apache_beam.typehints.decorators import WithTypeHints from apache_beam.typehints.decorators import get_type_hints +from apache_beam.typehints.decorators import getfullargspec from apache_beam.typehints.trivial_inference import element_type from apache_beam.typehints.typehints import is_consistent_with from apache_beam.utils import urns @@ -275,14 +275,14 @@ def restriction_coder(self): def get_function_arguments(obj, func): """Return the function arguments based on the name provided. If they have a _inspect_function attached to the class then use that otherwise default - to the python inspect library. + to the modified version of python inspect library. """ func_name = '_inspect_%s' % func if hasattr(obj, func_name): f = getattr(obj, func_name) return f() f = getattr(obj, func) - return inspect.getargspec(f) + return getfullargspec(f) class _DoFnParam(object): @@ -442,12 +442,21 @@ def is_process_bounded(self): def _fn_takes_side_inputs(fn): try: -argspec = inspect.getargspec(fn) +argspec = getfullargspec(fn) except TypeError: # We can't tell; maybe it does. return True is_bound = isinstance(fn, types.MethodType) and fn.__self__ is not None - return len(argspec.args) > 1 + is_bound or argspec.varargs or argspec.keywords + + try: +varkw = argspec.varkw +kwonlyargs = argspec.kwonlyargs + except AttributeError: # Python 2 +varkw = argspec.keywords +kwonlyargs = [] + + return (len(argspec.args) + len(kwonlyargs) > 1 + is_bound or + argspec.varargs or varkw) class CallableWrapperDoFn(DoFn): @@ -515,7 +524,7 @@ def _process_argspec_fn(self): return getattr(self._fn, '_argspec_fn', self._fn) def _inspect_process(self): -return inspect.getargspec(self._process_argspec_fn()) +return getfullargspec(self._process_argspec_fn()) class CombineFn(WithTypeHints, HasDisplayData, urns.RunnerApiFn): diff --git a/sdks/python/apache_beam/transforms/ptransform.py b/sdks/python/apache_beam/transforms/ptransform.py index 9939278d380..c512d9f3aa6 100644 --- a/sdks/python/apache_beam/transforms/ptransform.py +++ b/sdks/python/apache_beam/transforms/ptransform.py @@ -37,7 +37,6 @@ class and wrapper class that allows lambda functions to be used as from __future__ import absolute_import import copy -import inspect import itertools import operator import os @@ -61,6 +60,7 @@ class and wrapper class that allows lambda functions to be used as from apache_beam.typehints.decorators import TypeCheckError from apache_beam.typehints.decorators import WithTypeHints from apache_beam.typehints.decorators import getcallargs_forhints +from apache_beam.typehints.decorators import getfullargspec from apache_beam.typehints.trivial_inference import instance_to_type from apache_beam.typehints.typehints impor
[jira] [Work logged] (BEAM-5930) Java SDK harness fails to access state during timer callbacks
[ https://issues.apache.org/jira/browse/BEAM-5930?focusedWorklogId=161284&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-161284 ] ASF GitHub Bot logged work on BEAM-5930: Author: ASF GitHub Bot Created on: 31/Oct/18 19:09 Start Date: 31/Oct/18 19:09 Worklog Time Spent: 10m Work Description: lukecwik opened a new pull request #6902: [BEAM-5930] Fix Java SDK harness to be able to access state on timer callbacks. URL: https://github.com/apache/beam/pull/6902 Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). It will help us expedite review of your Pull Request if you tag someone (e.g. `@username`) to look at it. Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/) | --- | --- | --- | --- | --- | --- Java | [](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/) [](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/) [](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Python_VR_Flink/lastCompletedBuild/) | --- | --- | --- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 161284) Time Spent: 10m Remaining Estimate: 0h > Java SDK harness fails to access state during timer callbacks > - > > Key: BEAM-5930 >
[jira] [Work logged] (BEAM-5930) Java SDK harness fails to access state during timer callbacks
[
https://issues.apache.org/jira/browse/BEAM-5930?focusedWorklogId=161285&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-161285
]
ASF GitHub Bot logged work on BEAM-5930:
Author: ASF GitHub Bot
Created on: 31/Oct/18 19:09
Start Date: 31/Oct/18 19:09
Worklog Time Spent: 10m
Work Description: lukecwik commented on issue #6902: [BEAM-5930] Fix Java
SDK harness to be able to access state on timer callbacks.
URL: https://github.com/apache/beam/pull/6902#issuecomment-434811273
R: @mxm
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 161285)
Time Spent: 20m (was: 10m)
> Java SDK harness fails to access state during timer callbacks
> -
>
> Key: BEAM-5930
> URL: https://issues.apache.org/jira/browse/BEAM-5930
> Project: Beam
> Issue Type: Bug
> Components: sdk-java-harness
>Reporter: Luke Cwik
>Assignee: Luke Cwik
>Priority: Major
> Labels: portability
> Time Spent: 20m
> Remaining Estimate: 0h
>
> The FnApiStateAccessor is given a callback to get the current element and it
> is not handling the case where the current element is a timer.
> callback:
> https://github.com/apache/beam/blob/29c443162a2fe4c89d26336b30aa6e3a3bfbade8/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/FnApiDoFnRunner.java#L212
> where the current "element" gets set:
> https://github.com/apache/beam/blob/29c443162a2fe4c89d26336b30aa6e3a3bfbade8/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/FnApiDoFnRunner.java#L220
> where the current "timer" gets set:
> https://github.com/apache/beam/blob/29c443162a2fe4c89d26336b30aa6e3a3bfbade8/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/FnApiDoFnRunner.java#L237
> The easiest fix would be to have the callback return the first non null from
> currentElement/currentTimer but longer term I think we'll want a different
> solution. Alternatively, we could collapse currentElement and currentTimer to
> be currentElementOrTimer which would solve the accessor issue.
> {code:java}
> Caused by: java.util.concurrent.ExecutionException:
> java.lang.RuntimeException: Error received from SDK harness for instruction
> 72: java.util.concurrent.ExecutionException: java.lang.NullPointerException
> at
> java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
> at
> java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895)
> at
> org.apache.beam.sdk.fn.data.CompletableFutureInboundDataClient.awaitCompletion(CompletableFutureInboundDataClient.java:49)
> at
> org.apache.beam.sdk.fn.data.BeamFnDataInboundObserver.awaitCompletion(BeamFnDataInboundObserver.java:90)
> at
> org.apache.beam.fn.harness.BeamFnDataReadRunner.blockTillReadFinishes(BeamFnDataReadRunner.java:185)
> at
> org.apache.beam.fn.harness.control.ProcessBundleHandler.processBundle(ProcessBundleHandler.java:292)
> at
> org.apache.beam.fn.harness.control.BeamFnControlClient.delegateOnInstructionRequestType(BeamFnControlClient.java:161)
> at
> org.apache.beam.fn.harness.control.BeamFnControlClient.lambda$processInstructionRequests$0(BeamFnControlClient.java:145)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.NullPointerException
> at
> org.apache.beam.model.fnexecution.v1.BeamFnApi$StateKey$BagUserState$Builder.setKey(BeamFnApi.java:49694)
> at
> org.apache.beam.fn.harness.state.FnApiStateAccessor.createBagUserStateKey(FnApiStateAccessor.java:451)
> at
> org.apache.beam.fn.harness.state.FnApiStateAccessor.bindBag(FnApiStateAccessor.java:244)
> at
> org.apache.beam.sdk.state.StateSpecs$BagStateSpec.bind(StateSpecs.java:487)
> at
> org.apache.beam.sdk.state.StateSpecs$BagStateSpec.bind(StateSpecs.java:477)
> at
> org.apache.beam.fn.harness.FnApiDoFnRunner$OnTimerContext.state(FnApiDoFnRunner.java:671)
> at StateTest$5$OnTimerInvoker$expiry$ZXhwaXJ5.invokeOnTimer(Unknown
> Source)
> at
> org.apache.beam.sdk.transforms.reflect.ByteBuddyDoFnInvokerFactory$DoFnInvokerBase.invokeOnTimer(ByteBuddyDoFnInvokerFactory.java:187)
> at
> org.apache.beam.fn.harness.FnApiDoFnRunner.proc
[jira] [Resolved] (BEAM-5912) The python dependency check report shows same release dates for different versions of libraries.
[ https://issues.apache.org/jira/browse/BEAM-5912?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] yifan zou resolved BEAM-5912. - Resolution: Fixed Fix Version/s: 2.9.0 > The python dependency check report shows same release dates for different > versions of libraries. > > > Key: BEAM-5912 > URL: https://issues.apache.org/jira/browse/BEAM-5912 > Project: Beam > Issue Type: Bug > Components: dependencies >Reporter: yifan zou >Assignee: yifan zou >Priority: Major > Fix For: 2.9.0 > > Time Spent: 2.5h > Remaining Estimate: 0h > > example: > [https://builds.apache.org/job/beam_Dependency_Check/160/artifact/src/build/dependencyUpdates/beam-dependency-check-report.html] > > |*Dependency Name*|*Current Version*|*Latest Version*|*Release Date Of the > Current Used Version*|*Release Date Of The Latest Release*|*JIRA Issue*| > |[google-cloud-pubsub|https://pypi.org/project/google-cloud-pubsub]|0.35.4|0.38.0|2018-06-06|2018-06-06|BEAM-5539| > |[pytz|https://pypi.org/project/pytz]|2018.4|2018.7|2018-04-10|2018-04-10|BEAM-5893| -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5918) Add Cast transform for Rows
[ https://issues.apache.org/jira/browse/BEAM-5918?focusedWorklogId=161296&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-161296 ] ASF GitHub Bot logged work on BEAM-5918: Author: ASF GitHub Bot Created on: 31/Oct/18 20:17 Start Date: 31/Oct/18 20:17 Worklog Time Spent: 10m Work Description: kanterov commented on issue #6888: [BEAM-5918] Add Cast transform for Rows URL: https://github.com/apache/beam/pull/6888#issuecomment-434832598 @kennknowles Thanks for the feedback. I've simplified implementation a lot. Please take a look. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 161296) Time Spent: 3h 20m (was: 3h 10m) > Add Cast transform for Rows > --- > > Key: BEAM-5918 > URL: https://issues.apache.org/jira/browse/BEAM-5918 > Project: Beam > Issue Type: Improvement > Components: sdk-java-core >Reporter: Gleb Kanterov >Assignee: Gleb Kanterov >Priority: Major > Time Spent: 3h 20m > Remaining Estimate: 0h > > There is a need for a generic transform that given two Row schemas will > convert rows between them. There must be a possibility to opt-out from > certain kind of conversions, for instance, converting ints to shorts can > cause overflow. Another example, a schema could have a nullable field, but > never have NULL value in practice, because it was filtered out. > What is needed: > - widening values (e.g., int -> long) > - narrowwing (e.g., int -> short) > - runtime check for overflow while narrowing > - ignoring nullability (nullable=true -> nullable=false) > - weakening nullability (nullable=false -> nullable=true) > - projection (Schema(a: Int32, b: Int32) -> Schema(a: Int32)) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (BEAM-5931) Rollback PR/6899
Luke Cwik created BEAM-5931:
---
Summary: Rollback PR/6899
Key: BEAM-5931
URL: https://issues.apache.org/jira/browse/BEAM-5931
Project: Beam
Issue Type: Task
Components: beam-model, runner-dataflow
Reporter: Luke Cwik
To rollback this change, one must either:
1) Update nexmark / perf test framework to use Dataflow worker jar.
This requires adding the
{code}
"--dataflowWorkerJar=${dataflowWorkerJar}",
"--workerHarnessContainerImage=",
{code}
when running the tests.
OR
2) Update the dataflow worker image.
#1 is preferable since we will no longer have tests running that don't use a
Dataflow worker jar built from Github HEAD.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (BEAM-5931) Rollback PR/6899
[
https://issues.apache.org/jira/browse/BEAM-5931?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Luke Cwik updated BEAM-5931:
Description:
To rollback this change, one must either:
1) Update nexmark / perf test framework to use Dataflow worker jar.
This requires adding the
{code}
"--dataflowWorkerJar=${dataflowWorkerJar}",
"--workerHarnessContainerImage=",
{code}
when running the tests.
OR
2) Update the dataflow worker image with code that contains the rollback of
PR/6899 and then rollback PR/6899 in Github with the updated Dataflow worker
image.
#1 is preferable since we will no longer have tests running that don't use a
Dataflow worker jar built from Github HEAD.
was:
To rollback this change, one must either:
1) Update nexmark / perf test framework to use Dataflow worker jar.
This requires adding the
{code}
"--dataflowWorkerJar=${dataflowWorkerJar}",
"--workerHarnessContainerImage=",
{code}
when running the tests.
OR
2) Update the dataflow worker image.
#1 is preferable since we will no longer have tests running that don't use a
Dataflow worker jar built from Github HEAD.
> Rollback PR/6899
>
>
> Key: BEAM-5931
> URL: https://issues.apache.org/jira/browse/BEAM-5931
> Project: Beam
> Issue Type: Task
> Components: beam-model, runner-dataflow
>Reporter: Luke Cwik
>Priority: Major
>
> To rollback this change, one must either:
> 1) Update nexmark / perf test framework to use Dataflow worker jar.
> This requires adding the
> {code}
> "--dataflowWorkerJar=${dataflowWorkerJar}",
> "--workerHarnessContainerImage=",
> {code}
> when running the tests.
> OR
> 2) Update the dataflow worker image with code that contains the rollback of
> PR/6899 and then rollback PR/6899 in Github with the updated Dataflow worker
> image.
> #1 is preferable since we will no longer have tests running that don't use a
> Dataflow worker jar built from Github HEAD.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-5299) Define max global window as a shared value in protos like URN enums.
[ https://issues.apache.org/jira/browse/BEAM-5299?focusedWorklogId=161307&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-161307 ] ASF GitHub Bot logged work on BEAM-5299: Author: ASF GitHub Bot Created on: 31/Oct/18 20:38 Start Date: 31/Oct/18 20:38 Worklog Time Spent: 10m Work Description: lukecwik commented on issue #6381: [BEAM-5299] Define max timestamp for global window in proto URL: https://github.com/apache/beam/pull/6381#issuecomment-434839304 Filed https://issues.apache.org/jira/browse/BEAM-5931 for the rollback of the rollback. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 161307) Time Spent: 10h 20m (was: 10h 10m) > Define max global window as a shared value in protos like URN enums. > > > Key: BEAM-5299 > URL: https://issues.apache.org/jira/browse/BEAM-5299 > Project: Beam > Issue Type: Improvement > Components: beam-model, sdk-go, sdk-java-core, sdk-py-core >Reporter: Luke Cwik >Assignee: Maximilian Michels >Priority: Minor > Labels: portability > Time Spent: 10h 20m > Remaining Estimate: 0h > > Instead of having each language define a max timestamp themselves, define the > max timestamps within proto to be shared across different languages. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5892) Allow registering UDF with the same method name but different argument list
[ https://issues.apache.org/jira/browse/BEAM-5892?focusedWorklogId=161308&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-161308 ] ASF GitHub Bot logged work on BEAM-5892: Author: ASF GitHub Bot Created on: 31/Oct/18 20:40 Start Date: 31/Oct/18 20:40 Worklog Time Spent: 10m Work Description: amaliujia commented on issue #6865: [BEAM-5892] Improve UDF registration URL: https://github.com/apache/beam/pull/6865#issuecomment-434839953 This JIRA's work is implemented in https://github.com/apache/beam/pull/6898 now. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 161308) Time Spent: 40m (was: 0.5h) > Allow registering UDF with the same method name but different argument list > --- > > Key: BEAM-5892 > URL: https://issues.apache.org/jira/browse/BEAM-5892 > Project: Beam > Issue Type: Improvement > Components: dsl-sql >Reporter: Rui Wang >Assignee: Rui Wang >Priority: Major > Time Spent: 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5892) Allow registering UDF with the same method name but different argument list
[
https://issues.apache.org/jira/browse/BEAM-5892?focusedWorklogId=161309&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-161309
]
ASF GitHub Bot logged work on BEAM-5892:
Author: ASF GitHub Bot
Created on: 31/Oct/18 20:40
Start Date: 31/Oct/18 20:40
Worklog Time Spent: 10m
Work Description: amaliujia closed pull request #6865: [BEAM-5892]
Improve UDF registration
URL: https://github.com/apache/beam/pull/6865
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git
a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/BeamSqlEnv.java
b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/BeamSqlEnv.java
index 093961c59d2..6e305b21094 100644
---
a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/BeamSqlEnv.java
+++
b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/BeamSqlEnv.java
@@ -17,6 +17,7 @@
*/
package org.apache.beam.sdk.extensions.sql.impl;
+import java.lang.reflect.Method;
import java.util.Map;
import java.util.ServiceLoader;
import org.apache.beam.sdk.annotations.Experimental;
@@ -77,7 +78,12 @@ public static BeamSqlEnv inMemory(TableProvider...
tableProviders) {
/** Register a UDF function which can be used in SQL expression. */
public void registerUdf(String functionName, Class clazz, String method) {
-defaultSchema.add(functionName, ScalarFunctionImpl.create(clazz, method));
+Method[] methods = clazz.getMethods();
+for (int i = 0; i < method.length(); i++) {
+ if (methods[i].getName().equals(method) && !methods[i].isBridge()) {
+defaultSchema.add(functionName, ScalarFunctionImpl.create(methods[i]));
+ }
+}
}
/** Register a UDF function which can be used in SQL expression. */
diff --git
a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamSqlDslUdfUdafTest.java
b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamSqlDslUdfUdafTest.java
index 5b23151e795..8538d62591b 100644
---
a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamSqlDslUdfUdafTest.java
+++
b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamSqlDslUdfUdafTest.java
@@ -108,12 +108,17 @@ public void testRawCombineFnSubclass() {
@Test
public void testUdf() throws Exception {
Schema resultType =
Schema.builder().addInt32Field("f_int").addInt32Field("cubicvalue").build();
+Schema resultType2 =
+
Schema.builder().addInt64Field("f_long").addInt64Field("cubicLongValue").build();
+Schema resultType3 =
+
Schema.builder().addDoubleField("f_double").addDoubleField("cubicDoubleValue").build();
+
Row row = Row.withSchema(resultType).addValues(2, 8).build();
String sql1 = "SELECT f_int, cubic1(f_int) as cubicvalue FROM PCOLLECTION
WHERE f_int = 2";
PCollection result1 =
boundedInput1.apply(
-"testUdf1", SqlTransform.query(sql1).registerUdf("cubic1",
CubicInteger.class));
+"testUdf1", SqlTransform.query(sql1).registerUdf("cubic1",
Cubic.class));
PAssert.that(result1).containsInAnyOrder(row);
String sql2 = "SELECT f_int, cubic2(f_int) as cubicvalue FROM PCOLLECTION
WHERE f_int = 2";
@@ -134,6 +139,25 @@ public void testUdf() throws Exception {
Row subStrRow = Row.withSchema(subStrSchema).addValues(2, "s").build();
PAssert.that(result3).containsInAnyOrder(subStrRow);
+String sql4 =
+"SELECT f_long, cubic1(f_long) as cubicLongValue FROM PCOLLECTION
WHERE f_long = 1000";
+PCollection result4 =
+PCollectionTuple.of(new TupleTag<>("PCOLLECTION"), boundedInput1)
+.apply("testUdf4", SqlTransform.query(sql4).registerUdf("cubic1",
Cubic.class));
+
+Row longRow = Row.withSchema(resultType2).addValues(1000L, 1000L * 1000L *
1000L).build();
+PAssert.that(result4).containsInAnyOrder(longRow);
+
+String sql5 =
+"SELECT f_double, cubic1(f_double) as cubicDoubleValue FROM
PCOLLECTION WHERE f_double = 1.0";
+PCollection result5 =
+PCollectionTuple.of(new TupleTag<>("PCOLLECTION"), boundedInput1)
+.apply("testUdf5", SqlTransform.query(sql5).registerUdf("cubic1",
Cubic.class));
+
+Row doubleRow = Row.withSchema(resultType3).addValues(1.0d, 1.0d * 1.0d *
1.0d).build();
+
+PAssert.that(result5).containsInAnyOrder(doubleRow);
+
pipeline.run().waitUntilFinish();
}
@@ -161,7 +185,7 @@ public void testAutoUdfUdaf() throws Exception {
@Override
public Map> getBeamSqlUdfs() {
- return ImmutableMap.o
[jira] [Work logged] (BEAM-5884) Allow nested types have null value.
[ https://issues.apache.org/jira/browse/BEAM-5884?focusedWorklogId=161310&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-161310 ] ASF GitHub Bot logged work on BEAM-5884: Author: ASF GitHub Bot Created on: 31/Oct/18 20:42 Start Date: 31/Oct/18 20:42 Worklog Time Spent: 10m Work Description: amaliujia commented on issue #6861: [BEAM-5884] support null values in Array and Map. URL: https://github.com/apache/beam/pull/6861#issuecomment-434840465 @kennknowles 's suggestion actually makes lots of sense. Following that suggestion require a large change in Beam repo. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 161310) Time Spent: 3h 10m (was: 3h) > Allow nested types have null value. > --- > > Key: BEAM-5884 > URL: https://issues.apache.org/jira/browse/BEAM-5884 > Project: Beam > Issue Type: Bug > Components: dsl-sql >Reporter: Rui Wang >Assignee: Rui Wang >Priority: Major > Time Spent: 3h 10m > Remaining Estimate: 0h > > We could allow arbitrary combination of nested types have null value. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-5243) beam_Release_Gradle_NightlySnapshot InvocationError py27-cython/bin/python setup.py nosetests
[ https://issues.apache.org/jira/browse/BEAM-5243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16670707#comment-16670707 ] Yueyang Qiu commented on BEAM-5243: --- I created a shell script wrapper around tox to retry exit code -11 once when this failure happens. The long term solution should be migrating to py.test, then we can remove this usage of the wrapper. > beam_Release_Gradle_NightlySnapshot InvocationError py27-cython/bin/python > setup.py nosetests > - > > Key: BEAM-5243 > URL: https://issues.apache.org/jira/browse/BEAM-5243 > Project: Beam > Issue Type: Bug > Components: sdk-py-core, test-failures >Reporter: Andrew Pilloud >Assignee: Yueyang Qiu >Priority: Major > Labels: currently-failing > > It isn't clear to me what exactly failed, logs are full of stack traces. > [https://builds.apache.org/job/beam_Release_Gradle_NightlySnapshot/151/] > [https://builds.apache.org/job/beam_PostCommit_Python_Verify/5844/] > > *01:00:38* ERROR: InvocationError for command > '/home/jenkins/jenkins-slave/workspace/beam_Release_Gradle_NightlySnapshot/src/sdks/python/target/.tox/py27-cython/bin/python > setup.py nosetests' (exited with code -11)*01:00:38* > ___ summary > *01:00:38* ERROR: py27-cython: commands > failed*01:00:38* -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5918) Add Cast transform for Rows
[
https://issues.apache.org/jira/browse/BEAM-5918?focusedWorklogId=161320&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-161320
]
ASF GitHub Bot logged work on BEAM-5918:
Author: ASF GitHub Bot
Created on: 31/Oct/18 21:21
Start Date: 31/Oct/18 21:21
Worklog Time Spent: 10m
Work Description: kanterov commented on a change in pull request #6888:
[BEAM-5918] Add Cast transform for Rows
URL: https://github.com/apache/beam/pull/6888#discussion_r229872246
##
File path:
sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/transforms/Cast.java
##
@@ -0,0 +1,440 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.schemas.transforms;
+
+import com.google.auto.value.AutoValue;
+import com.google.common.base.Joiner;
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.Maps;
+import java.io.Serializable;
+import java.math.BigDecimal;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.stream.Collectors;
+import org.apache.beam.sdk.annotations.Experimental;
+import org.apache.beam.sdk.schemas.FieldAccessDescriptor;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.schemas.Schema.Field;
+import org.apache.beam.sdk.schemas.Schema.FieldType;
+import org.apache.beam.sdk.schemas.Schema.TypeName;
+import org.apache.beam.sdk.schemas.utils.SchemaZipFold;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.Row;
+
+/** Set of utilities for casting rows between schemas. */
+@Experimental(Experimental.Kind.SCHEMAS)
+@AutoValue
+public abstract class Cast extends PTransform,
PCollection> {
+
+ public abstract Schema outputSchema();
+
+ public abstract Validator validator();
+
+ public static Cast of(Schema outputSchema, Validator validator) {
+return new AutoValue_Cast<>(outputSchema, validator);
+ }
+
+ public static Cast widening(Schema outputSchema) {
+return new AutoValue_Cast<>(outputSchema, Widening.of());
+ }
+
+ public static Cast narrowing(Schema outputSchema) {
+return new AutoValue_Cast<>(outputSchema, Narrowing.of());
+ }
+
+ /** Describes compatibility errors during casting. */
+ @AutoValue
+ public abstract static class CompatibilityError implements Serializable {
+
+public abstract List path();
+
+public abstract String message();
+
+public static CompatibilityError create(List path, String message)
{
+ return new AutoValue_Cast_CompatibilityError(path, message);
+}
+ }
+
+ /** Interface for statically validating casts. */
+ public interface Validator extends Serializable {
+List apply(Schema input, Schema output);
+ }
+
+ /**
+ * Widening changes to type that can represent any possible value of the
original type.
+ *
+ * Standard widening conversions:
+ *
+ *
+ * BYTE to INT16, INT32, INT64, FLOAT, DOUBLE, DECIMAL
+ * INT16 to INT32, INT64, FLOAT, DOUBLE, DECIMAL
+ * INT32 to INT64, FLOAT, DOUBLE, DECIMAL
+ * INT64 to FLOAT, DOUBLE, DECIMAL
+ * FLOAT to DOUBLE, DECIMAL
+ * DOUBLE to DECIMAL
+ *
+ *
+ * Row widening:
+ *
+ *
+ * wider schema to schema with a subset of fields
Review comment:
Doesn't match with the definition of widening, probably should be only in
narrowing.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 161320)
Time Spent: 3.5h (was: 3h 20m)
> Add Cast transform for Rows
> ---
>
> Key: BEAM-5918
> URL: https://issues.apache.o
[jira] [Work logged] (BEAM-5852) Function extension in BeamSQL
[
https://issues.apache.org/jira/browse/BEAM-5852?focusedWorklogId=161335&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-161335
]
ASF GitHub Bot logged work on BEAM-5852:
Author: ASF GitHub Bot
Created on: 31/Oct/18 21:45
Start Date: 31/Oct/18 21:45
Worklog Time Spent: 10m
Work Description: akedin commented on a change in pull request #6898:
[BEAM-5852] [BEAM-5892] BeamSQL functions
URL: https://github.com/apache/beam/pull/6898#discussion_r229863400
##
File path:
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/udf/IsNan.java
##
@@ -0,0 +1,50 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.impl.udf;
+
+import com.google.auto.service.AutoService;
+
+/**
+ * IS_NAN(X)
+ *
+ * Returns TRUE if the value is a NaN value. Returns NULL for NULL inputs.
input: Float, Double
+ *
+ * Output: Boolean
+ */
+@AutoService(BeamBuiltinFunctionClass.class)
+public class IsNan implements BeamBuiltinFunctionClass {
+ private static final String SQL_FUNCTION_NAME = "IS_NAN";
+
+ @UserDefinedFunctionAnnotation(
+funcName = SQL_FUNCTION_NAME,
+parameterArray = {Float.class},
+returnType = Boolean.class
+ )
+ public Boolean isNan(Float value) {
+return Float.isNaN(value);
+ }
+
+ @UserDefinedFunctionAnnotation(
+funcName = SQL_FUNCTION_NAME,
+parameterArray = {Double.class},
Review comment:
These should be `Schema.FieldType`, not java types.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 161335)
Time Spent: 40m (was: 0.5h)
> Function extension in BeamSQL
> -
>
> Key: BEAM-5852
> URL: https://issues.apache.org/jira/browse/BEAM-5852
> Project: Beam
> Issue Type: New Feature
> Components: dsl-sql
>Reporter: Rui Wang
>Assignee: Rui Wang
>Priority: Major
> Time Spent: 40m
> Remaining Estimate: 0h
>
> We could add more functions to BeamSQL (as UDFs) to provide rich
> functionalities than standard/Calcite functions.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-5852) Function extension in BeamSQL
[
https://issues.apache.org/jira/browse/BEAM-5852?focusedWorklogId=161337&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-161337
]
ASF GitHub Bot logged work on BEAM-5852:
Author: ASF GitHub Bot
Created on: 31/Oct/18 21:45
Start Date: 31/Oct/18 21:45
Worklog Time Spent: 10m
Work Description: akedin commented on a change in pull request #6898:
[BEAM-5852] [BEAM-5892] BeamSQL functions
URL: https://github.com/apache/beam/pull/6898#discussion_r229871421
##
File path:
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/BeamSqlEnv.java
##
@@ -75,6 +77,14 @@ public static BeamSqlEnv inMemory(TableProvider...
tableProviders) {
return withTableProvider(inMemoryMetaStore);
}
+ public void registerBuiltinUdf(Map> methods) {
+for (Map.Entry> entry : methods.entrySet()) {
+ for (Method method : entry.getValue()) {
+defaultSchema.add(entry.getKey(), ScalarFunctionImpl.create(method));
Review comment:
This PR looks like it's one step from implementing own subclass of
`Function`, or `ScalarFunction`, it's probably worth doing it in this PR or
immediately follow up in another. That will make it a complete solution in my
opinion.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 161337)
Time Spent: 1h (was: 50m)
> Function extension in BeamSQL
> -
>
> Key: BEAM-5852
> URL: https://issues.apache.org/jira/browse/BEAM-5852
> Project: Beam
> Issue Type: New Feature
> Components: dsl-sql
>Reporter: Rui Wang
>Assignee: Rui Wang
>Priority: Major
> Time Spent: 1h
> Remaining Estimate: 0h
>
> We could add more functions to BeamSQL (as UDFs) to provide rich
> functionalities than standard/Calcite functions.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-5852) Function extension in BeamSQL
[
https://issues.apache.org/jira/browse/BEAM-5852?focusedWorklogId=161338&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-161338
]
ASF GitHub Bot logged work on BEAM-5852:
Author: ASF GitHub Bot
Created on: 31/Oct/18 21:45
Start Date: 31/Oct/18 21:45
Worklog Time Spent: 10m
Work Description: akedin commented on a change in pull request #6898:
[BEAM-5852] [BEAM-5892] BeamSQL functions
URL: https://github.com/apache/beam/pull/6898#discussion_r229878799
##
File path:
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/udf/BeamBuiltinFunctionClass.java
##
@@ -0,0 +1,21 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.impl.udf;
+
+/** BeamBuiltinFunctionClass interface. */
+interface BeamBuiltinFunctionClass {}
Review comment:
I think this should be called `BuiltinFunctionsProvider` so that it doesn't
look weird if there are multiple functions implemented here. Plus it somewhat
aligns with service loader/provider terminology this way.
I would also make it an abstract class that can extract UDFs from itself:
```
abstract BuiltinFunctionsProvider {
protected List loadUDFs() {
this.getClass().getDeclaredMethods().stream().filter(BuiltinFunctionsProvider::isUDF).map(BuiltinFunctionsProvider::toUDF).collect(toList());
}
}
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 161338)
Time Spent: 1h 10m (was: 1h)
> Function extension in BeamSQL
> -
>
> Key: BEAM-5852
> URL: https://issues.apache.org/jira/browse/BEAM-5852
> Project: Beam
> Issue Type: New Feature
> Components: dsl-sql
>Reporter: Rui Wang
>Assignee: Rui Wang
>Priority: Major
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> We could add more functions to BeamSQL (as UDFs) to provide rich
> functionalities than standard/Calcite functions.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-5852) Function extension in BeamSQL
[
https://issues.apache.org/jira/browse/BEAM-5852?focusedWorklogId=161340&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-161340
]
ASF GitHub Bot logged work on BEAM-5852:
Author: ASF GitHub Bot
Created on: 31/Oct/18 21:45
Start Date: 31/Oct/18 21:45
Worklog Time Spent: 10m
Work Description: akedin commented on a change in pull request #6898:
[BEAM-5852] [BEAM-5892] BeamSQL functions
URL: https://github.com/apache/beam/pull/6898#discussion_r229869088
##
File path:
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/udf/HyperbolicSine.java
##
@@ -0,0 +1,40 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.impl.udf;
+
+import com.google.auto.service.AutoService;
+
+/**
+ * SINH(X)
+ *
+ * Computes the hyperbolic sine of X. Generates an error if an overflow
occurs.
+ */
+@AutoService(BeamBuiltinFunctionClass.class)
+public class HyperbolicSine implements BeamBuiltinFunctionClass {
Review comment:
I would probably move all of these into a common class, e.g.
`BuiltinTrigonometry`
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 161340)
Time Spent: 1.5h (was: 1h 20m)
> Function extension in BeamSQL
> -
>
> Key: BEAM-5852
> URL: https://issues.apache.org/jira/browse/BEAM-5852
> Project: Beam
> Issue Type: New Feature
> Components: dsl-sql
>Reporter: Rui Wang
>Assignee: Rui Wang
>Priority: Major
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> We could add more functions to BeamSQL (as UDFs) to provide rich
> functionalities than standard/Calcite functions.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-5852) Function extension in BeamSQL
[
https://issues.apache.org/jira/browse/BEAM-5852?focusedWorklogId=161339&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-161339
]
ASF GitHub Bot logged work on BEAM-5852:
Author: ASF GitHub Bot
Created on: 31/Oct/18 21:45
Start Date: 31/Oct/18 21:45
Worklog Time Spent: 10m
Work Description: akedin commented on a change in pull request #6898:
[BEAM-5852] [BEAM-5892] BeamSQL functions
URL: https://github.com/apache/beam/pull/6898#discussion_r229863229
##
File path:
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/udf/IsNan.java
##
@@ -0,0 +1,50 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.impl.udf;
+
+import com.google.auto.service.AutoService;
+
+/**
+ * IS_NAN(X)
+ *
+ * Returns TRUE if the value is a NaN value. Returns NULL for NULL inputs.
input: Float, Double
+ *
+ * Output: Boolean
+ */
+@AutoService(BeamBuiltinFunctionClass.class)
+public class IsNan implements BeamBuiltinFunctionClass {
+ private static final String SQL_FUNCTION_NAME = "IS_NAN";
+
+ @UserDefinedFunctionAnnotation(
+funcName = SQL_FUNCTION_NAME,
+parameterArray = {Float.class},
+returnType = Boolean.class
+ )
+ public Boolean isNan(Float value) {
+return Float.isNaN(value);
+ }
+
+ @UserDefinedFunctionAnnotation(
Review comment:
`@UserDefinedFunction` should work as is. I would change it to `@UDF`
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 161339)
Time Spent: 1h 20m (was: 1h 10m)
> Function extension in BeamSQL
> -
>
> Key: BEAM-5852
> URL: https://issues.apache.org/jira/browse/BEAM-5852
> Project: Beam
> Issue Type: New Feature
> Components: dsl-sql
>Reporter: Rui Wang
>Assignee: Rui Wang
>Priority: Major
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> We could add more functions to BeamSQL (as UDFs) to provide rich
> functionalities than standard/Calcite functions.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-5852) Function extension in BeamSQL
[
https://issues.apache.org/jira/browse/BEAM-5852?focusedWorklogId=161336&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-161336
]
ASF GitHub Bot logged work on BEAM-5852:
Author: ASF GitHub Bot
Created on: 31/Oct/18 21:45
Start Date: 31/Oct/18 21:45
Worklog Time Spent: 10m
Work Description: akedin commented on a change in pull request #6898:
[BEAM-5852] [BEAM-5892] BeamSQL functions
URL: https://github.com/apache/beam/pull/6898#discussion_r229863674
##
File path:
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/udf/UserDefinedFunctionAnnotation.java
##
@@ -0,0 +1,34 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.impl.udf;
+
+import java.lang.annotation.ElementType;
+import java.lang.annotation.Retention;
+import java.lang.annotation.RetentionPolicy;
+import java.lang.annotation.Target;
+
+/** Make UserDefinedFunction annotation as package private. */
+@Retention(RetentionPolicy.RUNTIME)
+@Target({ElementType.METHOD})
+@interface UserDefinedFunctionAnnotation {
+ String funcName();
+
+ Class[] parameterArray();
Review comment:
`default []`, `default null`
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 161336)
Time Spent: 50m (was: 40m)
> Function extension in BeamSQL
> -
>
> Key: BEAM-5852
> URL: https://issues.apache.org/jira/browse/BEAM-5852
> Project: Beam
> Issue Type: New Feature
> Components: dsl-sql
>Reporter: Rui Wang
>Assignee: Rui Wang
>Priority: Major
> Time Spent: 50m
> Remaining Estimate: 0h
>
> We could add more functions to BeamSQL (as UDFs) to provide rich
> functionalities than standard/Calcite functions.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-5852) Function extension in BeamSQL
[
https://issues.apache.org/jira/browse/BEAM-5852?focusedWorklogId=161341&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-161341
]
ASF GitHub Bot logged work on BEAM-5852:
Author: ASF GitHub Bot
Created on: 31/Oct/18 21:45
Start Date: 31/Oct/18 21:45
Worklog Time Spent: 10m
Work Description: akedin commented on a change in pull request #6898:
[BEAM-5852] [BEAM-5892] BeamSQL functions
URL: https://github.com/apache/beam/pull/6898#discussion_r229872663
##
File path:
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/udf/BeamBuiltinFunctions.java
##
@@ -0,0 +1,47 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.impl.udf;
+
+import java.lang.reflect.Method;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.ServiceLoader;
+
+/** The entry to get all methods annotated by UserDefinedFunctionAnnotation. */
+public class BeamBuiltinFunctions {
+ public static Map> getBeamBuiltinFunctions() throws
ClassNotFoundException {
+Map> ret = new HashMap<>();
+
+for (BeamBuiltinFunctionClass functionClass :
Review comment:
nit: I would rewrite this with streams:
```
StreamSupport
.stream(ServiceLoader.load(BeamBuiltinFunctionClass.class))
.map(BeamBuiltinFunction::loadUDFs);
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 161341)
Time Spent: 1.5h (was: 1h 20m)
> Function extension in BeamSQL
> -
>
> Key: BEAM-5852
> URL: https://issues.apache.org/jira/browse/BEAM-5852
> Project: Beam
> Issue Type: New Feature
> Components: dsl-sql
>Reporter: Rui Wang
>Assignee: Rui Wang
>Priority: Major
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> We could add more functions to BeamSQL (as UDFs) to provide rich
> functionalities than standard/Calcite functions.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-5058) Python precommits should run E2E tests
[
https://issues.apache.org/jira/browse/BEAM-5058?focusedWorklogId=161342&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-161342
]
ASF GitHub Bot logged work on BEAM-5058:
Author: ASF GitHub Bot
Created on: 31/Oct/18 21:50
Start Date: 31/Oct/18 21:50
Worklog Time Spent: 10m
Work Description: udim commented on a change in pull request #6707:
[BEAM-5058] Parallelize Python Precommit and add basic ITs
URL: https://github.com/apache/beam/pull/6707#discussion_r229444669
##
File path:
buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
##
@@ -1580,5 +1580,76 @@ artifactId=${project.name}
dependsOn ':beam-sdks-java-container:docker'
}
}
+
+/**
***/
+
+project.ext.applyPythonNature = {
+
+ // Define common lifecycle tasks and artifact types
+ project.apply plugin: "base"
+
+ // For some reason base doesn't define a test task so we define it
below and make
+ // check depend on it. This makes the Python project similar to the task
layout like
+ // Java projects, see
https://docs.gradle.org/4.2.1/userguide/img/javaPluginTasks.png
+ project.task('test', type: Test) {}
+ project.check.dependsOn project.test
Review comment:
I don't believe that projects under subdirs define tasks for `test`.
It doesn't look right to define a new task named `test` more than once. How
does Gradle handle this? Will we have 2 or more `test` tasks?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 161342)
Time Spent: 2h 50m (was: 2h 40m)
> Python precommits should run E2E tests
> --
>
> Key: BEAM-5058
> URL: https://issues.apache.org/jira/browse/BEAM-5058
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core, testing
>Reporter: Udi Meiri
>Assignee: Mark Liu
>Priority: Major
> Time Spent: 2h 50m
> Remaining Estimate: 0h
>
> According to [https://beam.apache.org/contribute/testing/] (which I'm working
> on), end-to-end tests should be run in precommit on each combination of
> \{batch, streaming}x\{SDK language}x\{supported runner}.
> At least 2 tests need to be added to Python's precommit: wordcount and
> wordcount_streaming on Dataflow, and possibly on other supported runners
> (direct runner and new runners plz).
> These tests should be configured to run from a Gradle sub-project, so that
> they're run in parallel to the unit tests.
> Example that parallelizes Java precommit integration tests:
> [https://github.com/apache/beam/pull/5731]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-5304) Support running user pipelines with the Universal Local Runner in Java.
[ https://issues.apache.org/jira/browse/BEAM-5304?focusedWorklogId=161351&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-161351 ] ASF GitHub Bot logged work on BEAM-5304: Author: ASF GitHub Bot Created on: 31/Oct/18 21:58 Start Date: 31/Oct/18 21:58 Worklog Time Spent: 10m Work Description: youngoli opened a new pull request #6904: [BEAM-5304] Adding default and dynamic ports to ULR. URL: https://github.com/apache/beam/pull/6904 Small change to improve convenience with ULR. Follow this checklist to help us incorporate your contribution quickly and easily: - [x] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). It will help us expedite review of your Pull Request if you tag someone (e.g. `@username`) to look at it. Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/) | --- | --- | --- | --- | --- | --- Java | [](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/) [](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/) [](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Python_VR_Flink/lastCompletedBuild/) | --- | --- | --- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 161351) Time Spent: 2h 50m (was: 2h 40m) > Support running user pipelines with the Universal Local Runner in Java. > --- > >
[jira] [Work logged] (BEAM-5304) Support running user pipelines with the Universal Local Runner in Java.
[ https://issues.apache.org/jira/browse/BEAM-5304?focusedWorklogId=161352&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-161352 ] ASF GitHub Bot logged work on BEAM-5304: Author: ASF GitHub Bot Created on: 31/Oct/18 21:59 Start Date: 31/Oct/18 21:59 Worklog Time Spent: 10m Work Description: youngoli commented on issue #6904: [BEAM-5304] Adding default and dynamic ports to ULR. URL: https://github.com/apache/beam/pull/6904#issuecomment-434862549 R: @HuangLED This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 161352) Time Spent: 3h (was: 2h 50m) > Support running user pipelines with the Universal Local Runner in Java. > --- > > Key: BEAM-5304 > URL: https://issues.apache.org/jira/browse/BEAM-5304 > Project: Beam > Issue Type: Sub-task > Components: runner-direct >Reporter: Daniel Oliveira >Assignee: Daniel Oliveira >Priority: Major > Time Spent: 3h > Remaining Estimate: 0h > > In order to aid testing, devs should be able to write pipelines and then > easily run them with the ULR. The Flink runner seems to have this > functionality, so the implementation could be based on that. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5058) Python precommits should run E2E tests
[
https://issues.apache.org/jira/browse/BEAM-5058?focusedWorklogId=161353&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-161353
]
ASF GitHub Bot logged work on BEAM-5058:
Author: ASF GitHub Bot
Created on: 31/Oct/18 21:59
Start Date: 31/Oct/18 21:59
Worklog Time Spent: 10m
Work Description: udim commented on issue #6707: [BEAM-5058] Parallelize
Python Precommit and add basic ITs
URL: https://github.com/apache/beam/pull/6707#issuecomment-434862676
setupVirtualenv is probably failing due to the path being too long:
https://issues.apache.org/jira/browse/BEAM-4256
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 161353)
Time Spent: 3h (was: 2h 50m)
> Python precommits should run E2E tests
> --
>
> Key: BEAM-5058
> URL: https://issues.apache.org/jira/browse/BEAM-5058
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core, testing
>Reporter: Udi Meiri
>Assignee: Mark Liu
>Priority: Major
> Time Spent: 3h
> Remaining Estimate: 0h
>
> According to [https://beam.apache.org/contribute/testing/] (which I'm working
> on), end-to-end tests should be run in precommit on each combination of
> \{batch, streaming}x\{SDK language}x\{supported runner}.
> At least 2 tests need to be added to Python's precommit: wordcount and
> wordcount_streaming on Dataflow, and possibly on other supported runners
> (direct runner and new runners plz).
> These tests should be configured to run from a Gradle sub-project, so that
> they're run in parallel to the unit tests.
> Example that parallelizes Java precommit integration tests:
> [https://github.com/apache/beam/pull/5731]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (BEAM-4256) Gradle setupVirtualenv task fails on Jenkins
[
https://issues.apache.org/jira/browse/BEAM-4256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16670820#comment-16670820
]
Udi Meiri commented on BEAM-4256:
-
I can verify that shortening the path solved the issue, as we just came across
it again when the path got lengthened in
https://github.com/apache/beam/pull/6707
> Gradle setupVirtualenv task fails on Jenkins
>
>
> Key: BEAM-4256
> URL: https://issues.apache.org/jira/browse/BEAM-4256
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core
>Reporter: Lukasz Gajowy
>Assignee: Ahmet Altay
>Priority: Critical
> Fix For: Not applicable
>
>
> AFAIK the setupVirtualenv task should run "pip install" command in the
> virtual environment. This works on my local machine, yet on Jenkins, when we
> run:
> {code:java}
> gradlew clean install -xcheck --stacktrace{code}
> from within PerfkitBenchmarker tool, we get the following stacktrace:
> {code:java}
> > Task :beam-sdks-python:setupVirtualenv
> Collecting tox==3.0.0
> Using cached
> https://files.pythonhosted.org/packages/e6/41/4dcfd713282bf3213b0384320fa8841e4db032ddcb80bc08a540159d42a8/tox-3.0.0-py2.py3-none-any.whl
> Collecting grpcio-tools==1.3.5
> Using cached
> https://files.pythonhosted.org/packages/05/f6/0296e29b1bac6f85d2a8556d48adf825307f73109a3c2c17fb734292db0a/grpcio_tools-1.3.5-cp27-cp27mu-manylinux1_x86_64.whl
> Collecting pluggy<1.0,>=0.3.0 (from tox==3.0.0)
> Using cached
> https://files.pythonhosted.org/packages/82/05/43e3947125a2137cba4746135c75934ceed1863f27e050fc560052104a71/pluggy-0.6.0-py2-none-any.whl
> Requirement not upgraded as not directly required: six in
> /usr/local/lib/python2.7/dist-packages (from tox==3.0.0) (1.11.0)
> Requirement not upgraded as not directly required: virtualenv>=1.11.2 in
> /usr/lib/python2.7/dist-packages (from tox==3.0.0) (15.0.1)
> Collecting py>=1.4.17 (from tox==3.0.0)
> Using cached
> https://files.pythonhosted.org/packages/67/a5/f77982214dd4c8fd104b066f249adea2c49e25e8703d284382eb5e9ab35a/py-1.5.3-py2.py3-none-any.whl
> Collecting grpcio>=1.3.5 (from grpcio-tools==1.3.5)
> Using cached
> https://files.pythonhosted.org/packages/0d/54/b647a6323be6526be27b2c90bb042769f1a7a6e59bd1a5f2eeb795bfece4/grpcio-1.11.0-cp27-cp27mu-manylinux1_x86_64.whl
> Collecting protobuf>=3.2.0 (from grpcio-tools==1.3.5)
> Using cached
> https://files.pythonhosted.org/packages/9d/61/54c3a9cfde6ffe0ca6a1786ddb8874263f4ca32e7693ad383bd8cf935015/protobuf-3.5.2.post1-cp27-cp27mu-manylinux1_x86_64.whl
> Requirement not upgraded as not directly required: enum34>=1.0.4 in
> /usr/local/lib/python2.7/dist-packages (from
> grpcio>=1.3.5->grpcio-tools==1.3.5) (1.1.6)
> Collecting futures>=2.2.0 (from grpcio>=1.3.5->grpcio-tools==1.3.5)
> Using cached
> https://files.pythonhosted.org/packages/2d/99/b2c4e9d5a30f6471e410a146232b4118e697fa3ffc06d6a65efde84debd0/futures-3.2.0-py2-none-any.whl
> Requirement not upgraded as not directly required: setuptools in
> /usr/local/lib/python2.7/dist-packages (from
> protobuf>=3.2.0->grpcio-tools==1.3.5) (39.0.1)
> Installing collected packages: pluggy, py, tox, protobuf, futures, grpcio,
> grpcio-tools
> Could not install packages due to an EnvironmentError: [Errno 13] Permission
> denied: '/usr/local/lib/python2.7/dist-packages/pluggy-0.6.0.dist-info'
> Consider using the `--user` option or check the permissions.
> > Task :beam-model-job-management:shadowJar
> > Task :beam-sdks-python:setupVirtualenv FAILED
> > Task :beam-model-job-management:jar
> > Task :beam-model-fn-execution:shadowJar
> > Task :beam-sdks-java-core:shadowJar
> Deprecated Gradle features were used in this build, making it incompatible
> with Gradle 5.0.
> See
> https://docs.gradle.org/4.7/userguide/command_line_interface.html#sec:command_line_warnings
> BUILD FAILED in 44s
> 133 actionable tasks: 126 executed, 5 from cache, 2 up-to-date{code}
> It looks as if the virtualenv is not used while installing the needed
> dependencies (why is it trying to install pluggy in /usr/local/lib?).
> Therefore it throws "PermissionDenied" exception.
> This affects all Performance tests on Beam's Jenkins. Currently none of the
> Jenkins jobs can build the whole repository code before the benchmark is
> started.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
