date:20181028

[jira] [Work logged] (BEAM-5058) Python precommits should run E2E tests

2018-10-28 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-5058?focusedWorklogId=159857=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-159857
 ]

ASF GitHub Bot logged work on BEAM-5058:


Author: ASF GitHub Bot
Created on: 29/Oct/18 05:10
Start Date: 29/Oct/18 05:10
Worklog Time Spent: 10m 
  Work Description: markflyhigh commented on a change in pull request 
#6707: [BEAM-5058] Run basic ITs in Python Precommit
URL: https://github.com/apache/beam/pull/6707#discussion_r228802561
 
 

 ##
 File path: sdks/python/build.gradle
 ##
 @@ -226,6 +228,26 @@ task directRunnerIT(dependsOn: 'installGcpTest') {
   }
 }
 
+task precommitIT(dependsOn: ['installGcpTest', 'sdist']) {
 
 Review comment:
   Leverage Gradle parallel mechanism is a great idea. I move some basic setup 
tasks into `BeamModulePlugins` and move `precommitIT` under 
`.../python/precommit/dataflow`. Let's see how this effects PreCommit build 
time.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 159857)
Time Spent: 1h 20m  (was: 1h 10m)

> Python precommits should run E2E tests
> --
>
> Key: BEAM-5058
> URL: https://issues.apache.org/jira/browse/BEAM-5058
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core, testing
>Reporter: Udi Meiri
>Assignee: Mark Liu
>Priority: Major
>  Time Spent: 1h 20m
>  Remaining Estimate: 0h
>
> According to [https://beam.apache.org/contribute/testing/] (which I'm working 
> on), end-to-end tests should be run in precommit on each combination of 
> \{batch, streaming}x\{SDK language}x\{supported runner}.
> At least 2 tests need to be added to Python's precommit: wordcount and 
> wordcount_streaming on Dataflow, and possibly on other supported runners 
> (direct runner and new runners plz).
>  These tests should be configured to run from a Gradle sub-project, so that 
> they're run in parallel to the unit tests.
> Example that parallelizes Java precommit integration tests: 
> [https://github.com/apache/beam/pull/5731]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5058) Python precommits should run E2E tests

2018-10-28 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-5058?focusedWorklogId=159854=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-159854
 ]

ASF GitHub Bot logged work on BEAM-5058:


Author: ASF GitHub Bot
Created on: 29/Oct/18 05:05
Start Date: 29/Oct/18 05:05
Worklog Time Spent: 10m 
  Work Description: markflyhigh commented on a change in pull request 
#6707: [BEAM-5058] Run basic ITs in Python Precommit
URL: https://github.com/apache/beam/pull/6707#discussion_r228802133
 
 

 ##
 File path: sdks/python/build.gradle
 ##
 @@ -226,6 +228,26 @@ task directRunnerIT(dependsOn: 'installGcpTest') {
   }
 }
 
+task precommitIT(dependsOn: ['installGcpTest', 'sdist']) {
+  doLast {
+// List of integration tests running in Python PreCommit.
+def precommitTests = [
+"apache_beam.examples.wordcount_it_test:WordCountIT.test_wordcount_it",
+
"apache_beam.examples.streaming_wordcount_it_test:StreamingWordCountIT.test_streaming_wordcount_it",
+]
+def testOpts = [
 
 Review comment:
   done


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 159854)
Time Spent: 1h 10m  (was: 1h)

> Python precommits should run E2E tests
> --
>
> Key: BEAM-5058
> URL: https://issues.apache.org/jira/browse/BEAM-5058
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core, testing
>Reporter: Udi Meiri
>Assignee: Mark Liu
>Priority: Major
>  Time Spent: 1h 10m
>  Remaining Estimate: 0h
>
> According to [https://beam.apache.org/contribute/testing/] (which I'm working 
> on), end-to-end tests should be run in precommit on each combination of 
> \{batch, streaming}x\{SDK language}x\{supported runner}.
> At least 2 tests need to be added to Python's precommit: wordcount and 
> wordcount_streaming on Dataflow, and possibly on other supported runners 
> (direct runner and new runners plz).
>  These tests should be configured to run from a Gradle sub-project, so that 
> they're run in parallel to the unit tests.
> Example that parallelizes Java precommit integration tests: 
> [https://github.com/apache/beam/pull/5731]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5892) Allow registering UDF with the same method name but different argument list

2018-10-28 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-5892?focusedWorklogId=159850=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-159850
 ]

ASF GitHub Bot logged work on BEAM-5892:


Author: ASF GitHub Bot
Created on: 29/Oct/18 05:04
Start Date: 29/Oct/18 05:04
Worklog Time Spent: 10m 
  Work Description: amaliujia opened a new pull request #6865: [BEAM-5892] 
Improve UDF
URL: https://github.com/apache/beam/pull/6865
 
 
   Allow registering UDF with the same method name but different argument lis. 
This allows us define a UDF which can handle different argument types.
   
   
   
   
   Follow this checklist to help us incorporate your contribution quickly and 
easily:
   
- [x] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue, if applicable. This will automatically link the pull request to the 
issue.
- [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   It will help us expedite review of your Pull Request if you tag someone 
(e.g. `@username`) to look at it.
   
   Post-Commit Tests Status (on master branch)
   

   
   Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
   --- | --- | --- | --- | --- | --- | --- | ---
   Go | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/)
 | --- | --- | --- | --- | --- | ---
   Java | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/)
 [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/)
   Python | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/)
 | --- | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)
  [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Python_VR_Flink/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python_VR_Flink/lastCompletedBuild/)
 | --- | --- | ---
   
   
   
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 159850)
Time Spent: 10m
Remaining Estimate: 0h

> Allow registering UDF with the same method name but different

[jira] [Work logged] (BEAM-5892) Allow registering UDF with the same method name but different argument list

2018-10-28 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-5892?focusedWorklogId=159851=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-159851
 ]

ASF GitHub Bot logged work on BEAM-5892:


Author: ASF GitHub Bot
Created on: 29/Oct/18 05:04
Start Date: 29/Oct/18 05:04
Worklog Time Spent: 10m 
  Work Description: amaliujia commented on issue #6865: [BEAM-5892] Improve 
UDF
URL: https://github.com/apache/beam/pull/6865#issuecomment-433789293
 
 
   R: @kennknowles @akedin @XuMingmin 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 159851)
Time Spent: 20m  (was: 10m)

> Allow registering UDF with the same method name but different argument list
> ---
>
> Key: BEAM-5892
> URL: https://issues.apache.org/jira/browse/BEAM-5892
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql
>Reporter: Rui Wang
>Assignee: Rui Wang
>Priority: Major
>  Time Spent: 20m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work started] (BEAM-5892) Allow registering UDF with the same method name but different argument list

2018-10-28 Thread Rui Wang (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-5892?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on BEAM-5892 started by Rui Wang.
--
> Allow registering UDF with the same method name but different argument list
> ---
>
> Key: BEAM-5892
> URL: https://issues.apache.org/jira/browse/BEAM-5892
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql
>Reporter: Rui Wang
>Assignee: Rui Wang
>Priority: Major
>  Time Spent: 20m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5058) Python precommits should run E2E tests

2018-10-28 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-5058?focusedWorklogId=159852=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-159852
 ]

ASF GitHub Bot logged work on BEAM-5058:


Author: ASF GitHub Bot
Created on: 29/Oct/18 05:04
Start Date: 29/Oct/18 05:04
Worklog Time Spent: 10m 
  Work Description: markflyhigh commented on a change in pull request 
#6707: [BEAM-5058] Run basic ITs in Python Precommit
URL: https://github.com/apache/beam/pull/6707#discussion_r228802106
 
 

 ##
 File path: sdks/python/build.gradle
 ##
 @@ -226,6 +228,26 @@ task directRunnerIT(dependsOn: 'installGcpTest') {
   }
 }
 
+task precommitIT(dependsOn: ['installGcpTest', 'sdist']) {
+  doLast {
+// List of integration tests running in Python PreCommit.
+def precommitTests = [
+"apache_beam.examples.wordcount_it_test:WordCountIT.test_wordcount_it",
+
"apache_beam.examples.streaming_wordcount_it_test:StreamingWordCountIT.test_streaming_wordcount_it",
+]
+def testOpts = [
+"--tests=${precommitTests.join(',')}",
+"--processes=4",
+"--process-timeout=1800",   // Total timeout includes all tests run.
+]
+
+exec {
+  executable 'sh'
+  args '-c', ". ${envdir}/bin/activate && 
./scripts/run_integration_test.sh --test_opts \"${testOpts.join(' ')}\""
 
 Review comment:
   sg. I'd love to put it in next PR.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 159852)
Time Spent: 50m  (was: 40m)

> Python precommits should run E2E tests
> --
>
> Key: BEAM-5058
> URL: https://issues.apache.org/jira/browse/BEAM-5058
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core, testing
>Reporter: Udi Meiri
>Assignee: Mark Liu
>Priority: Major
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> According to [https://beam.apache.org/contribute/testing/] (which I'm working 
> on), end-to-end tests should be run in precommit on each combination of 
> \{batch, streaming}x\{SDK language}x\{supported runner}.
> At least 2 tests need to be added to Python's precommit: wordcount and 
> wordcount_streaming on Dataflow, and possibly on other supported runners 
> (direct runner and new runners plz).
>  These tests should be configured to run from a Gradle sub-project, so that 
> they're run in parallel to the unit tests.
> Example that parallelizes Java precommit integration tests: 
> [https://github.com/apache/beam/pull/5731]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5058) Python precommits should run E2E tests

2018-10-28 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-5058?focusedWorklogId=159853=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-159853
 ]

ASF GitHub Bot logged work on BEAM-5058:


Author: ASF GitHub Bot
Created on: 29/Oct/18 05:04
Start Date: 29/Oct/18 05:04
Worklog Time Spent: 10m 
  Work Description: markflyhigh commented on a change in pull request 
#6707: [BEAM-5058] Run basic ITs in Python Precommit
URL: https://github.com/apache/beam/pull/6707#discussion_r228802125
 
 

 ##
 File path: sdks/python/build.gradle
 ##
 @@ -226,6 +228,26 @@ task directRunnerIT(dependsOn: 'installGcpTest') {
   }
 }
 
+task precommitIT(dependsOn: ['installGcpTest', 'sdist']) {
 
 Review comment:
   done


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 159853)
Time Spent: 1h  (was: 50m)

> Python precommits should run E2E tests
> --
>
> Key: BEAM-5058
> URL: https://issues.apache.org/jira/browse/BEAM-5058
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core, testing
>Reporter: Udi Meiri
>Assignee: Mark Liu
>Priority: Major
>  Time Spent: 1h
>  Remaining Estimate: 0h
>
> According to [https://beam.apache.org/contribute/testing/] (which I'm working 
> on), end-to-end tests should be run in precommit on each combination of 
> \{batch, streaming}x\{SDK language}x\{supported runner}.
> At least 2 tests need to be added to Python's precommit: wordcount and 
> wordcount_streaming on Dataflow, and possibly on other supported runners 
> (direct runner and new runners plz).
>  These tests should be configured to run from a Gradle sub-project, so that 
> they're run in parallel to the unit tests.
> Example that parallelizes Java precommit integration tests: 
> [https://github.com/apache/beam/pull/5731]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-5892) Allow registering UDF with the same method name but different argument list

2018-10-28 Thread Rui Wang (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-5892?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rui Wang updated BEAM-5892:
---
Summary: Allow registering UDF with the same method name but different 
argument list  (was: Allow registering UDF by method name and argument list)

> Allow registering UDF with the same method name but different argument list
> ---
>
> Key: BEAM-5892
> URL: https://issues.apache.org/jira/browse/BEAM-5892
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql
>Reporter: Rui Wang
>Assignee: Rui Wang
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-5892) Allow registering UDF by method name and argument list

2018-10-28 Thread Rui Wang (JIRA)

Rui Wang created BEAM-5892:
--

 Summary: Allow registering UDF by method name and argument list
 Key: BEAM-5892
 URL: https://issues.apache.org/jira/browse/BEAM-5892
 Project: Beam
  Issue Type: Improvement
  Components: dsl-sql
Reporter: Rui Wang
Assignee: Rui Wang






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-5843) Expand Grouped Window Functions to window_start and window_end in Aggregation

2018-10-28 Thread Rui Wang (JIRA)



[ 
https://issues.apache.org/jira/browse/BEAM-5843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1719#comment-1719
 ] 

Rui Wang commented on BEAM-5843:


Due to CALCITE-2645, even we expand the window_end in GROUP BY fields, we still 
cannot establish the correct input reference for session_end.

> Expand Grouped Window Functions to window_start and window_end in Aggregation 
> --
>
> Key: BEAM-5843
> URL: https://issues.apache.org/jira/browse/BEAM-5843
> Project: Beam
>  Issue Type: Bug
>  Components: dsl-sql
>Reporter: Rui Wang
>Assignee: Rui Wang
>Priority: Major
>
> Right now, the session window_end is always equal to window_start due to 
> missing implementation window_end for grouped window functions. Should fix 
> this bug by adding window_end to BeamAggregationRel.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work stopped] (BEAM-5843) Expand Grouped Window Functions to window_start and window_end in Aggregation

2018-10-28 Thread Rui Wang (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-5843?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on BEAM-5843 stopped by Rui Wang.
--
> Expand Grouped Window Functions to window_start and window_end in Aggregation 
> --
>
> Key: BEAM-5843
> URL: https://issues.apache.org/jira/browse/BEAM-5843
> Project: Beam
>  Issue Type: Bug
>  Components: dsl-sql
>Reporter: Rui Wang
>Assignee: Rui Wang
>Priority: Major
>
> Right now, the session window_end is always equal to window_start due to 
> missing implementation window_end for grouped window functions. Should fix 
> this bug by adding window_end to BeamAggregationRel.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3573) Test jars should export only tests, and only be exported for select modules

2018-10-28 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-3573?focusedWorklogId=159824=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-159824
 ]

ASF GitHub Bot logged work on BEAM-3573:


Author: ASF GitHub Bot
Created on: 29/Oct/18 03:32
Start Date: 29/Oct/18 03:32
Worklog Time Spent: 10m 
  Work Description: kennknowles commented on issue #6838: [BEAM-3573] Move 
GCPIO test utilities to main jar
URL: https://github.com/apache/beam/pull/6838#issuecomment-433778641
 
 
   run java postcommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 159824)
Time Spent: 50m  (was: 40m)

> Test jars should export only tests, and only be exported for select modules
> ---
>
> Key: BEAM-3573
> URL: https://issues.apache.org/jira/browse/BEAM-3573
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>Priority: Major
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> Today, we have test-jars that are used as libraries for testing. That is not 
> what "test jar" means, and dependency management actually does not work 
> correctly for this. It is OK to depend on a test jar in order to run the 
> tests therein, and not really OK to depend on one for another reason.
> This ticket is a bucket ticket for fixes to this situation.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5887) packageTests and shadowTestJar write the same file

2018-10-28 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-5887?focusedWorklogId=159792=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-159792
 ]

ASF GitHub Bot logged work on BEAM-5887:


Author: ASF GitHub Bot
Created on: 29/Oct/18 01:43
Start Date: 29/Oct/18 01:43
Worklog Time Spent: 10m 
  Work Description: kennknowles closed pull request #6864: [BEAM-5887] Fix 
classifier for unshaded tests jars
URL: https://github.com/apache/beam/pull/6864
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/.test-infra/jenkins/job_PostCommit_Java_GradleBuild.groovy 
b/.test-infra/jenkins/job_PostCommit_Java_GradleBuild.groovy
index 81a47631ec1..e2b1515bad0 100644
--- a/.test-infra/jenkins/job_PostCommit_Java_GradleBuild.groovy
+++ b/.test-infra/jenkins/job_PostCommit_Java_GradleBuild.groovy
@@ -43,8 +43,6 @@ 
PostcommitJobBuilder.postCommitJob('beam_PostCommit_Java_GradleBuild', 'Run Java
   commonJobProperties.setGradleSwitches(delegate)
   // Specify maven home on Jenkins, needed by Maven archetype integration 
tests.
   switches('-Pmaven_home=/home/jenkins/tools/maven/apache-maven-3.5.2')
-  // BEAM-5035: Parallel builds are very flaky
-  switches('--no-parallel')
 }
   }
 }
diff --git 
a/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy 
b/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
index 18543ced40d..edeafb73703 100644
--- a/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
+++ b/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
@@ -608,7 +608,7 @@ class BeamModulePlugin implements Plugin {
 
   // Ensure that tests are packaged and part of the artifact set.
   project.task('packageTests', type: Jar) {
-classifier = 'tests'
+classifier = 'tests-unshaded'
 from project.sourceSets.test.output
   }
   project.artifacts.archives project.packageTests


 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 159792)
Time Spent: 1h 40m  (was: 1.5h)

> packageTests and shadowTestJar write the same file
> --
>
> Key: BEAM-5887
> URL: https://issues.apache.org/jira/browse/BEAM-5887
> Project: Beam
>  Issue Type: Bug
>  Components: build-system
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>Priority: Major
>  Time Spent: 1h 40m
>  Remaining Estimate: 0h
>
> Pointed out by [~michel], we should be packaging unshaded tests to e.g. 
> {{tests-unshaded}} classifier. This is likely the root cause of BEAM-5035, 
> BEAM-5116, BEAM-5207.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5887) packageTests and shadowTestJar write the same file

2018-10-28 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-5887?focusedWorklogId=159790=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-159790
 ]

ASF GitHub Bot logged work on BEAM-5887:


Author: ASF GitHub Bot
Created on: 29/Oct/18 01:43
Start Date: 29/Oct/18 01:43
Worklog Time Spent: 10m 
  Work Description: kennknowles commented on issue #6864: [BEAM-5887] Fix 
classifier for unshaded tests jars
URL: https://github.com/apache/beam/pull/6864#issuecomment-433764128
 
 
   I think this is clear enough to merge.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 159790)
Time Spent: 1.5h  (was: 1h 20m)

> packageTests and shadowTestJar write the same file
> --
>
> Key: BEAM-5887
> URL: https://issues.apache.org/jira/browse/BEAM-5887
> Project: Beam
>  Issue Type: Bug
>  Components: build-system
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>Priority: Major
>  Time Spent: 1.5h
>  Remaining Estimate: 0h
>
> Pointed out by [~michel], we should be packaging unshaded tests to e.g. 
> {{tests-unshaded}} classifier. This is likely the root cause of BEAM-5035, 
> BEAM-5116, BEAM-5207.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-5890) TANH

2018-10-28 Thread Rui Wang (JIRA)

Rui Wang created BEAM-5890:
--

 Summary: TANH
 Key: BEAM-5890
 URL: https://issues.apache.org/jira/browse/BEAM-5890
 Project: Beam
  Issue Type: Sub-task
  Components: dsl-sql
Reporter: Rui Wang
Assignee: Rui Wang


TANH(X)

Computes hyperbolic tangent of X. Does not fail.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-5888) ACOSH

2018-10-28 Thread Rui Wang (JIRA)

Rui Wang created BEAM-5888:
--

 Summary: ACOSH
 Key: BEAM-5888
 URL: https://issues.apache.org/jira/browse/BEAM-5888
 Project: Beam
  Issue Type: Sub-task
  Components: dsl-sql
Reporter: Rui Wang
Assignee: Rui Wang


ACOSH(X)

Computes the inverse hyperbolic cosine of X. Generates an error if X is a 
finite value less than 1.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5887) packageTests and shadowTestJar write the same file

2018-10-28 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-5887?focusedWorklogId=159782=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-159782
 ]

ASF GitHub Bot logged work on BEAM-5887:


Author: ASF GitHub Bot
Created on: 29/Oct/18 00:03
Start Date: 29/Oct/18 00:03
Worklog Time Spent: 10m 
  Work Description: adude3141 commented on issue #6864: [BEAM-5887] Fix 
classifier for unshaded tests jars
URL: https://github.com/apache/beam/pull/6864#issuecomment-433754989
 
 
   Ah... no sorry. forgot to mention:
   
   I ve also that change on google cloud data flow worker thingy switching to 
osx libraries, as otherwise it will bark at me on unsatisfied links.
   
   But that should not have any impact on this issue at all.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 159782)
Time Spent: 1h 20m  (was: 1h 10m)

> packageTests and shadowTestJar write the same file
> --
>
> Key: BEAM-5887
> URL: https://issues.apache.org/jira/browse/BEAM-5887
> Project: Beam
>  Issue Type: Bug
>  Components: build-system
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>Priority: Major
>  Time Spent: 1h 20m
>  Remaining Estimate: 0h
>
> Pointed out by [~michel], we should be packaging unshaded tests to e.g. 
> {{tests-unshaded}} classifier. This is likely the root cause of BEAM-5035, 
> BEAM-5116, BEAM-5207.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5887) packageTests and shadowTestJar write the same file

2018-10-28 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-5887?focusedWorklogId=159781=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-159781
 ]

ASF GitHub Bot logged work on BEAM-5887:


Author: ASF GitHub Bot
Created on: 28/Oct/18 23:58
Start Date: 28/Oct/18 23:58
Worklog Time Spent: 10m 
  Work Description: adude3141 commented on issue #6864: [BEAM-5887] Fix 
classifier for unshaded tests jars
URL: https://github.com/apache/beam/pull/6864#issuecomment-433754633
 
 
   LGTM
   
   @kennknowles 
   I can confirm that
   a: The change in commit '53753c8' is the only change I did to get to the 
improvements with incremental build (i.e. rebuilding time without any changes 
cut to 50% on my machine)
   b: after applying that change I was able to enable parallel build (excluding 
python/go/website which still cause me headaches), whereas without that fix I m 
required to disable parallel build by adding option '--no-parallel' 
   
   Of course, that's not a proper testing so should be seen as a hint to 
improvement at best.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 159781)
Time Spent: 1h 10m  (was: 1h)

> packageTests and shadowTestJar write the same file
> --
>
> Key: BEAM-5887
> URL: https://issues.apache.org/jira/browse/BEAM-5887
> Project: Beam
>  Issue Type: Bug
>  Components: build-system
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>Priority: Major
>  Time Spent: 1h 10m
>  Remaining Estimate: 0h
>
> Pointed out by [~michel], we should be packaging unshaded tests to e.g. 
> {{tests-unshaded}} classifier. This is likely the root cause of BEAM-5035, 
> BEAM-5116, BEAM-5207.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5887) packageTests and shadowTestJar write the same file

2018-10-28 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-5887?focusedWorklogId=159780=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-159780
 ]

ASF GitHub Bot logged work on BEAM-5887:


Author: ASF GitHub Bot
Created on: 28/Oct/18 22:54
Start Date: 28/Oct/18 22:54
Worklog Time Spent: 10m 
  Work Description: kennknowles edited a comment on issue #6864: 
[BEAM-5887] Fix classifier for unshaded tests jars
URL: https://github.com/apache/beam/pull/6864#issuecomment-433750070
 
 
   First, this is clearly a bugfix whether or not it works (so only my second 
commit is questionable). We shouldn't be writing unshaded artifacts to 
classifiers that don't scream `unshaded`. Ultimately "unshaded" sort of means 
"funky invalid artifact, use at your own risk".
   
   Now, actually trying to verify: It is crude, but I ran `/gradlew 
:beam-sdks-java-core:shadowTestJar :beam-sdks-java-core:packageTests` on master 
and on this PR with `fswatch sdks/java/core/build | grep jar` in another shell.
   
   `master`:
   
   ```
   sdks/java/core/build/libs/beam-sdks-java-core-2.9.0-SNAPSHOT-tests.jar
   sdks/java/core/build/libs/beam-sdks-java-core-2.9.0-SNAPSHOT-tests.jar
   sdks/java/core/build/libs/beam-sdks-java-core-2.9.0-SNAPSHOT-tests.jar
   ```
   
   This PR:
   
   ```
   sdks/java/core/build/libs/beam-sdks-java-core-2.9.0-SNAPSHOT-tests.jar
   sdks/java/core/build/libs/beam-sdks-java-core-2.9.0-SNAPSHOT-tests.jar
   
sdks/java/core/build/libs/beam-sdks-java-core-2.9.0-SNAPSHOT-tests-unshaded.jar
   ```
   
   It doesn't tell me much; I did not work super hard to associate the timings 
with tasks, etc.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 159780)
Time Spent: 1h  (was: 50m)

> packageTests and shadowTestJar write the same file
> --
>
> Key: BEAM-5887
> URL: https://issues.apache.org/jira/browse/BEAM-5887
> Project: Beam
>  Issue Type: Bug
>  Components: build-system
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>Priority: Major
>  Time Spent: 1h
>  Remaining Estimate: 0h
>
> Pointed out by [~michel], we should be packaging unshaded tests to e.g. 
> {{tests-unshaded}} classifier. This is likely the root cause of BEAM-5035, 
> BEAM-5116, BEAM-5207.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5887) packageTests and shadowTestJar write the same file

2018-10-28 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-5887?focusedWorklogId=159779=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-159779
 ]

ASF GitHub Bot logged work on BEAM-5887:


Author: ASF GitHub Bot
Created on: 28/Oct/18 22:52
Start Date: 28/Oct/18 22:52
Worklog Time Spent: 10m 
  Work Description: kennknowles edited a comment on issue #6864: 
[BEAM-5887] Fix classifier for unshaded tests jars
URL: https://github.com/apache/beam/pull/6864#issuecomment-433750070
 
 
   First, this is clearly a bugfix whether or not it works (so only my second 
commit is questionable). We shouldn't be writing unshaded artifacts to 
classifiers that don't scream `unshaded`. Ultimately "unshaded" sort of means 
"funky invalid artifact, use at your own risk".
   
   Now, actually trying to verify: It is crude, but I ran `/gradlew 
:beam-sdks-java-core:shadowTestJar :beam-sdks-java-core:packageTests` on master 
and on this PR with `fswatch sdks/java/core/build | grep jar` in another shell.
   
   `master`:
   
   ```
   sdks/java/core/build/libs/beam-sdks-java-core-2.9.0-SNAPSHOT-tests.jar
   sdks/java/core/build/libs/beam-sdks-java-core-2.9.0-SNAPSHOT-tests.jar
   sdks/java/core/build/libs/beam-sdks-java-core-2.9.0-SNAPSHOT-tests.jar
   ```
   
   This PR:
   
   ```
   sdks/java/core/build/libs/beam-sdks-java-core-2.9.0-SNAPSHOT-tests.jar
   sdks/java/core/build/libs/beam-sdks-java-core-2.9.0-SNAPSHOT-tests.jar
   
sdks/java/core/build/libs/beam-sdks-java-core-2.9.0-SNAPSHOT-tests-unshaded.jar
   ```


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 159779)
Time Spent: 50m  (was: 40m)

> packageTests and shadowTestJar write the same file
> --
>
> Key: BEAM-5887
> URL: https://issues.apache.org/jira/browse/BEAM-5887
> Project: Beam
>  Issue Type: Bug
>  Components: build-system
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>Priority: Major
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> Pointed out by [~michel], we should be packaging unshaded tests to e.g. 
> {{tests-unshaded}} classifier. This is likely the root cause of BEAM-5035, 
> BEAM-5116, BEAM-5207.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5887) packageTests and shadowTestJar write the same file

2018-10-28 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-5887?focusedWorklogId=159778=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-159778
 ]

ASF GitHub Bot logged work on BEAM-5887:


Author: ASF GitHub Bot
Created on: 28/Oct/18 22:52
Start Date: 28/Oct/18 22:52
Worklog Time Spent: 10m 
  Work Description: kennknowles commented on issue #6864: [BEAM-5887] Fix 
classifier for unshaded tests jars
URL: https://github.com/apache/beam/pull/6864#issuecomment-433750070
 
 
   First, this is clearly a bugfix whether or not it works (so only my second 
commit is questionable). We shouldn't be writing unshaded artifacts to 
classifiers that don't scream `unshaded`. Ultimately "unshaded" sort of means 
"funky invalid artifact, use at your own risk".
   
   Now, actually trying to verify: It is crude, but I ran `/gradlew 
:beam-sdks-java-core:shadowTestJar :beam-sdks-java-core:packageTests` on master 
and on this PR with `fswatch sdks/java/core/build | grep jar` in another shell.
   
   `master`:
   
   ```
   
/Users/klk/GitHub/apache/beam/sdks/java/core/build/libs/beam-sdks-java-core-2.9.0-SNAPSHOT-tests.jar
   
/Users/klk/GitHub/apache/beam/sdks/java/core/build/libs/beam-sdks-java-core-2.9.0-SNAPSHOT-tests.jar
   
/Users/klk/GitHub/apache/beam/sdks/java/core/build/libs/beam-sdks-java-core-2.9.0-SNAPSHOT-tests.jar
   ```
   
   This PR:
   
   ```
   
/Users/klk/GitHub/apache/beam/sdks/java/core/build/libs/beam-sdks-java-core-2.9.0-SNAPSHOT-tests.jar
   
/Users/klk/GitHub/apache/beam/sdks/java/core/build/libs/beam-sdks-java-core-2.9.0-SNAPSHOT-tests.jar
   
/Users/klk/GitHub/apache/beam/sdks/java/core/build/libs/beam-sdks-java-core-2.9.0-SNAPSHOT-tests-unshaded.jar
   ```


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 159778)
Time Spent: 40m  (was: 0.5h)

> packageTests and shadowTestJar write the same file
> --
>
> Key: BEAM-5887
> URL: https://issues.apache.org/jira/browse/BEAM-5887
> Project: Beam
>  Issue Type: Bug
>  Components: build-system
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>Priority: Major
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> Pointed out by [~michel], we should be packaging unshaded tests to e.g. 
> {{tests-unshaded}} classifier. This is likely the root cause of BEAM-5035, 
> BEAM-5116, BEAM-5207.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5887) packageTests and shadowTestJar write the same file

2018-10-28 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-5887?focusedWorklogId=159777=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-159777
 ]

ASF GitHub Bot logged work on BEAM-5887:


Author: ASF GitHub Bot
Created on: 28/Oct/18 22:32
Start Date: 28/Oct/18 22:32
Worklog Time Spent: 10m 
  Work Description: swegner commented on issue #6864: [BEAM-5887] Fix 
classifier for unshaded tests jars
URL: https://github.com/apache/beam/pull/6864#issuecomment-433747952
 
 
   I'm very excited about this. Any thoughts on how we can validate that this 
actually fixed the flakiness? The linked JIRA mentions using 
[inotifywatch](https://linux.die.net/man/1/inotifywatch)


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 159777)
Time Spent: 0.5h  (was: 20m)

> packageTests and shadowTestJar write the same file
> --
>
> Key: BEAM-5887
> URL: https://issues.apache.org/jira/browse/BEAM-5887
> Project: Beam
>  Issue Type: Bug
>  Components: build-system
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>Priority: Major
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> Pointed out by [~michel], we should be packaging unshaded tests to e.g. 
> {{tests-unshaded}} classifier. This is likely the root cause of BEAM-5035, 
> BEAM-5116, BEAM-5207.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5887) packageTests and shadowTestJar write the same file

2018-10-28 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-5887?focusedWorklogId=159776=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-159776
 ]

ASF GitHub Bot logged work on BEAM-5887:


Author: ASF GitHub Bot
Created on: 28/Oct/18 22:29
Start Date: 28/Oct/18 22:29
Worklog Time Spent: 10m 
  Work Description: kennknowles commented on issue #6864: [BEAM-5887] Fix 
classifier for unshaded tests jars
URL: https://github.com/apache/beam/pull/6864#issuecomment-433747598
 
 
   R: @adude3141 
   
   If you give me an LGTM that this is all you did to make the improvement, 
I'll merge it.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 159776)
Time Spent: 20m  (was: 10m)

> packageTests and shadowTestJar write the same file
> --
>
> Key: BEAM-5887
> URL: https://issues.apache.org/jira/browse/BEAM-5887
> Project: Beam
>  Issue Type: Bug
>  Components: build-system
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>Priority: Major
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> Pointed out by [~michel], we should be packaging unshaded tests to e.g. 
> {{tests-unshaded}} classifier. This is likely the root cause of BEAM-5035, 
> BEAM-5116, BEAM-5207.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-5868) Row equality does not work for java data structures (e.g. List).

2018-10-28 Thread Rui Wang (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-5868?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rui Wang updated BEAM-5868:
---
Description: 
In current implementation, Row's equals uses Objects.deepEquals, which only 
deeply compare array. For example, List will not be deeply compared because 
this function will just call List.equals.

Map might not have the correct equals as well. 


If there is no other option,  we might need to have a customized deep equal to 
handle all cases we have. 

  was:
In current implementation, Row's equals uses Objects.deepEquals, which only 
deeply compare array. For example, List will be deeply compared because this 
function will just call List.equals.

Map might not have the correct equals as well. 


If there is no other option,  we might need to have a customized deep equal to 
handle all cases we have. 


> Row equality does not work for java data structures (e.g. List).
> 
>
> Key: BEAM-5868
> URL: https://issues.apache.org/jira/browse/BEAM-5868
> Project: Beam
>  Issue Type: Bug
>  Components: dsl-sql
>Reporter: Rui Wang
>Priority: Major
>
> In current implementation, Row's equals uses Objects.deepEquals, which only 
> deeply compare array. For example, List will not be deeply compared because 
> this function will just call List.equals.
> Map might not have the correct equals as well. 
> If there is no other option,  we might need to have a customized deep equal 
> to handle all cases we have. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5884) Allow nested types have null value.

2018-10-28 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-5884?focusedWorklogId=159775=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-159775
 ]

ASF GitHub Bot logged work on BEAM-5884:


Author: ASF GitHub Bot
Created on: 28/Oct/18 21:48
Start Date: 28/Oct/18 21:48
Worklog Time Spent: 10m 
  Work Description: amaliujia edited a comment on issue #6861: [BEAM-5884] 
support null values in Array and Map.
URL: https://github.com/apache/beam/pull/6861#issuecomment-433744006
 
 
   Yes. It is adding `ARRAY>`, and `NULLABLE>` exists in 
current Beam implementation as a nullable flag of `Field`.
   
   The reason I think container type should has `nullable`, is there is a 
difference between `NULLABLE>` (which is a flag in `Array Field`) and 
`ARRAY>` (which is flag for container type `T`).  
   
   It's two difference concepts:
   1. `ARRAY` as a field (or type) can be `null`. E.g. `a null array`.
   2. `ARRAY element type` can be `null`. E.g. `array(1, 2, null, 3, 
null)`.
   
   So there could be a `not nullable ARRAY` field with a `nullable element 
type`, which is not a conflict.
   
   I also think both nullable flag has use cases:
   1. messages ingested from message queue contains `null` for `Array Field`.
   2. In the intermediate stages, it generates `null` for `array element`.
   
   In this PR, I make `not nullable` as the default setting for `nullable` of 
`content/element type`.
   
   
   I added `nullable` for `Map`'s `value type` for the same reason.
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 159775)
Time Spent: 3h  (was: 2h 50m)

> Allow nested types have null value.
> ---
>
> Key: BEAM-5884
> URL: https://issues.apache.org/jira/browse/BEAM-5884
> Project: Beam
>  Issue Type: Bug
>  Components: dsl-sql
>Reporter: Rui Wang
>Assignee: Rui Wang
>Priority: Major
>  Time Spent: 3h
>  Remaining Estimate: 0h
>
> We could allow arbitrary combination of nested types have null value.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5884) Allow nested types have null value.

2018-10-28 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-5884?focusedWorklogId=159774=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-159774
 ]

ASF GitHub Bot logged work on BEAM-5884:


Author: ASF GitHub Bot
Created on: 28/Oct/18 21:44
Start Date: 28/Oct/18 21:44
Worklog Time Spent: 10m 
  Work Description: amaliujia edited a comment on issue #6861: [BEAM-5884] 
support null values in Array and Map.
URL: https://github.com/apache/beam/pull/6861#issuecomment-433744006
 
 
   Yes. It is adding `ARRAY>`, and `NULLABLE>` exists in 
current Beam implementation as a nullable flag of `Field`.
   
   The reason I think container type should has `nullable`, is there is a 
difference between `NULLABLE>` (which is a flag in `Array Field`) and 
`ARRAY>` (which is flag for container type `T`).  
   
   It's two difference concepts:
   1. `ARRAY` as a field (or type) can be `null`. E.g. `a null array`.
   2. `ARRAY element type` can be `null`. E.g. `array(1, 2, null, 3, 
null)`.
   
   So there could be a `not nullable ARRAY` field with a `nullable element 
type`, which is not a conflict.
   
   I also think both nullable flag has use cases:
   1. messages ingested from message queue contains `null` for `Array Field`.
   2. In the intermediate stages, it generates `null` for `array element`.
   
   In this PR, I make `not nullable` as the default setting.
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 159774)
Time Spent: 2h 50m  (was: 2h 40m)

> Allow nested types have null value.
> ---
>
> Key: BEAM-5884
> URL: https://issues.apache.org/jira/browse/BEAM-5884
> Project: Beam
>  Issue Type: Bug
>  Components: dsl-sql
>Reporter: Rui Wang
>Assignee: Rui Wang
>Priority: Major
>  Time Spent: 2h 50m
>  Remaining Estimate: 0h
>
> We could allow arbitrary combination of nested types have null value.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5884) Allow nested types have null value.

2018-10-28 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-5884?focusedWorklogId=159773=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-159773
 ]

ASF GitHub Bot logged work on BEAM-5884:


Author: ASF GitHub Bot
Created on: 28/Oct/18 21:43
Start Date: 28/Oct/18 21:43
Worklog Time Spent: 10m 
  Work Description: amaliujia edited a comment on issue #6861: [BEAM-5884] 
support null values in Array and Map.
URL: https://github.com/apache/beam/pull/6861#issuecomment-433744006
 
 
   Yes. It is adding `ARRAY>`.
   
   The reason I think container type should has `nullable`, is there is a 
difference between `NULLABLE>` (which is a flag in `Array Field`) and 
`ARRAY>` (which is flag for container type `T`).  
   
   It's two difference concepts:
   1. `ARRAY` as a field (or type) can be `null`. E.g. `a null array`.
   2. `ARRAY element type` can be `null`. E.g. `array(1, 2, null, 3, 
null)`.
   
   So there could be a `not nullable ARRAY` field with a `nullable element 
type`, which is not a conflict.
   
   I also think both nullable flag has use cases:
   1. messages ingested from message queue contains `null` for `array field`.
   2. In the intermediate stages, it generates `null` for `array element`.
   
   In this PR, I make `not nullable` as the default setting.
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 159773)
Time Spent: 2h 40m  (was: 2.5h)

> Allow nested types have null value.
> ---
>
> Key: BEAM-5884
> URL: https://issues.apache.org/jira/browse/BEAM-5884
> Project: Beam
>  Issue Type: Bug
>  Components: dsl-sql
>Reporter: Rui Wang
>Assignee: Rui Wang
>Priority: Major
>  Time Spent: 2h 40m
>  Remaining Estimate: 0h
>
> We could allow arbitrary combination of nested types have null value.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5884) Allow nested types have null value.

2018-10-28 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-5884?focusedWorklogId=159772=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-159772
 ]

ASF GitHub Bot logged work on BEAM-5884:


Author: ASF GitHub Bot
Created on: 28/Oct/18 21:41
Start Date: 28/Oct/18 21:41
Worklog Time Spent: 10m 
  Work Description: amaliujia edited a comment on issue #6861: [BEAM-5884] 
support null values in Array and Map.
URL: https://github.com/apache/beam/pull/6861#issuecomment-433744006
 
 
   Yes. It is adding `ARRAY>`.
   
   The reason I think container type should has `nullable`, is there is a 
difference between `NULLABLE>` (which is a flag in `Array Field`) and 
`ARRAY>` (which is flag for container type `T`).  
   
   It's two difference concepts:
   1. `ARRAY` as a field (or type) can be `null`. E.g. `a null array`.
   2. `ARRAY element type` can be `null`. E.g. `array(1, 2, null, 3, 
null)`.
   
   So there could be a `not nullable ARRAY` field with a `nullable element 
type`, which is not a conflict.
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 159772)
Time Spent: 2.5h  (was: 2h 20m)

> Allow nested types have null value.
> ---
>
> Key: BEAM-5884
> URL: https://issues.apache.org/jira/browse/BEAM-5884
> Project: Beam
>  Issue Type: Bug
>  Components: dsl-sql
>Reporter: Rui Wang
>Assignee: Rui Wang
>Priority: Major
>  Time Spent: 2.5h
>  Remaining Estimate: 0h
>
> We could allow arbitrary combination of nested types have null value.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5884) Allow nested types have null value.

2018-10-28 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-5884?focusedWorklogId=159771=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-159771
 ]

ASF GitHub Bot logged work on BEAM-5884:


Author: ASF GitHub Bot
Created on: 28/Oct/18 21:40
Start Date: 28/Oct/18 21:40
Worklog Time Spent: 10m 
  Work Description: amaliujia edited a comment on issue #6861: [BEAM-5884] 
support null values in Array and Map.
URL: https://github.com/apache/beam/pull/6861#issuecomment-433744006
 
 
   Yes. It is adding `ARRAY>`.
   
   The reason I think container type should has `nullable`, is there is a 
difference between `NULLABLE>` (which is a flag in `Array Field`) and 
`ARRAY>` (which is flag for container type `T`).  
   
   It's two difference concepts:
   1. `ARRAY` as a field (or type) can be `null`. E.g. `a null array`.
   2. `ARRAY element type` can be `null`. E.g. `array(1, 2, null, 3, 
null)`.
   
   So there could be a `not nullable ARRAY` field with a `nullable content 
type`, which is not a conflict.
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 159771)
Time Spent: 2h 20m  (was: 2h 10m)

> Allow nested types have null value.
> ---
>
> Key: BEAM-5884
> URL: https://issues.apache.org/jira/browse/BEAM-5884
> Project: Beam
>  Issue Type: Bug
>  Components: dsl-sql
>Reporter: Rui Wang
>Assignee: Rui Wang
>Priority: Major
>  Time Spent: 2h 20m
>  Remaining Estimate: 0h
>
> We could allow arbitrary combination of nested types have null value.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5884) Allow nested types have null value.

2018-10-28 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-5884?focusedWorklogId=159770=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-159770
 ]

ASF GitHub Bot logged work on BEAM-5884:


Author: ASF GitHub Bot
Created on: 28/Oct/18 21:40
Start Date: 28/Oct/18 21:40
Worklog Time Spent: 10m 
  Work Description: amaliujia commented on issue #6861: [BEAM-5884] support 
null values in Array and Map.
URL: https://github.com/apache/beam/pull/6861#issuecomment-433744006
 
 
   Yes. It is adding `ARRAY>`.
   
   The reason I think container type should has `nullable`, is there is a 
difference between `NULLABLE>` (which is a flag in `Array Field`) and 
`ARRAY>` (which is flag for container type `T`).  
   
   It's two difference concepts:
   1. `ARRAY` as a field (or type) can be `null`. E.g. a null array`.
   2. `ARRAY element type` can be `null`. E.g. `array(1, 2, null, 3, 
null)`.
   
   So there could be a `not nullable ARRAY` field with a `nullable content 
type`, which is not a conflict.
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 159770)
Time Spent: 2h 10m  (was: 2h)

> Allow nested types have null value.
> ---
>
> Key: BEAM-5884
> URL: https://issues.apache.org/jira/browse/BEAM-5884
> Project: Beam
>  Issue Type: Bug
>  Components: dsl-sql
>Reporter: Rui Wang
>Assignee: Rui Wang
>Priority: Major
>  Time Spent: 2h 10m
>  Remaining Estimate: 0h
>
> We could allow arbitrary combination of nested types have null value.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5886) Nexmark failing due to IllegalStateException

2018-10-28 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-5886?focusedWorklogId=159769=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-159769
 ]

ASF GitHub Bot logged work on BEAM-5886:


Author: ASF GitHub Bot
Created on: 28/Oct/18 21:36
Start Date: 28/Oct/18 21:36
Worklog Time Spent: 10m 
  Work Description: lgajowy closed pull request #6863: [BEAM-5886] Fix 
incorrectly formulated condition in checkState method
URL: https://github.com/apache/beam/pull/6863
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git 
a/sdks/java/testing/test-utils/src/main/java/org/apache/beam/sdk/testutils/metrics/MetricsReader.java
 
b/sdks/java/testing/test-utils/src/main/java/org/apache/beam/sdk/testutils/metrics/MetricsReader.java
index 77dccc03261..7dd8bcb5785 100644
--- 
a/sdks/java/testing/test-utils/src/main/java/org/apache/beam/sdk/testutils/metrics/MetricsReader.java
+++ 
b/sdks/java/testing/test-utils/src/main/java/org/apache/beam/sdk/testutils/metrics/MetricsReader.java
@@ -124,9 +124,13 @@ private long getDistributionMetric(String name, 
DistributionType distType, long
   private  void checkIfMetricResultIsUnique(String name, 
Iterable> metricResult)
   throws IllegalStateException {
 
+int resultCount = Iterables.size(metricResult);
 Preconditions.checkState(
-Iterables.size(metricResult) == 1,
-String.format("More than one metric matches name: %s in namespace 
%s.", name, namespace));
+resultCount <= 1,
+"More than one metric result matches name: %s in namespace %s. Metric 
results count: %s",
+name,
+namespace,
+resultCount);
   }
 
   /** Return the current value for a time counter, or -1 if can't be 
retrieved. */
diff --git 
a/sdks/java/testing/test-utils/src/test/java/org/apache/beam/sdk/testutils/metrics/MetricsReaderTest.java
 
b/sdks/java/testing/test-utils/src/test/java/org/apache/beam/sdk/testutils/metrics/MetricsReaderTest.java
index 521317f2665..d0ad4a74524 100644
--- 
a/sdks/java/testing/test-utils/src/test/java/org/apache/beam/sdk/testutils/metrics/MetricsReaderTest.java
+++ 
b/sdks/java/testing/test-utils/src/test/java/org/apache/beam/sdk/testutils/metrics/MetricsReaderTest.java
@@ -79,14 +79,11 @@ public void testEndTimeIsTheMaximumOfTheDistribution() {
 assertEquals(5, reader.getEndTimeMetric(0, "timeDist"));
   }
 
-  @Test(expected = IllegalStateException.class)
-  public void 
throwsIllegalStateExceptionWhenThereAreMultipleCountersOfTheSameNameAndType() {
-Metrics.counter(NAMESPACE, "counter");
-Metrics.counter(NAMESPACE, "counter");
-
+  @Test
+  public void doesntThrowIllegalStateExceptionWhenThereIsNoMetricFound() {
 PipelineResult result = testPipeline.run();
 MetricsReader reader = new MetricsReader(result, NAMESPACE);
-reader.getCounterMetric("counter", -1);
+reader.getCounterMetric("nonexistent", -1);
   }
 
   @Test


 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 159769)
Time Spent: 1h 40m  (was: 1.5h)

> Nexmark failing due to IllegalStateException
> 
>
> Key: BEAM-5886
> URL: https://issues.apache.org/jira/browse/BEAM-5886
> Project: Beam
>  Issue Type: Task
>  Components: test-failures, testing
>Reporter: Lukasz Gajowy
>Assignee: Lukasz Gajowy
>Priority: Major
>  Time Spent: 1h 40m
>  Remaining Estimate: 0h
>
> Nexmark fails due to the following error:
>  
> {code:java}
> java.lang.RuntimeException: java.lang.IllegalStateException: More than one 
> metric matches name: fatal in namespace Query0.
> 13:05:49   0012  *** not run ***
> 13:05:49 
> ==
> 13:05:49  at org.apache.beam.sdk.nexmark.Main.runAll(Main.java:144)
> 13:05:49 
> 13:05:49  at org.apache.beam.sdk.nexmark.Main.main(Main.java:477)
> 13:05:49 Caused by: java.lang.IllegalStateException: More than one metric 
> matches name: fatal in namespace Query0.
> 13:05:49  at 
> org.apache.beam.repackaged.beam_sdks_java_test_utils.com.google.common.base.Preconditions.checkState(Preconditions.java:444)
> 13:05:49  at 
>

[jira] [Work logged] (BEAM-5265) Can not test Timer with processing time domain

2018-10-28 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-5265?focusedWorklogId=159766=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-159766
 ]

ASF GitHub Bot logged work on BEAM-5265:


Author: ASF GitHub Bot
Created on: 28/Oct/18 20:46
Start Date: 28/Oct/18 20:46
Worklog Time Spent: 10m 
  Work Description: kennknowles commented on a change in pull request 
#6305: [BEAM-5265] Use currentProcessingTime() for onTime with processing time 
domain
URL: https://github.com/apache/beam/pull/6305#discussion_r228764446
 
 

 ##
 File path: 
runners/core-java/src/test/java/org/apache/beam/runners/core/SimpleDoFnRunnerTest.java
 ##
 @@ -198,44 +218,185 @@ public void 
testFinishBundleExceptionsWrappedAsUserCodeException() {
   }
 
   /**
-   * Tests that {@link SimpleDoFnRunner#onTimer} properly dispatches to the 
underlying {@link DoFn}.
+   * Tests that {@link SimpleDoFnRunner#onTimer} properly dispatches to the 
underlying {@link DoFn}
+   * on appropriate time domains.
*/
   @Test
-  public void testOnTimerCalled() {
+  @Category(NeedsRunner.class)
+  public void testOnTimerCalledWithGlobalWindow() {
+
+// TIMESTAMP_MIN_VALUE is initial value for processing time used done by 
TestClock
+Instant currentProcessingTime = BoundedWindow.TIMESTAMP_MIN_VALUE;
+Instant currentEventTime = new Instant(42);
+
+TestStream> testStream =
+TestStream.create(KvCoder.of(StringUtf8Coder.of(), 
StringUtf8Coder.of()))
+.advanceWatermarkTo(currentEventTime)
+.addElements(TimestampedValue.of(KV.of("anyKey", "anyValue"), new 
Instant(99)))
+.advanceProcessingTime(DoFnWithTimers.TIMER_OFFSET.plus(1))
+.advanceWatermarkToInfinity();
+
 WindowFn windowFn = new GlobalWindows();
-DoFnWithTimers fn = new 
DoFnWithTimers(windowFn.windowCoder());
-DoFnRunner runner =
-new SimpleDoFnRunner<>(
-null,
-fn,
-NullSideInputReader.empty(),
-null,
-null,
-Collections.emptyList(),
-mockStepContext,
-null,
-Collections.emptyMap(),
-WindowingStrategy.of(windowFn));
+DoFnWithTimers fn = new 
DoFnWithTimers<>(windowFn.windowCoder());
 
-Instant currentTime = new Instant(42);
-Duration offset = Duration.millis(37);
+PCollection output =
+pipeline
+.apply(testStream)
+.apply(Window.into(new GlobalWindows()))
+.apply(ParDo.of(fn))
+
.setCoder(TimerInternals.TimerDataCoder.of(windowFn.windowCoder()));
 
-// Mocking is not easily compatible with annotation analysis, so we 
manually record
-// the method call.
-runner.onTimer(
-DoFnWithTimers.TIMER_ID,
-GlobalWindow.INSTANCE,
-currentTime.plus(offset),
-TimeDomain.EVENT_TIME);
-
-assertThat(
-fn.onTimerInvocations,
-contains(
+PAssert.that(output)
+.containsInAnyOrder(
 TimerData.of(
-DoFnWithTimers.TIMER_ID,
+DoFnWithTimers.PROCESSING_TIMER_ID,
 StateNamespaces.window(windowFn.windowCoder(), 
GlobalWindow.INSTANCE),
-currentTime.plus(offset),
-TimeDomain.EVENT_TIME)));
+
currentProcessingTime.plus(DoFnWithTimers.TIMER_OFFSET).plus(1),
+TimeDomain.PROCESSING_TIME),
+TimerData.of(
+DoFnWithTimers.EVENT_TIMER_ID,
+StateNamespaces.window(windowFn.windowCoder(), 
GlobalWindow.INSTANCE),
+currentEventTime.plus(DoFnWithTimers.TIMER_OFFSET),
+TimeDomain.EVENT_TIME));
+
+pipeline.run();
+  }
+
+  /**
+   * Tests that {@link SimpleDoFnRunner#onTimer} properly dispatches to the 
underlying {@link DoFn}
+   * on appropriate time domains. With {@link IntervalWindow}, we check 
behavior of emitted events
+   * when time is inside and outside of window boundaries.
+   */
+  @Test
+  @Category(NeedsRunner.class)
+  public void testOnTimerCalledWithIntervalWindow() {
+
+// TIMESTAMP_MIN_VALUE is initial value for processing time used done by 
TestClock
+Instant baseTime = new Instant(0);
+
+Duration windowDuration = Duration.standardHours(1);
+Duration windowLateness = Duration.standardMinutes(1);
+IntervalWindow window = new IntervalWindow(baseTime, windowDuration);
+FixedWindows windowFn = FixedWindows.of(windowDuration);
+DoFnWithTimers fn = new 
DoFnWithTimers<>(windowFn.windowCoder());
+
+TimestampedValue> event =
+TimestampedValue.of(KV.of("anyKey", "anyValue"), window.start());
+
+TestStream> testStream =
+TestStream.create(KvCoder.of(StringUtf8Coder.of(), 
StringUtf8Coder.of()))
+// watermark in window, processing time far behind
+

[jira] [Work logged] (BEAM-5887) packageTests and shadowTestJar write the same file

2018-10-28 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-5887?focusedWorklogId=159759=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-159759
 ]

ASF GitHub Bot logged work on BEAM-5887:


Author: ASF GitHub Bot
Created on: 28/Oct/18 20:35
Start Date: 28/Oct/18 20:35
Worklog Time Spent: 10m 
  Work Description: kennknowles opened a new pull request #6864: 
[BEAM-5887] Fix classifier for unshaded tests jars
URL: https://github.com/apache/beam/pull/6864
 
 
   Unshaded and shaded tests are being written to the same classifier. This 
fixes that and restores parallel builds.
   
   
   
   Follow this checklist to help us incorporate your contribution quickly and 
easily:
   
- [x] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue, if applicable. This will automatically link the pull request to the 
issue.
- [x] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   It will help us expedite review of your Pull Request if you tag someone 
(e.g. `@username`) to look at it.
   
   Post-Commit Tests Status (on master branch)
   

   
   Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
   --- | --- | --- | --- | --- | --- | --- | ---
   Go | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/)
 | --- | --- | --- | --- | --- | ---
   Java | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/)
 [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/)
   Python | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/)
 | --- | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)
  [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Python_VR_Flink/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python_VR_Flink/lastCompletedBuild/)
 | --- | --- | ---
   
   
   
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 159759)
Time Spent: 10m
Remaining Estimate: 0h

> packageTests and shadowTestJar write the same file
>

[jira] [Created] (BEAM-5887) packageTests and shadowTestJar write the same file

2018-10-28 Thread Kenneth Knowles (JIRA)

Kenneth Knowles created BEAM-5887:
-

 Summary: packageTests and shadowTestJar write the same file
 Key: BEAM-5887
 URL: https://issues.apache.org/jira/browse/BEAM-5887
 Project: Beam
  Issue Type: Bug
  Components: build-system
Reporter: Kenneth Knowles
Assignee: Kenneth Knowles


Pointed out by [~michel], we should be packaging unshaded tests to e.g. 
{{tests-unshaded}} classifier. This is likely the root cause of BEAM-5035, 
BEAM-5116, BEAM-5207.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-3573) Test jars should export only tests, and only be exported for select modules

2018-10-28 Thread Kenneth Knowles (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-3573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-3573:
--
Summary: Test jars should export only tests, and only be exported for 
select modules  (was: Test jars should export only tests)

> Test jars should export only tests, and only be exported for select modules
> ---
>
> Key: BEAM-3573
> URL: https://issues.apache.org/jira/browse/BEAM-3573
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>Priority: Major
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> Today, we have test-jars that are used as libraries for testing. That is not 
> what "test jar" means, and dependency management actually does not work 
> correctly for this. It is OK to depend on a test jar in order to run the 
> tests therein, and not really OK to depend on one for another reason.
> This ticket is a bucket ticket for fixes to this situation.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5886) Nexmark failing due to IllegalStateException

2018-10-28 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-5886?focusedWorklogId=159758=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-159758
 ]

ASF GitHub Bot logged work on BEAM-5886:


Author: ASF GitHub Bot
Created on: 28/Oct/18 20:29
Start Date: 28/Oct/18 20:29
Worklog Time Spent: 10m 
  Work Description: kennknowles commented on issue #6863: [BEAM-5886] Fix 
incorrectly formulated condition in checkState method
URL: https://github.com/apache/beam/pull/6863#issuecomment-433738669
 
 
   When tests pass, looks like this is a squash and merge case, yea?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 159758)
Time Spent: 1h 20m  (was: 1h 10m)

> Nexmark failing due to IllegalStateException
> 
>
> Key: BEAM-5886
> URL: https://issues.apache.org/jira/browse/BEAM-5886
> Project: Beam
>  Issue Type: Task
>  Components: test-failures, testing
>Reporter: Lukasz Gajowy
>Assignee: Lukasz Gajowy
>Priority: Major
>  Time Spent: 1h 20m
>  Remaining Estimate: 0h
>
> Nexmark fails due to the following error:
>  
> {code:java}
> java.lang.RuntimeException: java.lang.IllegalStateException: More than one 
> metric matches name: fatal in namespace Query0.
> 13:05:49   0012  *** not run ***
> 13:05:49 
> ==
> 13:05:49  at org.apache.beam.sdk.nexmark.Main.runAll(Main.java:144)
> 13:05:49 
> 13:05:49  at org.apache.beam.sdk.nexmark.Main.main(Main.java:477)
> 13:05:49 Caused by: java.lang.IllegalStateException: More than one metric 
> matches name: fatal in namespace Query0.
> 13:05:49  at 
> org.apache.beam.repackaged.beam_sdks_java_test_utils.com.google.common.base.Preconditions.checkState(Preconditions.java:444)
> 13:05:49  at 
> org.apache.beam.sdk.testutils.metrics.MetricsReader.checkIfMetricResultIsUnique(MetricsReader.java:127)
> 13:05:49  at 
> org.apache.beam.sdk.testutils.metrics.MetricsReader.getCounterMetric(MetricsReader.java:65)
> 13:05:49  at 
> org.apache.beam.sdk.nexmark.NexmarkLauncher.monitor(NexmarkLauncher.java:447)
> 13:05:49  at 
> org.apache.beam.sdk.nexmark.NexmarkLauncher.run(NexmarkLauncher.java:1156)
> 13:05:49  at org.apache.beam.sdk.nexmark.Main$Run.call(Main.java:108)
> 13:05:49  at org.apache.beam.sdk.nexmark.Main$Run.call(Main.java:96)
> 13:05:49  at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> 13:05:49  at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> 13:05:49  at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> 13:05:49  at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> 13:05:49  at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 13:05:49  at java.lang.Thread.run(Thread.java:748)
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5886) Nexmark failing due to IllegalStateException

2018-10-28 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-5886?focusedWorklogId=159757=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-159757
 ]

ASF GitHub Bot logged work on BEAM-5886:


Author: ASF GitHub Bot
Created on: 28/Oct/18 20:25
Start Date: 28/Oct/18 20:25
Worklog Time Spent: 10m 
  Work Description: lgajowy commented on issue #6863: [BEAM-5886] Fix 
incorrectly formulated condition in checkState method
URL: https://github.com/apache/beam/pull/6863#issuecomment-433738335
 
 
   Thank you. I applied the suggestion. PTAL


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 159757)
Time Spent: 1h 10m  (was: 1h)

> Nexmark failing due to IllegalStateException
> 
>
> Key: BEAM-5886
> URL: https://issues.apache.org/jira/browse/BEAM-5886
> Project: Beam
>  Issue Type: Task
>  Components: test-failures, testing
>Reporter: Lukasz Gajowy
>Assignee: Lukasz Gajowy
>Priority: Major
>  Time Spent: 1h 10m
>  Remaining Estimate: 0h
>
> Nexmark fails due to the following error:
>  
> {code:java}
> java.lang.RuntimeException: java.lang.IllegalStateException: More than one 
> metric matches name: fatal in namespace Query0.
> 13:05:49   0012  *** not run ***
> 13:05:49 
> ==
> 13:05:49  at org.apache.beam.sdk.nexmark.Main.runAll(Main.java:144)
> 13:05:49 
> 13:05:49  at org.apache.beam.sdk.nexmark.Main.main(Main.java:477)
> 13:05:49 Caused by: java.lang.IllegalStateException: More than one metric 
> matches name: fatal in namespace Query0.
> 13:05:49  at 
> org.apache.beam.repackaged.beam_sdks_java_test_utils.com.google.common.base.Preconditions.checkState(Preconditions.java:444)
> 13:05:49  at 
> org.apache.beam.sdk.testutils.metrics.MetricsReader.checkIfMetricResultIsUnique(MetricsReader.java:127)
> 13:05:49  at 
> org.apache.beam.sdk.testutils.metrics.MetricsReader.getCounterMetric(MetricsReader.java:65)
> 13:05:49  at 
> org.apache.beam.sdk.nexmark.NexmarkLauncher.monitor(NexmarkLauncher.java:447)
> 13:05:49  at 
> org.apache.beam.sdk.nexmark.NexmarkLauncher.run(NexmarkLauncher.java:1156)
> 13:05:49  at org.apache.beam.sdk.nexmark.Main$Run.call(Main.java:108)
> 13:05:49  at org.apache.beam.sdk.nexmark.Main$Run.call(Main.java:96)
> 13:05:49  at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> 13:05:49  at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> 13:05:49  at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> 13:05:49  at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> 13:05:49  at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 13:05:49  at java.lang.Thread.run(Thread.java:748)
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5886) Nexmark failing due to IllegalStateException

2018-10-28 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-5886?focusedWorklogId=159755=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-159755
 ]

ASF GitHub Bot logged work on BEAM-5886:


Author: ASF GitHub Bot
Created on: 28/Oct/18 20:22
Start Date: 28/Oct/18 20:22
Worklog Time Spent: 10m 
  Work Description: lgajowy commented on a change in pull request #6863: 
[BEAM-5886] Fix incorrectly formulated condition in checkState method
URL: https://github.com/apache/beam/pull/6863#discussion_r228763482
 
 

 ##
 File path: 
sdks/java/testing/test-utils/src/main/java/org/apache/beam/sdk/testutils/metrics/MetricsReader.java
 ##
 @@ -124,9 +124,12 @@ private long getDistributionMetric(String name, 
DistributionType distType, long
   private  void checkIfMetricResultIsUnique(String name, 
Iterable> metricResult)
   throws IllegalStateException {
 
+int resultCount = Iterables.size(metricResult);
 Preconditions.checkState(
-Iterables.size(metricResult) == 1,
-String.format("More than one metric matches name: %s in namespace 
%s.", name, namespace));
+resultCount <= 1,
+String.format(
 
 Review comment:
   ok


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 159755)
Time Spent: 1h  (was: 50m)

> Nexmark failing due to IllegalStateException
> 
>
> Key: BEAM-5886
> URL: https://issues.apache.org/jira/browse/BEAM-5886
> Project: Beam
>  Issue Type: Task
>  Components: test-failures, testing
>Reporter: Lukasz Gajowy
>Assignee: Lukasz Gajowy
>Priority: Major
>  Time Spent: 1h
>  Remaining Estimate: 0h
>
> Nexmark fails due to the following error:
>  
> {code:java}
> java.lang.RuntimeException: java.lang.IllegalStateException: More than one 
> metric matches name: fatal in namespace Query0.
> 13:05:49   0012  *** not run ***
> 13:05:49 
> ==
> 13:05:49  at org.apache.beam.sdk.nexmark.Main.runAll(Main.java:144)
> 13:05:49 
> 13:05:49  at org.apache.beam.sdk.nexmark.Main.main(Main.java:477)
> 13:05:49 Caused by: java.lang.IllegalStateException: More than one metric 
> matches name: fatal in namespace Query0.
> 13:05:49  at 
> org.apache.beam.repackaged.beam_sdks_java_test_utils.com.google.common.base.Preconditions.checkState(Preconditions.java:444)
> 13:05:49  at 
> org.apache.beam.sdk.testutils.metrics.MetricsReader.checkIfMetricResultIsUnique(MetricsReader.java:127)
> 13:05:49  at 
> org.apache.beam.sdk.testutils.metrics.MetricsReader.getCounterMetric(MetricsReader.java:65)
> 13:05:49  at 
> org.apache.beam.sdk.nexmark.NexmarkLauncher.monitor(NexmarkLauncher.java:447)
> 13:05:49  at 
> org.apache.beam.sdk.nexmark.NexmarkLauncher.run(NexmarkLauncher.java:1156)
> 13:05:49  at org.apache.beam.sdk.nexmark.Main$Run.call(Main.java:108)
> 13:05:49  at org.apache.beam.sdk.nexmark.Main$Run.call(Main.java:96)
> 13:05:49  at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> 13:05:49  at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> 13:05:49  at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> 13:05:49  at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> 13:05:49  at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 13:05:49  at java.lang.Thread.run(Thread.java:748)
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5884) Allow nested types have null value.

2018-10-28 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-5884?focusedWorklogId=159754=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-159754
 ]

ASF GitHub Bot logged work on BEAM-5884:


Author: ASF GitHub Bot
Created on: 28/Oct/18 20:02
Start Date: 28/Oct/18 20:02
Worklog Time Spent: 10m 
  Work Description: kennknowles commented on issue #6861: [BEAM-5884] 
support null values in Array and Map.
URL: https://github.com/apache/beam/pull/6861#issuecomment-433736576
 
 
   It is admittedly much less likely to have unknown data not at the top level, 
so this is mostly relevant for intermediate stages of a computation.
   
   I'm not sure this is the best code approach to the feature; we should just 
add a nullability flag to types in general rather than having the flag on the 
container type.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 159754)
Time Spent: 2h  (was: 1h 50m)

> Allow nested types have null value.
> ---
>
> Key: BEAM-5884
> URL: https://issues.apache.org/jira/browse/BEAM-5884
> Project: Beam
>  Issue Type: Bug
>  Components: dsl-sql
>Reporter: Rui Wang
>Assignee: Rui Wang
>Priority: Major
>  Time Spent: 2h
>  Remaining Estimate: 0h
>
> We could allow arbitrary combination of nested types have null value.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5884) Allow nested types have null value.

2018-10-28 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-5884?focusedWorklogId=159753=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-159753
 ]

ASF GitHub Bot logged work on BEAM-5884:


Author: ASF GitHub Bot
Created on: 28/Oct/18 20:00
Start Date: 28/Oct/18 20:00
Worklog Time Spent: 10m 
  Work Description: kennknowles commented on issue #6861: [BEAM-5884] 
support null values in Array and Map.
URL: https://github.com/apache/beam/pull/6861#issuecomment-433736409
 
 
   The default for everything should be non-nullable except in cases where 
compatibility with some standard requires us to be nullable by default.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 159753)
Time Spent: 1h 50m  (was: 1h 40m)

> Allow nested types have null value.
> ---
>
> Key: BEAM-5884
> URL: https://issues.apache.org/jira/browse/BEAM-5884
> Project: Beam
>  Issue Type: Bug
>  Components: dsl-sql
>Reporter: Rui Wang
>Assignee: Rui Wang
>Priority: Major
>  Time Spent: 1h 50m
>  Remaining Estimate: 0h
>
> We could allow arbitrary combination of nested types have null value.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5884) Allow nested types have null value.

2018-10-28 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-5884?focusedWorklogId=159751=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-159751
 ]

ASF GitHub Bot logged work on BEAM-5884:


Author: ASF GitHub Bot
Created on: 28/Oct/18 19:59
Start Date: 28/Oct/18 19:59
Worklog Time Spent: 10m 
  Work Description: kennknowles commented on issue #6861: [BEAM-5884] 
support null values in Array and Map.
URL: https://github.com/apache/beam/pull/6861#issuecomment-433736320
 
 
   Ah, I thought this was adding support for `NULLABLE>` but it is 
adding support for `ARRAY>`. This is exactly what is necessary for 
composability. We should do this regardless, but I'm curious what the SQL 
standard and other engines support.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 159751)
Time Spent: 1h 40m  (was: 1.5h)

> Allow nested types have null value.
> ---
>
> Key: BEAM-5884
> URL: https://issues.apache.org/jira/browse/BEAM-5884
> Project: Beam
>  Issue Type: Bug
>  Components: dsl-sql
>Reporter: Rui Wang
>Assignee: Rui Wang
>Priority: Major
>  Time Spent: 1h 40m
>  Remaining Estimate: 0h
>
> We could allow arbitrary combination of nested types have null value.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5884) Allow nested types have null value.

2018-10-28 Thread ASF GitHub Bot (JIRA)

 [ 
https://issues.apache.org/jira/browse/BEAM-5884?focusedWorklogId=159750=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-159750
 ]

ASF GitHub Bot logged work on BEAM-5884:

Author: ASF GitHub Bot
Created on: 28/Oct/18 19:57
Start Date: 28/Oct/18 19:57
Worklog Time Spent: 10m 
  Work Description: reuvenlax commented on issue #6861: [BEAM-5884] support 
null values in Array and Map.
URL: https://github.com/apache/beam/pull/6861#issuecomment-433736172

   What should the default for list entries be? Nullable or not nullable?

   On Sun, Oct 28, 2018 at 12:55 PM Kenn Knowles 
   wrote:

   > Use case: if a field is missing from an input data set, i.e. unknown, that
   > is a different situation that a known empty array. Every data type needs to
   > have the ability to be nullable.
   >
   > —
   > You are receiving this because you were mentioned.
   > Reply to this email directly, view it on GitHub
   > , or mute
   > the thread
   > 

   > .
   >

This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Issue Time Tracking
---

Worklog Id: (was: 159750)
Time Spent: 1.5h  (was: 1h 20m)

> Allow nested types have null value.
> ---
>
> Key: BEAM-5884
> URL: https://issues.apache.org/jira/browse/BEAM-5884
> Project: Beam
>  Issue Type: Bug
>  Components: dsl-sql
>Reporter: Rui Wang
>Assignee: Rui Wang
>Priority: Major
>  Time Spent: 1.5h
>  Remaining Estimate: 0h
>
> We could allow arbitrary combination of nested types have null value.

--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5884) Allow nested types have null value.

2018-10-28 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-5884?focusedWorklogId=159749=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-159749
 ]

ASF GitHub Bot logged work on BEAM-5884:


Author: ASF GitHub Bot
Created on: 28/Oct/18 19:56
Start Date: 28/Oct/18 19:56
Worklog Time Spent: 10m 
  Work Description: kennknowles commented on issue #6861: [BEAM-5884] 
support null values in Array and Map.
URL: https://github.com/apache/beam/pull/6861#issuecomment-433736137
 
 
   TBH nullability doesn't compose, hence `Optional` or equivalent in every 
language with good support for complex types or generic types (incl. built-in 
arrays and maps). I don't know if SQL has caught up in that regard.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 159749)
Time Spent: 1h 20m  (was: 1h 10m)

> Allow nested types have null value.
> ---
>
> Key: BEAM-5884
> URL: https://issues.apache.org/jira/browse/BEAM-5884
> Project: Beam
>  Issue Type: Bug
>  Components: dsl-sql
>Reporter: Rui Wang
>Assignee: Rui Wang
>Priority: Major
>  Time Spent: 1h 20m
>  Remaining Estimate: 0h
>
> We could allow arbitrary combination of nested types have null value.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5886) Nexmark failing due to IllegalStateException

2018-10-28 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-5886?focusedWorklogId=159747=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-159747
 ]

ASF GitHub Bot logged work on BEAM-5886:


Author: ASF GitHub Bot
Created on: 28/Oct/18 19:46
Start Date: 28/Oct/18 19:46
Worklog Time Spent: 10m 
  Work Description: kennknowles commented on a change in pull request 
#6863: [BEAM-5886] Fix incorrectly formulated condition in checkState method
URL: https://github.com/apache/beam/pull/6863#discussion_r228762037
 
 

 ##
 File path: 
sdks/java/testing/test-utils/src/main/java/org/apache/beam/sdk/testutils/metrics/MetricsReader.java
 ##
 @@ -124,9 +124,12 @@ private long getDistributionMetric(String name, 
DistributionType distType, long
   private  void checkIfMetricResultIsUnique(String name, 
Iterable> metricResult)
   throws IllegalStateException {
 
+int resultCount = Iterables.size(metricResult);
 Preconditions.checkState(
-Iterables.size(metricResult) == 1,
-String.format("More than one metric matches name: %s in namespace 
%s.", name, namespace));
+resultCount <= 1,
+String.format(
 
 Review comment:
   Actually this should be just `checkState(resultCount <= 1, "..", name, 
namespace, resultCount)` so that it does not execute the String.format when the 
check passes.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 159747)
Time Spent: 50m  (was: 40m)

> Nexmark failing due to IllegalStateException
> 
>
> Key: BEAM-5886
> URL: https://issues.apache.org/jira/browse/BEAM-5886
> Project: Beam
>  Issue Type: Task
>  Components: test-failures, testing
>Reporter: Lukasz Gajowy
>Assignee: Lukasz Gajowy
>Priority: Major
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> Nexmark fails due to the following error:
>  
> {code:java}
> java.lang.RuntimeException: java.lang.IllegalStateException: More than one 
> metric matches name: fatal in namespace Query0.
> 13:05:49   0012  *** not run ***
> 13:05:49 
> ==
> 13:05:49  at org.apache.beam.sdk.nexmark.Main.runAll(Main.java:144)
> 13:05:49 
> 13:05:49  at org.apache.beam.sdk.nexmark.Main.main(Main.java:477)
> 13:05:49 Caused by: java.lang.IllegalStateException: More than one metric 
> matches name: fatal in namespace Query0.
> 13:05:49  at 
> org.apache.beam.repackaged.beam_sdks_java_test_utils.com.google.common.base.Preconditions.checkState(Preconditions.java:444)
> 13:05:49  at 
> org.apache.beam.sdk.testutils.metrics.MetricsReader.checkIfMetricResultIsUnique(MetricsReader.java:127)
> 13:05:49  at 
> org.apache.beam.sdk.testutils.metrics.MetricsReader.getCounterMetric(MetricsReader.java:65)
> 13:05:49  at 
> org.apache.beam.sdk.nexmark.NexmarkLauncher.monitor(NexmarkLauncher.java:447)
> 13:05:49  at 
> org.apache.beam.sdk.nexmark.NexmarkLauncher.run(NexmarkLauncher.java:1156)
> 13:05:49  at org.apache.beam.sdk.nexmark.Main$Run.call(Main.java:108)
> 13:05:49  at org.apache.beam.sdk.nexmark.Main$Run.call(Main.java:96)
> 13:05:49  at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> 13:05:49  at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> 13:05:49  at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> 13:05:49  at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> 13:05:49  at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 13:05:49  at java.lang.Thread.run(Thread.java:748)
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5866) RowCoder doesn't implement structuralValue

2018-10-28 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-5866?focusedWorklogId=159746=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-159746
 ]

ASF GitHub Bot logged work on BEAM-5866:


Author: ASF GitHub Bot
Created on: 28/Oct/18 19:44
Start Date: 28/Oct/18 19:44
Worklog Time Spent: 10m 
  Work Description: kennknowles commented on issue #6845: [BEAM-5866] Fix 
`Row#equals`
URL: https://github.com/apache/beam/pull/6845#issuecomment-433735267
 
 
   SGTM to try different map implementations. I would like to preserve our 
ability to change the decision of what map data structure is used in a Java 
implementation of rows without any breaking change for users. It may just need 
to be altered over time, or it may need to be tunable based on application.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 159746)
Time Spent: 4.5h  (was: 4h 20m)

> RowCoder doesn't implement structuralValue
> --
>
> Key: BEAM-5866
> URL: https://issues.apache.org/jira/browse/BEAM-5866
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Gleb Kanterov
>Assignee: Gleb Kanterov
>Priority: Major
>  Time Spent: 4.5h
>  Remaining Estimate: 0h
>
> These two properties fail for RowCoder with `BYTES` field, or `Map` 
> field. 
> {code}
>   public static  void testConsistentWithEquals(Coder coder, T example) {
> assumeTrue(coder.consistentWithEquals());
> byte[] bytes = encodeBytes(coder, example);
> // even if the coder is non-deterministic, if the encoded bytes match,
> // coder is consistent with equals, decoded values must be equal
> T out0 = decodeBytes(coder, bytes);
> T out1 = decodeBytes(coder, bytes);
> assertEquals("If the encoded bytes match, decoded values must be equal", 
> out0, out1);
> assertEquals(
> "If two values are equal, their hash codes must be equal",
> out0.hashCode(),
> out1.hashCode());
>   }
>   public static  void testStructuralValueConsistentWithEquals(Coder 
> coder, T example) {
> byte[] bytes = encodeBytes(coder, example);
> // even if coder is non-deterministic, if the encoded bytes match,
> // structural values must be equal
> Object out0 = coder.structuralValue(decodeBytes(coder, bytes));
> Object out1 = coder.structuralValue(decodeBytes(coder, bytes));
> assertEquals("If the encoded bytes match, structural values must be 
> equal", out0, out1);
> assertEquals(
> "If two values are equal, their hash codes must be equal",
> out0.hashCode(),
> out1.hashCode());
>   }
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5866) RowCoder doesn't implement structuralValue

2018-10-28 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-5866?focusedWorklogId=159740=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-159740
 ]

ASF GitHub Bot logged work on BEAM-5866:


Author: ASF GitHub Bot
Created on: 28/Oct/18 18:49
Start Date: 28/Oct/18 18:49
Worklog Time Spent: 10m 
  Work Description: kanterov commented on issue #6845: [BEAM-5866] Fix 
`Row#equals`
URL: https://github.com/apache/beam/pull/6845#issuecomment-433731101
 
 
   @reuvenlax Yes, the problem with ByteBuffer that it has too many fields we 
don't need, like `positions`, `limit`, `offset`, `mark`, `capacity`, 
`isReadonly`. They make sense when implementing protocols, but aren't so much 
useful in data pipelines.
   
   @kennknowles do we even want hashmap as implementation? In Spark, it's done 
with ArrayBasedMaps. I was thinking similar could be done, for instance, for 
arrays, to use primitive non-boxing collections where applicable.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 159740)
Time Spent: 4h 20m  (was: 4h 10m)

> RowCoder doesn't implement structuralValue
> --
>
> Key: BEAM-5866
> URL: https://issues.apache.org/jira/browse/BEAM-5866
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Gleb Kanterov
>Assignee: Gleb Kanterov
>Priority: Major
>  Time Spent: 4h 20m
>  Remaining Estimate: 0h
>
> These two properties fail for RowCoder with `BYTES` field, or `Map` 
> field. 
> {code}
>   public static  void testConsistentWithEquals(Coder coder, T example) {
> assumeTrue(coder.consistentWithEquals());
> byte[] bytes = encodeBytes(coder, example);
> // even if the coder is non-deterministic, if the encoded bytes match,
> // coder is consistent with equals, decoded values must be equal
> T out0 = decodeBytes(coder, bytes);
> T out1 = decodeBytes(coder, bytes);
> assertEquals("If the encoded bytes match, decoded values must be equal", 
> out0, out1);
> assertEquals(
> "If two values are equal, their hash codes must be equal",
> out0.hashCode(),
> out1.hashCode());
>   }
>   public static  void testStructuralValueConsistentWithEquals(Coder 
> coder, T example) {
> byte[] bytes = encodeBytes(coder, example);
> // even if coder is non-deterministic, if the encoded bytes match,
> // structural values must be equal
> Object out0 = coder.structuralValue(decodeBytes(coder, bytes));
> Object out1 = coder.structuralValue(decodeBytes(coder, bytes));
> assertEquals("If the encoded bytes match, structural values must be 
> equal", out0, out1);
> assertEquals(
> "If two values are equal, their hash codes must be equal",
> out0.hashCode(),
> out1.hashCode());
>   }
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5866) RowCoder doesn't implement structuralValue

2018-10-28 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-5866?focusedWorklogId=159739=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-159739
 ]

ASF GitHub Bot logged work on BEAM-5866:


Author: ASF GitHub Bot
Created on: 28/Oct/18 18:25
Start Date: 28/Oct/18 18:25
Worklog Time Spent: 10m 
  Work Description: reuvenlax commented on issue #6845: [BEAM-5866] Fix 
`Row#equals`
URL: https://github.com/apache/beam/pull/6845#issuecomment-433729161
 
 
   Does ByteBuffer implement an appropriate equals?
   
   On Sun, Oct 28, 2018 at 11:22 AM Kenn Knowles 
   wrote:
   
   > I guess the problem with Map goes all the way to Java. We
   > should probably model the schema type MAP with the Java type 
Map T>
   >
   > —
   > You are receiving this because you were mentioned.
   > Reply to this email directly, view it on GitHub
   > , or mute
   > the thread
   > 

   > .
   >
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 159739)
Time Spent: 4h 10m  (was: 4h)

> RowCoder doesn't implement structuralValue
> --
>
> Key: BEAM-5866
> URL: https://issues.apache.org/jira/browse/BEAM-5866
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Gleb Kanterov
>Assignee: Gleb Kanterov
>Priority: Major
>  Time Spent: 4h 10m
>  Remaining Estimate: 0h
>
> These two properties fail for RowCoder with `BYTES` field, or `Map` 
> field. 
> {code}
>   public static  void testConsistentWithEquals(Coder coder, T example) {
> assumeTrue(coder.consistentWithEquals());
> byte[] bytes = encodeBytes(coder, example);
> // even if the coder is non-deterministic, if the encoded bytes match,
> // coder is consistent with equals, decoded values must be equal
> T out0 = decodeBytes(coder, bytes);
> T out1 = decodeBytes(coder, bytes);
> assertEquals("If the encoded bytes match, decoded values must be equal", 
> out0, out1);
> assertEquals(
> "If two values are equal, their hash codes must be equal",
> out0.hashCode(),
> out1.hashCode());
>   }
>   public static  void testStructuralValueConsistentWithEquals(Coder 
> coder, T example) {
> byte[] bytes = encodeBytes(coder, example);
> // even if coder is non-deterministic, if the encoded bytes match,
> // structural values must be equal
> Object out0 = coder.structuralValue(decodeBytes(coder, bytes));
> Object out1 = coder.structuralValue(decodeBytes(coder, bytes));
> assertEquals("If the encoded bytes match, structural values must be 
> equal", out0, out1);
> assertEquals(
> "If two values are equal, their hash codes must be equal",
> out0.hashCode(),
> out1.hashCode());
>   }
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5866) RowCoder doesn't implement structuralValue

2018-10-28 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-5866?focusedWorklogId=159738=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-159738
 ]

ASF GitHub Bot logged work on BEAM-5866:


Author: ASF GitHub Bot
Created on: 28/Oct/18 18:22
Start Date: 28/Oct/18 18:22
Worklog Time Spent: 10m 
  Work Description: kennknowles commented on issue #6845: [BEAM-5866] Fix 
`Row#equals`
URL: https://github.com/apache/beam/pull/6845#issuecomment-433728949
 
 
   I guess the problem with `Map` goes all the way to Java. We 
should probably model the schema type `MAP` with the Java type 
`Map`


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 159738)
Time Spent: 4h  (was: 3h 50m)

> RowCoder doesn't implement structuralValue
> --
>
> Key: BEAM-5866
> URL: https://issues.apache.org/jira/browse/BEAM-5866
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Gleb Kanterov
>Assignee: Gleb Kanterov
>Priority: Major
>  Time Spent: 4h
>  Remaining Estimate: 0h
>
> These two properties fail for RowCoder with `BYTES` field, or `Map` 
> field. 
> {code}
>   public static  void testConsistentWithEquals(Coder coder, T example) {
> assumeTrue(coder.consistentWithEquals());
> byte[] bytes = encodeBytes(coder, example);
> // even if the coder is non-deterministic, if the encoded bytes match,
> // coder is consistent with equals, decoded values must be equal
> T out0 = decodeBytes(coder, bytes);
> T out1 = decodeBytes(coder, bytes);
> assertEquals("If the encoded bytes match, decoded values must be equal", 
> out0, out1);
> assertEquals(
> "If two values are equal, their hash codes must be equal",
> out0.hashCode(),
> out1.hashCode());
>   }
>   public static  void testStructuralValueConsistentWithEquals(Coder 
> coder, T example) {
> byte[] bytes = encodeBytes(coder, example);
> // even if coder is non-deterministic, if the encoded bytes match,
> // structural values must be equal
> Object out0 = coder.structuralValue(decodeBytes(coder, bytes));
> Object out1 = coder.structuralValue(decodeBytes(coder, bytes));
> assertEquals("If the encoded bytes match, structural values must be 
> equal", out0, out1);
> assertEquals(
> "If two values are equal, their hash codes must be equal",
> out0.hashCode(),
> out1.hashCode());
>   }
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5886) Nexmark failing due to IllegalStateException

2018-10-28 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-5886?focusedWorklogId=159737=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-159737
 ]

ASF GitHub Bot logged work on BEAM-5886:


Author: ASF GitHub Bot
Created on: 28/Oct/18 17:23
Start Date: 28/Oct/18 17:23
Worklog Time Spent: 10m 
  Work Description: lgajowy edited a comment on issue #6863: [BEAM-5886] 
Fix incorrectly formulated condition in checkState method
URL: https://github.com/apache/beam/pull/6863#issuecomment-433723542
 
 
   I think that there's also a problem with creating the "fatal" metric 
counter. Even though it was created using `Metrics.counter()` method, it was 
not found  (hence 0 counters with the name "fatal" and the error this PR fixes).
   
   One line from the logs caught my attention (from a successful run): 
   ```
   SEVERE: Failed to get metric fatal, from namespace Query9
   ```
   This looks analogously for every query. I can take a closer look at this 
later.
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 159737)
Time Spent: 40m  (was: 0.5h)

> Nexmark failing due to IllegalStateException
> 
>
> Key: BEAM-5886
> URL: https://issues.apache.org/jira/browse/BEAM-5886
> Project: Beam
>  Issue Type: Task
>  Components: test-failures, testing
>Reporter: Lukasz Gajowy
>Assignee: Lukasz Gajowy
>Priority: Major
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> Nexmark fails due to the following error:
>  
> {code:java}
> java.lang.RuntimeException: java.lang.IllegalStateException: More than one 
> metric matches name: fatal in namespace Query0.
> 13:05:49   0012  *** not run ***
> 13:05:49 
> ==
> 13:05:49  at org.apache.beam.sdk.nexmark.Main.runAll(Main.java:144)
> 13:05:49 
> 13:05:49  at org.apache.beam.sdk.nexmark.Main.main(Main.java:477)
> 13:05:49 Caused by: java.lang.IllegalStateException: More than one metric 
> matches name: fatal in namespace Query0.
> 13:05:49  at 
> org.apache.beam.repackaged.beam_sdks_java_test_utils.com.google.common.base.Preconditions.checkState(Preconditions.java:444)
> 13:05:49  at 
> org.apache.beam.sdk.testutils.metrics.MetricsReader.checkIfMetricResultIsUnique(MetricsReader.java:127)
> 13:05:49  at 
> org.apache.beam.sdk.testutils.metrics.MetricsReader.getCounterMetric(MetricsReader.java:65)
> 13:05:49  at 
> org.apache.beam.sdk.nexmark.NexmarkLauncher.monitor(NexmarkLauncher.java:447)
> 13:05:49  at 
> org.apache.beam.sdk.nexmark.NexmarkLauncher.run(NexmarkLauncher.java:1156)
> 13:05:49  at org.apache.beam.sdk.nexmark.Main$Run.call(Main.java:108)
> 13:05:49  at org.apache.beam.sdk.nexmark.Main$Run.call(Main.java:96)
> 13:05:49  at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> 13:05:49  at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> 13:05:49  at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> 13:05:49  at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> 13:05:49  at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 13:05:49  at java.lang.Thread.run(Thread.java:748)
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5886) Nexmark failing due to IllegalStateException

2018-10-28 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-5886?focusedWorklogId=159735=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-159735
 ]

ASF GitHub Bot logged work on BEAM-5886:


Author: ASF GitHub Bot
Created on: 28/Oct/18 17:13
Start Date: 28/Oct/18 17:13
Worklog Time Spent: 10m 
  Work Description: lgajowy commented on issue #6863: [BEAM-5886] Fix 
incorrectly formulated condition in checkState method
URL: https://github.com/apache/beam/pull/6863#issuecomment-433723542
 
 
   I think that there's also a problem with creating the "fatal" metric 
counter. Even though it was created using `Metrics.counter()` method, it was 
not found  (hence 0 counters with the name "fatal" and the error this PR fixes).
   
   One line from the logs caught my attention (from a successful run): 
   ```
   SEVERE: Failed to get metric fatal, from namespace Query9
   ```
   This looks analogously for every metrics. I can take a closer look at this 
later.
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 159735)
Time Spent: 0.5h  (was: 20m)

> Nexmark failing due to IllegalStateException
> 
>
> Key: BEAM-5886
> URL: https://issues.apache.org/jira/browse/BEAM-5886
> Project: Beam
>  Issue Type: Task
>  Components: test-failures, testing
>Reporter: Lukasz Gajowy
>Assignee: Lukasz Gajowy
>Priority: Major
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> Nexmark fails due to the following error:
>  
> {code:java}
> java.lang.RuntimeException: java.lang.IllegalStateException: More than one 
> metric matches name: fatal in namespace Query0.
> 13:05:49   0012  *** not run ***
> 13:05:49 
> ==
> 13:05:49  at org.apache.beam.sdk.nexmark.Main.runAll(Main.java:144)
> 13:05:49 
> 13:05:49  at org.apache.beam.sdk.nexmark.Main.main(Main.java:477)
> 13:05:49 Caused by: java.lang.IllegalStateException: More than one metric 
> matches name: fatal in namespace Query0.
> 13:05:49  at 
> org.apache.beam.repackaged.beam_sdks_java_test_utils.com.google.common.base.Preconditions.checkState(Preconditions.java:444)
> 13:05:49  at 
> org.apache.beam.sdk.testutils.metrics.MetricsReader.checkIfMetricResultIsUnique(MetricsReader.java:127)
> 13:05:49  at 
> org.apache.beam.sdk.testutils.metrics.MetricsReader.getCounterMetric(MetricsReader.java:65)
> 13:05:49  at 
> org.apache.beam.sdk.nexmark.NexmarkLauncher.monitor(NexmarkLauncher.java:447)
> 13:05:49  at 
> org.apache.beam.sdk.nexmark.NexmarkLauncher.run(NexmarkLauncher.java:1156)
> 13:05:49  at org.apache.beam.sdk.nexmark.Main$Run.call(Main.java:108)
> 13:05:49  at org.apache.beam.sdk.nexmark.Main$Run.call(Main.java:96)
> 13:05:49  at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> 13:05:49  at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> 13:05:49  at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> 13:05:49  at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> 13:05:49  at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 13:05:49  at java.lang.Thread.run(Thread.java:748)
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5735) Contributor Guide Improvements

2018-10-28 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-5735?focusedWorklogId=159734=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-159734
 ]

ASF GitHub Bot logged work on BEAM-5735:


Author: ASF GitHub Bot
Created on: 28/Oct/18 17:09
Start Date: 28/Oct/18 17:09
Worklog Time Spent: 10m 
  Work Description: tweise commented on issue #6857: [BEAM-5735] 
Contributor guide improvements
URL: https://github.com/apache/beam/pull/6857#issuecomment-433723244
 
 
   @alanmyrvold Thanks for working on this! I think it would be good to bring 
more visibility to this effort by sending an email to the list. I also think 
that most of this material should live on the Wiki. 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 159734)
Time Spent: 40m  (was: 0.5h)

> Contributor Guide Improvements
> --
>
> Key: BEAM-5735
> URL: https://issues.apache.org/jira/browse/BEAM-5735
> Project: Beam
>  Issue Type: Improvement
>  Components: website
>Reporter: Scott Wegner
>Assignee: Alan Myrvold
>Priority: Major
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> This is a wish-list for improvements to the Beam contributor guide.
> Many thanks to [~rohdesam] for the feedback which helped shape this list.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5886) Nexmark failing due to IllegalStateException

2018-10-28 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-5886?focusedWorklogId=159733=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-159733
 ]

ASF GitHub Bot logged work on BEAM-5886:


Author: ASF GitHub Bot
Created on: 28/Oct/18 17:05
Start Date: 28/Oct/18 17:05
Worklog Time Spent: 10m 
  Work Description: lgajowy commented on issue #6863: [BEAM-5886] Fix 
incorrectly formulated condition in checkState method
URL: https://github.com/apache/beam/pull/6863#issuecomment-433722941
 
 
   @kennknowles could you take a look again? 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 159733)
Time Spent: 20m  (was: 10m)

> Nexmark failing due to IllegalStateException
> 
>
> Key: BEAM-5886
> URL: https://issues.apache.org/jira/browse/BEAM-5886
> Project: Beam
>  Issue Type: Task
>  Components: test-failures, testing
>Reporter: Lukasz Gajowy
>Assignee: Lukasz Gajowy
>Priority: Major
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> Nexmark fails due to the following error:
>  
> {code:java}
> java.lang.RuntimeException: java.lang.IllegalStateException: More than one 
> metric matches name: fatal in namespace Query0.
> 13:05:49   0012  *** not run ***
> 13:05:49 
> ==
> 13:05:49  at org.apache.beam.sdk.nexmark.Main.runAll(Main.java:144)
> 13:05:49 
> 13:05:49  at org.apache.beam.sdk.nexmark.Main.main(Main.java:477)
> 13:05:49 Caused by: java.lang.IllegalStateException: More than one metric 
> matches name: fatal in namespace Query0.
> 13:05:49  at 
> org.apache.beam.repackaged.beam_sdks_java_test_utils.com.google.common.base.Preconditions.checkState(Preconditions.java:444)
> 13:05:49  at 
> org.apache.beam.sdk.testutils.metrics.MetricsReader.checkIfMetricResultIsUnique(MetricsReader.java:127)
> 13:05:49  at 
> org.apache.beam.sdk.testutils.metrics.MetricsReader.getCounterMetric(MetricsReader.java:65)
> 13:05:49  at 
> org.apache.beam.sdk.nexmark.NexmarkLauncher.monitor(NexmarkLauncher.java:447)
> 13:05:49  at 
> org.apache.beam.sdk.nexmark.NexmarkLauncher.run(NexmarkLauncher.java:1156)
> 13:05:49  at org.apache.beam.sdk.nexmark.Main$Run.call(Main.java:108)
> 13:05:49  at org.apache.beam.sdk.nexmark.Main$Run.call(Main.java:96)
> 13:05:49  at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> 13:05:49  at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> 13:05:49  at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> 13:05:49  at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> 13:05:49  at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 13:05:49  at java.lang.Thread.run(Thread.java:748)
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5886) Nexmark failing due to IllegalStateException

2018-10-28 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-5886?focusedWorklogId=159732=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-159732
 ]

ASF GitHub Bot logged work on BEAM-5886:


Author: ASF GitHub Bot
Created on: 28/Oct/18 17:00
Start Date: 28/Oct/18 17:00
Worklog Time Spent: 10m 
  Work Description: lgajowy opened a new pull request #6863: [BEAM-5886] 
Fix incorrectly formulated condition in checkState method
URL: https://github.com/apache/beam/pull/6863
 
 
   The suites failed because 0 counters of the same name were created.
   I extended the error message to include a more descriptive information
   about this.
   
   Other than that a test for multiple counters with the same name was
   deleted. The reason it passed was because 0 counters were created
   (false-positive). As it turned out it is nontrivial to create a test
   that creates two metrics of the same name and type so I skipped this
   for now.
   
   
   
   
   Follow this checklist to help us incorporate your contribution quickly and 
easily:
   
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue, if applicable. This will automatically link the pull request to the 
issue.
- [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   It will help us expedite review of your Pull Request if you tag someone 
(e.g. `@username`) to look at it.
   
   Post-Commit Tests Status (on master branch)
   

   
   Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
   --- | --- | --- | --- | --- | --- | --- | ---
   Go | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/)
 | --- | --- | --- | --- | --- | ---
   Java | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/)
 [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/)
   Python | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/)
 | --- | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)
  [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Python_VR_Flink/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python_VR_Flink/lastCompletedBuild/)
 | --- | --- | ---
   
   
   
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub

[jira] [Created] (BEAM-5886) Nexmark failing due to IllegalStateException

2018-10-28 Thread Lukasz Gajowy (JIRA)

Lukasz Gajowy created BEAM-5886:
---

 Summary: Nexmark failing due to IllegalStateException
 Key: BEAM-5886
 URL: https://issues.apache.org/jira/browse/BEAM-5886
 Project: Beam
  Issue Type: Task
  Components: test-failures, testing
Reporter: Lukasz Gajowy
Assignee: Lukasz Gajowy


Nexmark fails due to the following error:

 
{code:java}
java.lang.RuntimeException: java.lang.IllegalStateException: More than one 
metric matches name: fatal in namespace Query0.
13:05:49   0012  *** not run ***
13:05:49 
==
13:05:49at org.apache.beam.sdk.nexmark.Main.runAll(Main.java:144)
13:05:49 
13:05:49at org.apache.beam.sdk.nexmark.Main.main(Main.java:477)
13:05:49 Caused by: java.lang.IllegalStateException: More than one metric 
matches name: fatal in namespace Query0.
13:05:49at 
org.apache.beam.repackaged.beam_sdks_java_test_utils.com.google.common.base.Preconditions.checkState(Preconditions.java:444)
13:05:49at 
org.apache.beam.sdk.testutils.metrics.MetricsReader.checkIfMetricResultIsUnique(MetricsReader.java:127)
13:05:49at 
org.apache.beam.sdk.testutils.metrics.MetricsReader.getCounterMetric(MetricsReader.java:65)
13:05:49at 
org.apache.beam.sdk.nexmark.NexmarkLauncher.monitor(NexmarkLauncher.java:447)
13:05:49at 
org.apache.beam.sdk.nexmark.NexmarkLauncher.run(NexmarkLauncher.java:1156)
13:05:49at org.apache.beam.sdk.nexmark.Main$Run.call(Main.java:108)
13:05:49at org.apache.beam.sdk.nexmark.Main$Run.call(Main.java:96)
13:05:49at java.util.concurrent.FutureTask.run(FutureTask.java:266)
13:05:49at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
13:05:49at java.util.concurrent.FutureTask.run(FutureTask.java:266)
13:05:49at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
13:05:49at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
13:05:49at java.lang.Thread.run(Thread.java:748)
{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5265) Can not test Timer with processing time domain

2018-10-28 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-5265?focusedWorklogId=159728=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-159728
 ]

ASF GitHub Bot logged work on BEAM-5265:


Author: ASF GitHub Bot
Created on: 28/Oct/18 14:32
Start Date: 28/Oct/18 14:32
Worklog Time Spent: 10m 
  Work Description: JozoVilcek commented on issue #6305: [BEAM-5265] Use 
currentProcessingTime() for onTime with processing time domain
URL: https://github.com/apache/beam/pull/6305#issuecomment-433710767
 
 
   I am not sure if result of tests are in line with your expectations. It 
seems that processing time being out of window of fine as long as watermark is 
within the window and it's lateness.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 159728)
Time Spent: 2h 50m  (was: 2h 40m)

> Can not test Timer with processing time domain
> --
>
> Key: BEAM-5265
> URL: https://issues.apache.org/jira/browse/BEAM-5265
> Project: Beam
>  Issue Type: Bug
>  Components: runner-core, runner-direct
>Reporter: Jozef Vilcek
>Assignee: Kenneth Knowles
>Priority: Major
>  Time Spent: 2h 50m
>  Remaining Estimate: 0h
>
> I have a stateful DoFn which has a timer on PROCESSING_TIME domain. While 
> writing tests, I noticed that it does not react to `advanceProcessingTime()` 
> on tests stream. Problem seems to be here:
> [https://github.com/apache/beam/blob/master/runners/core-java/src/main/java/org/apache/beam/runners/core/SimpleDoFnRunner.java#L260]
> I can only tell that patching this place works for direct runner tests. Not 
> sure about broader impact on other runners since it is in `runner-core`



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5866) RowCoder doesn't implement structuralValue

2018-10-28 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-5866?focusedWorklogId=159727=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-159727
 ]

ASF GitHub Bot logged work on BEAM-5866:


Author: ASF GitHub Bot
Created on: 28/Oct/18 14:24
Start Date: 28/Oct/18 14:24
Worklog Time Spent: 10m 
  Work Description: kanterov commented on issue #6845: [BEAM-5866] Fix 
`Row#equals`
URL: https://github.com/apache/beam/pull/6845#issuecomment-433710063
 
 
   @kennknowles as it was last implemented, `Row#equals` didn't do any 
allocations, or used `structuralValue`. It became a bit confusing because ghere 
are two different, but related issues: 
   1. `structuralValue` wasn't working properly for `Row`, `Map` and `List`
   2.`Row#equals` didn't handle lists or maps with BYTES
   
   I've put the fix for the first issue into a separate pull request: 
https://github.com/apache/beam/pull/6862. In this pull request, I removed 
everything not relevant for `Row#equals`
   
   As we already discussed, there are two approaches to fix equality:
   - rolling own `deepEquals` 
   - wrapping `byte[]` into object implementing equality, but not in 
`structuralValue`, but while we build rows
   
   Not to talk about performance without having any numbers, I did two 
implementations and benchmarked both with JMH. Approach with own `deepEquals` 
has better performance by almost 50%. There are two reasons:
   
   - we can abuse the fact that our lists have random access and don't need to 
allocate iterators, it makes it even more performant than anything using 
`AbstractList#equals`
   - getting field type from schema is faster than using reflection
   
   There is still a problem with `Map`, it could be fixed either by 
banning such schemas, or rolling own `ByteArrayMap extends Map`


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 159727)
Time Spent: 3h 50m  (was: 3h 40m)

> RowCoder doesn't implement structuralValue
> --
>
> Key: BEAM-5866
> URL: https://issues.apache.org/jira/browse/BEAM-5866
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Gleb Kanterov
>Assignee: Gleb Kanterov
>Priority: Major
>  Time Spent: 3h 50m
>  Remaining Estimate: 0h
>
> These two properties fail for RowCoder with `BYTES` field, or `Map` 
> field. 
> {code}
>   public static  void testConsistentWithEquals(Coder coder, T example) {
> assumeTrue(coder.consistentWithEquals());
> byte[] bytes = encodeBytes(coder, example);
> // even if the coder is non-deterministic, if the encoded bytes match,
> // coder is consistent with equals, decoded values must be equal
> T out0 = decodeBytes(coder, bytes);
> T out1 = decodeBytes(coder, bytes);
> assertEquals("If the encoded bytes match, decoded values must be equal", 
> out0, out1);
> assertEquals(
> "If two values are equal, their hash codes must be equal",
> out0.hashCode(),
> out1.hashCode());
>   }
>   public static  void testStructuralValueConsistentWithEquals(Coder 
> coder, T example) {
> byte[] bytes = encodeBytes(coder, example);
> // even if coder is non-deterministic, if the encoded bytes match,
> // structural values must be equal
> Object out0 = coder.structuralValue(decodeBytes(coder, bytes));
> Object out1 = coder.structuralValue(decodeBytes(coder, bytes));
> assertEquals("If the encoded bytes match, structural values must be 
> equal", out0, out1);
> assertEquals(
> "If two values are equal, their hash codes must be equal",
> out0.hashCode(),
> out1.hashCode());
>   }
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5036) Optimize FileBasedSink's WriteOperation.moveToOutput()

2018-10-28 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-5036?focusedWorklogId=159696=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-159696
 ]

ASF GitHub Bot logged work on BEAM-5036:


Author: ASF GitHub Bot
Created on: 28/Oct/18 07:44
Start Date: 28/Oct/18 07:44
Worklog Time Spent: 10m 
  Work Description: timrobertson100 edited a comment on issue #6843: 
[BEAM-5036] Optimize the FileBasedSink WriteOperation.moveToOutput()
URL: https://github.com/apache/beam/pull/6843#issuecomment-433612987
 
 
   Our Jenkins issued performance tests have 100% variation across runs so seem 
mostly unhelpful.  
   
   Attached are 2 screenshots showing the result of this PR for a simple Avro 
file rewrite on an en-premise Hadoop cluster.
   
   **Without this PR**
   https://user-images.githubusercontent.com/237221/47603493-ddb5f980-d9ec-11e8-9cfd-4f25f5402291.png;>
   
   **With this PR**
   https://user-images.githubusercontent.com/237221/47603498-e5759e00-d9ec-11e8-9c6a-9e28f4ff62c1.png;>
   
   This is using [this 
code](https://github.com/gbif/beam-perf/tree/master/avro-to-avro) and executed 
as 
   ```
   spark2-submit --class org.gbif.beam.perf.avrotoavro.BeamAvroIOTest  \
--conf spark.dynamicAllocation.enabled=false   --master yarn 
--executor-memory 16G \
--executor-cores 5 --num-executors 10   
avro-to-avro-0.1-SNAPSHOT-shaded.jar \
--runner=SparkRunner   
--source=hdfs:///user/hive/warehouse/tim.db/occurrence_avro/* \
--target=hdfs:///tmp/beam4
   ```


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 159696)
Time Spent: 13.5h  (was: 13h 20m)

> Optimize FileBasedSink's WriteOperation.moveToOutput()
> --
>
> Key: BEAM-5036
> URL: https://issues.apache.org/jira/browse/BEAM-5036
> Project: Beam
>  Issue Type: Improvement
>  Components: io-java-files
>Affects Versions: 2.5.0
>Reporter: Jozef Vilcek
>Assignee: Tim Robertson
>Priority: Major
>  Time Spent: 13.5h
>  Remaining Estimate: 0h
>
> moveToOutput() methods in FileBasedSink.WriteOperation implements move by 
> copy+delete. It would be better to use a rename() which can be much more 
> effective for some filesystems.
> Filesystem must support cross-directory rename. BEAM-4861 is related to this 
> for the case of HDFS filesystem.
> Feature was discussed here:
> http://mail-archives.apache.org/mod_mbox/beam-dev/201807.mbox/%3CCAF9t7_4Mp54pQ+vRrJrBh9Vx0=uaknupzd_qdh_qdm9vxll...@mail.gmail.com%3E



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5036) Optimize FileBasedSink's WriteOperation.moveToOutput()

2018-10-28 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-5036?focusedWorklogId=159695=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-159695
 ]

ASF GitHub Bot logged work on BEAM-5036:


Author: ASF GitHub Bot
Created on: 28/Oct/18 07:43
Start Date: 28/Oct/18 07:43
Worklog Time Spent: 10m 
  Work Description: timrobertson100 edited a comment on issue #6843: 
[BEAM-5036] Optimize the FileBasedSink WriteOperation.moveToOutput()
URL: https://github.com/apache/beam/pull/6843#issuecomment-433612987
 
 
   Our Jenkins issued performance tests have 100% variation across runs so seem 
mostly unhelpful.  
   
   Attached are 2 screenshots showing the result of this PR for a simple Avro 
file rewrite on an en-premise Hadoop cluster.
   
   **Without this PR**
   https://user-images.githubusercontent.com/237221/47603493-ddb5f980-d9ec-11e8-9cfd-4f25f5402291.png;>
   
   **With this PR**
   https://user-images.githubusercontent.com/237221/47603498-e5759e00-d9ec-11e8-9c6a-9e28f4ff62c1.png;>
   
   This is using [this 
code](https://github.com/gbif/beam-perf/tree/master/avro-to-avro) and executed 
as 
   ```
   spark2-submit --class org.gbif.beam.perf.avrotoavro.BeamAvroIOTest   --conf 
spark.dynamicAllocation.enabled=false   --master yarn --executor-memory 16G 
--executor-cores 5 --num-executors 10   avro-to-avro-0.1-SNAPSHOT-shaded.jar 
--runner=SparkRunner   
--source=hdfs:///user/hive/warehouse/tim.db/occurrence_avro/*   
--target=hdfs:///tmp/beam4
   ```


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 159695)
Time Spent: 13h 20m  (was: 13h 10m)

> Optimize FileBasedSink's WriteOperation.moveToOutput()
> --
>
> Key: BEAM-5036
> URL: https://issues.apache.org/jira/browse/BEAM-5036
> Project: Beam
>  Issue Type: Improvement
>  Components: io-java-files
>Affects Versions: 2.5.0
>Reporter: Jozef Vilcek
>Assignee: Tim Robertson
>Priority: Major
>  Time Spent: 13h 20m
>  Remaining Estimate: 0h
>
> moveToOutput() methods in FileBasedSink.WriteOperation implements move by 
> copy+delete. It would be better to use a rename() which can be much more 
> effective for some filesystems.
> Filesystem must support cross-directory rename. BEAM-4861 is related to this 
> for the case of HDFS filesystem.
> Feature was discussed here:
> http://mail-archives.apache.org/mod_mbox/beam-dev/201807.mbox/%3CCAF9t7_4Mp54pQ+vRrJrBh9Vx0=uaknupzd_qdh_qdm9vxll...@mail.gmail.com%3E



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5884) Allow nested types have null value.

2018-10-28 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-5884?focusedWorklogId=159693=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-159693
 ]

ASF GitHub Bot logged work on BEAM-5884:


Author: ASF GitHub Bot
Created on: 28/Oct/18 06:26
Start Date: 28/Oct/18 06:26
Worklog Time Spent: 10m 
  Work Description: amaliujia edited a comment on issue #6861: [BEAM-5884] 
support null values for Array and Map.
URL: https://github.com/apache/beam/pull/6861#issuecomment-433680062
 
 
   The current use case is I am testing a sql function that works with Array 
type. The function has special logical to deal with `null`. When I created test 
cases for that function, I found current implementation does not allow `null` 
in Array.
   
   So the use case is SQL's use case.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 159693)
Time Spent: 1h  (was: 50m)

> Allow nested types have null value.
> ---
>
> Key: BEAM-5884
> URL: https://issues.apache.org/jira/browse/BEAM-5884
> Project: Beam
>  Issue Type: Bug
>  Components: dsl-sql
>Reporter: Rui Wang
>Assignee: Rui Wang
>Priority: Major
>  Time Spent: 1h
>  Remaining Estimate: 0h
>
> We could allow arbitrary combination of nested types have null value.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5884) Allow nested types have null value.

2018-10-28 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-5884?focusedWorklogId=159692=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-159692
 ]

ASF GitHub Bot logged work on BEAM-5884:


Author: ASF GitHub Bot
Created on: 28/Oct/18 06:25
Start Date: 28/Oct/18 06:25
Worklog Time Spent: 10m 
  Work Description: amaliujia edited a comment on issue #6861: [BEAM-5884] 
support null values for Array and Map.
URL: https://github.com/apache/beam/pull/6861#issuecomment-433680062
 
 
   The current use case is I am testing a sql function that works with Array 
type. The function has special logical to deal with `null`. When I created test 
cases for that function, I found current implementation does not allow `null` 
in Array.
   
   So the use case is SQL.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 159692)
Time Spent: 50m  (was: 40m)

> Allow nested types have null value.
> ---
>
> Key: BEAM-5884
> URL: https://issues.apache.org/jira/browse/BEAM-5884
> Project: Beam
>  Issue Type: Bug
>  Components: dsl-sql
>Reporter: Rui Wang
>Assignee: Rui Wang
>Priority: Major
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> We could allow arbitrary combination of nested types have null value.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5884) Allow nested types have null value.

2018-10-28 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-5884?focusedWorklogId=159691=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-159691
 ]

ASF GitHub Bot logged work on BEAM-5884:


Author: ASF GitHub Bot
Created on: 28/Oct/18 06:25
Start Date: 28/Oct/18 06:25
Worklog Time Spent: 10m 
  Work Description: amaliujia commented on issue #6861: [BEAM-5884] support 
null values for Array and Map.
URL: https://github.com/apache/beam/pull/6861#issuecomment-433680062
 
 
   The current use case is I am testing a sql function that works with Array 
type. The function has special logical to deal with `null`. When I created test 
cases for that function, I found current implementation does not allow `null` 
in Array.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 159691)
Time Spent: 40m  (was: 0.5h)

> Allow nested types have null value.
> ---
>
> Key: BEAM-5884
> URL: https://issues.apache.org/jira/browse/BEAM-5884
> Project: Beam
>  Issue Type: Bug
>  Components: dsl-sql
>Reporter: Rui Wang
>Assignee: Rui Wang
>Priority: Major
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> We could allow arbitrary combination of nested types have null value.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5884) Allow nested types have null value.

2018-10-28 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-5884?focusedWorklogId=159690=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-159690
 ]

ASF GitHub Bot logged work on BEAM-5884:


Author: ASF GitHub Bot
Created on: 28/Oct/18 06:17
Start Date: 28/Oct/18 06:17
Worklog Time Spent: 10m 
  Work Description: reuvenlax commented on issue #6861: [BEAM-5884] support 
null values for Array and Map.
URL: https://github.com/apache/beam/pull/6861#issuecomment-433679746
 
 
   What is the use case for null array elements?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 159690)
Time Spent: 0.5h  (was: 20m)

> Allow nested types have null value.
> ---
>
> Key: BEAM-5884
> URL: https://issues.apache.org/jira/browse/BEAM-5884
> Project: Beam
>  Issue Type: Bug
>  Components: dsl-sql
>Reporter: Rui Wang
>Assignee: Rui Wang
>Priority: Major
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> We could allow arbitrary combination of nested types have null value.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

62 matches

Mail list logo