Build failed in Jenkins: beam_PerformanceTests_Python #916

[Apache Jenkins Server]

See 


Changes:

[batbat] Fixed a bug that timer ID was not used for comparing timer data. Added

[ccy] Use TestClock when TestStream is present in pipeline

--
[...truncated 450 B...]
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 89a92050d59185876c79e3ee3dc0a23dfcf8d2df (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 89a92050d59185876c79e3ee3dc0a23dfcf8d2df
Commit message: "Merge pull request #4670: Use timer ID to compare TimerData"
 > git rev-list e668688690325127becfaef7486d02a6dae3e101 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins8459751519462803759.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins3385301852785052320.sh
+ rm -rf .env
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins1667267981940655671.sh
+ virtualenv .env --system-site-packages
New python executable in .env/bin/python
Installing setuptools, pip...done.
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins7882977796079160188.sh
+ .env/bin/pip install --upgrade setuptools pip
Downloading/unpacking setuptools from 
https://pypi.python.org/packages/43/41/033a273f9a25cb63050a390ee8397acbc7eae2159195d85f06f17e7be45a/setuptools-38.5.1-py2.py3-none-any.whl#md5=908b8b5e50bf429e520b2b5fa1b350e5
Downloading/unpacking pip from 
https://pypi.python.org/packages/b6/ac/7015eb97dc749283ffdec1c3a88ddb8ae03b8fad0f0e611408f196358da3/pip-9.0.1-py2.py3-none-any.whl#md5=297dbd16ef53bcef0447d245815f5144
Installing collected packages: setuptools, pip
  Found existing installation: setuptools 2.2
Uninstalling setuptools:
  Successfully uninstalled setuptools
  Found existing installation: pip 1.5.4
Uninstalling pip:
  Successfully uninstalled pip
Successfully installed setuptools pip
Cleaning up...
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins5541250928182785281.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins3631430507054988394.sh
+ .env/bin/pip install -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied: absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: setuptools in ./.env/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 16))
Requirement already satisfied: colorlog[windows]==2.6.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied: futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied: PyYAML==3.12 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied: pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Requirement already satisfied: numpy==1.13.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 22))
Requirement already satisfied: functools32 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23))
Requirement already satisfied: contextlib2>=0.5.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 24))
Requirement already satisfied: pywinrm in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: six in /usr/local/lib/python2.7/dist-packages 
(from 

[jira] [Comment Edited] (BEAM-3647) Default Coder/Reading Coder From File

[Kishan Kumar (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16365147#comment-16365147
 ] 

Kishan Kumar edited comment on BEAM-3647 at 2/15/18 4:55 AM:
-

Thanks [~jkff] I need to Specify That 

PCollection inputTable = PBegin.in(p).apply(Create.of(row1, row2, row3)
.*withCoder(type.getRowCoder()*));

 

The Coder Can We Choose at Run Time Because the Same Query is Going to Run on 
Different DDL as Shown Above in Example Because Seeing the Above *Case* I have 
not Found Such Use Case.

 

And the Same Question I have Asked at StackOverflow -: 
https://stackoverflow.com/questions/47806368/running-beamsql-withoutcoder-or-making-coder-dynamic


was (Author: kishank):
[~jkff] I need to Specify That 

PCollection inputTable = PBegin.in(p).apply(Create.of(row1, row2, row3)
.*withCoder(type.getRowCoder()*));

 

The Coder Can We Choose at Run Time Because the Same Query is Going to Run on 
Different DDL as Shown Above in Example Because Seeing the Above *Case* I have 
not Found Such Use Case.

> Default Coder/Reading Coder From File 
> --
>
> Key: BEAM-3647
> URL: https://issues.apache.org/jira/browse/BEAM-3647
> Project: Beam
>  Issue Type: New Feature
>  Components: beam-model, dsl-sql
>Affects Versions: 2.2.0
>Reporter: Kishan Kumar
>Priority: Major
> Fix For: 2.1.0
>
>
> *Requirement*-: Need to Run Template With Same Logics on Different Tables 
> Data.(Example is Given Below)
>  
> *Need*: Default Coder is Required So According to Data It Make All Fields as 
> String and Read Data else Thier must be Dynamic Options to Read Coder From 
> GCS as JSON FILE and Parse Data on Basis of That (But We can Pass Location 
> Using ValueProvider) or SomeWhere Else so At Runtime Using ValueProvider.
>  
>  
> *Examples*: I Have Two Tables 1 is Having Column (NAME, CLASS, ROLL, 
> SUB_PRICE)
> And 2 Table is (NAME, ROLL, SUB, TEST_MARKS)
>  
> On Both Tables, I am Just Sorting Table on Basis Of Roll Number so if We can 
> Read Coder at Run Time The Same Template Can Be Used For Different Tables at 
> Run Time.
>  
> Such Situations Make Our Work Easy and Make Our job Easy.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-3647) Default Coder/Reading Coder From File

[Kishan Kumar (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16365147#comment-16365147
 ] 

Kishan Kumar commented on BEAM-3647:


[~jkff] I need to Specify That 

PCollection inputTable = PBegin.in(p).apply(Create.of(row1, row2, row3)
.*withCoder(type.getRowCoder()*));

 

The Coder Can We Choose at Run Time Because the Same Query is Going to Run on 
Different DDL as Shown Above in Example Because Seeing the Above *Case* I have 
not Found Such Use Case.

> Default Coder/Reading Coder From File 
> --
>
> Key: BEAM-3647
> URL: https://issues.apache.org/jira/browse/BEAM-3647
> Project: Beam
>  Issue Type: New Feature
>  Components: beam-model, dsl-sql
>Affects Versions: 2.2.0
>Reporter: Kishan Kumar
>Priority: Major
> Fix For: 2.1.0
>
>
> *Requirement*-: Need to Run Template With Same Logics on Different Tables 
> Data.(Example is Given Below)
>  
> *Need*: Default Coder is Required So According to Data It Make All Fields as 
> String and Read Data else Thier must be Dynamic Options to Read Coder From 
> GCS as JSON FILE and Parse Data on Basis of That (But We can Pass Location 
> Using ValueProvider) or SomeWhere Else so At Runtime Using ValueProvider.
>  
>  
> *Examples*: I Have Two Tables 1 is Having Column (NAME, CLASS, ROLL, 
> SUB_PRICE)
> And 2 Table is (NAME, ROLL, SUB, TEST_MARKS)
>  
> On Both Tables, I am Just Sorting Table on Basis Of Roll Number so if We can 
> Read Coder at Run Time The Same Template Can Be Used For Different Tables at 
> Run Time.
>  
> Such Situations Make Our Work Easy and Make Our job Easy.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-3690) Dependency Conflict problems: several conflicting classes exist in different JARs

[PandaMonkey (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16365088#comment-16365088
 ] 

PandaMonkey commented on BEAM-3690:
---

[~lcwik], thanks for your affirmation. I would like to contribute to BEAM. 
:)(*r)

> Dependency Conflict problems: several conflicting classes exist in different 
> JARs
> -
>
> Key: BEAM-3690
> URL: https://issues.apache.org/jira/browse/BEAM-3690
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Affects Versions: 2.2.0
>Reporter: PandaMonkey
>Assignee: PandaMonkey
>Priority: Major
> Fix For: 2.3.0
>
> Attachments: beam_conflicts.txt
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> Hi, we found that there are duplicate classes exist in different JARs, and 
> these classes have different features.
> The conflicting JAR pairs are:
> 1. 
> jar-pair:
> 2. 
> jar-pair:
> Some of method only exist in one version of duplicate classes.
> As the JVM only load the classes present first on the classpath and shadow 
> the other duplicate ones with the same names. The dependency conflict problem 
> brings high risks of "*NoSuchMethodException*" or "*NoSuchMethodError*"  
> issues at runtime. The conflicting details are listed in the attachment. 
> Please notice that. Thanks.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-3690) Dependency Conflict problems: several conflicting classes exist in different JARs

[PandaMonkey (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16365085#comment-16365085
 ] 

PandaMonkey commented on BEAM-3690:
---

[~kenn], [~iemejia] thanks for your interest about our tool. The conflict 
detection tool is in the development and evaluation phase. I'm so honored to 
help open source project to ensure long-term health and protect from 
regressions. You are welcome to try it when we release it later!

> Dependency Conflict problems: several conflicting classes exist in different 
> JARs
> -
>
> Key: BEAM-3690
> URL: https://issues.apache.org/jira/browse/BEAM-3690
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Affects Versions: 2.2.0
>Reporter: PandaMonkey
>Assignee: PandaMonkey
>Priority: Major
> Fix For: 2.3.0
>
> Attachments: beam_conflicts.txt
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> Hi, we found that there are duplicate classes exist in different JARs, and 
> these classes have different features.
> The conflicting JAR pairs are:
> 1. 
> jar-pair:
> 2. 
> jar-pair:
> Some of method only exist in one version of duplicate classes.
> As the JVM only load the classes present first on the classpath and shadow 
> the other duplicate ones with the same names. The dependency conflict problem 
> brings high risks of "*NoSuchMethodException*" or "*NoSuchMethodError*"  
> issues at runtime. The conflicting details are listed in the attachment. 
> Please notice that. Thanks.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Jenkins build is back to stable : beam_PostCommit_Java_MavenInstall #5970

[Apache Jenkins Server]

See 

Jenkins build became unstable: beam_PostCommit_Java_MavenInstall #5969

[Apache Jenkins Server]

See 

[jira] [Resolved] (BEAM-3420) TimerData#compareTo should respect Timer IDs

[Thomas Groh (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thomas Groh resolved BEAM-3420.
---
   Resolution: Fixed
Fix Version/s: 2.4.0

> TimerData#compareTo should respect Timer IDs
> 
>
> Key: BEAM-3420
> URL: https://issues.apache.org/jira/browse/BEAM-3420
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Thomas Groh
>Assignee: Batkhuyag Batsaikhan
>Priority: Major
> Fix For: 2.4.0
>
>  Time Spent: 1.5h
>  Remaining Estimate: 0h
>
> https://github.com/apache/beam/blob/master/runners/core-java/src/main/java/org/apache/beam/runners/core/TimerInternals.java#L209
> Two timers that have distinct IDs should not compare as equal, even if 
> they're in the same namespace, for the same time in the same domain. They 
> should appear in an arbitrary, but consistent, order.
> We should have more tests surrounding these code paths.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] branch master updated (33c349f -> 89a9205)

[tgroh]

This is an automated email from the ASF dual-hosted git repository.

tgroh pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 33c349f  Merge pull request #4650 Use TestClock when TestStream is 
present in pipeline
 add cdf6bc2  Fixed a bug that timer ID was not used for comparing timer 
data. Added and refactored unit tests for TimerData.
 new 89a9205  Merge pull request #4670: Use timer ID to compare TimerData

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../apache/beam/runners/core/TimerInternals.java   | 10 ++-
 .../beam/runners/core/TimerInternalsTest.java  | 82 ++
 2 files changed, 59 insertions(+), 33 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
tg...@apache.org.

[beam] 01/01: Merge pull request #4670: Use timer ID to compare TimerData

[tgroh]

This is an automated email from the ASF dual-hosted git repository.

tgroh pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 89a92050d59185876c79e3ee3dc0a23dfcf8d2df
Merge: 33c349f cdf6bc2
Author: Thomas Groh 
AuthorDate: Wed Feb 14 17:01:12 2018 -0800

Merge pull request #4670: Use timer ID to compare TimerData

[BEAM-3420] Uses timer ID for comparing TimerData

 .../apache/beam/runners/core/TimerInternals.java   | 10 ++-
 .../beam/runners/core/TimerInternalsTest.java  | 82 ++
 2 files changed, 59 insertions(+), 33 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
tg...@apache.org.

Jenkins build is back to stable : beam_PostCommit_Java_ValidatesRunner_Apex #3481

[Apache Jenkins Server]

See 

[jira] [Updated] (BEAM-3709) Implement the portable lifted Combiner transforms in Python SDK

[Daniel Oliveira (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3709?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Oliveira updated BEAM-3709:
--
Component/s: (was: sdk-java-harness)
 (was: sdk-java-core)
 sdk-py-harness
 sdk-py-core

> Implement the portable lifted Combiner transforms in Python SDK
> ---
>
> Key: BEAM-3709
> URL: https://issues.apache.org/jira/browse/BEAM-3709
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core, sdk-py-harness
>Reporter: Daniel Oliveira
>Assignee: Daniel Oliveira
>Priority: Major
>  Labels: portability
>
> Lifted combines are split into separate parts with different URNs. These 
> parts need to be implemented in the Python SDK harness so that the SDK can 
> actually execute them when receiving Combine transforms with the 
> corresponding URNs.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-3711) Implement portable Combiner lifting in Dataflow Runner

[Daniel Oliveira (JIRA)]

Daniel Oliveira created BEAM-3711:
-

 Summary: Implement portable Combiner lifting in Dataflow Runner
 Key: BEAM-3711
 URL: https://issues.apache.org/jira/browse/BEAM-3711
 Project: Beam
  Issue Type: Sub-task
  Components: runner-dataflow
Reporter: Daniel Oliveira
Assignee: Daniel Oliveira


This is the task to make sure that once other parts of portable Combiner 
lifting are complete, that the actual Combiner lifting can be done in the 
Dataflow Runner.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-3710) Implement portable Combiner lifting decision logic

[Daniel Oliveira (JIRA)]

Daniel Oliveira created BEAM-3710:
-

 Summary: Implement portable Combiner lifting decision logic
 Key: BEAM-3710
 URL: https://issues.apache.org/jira/browse/BEAM-3710
 Project: Beam
  Issue Type: Sub-task
  Components: sdk-java-core, sdk-java-harness
Reporter: Daniel Oliveira
Assignee: Daniel Oliveira


The logic for deciding if a Combiner should be lifted or not needs to be 
implemented somewhere. If it's possible to provide all the necessary 
information for runners to perform this check then it should be implemented 
runner-side. Otherwise it should be implemented SDK-side.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-3709) Implement the portable lifted Combiner transforms in Python SDK

[Daniel Oliveira (JIRA)]

Daniel Oliveira created BEAM-3709:
-

 Summary: Implement the portable lifted Combiner transforms in 
Python SDK
 Key: BEAM-3709
 URL: https://issues.apache.org/jira/browse/BEAM-3709
 Project: Beam
  Issue Type: Sub-task
  Components: sdk-java-core, sdk-java-harness
Reporter: Daniel Oliveira
Assignee: Daniel Oliveira


Lifted combines are split into separate parts with different URNs. These parts 
need to be implemented in the Python SDK harness so that the SDK can actually 
execute them when receiving Combine transforms with the corresponding URNs.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-3708) Implement the portable lifted Combiner transforms in Java SDK

[Daniel Oliveira (JIRA)]

Daniel Oliveira created BEAM-3708:
-

 Summary: Implement the portable lifted Combiner transforms in Java 
SDK
 Key: BEAM-3708
 URL: https://issues.apache.org/jira/browse/BEAM-3708
 Project: Beam
  Issue Type: Sub-task
  Components: sdk-java-core, sdk-java-harness
Reporter: Daniel Oliveira
Assignee: Daniel Oliveira


Lifted combines are split into separate parts with different URNs. These parts 
need to be implemented in the Java SDK harness so that the SDK can actually 
execute them when receiving Combine transforms with the corresponding URNs.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] branch master updated (e668688 -> 33c349f)

[robertwb]

This is an automated email from the ASF dual-hosted git repository.

robertwb pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from e668688  Merge pull request #4646: [SQL] Refactor BeamSql interface
 add 41311f5  Use TestClock when TestStream is present in pipeline
 new 33c349f  Merge pull request #4650 Use TestClock when TestStream is 
present in pipeline

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../apache_beam/runners/direct/direct_runner.py| 32 ++
 1 file changed, 21 insertions(+), 11 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
rober...@apache.org.

[beam] 01/01: Merge pull request #4650 Use TestClock when TestStream is present in pipeline

[robertwb]

This is an automated email from the ASF dual-hosted git repository.

robertwb pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 33c349fffb22478807034a6aec7f2a212116683b
Merge: e668688 41311f5
Author: Robert Bradshaw 
AuthorDate: Wed Feb 14 16:29:13 2018 -0800

Merge pull request #4650 Use TestClock when TestStream is present in 
pipeline

Use TestClock when TestStream is present in pipeline

 .../apache_beam/runners/direct/direct_runner.py| 32 ++
 1 file changed, 21 insertions(+), 11 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
rober...@apache.org.

Build failed in Jenkins: beam_PerformanceTests_Python #915

[Apache Jenkins Server]

See 


Changes:

[arnaudfournier021] Minor Javadoc corrections for SketchFrequencies

[arnaudfournier021] [BEAM-2728] Add Quantiles finding transforms to sketching 
extension

[arnaudfournier021] Change coder serialization + improve Javadoc comments + 
minor fixes

[arnaudfournier921] Improve Javadoc ° minor fixes

[XuMingmin] [BEAM-3345][SQL] Reject unsupported inputs into JOIN (#4642)

[kedin] Update 'PCollectionTuple.apply()' generic definition

[kedin] [SQL] Refactor BeamSql

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam4 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision e668688690325127becfaef7486d02a6dae3e101 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f e668688690325127becfaef7486d02a6dae3e101
Commit message: "Merge pull request #4646: [SQL] Refactor BeamSql interface"
 > git rev-list 359ace325ce117103718f73a2249c140ec135cbe # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins8462557798293645102.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins6335700391301765482.sh
+ rm -rf .env
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins3918622110123112220.sh
+ virtualenv .env --system-site-packages
New python executable in .env/bin/python
Installing setuptools, pip...done.
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins8230619075881100319.sh
+ .env/bin/pip install --upgrade setuptools pip
Downloading/unpacking setuptools from 
https://pypi.python.org/packages/43/41/033a273f9a25cb63050a390ee8397acbc7eae2159195d85f06f17e7be45a/setuptools-38.5.1-py2.py3-none-any.whl#md5=908b8b5e50bf429e520b2b5fa1b350e5
Downloading/unpacking pip from 
https://pypi.python.org/packages/b6/ac/7015eb97dc749283ffdec1c3a88ddb8ae03b8fad0f0e611408f196358da3/pip-9.0.1-py2.py3-none-any.whl#md5=297dbd16ef53bcef0447d245815f5144
Installing collected packages: setuptools, pip
  Found existing installation: setuptools 2.2
Uninstalling setuptools:
  Successfully uninstalled setuptools
  Found existing installation: pip 1.5.4
Uninstalling pip:
  Successfully uninstalled pip
Successfully installed setuptools pip
Cleaning up...
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins561514847844962.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins8823925773942255777.sh
+ .env/bin/pip install -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied: absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: setuptools in ./.env/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 16))
Requirement already satisfied: colorlog[windows]==2.6.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied: futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied: PyYAML==3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied: pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Collecting numpy==1.13.3 (from -r PerfKitBenchmarker/requirements.txt (line 22))
  Using cached numpy-1.13.3-cp27-cp27mu-manylinux1_x86_64.whl

[beam] branch master updated (e12500e -> e668688)

[kenn]

This is an automated email from the ASF dual-hosted git repository.

kenn pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from e12500e  Merge pull request #4611: [BEAM-2728] Add TDigest 
implementation for finding Quantiles to sketching extension
 add 90fdc69  Update 'PCollectionTuple.apply()' generic definition
 add 01bb580  [SQL] Refactor BeamSql
 new e668688  Merge pull request #4646: [SQL] Refactor BeamSql interface

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../apache/beam/sdk/values/PCollectionTuple.java   |   4 +-
 .../apache/beam/sdk/extensions/sql/BeamSql.java| 205 +++--
 .../beam/sdk/extensions/sql/QueryTransform.java| 197 
 .../sdk/extensions/sql/QueryValidationHelper.java  |  55 ++
 .../sdk/extensions/sql/example/BeamSqlExample.java |   2 +-
 .../beam/sdk/extensions/sql/impl/BeamSqlEnv.java   |  50 -
 .../extensions/sql/BeamSqlDslAggregationTest.java  |  11 +-
 .../sdk/extensions/sql/BeamSqlDslFilterTest.java   |  11 +-
 .../sdk/extensions/sql/BeamSqlDslJoinTest.java |  12 +-
 .../sdk/extensions/sql/BeamSqlDslProjectTest.java  |  10 +-
 .../sdk/extensions/sql/BeamSqlDslUdfUdafTest.java  |  18 +-
 .../sdk/extensions/sql/BeamSqlNonAsciiTest.java|   4 +-
 .../beam/sdk/nexmark/queries/sql/SqlQuery0.java|   3 +-
 .../beam/sdk/nexmark/queries/sql/SqlQuery1.java|   5 +-
 .../beam/sdk/nexmark/queries/sql/SqlQuery2.java|   3 +-
 .../beam/sdk/nexmark/queries/sql/SqlQuery3.java|   5 +-
 16 files changed, 375 insertions(+), 220 deletions(-)
 create mode 100644 
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/QueryTransform.java
 create mode 100644 
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/QueryValidationHelper.java

-- 
To stop receiving notification emails like this one, please contact
k...@apache.org.

[beam] 01/01: Merge pull request #4646: [SQL] Refactor BeamSql interface

[kenn]

This is an automated email from the ASF dual-hosted git repository.

kenn pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit e668688690325127becfaef7486d02a6dae3e101
Merge: e12500e 01bb580
Author: Kenn Knowles 
AuthorDate: Wed Feb 14 15:51:52 2018 -0800

Merge pull request #4646: [SQL] Refactor BeamSql interface

 .../apache/beam/sdk/values/PCollectionTuple.java   |   4 +-
 .../apache/beam/sdk/extensions/sql/BeamSql.java| 205 +++--
 .../beam/sdk/extensions/sql/QueryTransform.java| 197 
 .../sdk/extensions/sql/QueryValidationHelper.java  |  55 ++
 .../sdk/extensions/sql/example/BeamSqlExample.java |   2 +-
 .../beam/sdk/extensions/sql/impl/BeamSqlEnv.java   |  50 -
 .../extensions/sql/BeamSqlDslAggregationTest.java  |  11 +-
 .../sdk/extensions/sql/BeamSqlDslFilterTest.java   |  11 +-
 .../sdk/extensions/sql/BeamSqlDslJoinTest.java |  12 +-
 .../sdk/extensions/sql/BeamSqlDslProjectTest.java  |  10 +-
 .../sdk/extensions/sql/BeamSqlDslUdfUdafTest.java  |  18 +-
 .../sdk/extensions/sql/BeamSqlNonAsciiTest.java|   4 +-
 .../beam/sdk/nexmark/queries/sql/SqlQuery0.java|   3 +-
 .../beam/sdk/nexmark/queries/sql/SqlQuery1.java|   5 +-
 .../beam/sdk/nexmark/queries/sql/SqlQuery2.java|   3 +-
 .../beam/sdk/nexmark/queries/sql/SqlQuery3.java|   5 +-
 16 files changed, 375 insertions(+), 220 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
k...@apache.org.

[jira] [Created] (BEAM-3707) Implement portable non-lifted Combines in Java SDK

[Daniel Oliveira (JIRA)]

Daniel Oliveira created BEAM-3707:
-

 Summary: Implement portable non-lifted Combines in Java SDK
 Key: BEAM-3707
 URL: https://issues.apache.org/jira/browse/BEAM-3707
 Project: Beam
  Issue Type: Sub-task
  Components: sdk-java-core
Reporter: Daniel Oliveira
Assignee: Daniel Oliveira


Get a portable version of non-lifted Combines working in Java.

Non-lifted combines should be done as an opaque ParDo after a GBK, created 
SDK-side. Runners won't be able to identify it as a Combine, but it should work 
without any changes to any runners.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-3706) Update CombinePayload to improved model for Portability

[Daniel Oliveira (JIRA)]

Daniel Oliveira created BEAM-3706:
-

 Summary: Update CombinePayload to improved model for Portability
 Key: BEAM-3706
 URL: https://issues.apache.org/jira/browse/BEAM-3706
 Project: Beam
  Issue Type: Sub-task
  Components: beam-model
Reporter: Daniel Oliveira
Assignee: Daniel Oliveira


This will mean changing the proto definition in beam_runner_api, most likely 
trimming out fields that are no longer necessary and adding any new ones that 
could be useful. The majority of work will probably be in investigating if some 
existing fields can actually be removed (SideInputs and Parameters for example).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Reopened] (BEAM-2728) Extension for sketch-based statistics

[JIRA]

 [ 
https://issues.apache.org/jira/browse/BEAM-2728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismaël Mejía reopened BEAM-2728:


I am reopening this since we are still missing one of the sketches of the 
original proposal. KMost frequent, good work [~arnaudfnr]. Just one more and 
this will be done !

> Extension for sketch-based statistics
> -
>
> Key: BEAM-2728
> URL: https://issues.apache.org/jira/browse/BEAM-2728
> Project: Beam
>  Issue Type: New Feature
>  Components: z-do-not-use-sdk-java-extensions
>Reporter: Arnaud Fournier
>Assignee: Arnaud Fournier
>Priority: Minor
>  Time Spent: 6h 50m
>  Remaining Estimate: 0h
>
> Goal : Provide an extension library to compute approximate statistics on 
> streams.
> Interest : Probabilistic data structures can create an approximation (sketch) 
> of the current state of a stream without storing every element but rather 
> processing each observation quickly to summarize its current state and find 
> useful statistical insights.
> Implementation is here : 
> https://github.com/ArnaudFnr/beam/tree/sketching/sdks/java/extensions/sketching
> More info : 
> https://docs.google.com/document/d/1Xy6g5RPBYX_HadpIr_2WrUeusiwL0Jo2ACI5PEOP1kc/edit



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-2728) Extension for sketch-based statistics

[JIRA]

 [ 
https://issues.apache.org/jira/browse/BEAM-2728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismaël Mejía updated BEAM-2728:
---
Fix Version/s: (was: 2.3.0)

> Extension for sketch-based statistics
> -
>
> Key: BEAM-2728
> URL: https://issues.apache.org/jira/browse/BEAM-2728
> Project: Beam
>  Issue Type: New Feature
>  Components: z-do-not-use-sdk-java-extensions
>Reporter: Arnaud Fournier
>Assignee: Arnaud Fournier
>Priority: Minor
>  Time Spent: 6h 50m
>  Remaining Estimate: 0h
>
> Goal : Provide an extension library to compute approximate statistics on 
> streams.
> Interest : Probabilistic data structures can create an approximation (sketch) 
> of the current state of a stream without storing every element but rather 
> processing each observation quickly to summarize its current state and find 
> useful statistical insights.
> Implementation is here : 
> https://github.com/ArnaudFnr/beam/tree/sketching/sdks/java/extensions/sketching
> More info : 
> https://docs.google.com/document/d/1Xy6g5RPBYX_HadpIr_2WrUeusiwL0Jo2ACI5PEOP1kc/edit



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Jenkins build is unstable: beam_PostCommit_Java_ValidatesRunner_Apex #3480

[Apache Jenkins Server]

See 

[jira] [Assigned] (BEAM-3679) Upgrade calcite to release 1.16

[Anton Kedin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Anton Kedin reassigned BEAM-3679:
-

Assignee: Andrew Pilloud  (was: Kenneth Knowles)

> Upgrade calcite to release 1.16
> ---
>
> Key: BEAM-3679
> URL: https://issues.apache.org/jira/browse/BEAM-3679
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql
>Reporter: Ted Yu
>Assignee: Andrew Pilloud
>Priority: Minor
> Fix For: 2.4.0
>
>
> Currently Beam uses Calcite 1.13.0
> This issue is to upgrade to Calcite 1.16.0



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-3679) Upgrade calcite to release 1.16

[Anton Kedin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Anton Kedin reassigned BEAM-3679:
-

Assignee: Kenneth Knowles  (was: Xu Mingmin)

> Upgrade calcite to release 1.16
> ---
>
> Key: BEAM-3679
> URL: https://issues.apache.org/jira/browse/BEAM-3679
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql
>Reporter: Ted Yu
>Assignee: Kenneth Knowles
>Priority: Minor
> Fix For: 2.4.0
>
>
> Currently Beam uses Calcite 1.13.0
> This issue is to upgrade to Calcite 1.16.0



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] branch master updated (e9cd1f9 -> e12500e)

[jkff]

This is an automated email from the ASF dual-hosted git repository.

jkff pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from e9cd1f9  [BEAM-3345][SQL] Reject unsupported inputs into JOIN (#4642)
 add 80fab79  Minor Javadoc corrections for SketchFrequencies
 add fa94a50  [BEAM-2728] Add Quantiles finding transforms to sketching 
extension
 add 98ea654  Change coder serialization + improve Javadoc comments + minor 
fixes
 add 3313848  Improve Javadoc ° minor fixes
 new e12500e  Merge pull request #4611: [BEAM-2728] Add TDigest 
implementation for finding Quantiles to sketching extension

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 sdks/java/extensions/sketching/build.gradle|   2 +
 sdks/java/extensions/sketching/pom.xml |  10 +-
 .../extensions/sketching/ApproximateDistinct.java  |  50 ++-
 .../extensions/sketching/SketchFrequencies.java|  42 ++-
 .../sdk/extensions/sketching/TDigestQuantiles.java | 349 +
 .../extensions/sketching/TDigestQuantilesTest.java | 180 +++
 6 files changed, 621 insertions(+), 12 deletions(-)
 create mode 100644 
sdks/java/extensions/sketching/src/main/java/org/apache/beam/sdk/extensions/sketching/TDigestQuantiles.java
 create mode 100644 
sdks/java/extensions/sketching/src/test/java/org/apache/beam/sdk/extensions/sketching/TDigestQuantilesTest.java

-- 
To stop receiving notification emails like this one, please contact
j...@apache.org.

[beam] 01/01: Merge pull request #4611: [BEAM-2728] Add TDigest implementation for finding Quantiles to sketching extension

[jkff]

This is an automated email from the ASF dual-hosted git repository.

jkff pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit e12500e6ca03f661538a7bd21404c8f60012831b
Merge: e9cd1f9 3313848
Author: Eugene Kirpichov 
AuthorDate: Wed Feb 14 11:52:46 2018 -0800

Merge pull request #4611: [BEAM-2728] Add TDigest implementation for 
finding Quantiles to sketching extension

[BEAM-2728] Add TDigest implementation for finding Quantiles to sketching 
extension

 sdks/java/extensions/sketching/build.gradle|   2 +
 sdks/java/extensions/sketching/pom.xml |  10 +-
 .../extensions/sketching/ApproximateDistinct.java  |  50 ++-
 .../extensions/sketching/SketchFrequencies.java|  42 ++-
 .../sdk/extensions/sketching/TDigestQuantiles.java | 349 +
 .../extensions/sketching/TDigestQuantilesTest.java | 180 +++
 6 files changed, 621 insertions(+), 12 deletions(-)

diff --cc sdks/java/extensions/sketching/pom.xml
index 50e487d,eef5fd7..dfc21b2
--- a/sdks/java/extensions/sketching/pom.xml
+++ b/sdks/java/extensions/sketching/pom.xml
@@@ -46,7 -47,19 +47,14 @@@
${streamlib.version}
  
  
+ 
+ 
+   com.tdunning
+   t-digest
+   ${t-digest.version}
+ 
+ 
  
 -  org.slf4j
 -  slf4j-api
 -
 -
 -
com.google.guava
guava
  

-- 
To stop receiving notification emails like this one, please contact
j...@apache.org.

[jira] [Created] (BEAM-3705) ApproximateUnique discards accumulated data with multiple firings.

[Raghu Angadi (JIRA)]

Raghu Angadi created BEAM-3705:
--

 Summary: ApproximateUnique discards accumulated data with multiple 
firings. 
 Key: BEAM-3705
 URL: https://issues.apache.org/jira/browse/BEAM-3705
 Project: Beam
  Issue Type: Bug
  Components: sdk-java-core
Reporter: Raghu Angadi
Assignee: Raghu Angadi


`extractOutput()` on `ApproximateUniqueCombineFn` resets the accumulated value. 
This discards accumulated state when making subsequent firings incorrect.

 

First reported in https://stackoverflow.com/questions/48698028/. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-3621) HCatalog failing frequently on fetch of org.pentaho:pentaho-aggdesigner-algorithm:5.1.5-jhyde

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles reassigned BEAM-3621:
-

Assignee: (was: Reuven Lax)

> HCatalog failing frequently on fetch of 
> org.pentaho:pentaho-aggdesigner-algorithm:5.1.5-jhyde
> -
>
> Key: BEAM-3621
> URL: https://issues.apache.org/jira/browse/BEAM-3621
> Project: Beam
>  Issue Type: Bug
>  Components: io-java-hcatalog
>Reporter: Kenneth Knowles
>Priority: Critical
>  Labels: flake
>
> The download seems to fail. There are three aspects to this:
>  - The Gradle build should have a local persistent cache so it isn't 
> fetching. But as I understand it Gradle has a policy of still pinging the 
> repo even when cached.
>  - The Gradle build is probably re-fetching due to {{--rerun-tasks}} so we 
> need to stop doing that.
>  - The artifact is not in "Central" repository but in "Spring Plugins". That 
> is probably why all the failures are on this artifact and no other artifacts 
> cause failures.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] branch master updated: [BEAM-3345][SQL] Reject unsupported inputs into JOIN (#4642)

[mingmxu]

This is an automated email from the ASF dual-hosted git repository.

mingmxu pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git


The following commit(s) were added to refs/heads/master by this push:
 new e9cd1f9  [BEAM-3345][SQL] Reject unsupported inputs into JOIN (#4642)
e9cd1f9 is described below

commit e9cd1f9c92636e968cb0b550f2ee3a6fe5d103a5
Author: Anton Kedin <33067037+ake...@users.noreply.github.com>
AuthorDate: Wed Feb 14 11:16:19 2018 -0800

[BEAM-3345][SQL] Reject unsupported inputs into JOIN (#4642)

* [SQL] Reject unsupported inputs into JOIN

Allow only non-global windows with default trigger or end of window trigger.

* fixup! [SQL] Reject unsupported inputs into JOIN
---
 sdks/java/extensions/sql/pom.xml   |   5 +
 .../sdk/extensions/sql/impl/rel/BeamJoinRel.java   |  39 -
 .../sdk/extensions/sql/BeamSqlDslJoinTest.java | 195 +++--
 .../apache/beam/sdk/extensions/sql/TestUtils.java  |  19 ++
 4 files changed, 244 insertions(+), 14 deletions(-)

diff --git a/sdks/java/extensions/sql/pom.xml b/sdks/java/extensions/sql/pom.xml
index 16cfe77..46de596 100644
--- a/sdks/java/extensions/sql/pom.xml
+++ b/sdks/java/extensions/sql/pom.xml
@@ -409,6 +409,11 @@
   test
 
 
+  org.hamcrest
+  hamcrest-core
+  test
+
+
   org.mockito
   mockito-all
   ${mockito.version}
diff --git 
a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamJoinRel.java
 
b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamJoinRel.java
index bebfca3..89196ef 100644
--- 
a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamJoinRel.java
+++ 
b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamJoinRel.java
@@ -18,7 +18,9 @@
 
 package org.apache.beam.sdk.extensions.sql.impl.rel;
 
+import static org.apache.beam.sdk.values.PCollection.IsBounded.UNBOUNDED;
 import static org.apache.beam.sdk.values.RowType.toRowType;
+import static org.joda.time.Duration.ZERO;
 
 import com.google.common.base.Joiner;
 import java.util.ArrayList;
@@ -35,7 +37,10 @@ import 
org.apache.beam.sdk.extensions.sql.impl.utils.CalciteUtils;
 import org.apache.beam.sdk.transforms.MapElements;
 import org.apache.beam.sdk.transforms.ParDo;
 import org.apache.beam.sdk.transforms.View;
+import org.apache.beam.sdk.transforms.windowing.DefaultTrigger;
+import org.apache.beam.sdk.transforms.windowing.GlobalWindows;
 import org.apache.beam.sdk.transforms.windowing.IncompatibleWindowException;
+import org.apache.beam.sdk.transforms.windowing.Trigger;
 import org.apache.beam.sdk.transforms.windowing.WindowFn;
 import org.apache.beam.sdk.values.KV;
 import org.apache.beam.sdk.values.PCollection;
@@ -43,6 +48,7 @@ import org.apache.beam.sdk.values.PCollectionTuple;
 import org.apache.beam.sdk.values.PCollectionView;
 import org.apache.beam.sdk.values.Row;
 import org.apache.beam.sdk.values.RowType;
+import org.apache.beam.sdk.values.WindowingStrategy;
 import org.apache.calcite.plan.RelOptCluster;
 import org.apache.calcite.plan.RelTraitSet;
 import org.apache.calcite.rel.RelNode;
@@ -112,6 +118,9 @@ public class BeamJoinRel extends Join implements 
BeamRelNode {
 PCollection leftRows = 
leftRelNode.buildBeamPipeline(inputPCollections, sqlEnv);
 PCollection rightRows = 
rightRelNode.buildBeamPipeline(inputPCollections, sqlEnv);
 
+verifySupportedTrigger(leftRows);
+verifySupportedTrigger(rightRows);
+
 String stageName = BeamSqlRelUtils.getStageName(this);
 WindowFn leftWinFn = leftRows.getWindowingStrategy().getWindowFn();
 WindowFn rightWinFn = rightRows.getWindowingStrategy().getWindowFn();
@@ -151,8 +160,8 @@ public class BeamJoinRel extends Join implements 
BeamRelNode {
 // a regular join
 if ((leftRows.isBounded() == PCollection.IsBounded.BOUNDED
 && rightRows.isBounded() == PCollection.IsBounded.BOUNDED)
-   || (leftRows.isBounded() == PCollection.IsBounded.UNBOUNDED
-&& rightRows.isBounded() == PCollection.IsBounded.UNBOUNDED)) {
+   || (leftRows.isBounded() == UNBOUNDED
+&& rightRows.isBounded() == UNBOUNDED)) {
   try {
 leftWinFn.verifyCompatibility(rightWinFn);
   } catch (IncompatibleWindowException e) {
@@ -164,8 +173,8 @@ public class BeamJoinRel extends Join implements 
BeamRelNode {
   leftNullRow, rightNullRow, stageName);
 } else if (
 (leftRows.isBounded() == PCollection.IsBounded.BOUNDED
-&& rightRows.isBounded() == PCollection.IsBounded.UNBOUNDED)
-|| (leftRows.isBounded() == PCollection.IsBounded.UNBOUNDED
+&& rightRows.isBounded() == UNBOUNDED)
+|| (leftRows.isBounded() == UNBOUNDED
 && rightRows.isBounded() == PCollection.IsBounded.BOUNDED)
 ) {
   // if one of 

[jira] [Assigned] (BEAM-2597) FlinkRunner Fn API based ParDo operator

[Ben Sidhom (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ben Sidhom reassigned BEAM-2597:


Assignee: Ben Sidhom  (was: Aljoscha Krettek)

> FlinkRunner Fn API based ParDo operator
> ---
>
> Key: BEAM-2597
> URL: https://issues.apache.org/jira/browse/BEAM-2597
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-flink
>Reporter: Kenneth Knowles
>Assignee: Ben Sidhom
>Priority: Major
>  Labels: portability
>
> To run non-Java SDK code is to put together an operator that manages a Fn API 
> client DoFnRunner and an SDK harness Fn API server.
> (filing to organize steps, details of this may evolve as it is implemented)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-1254) SuppressWarnings("serial") for starter and example archetypes

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-1254:
--
Component/s: (was: sdk-java-extensions)
 examples-java

> SuppressWarnings("serial") for starter and example archetypes
> -
>
> Key: BEAM-1254
> URL: https://issues.apache.org/jira/browse/BEAM-1254
> Project: Beam
>  Issue Type: Bug
>  Components: examples-java
>Reporter: Kenneth Knowles
>Priority: Minor
>  Labels: easy, starter
>
> The usual case for using Beam is that every user `DoFn`, etc, is serialized 
> with autogenerated `serialVersionUID`. Our starter and example archetypes 
> should reflect this, so when someone loads them up in an IDE they are not 
> presented with warning-throwing code.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-1891) Java @Autovalue: currently doesn't have a good coder

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-1891:
--
Component/s: (was: sdk-ideas)
 sdk-java-core

> Java @Autovalue: currently doesn't have a good coder
> 
>
> Key: BEAM-1891
> URL: https://issues.apache.org/jira/browse/BEAM-1891
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Stephen Sisk
>Priority: Minor
>  Labels: newbie, starter
>
> In Java, @AutoValue classes are something that we would like developers to be 
> able to use in PCollections in Beam.
> However, there doesn't appear to be a good existing Beam Coder for Autovalue 
> generated classes:
> * Avrocoder doesn't work
> * Serializable Coder works, but has other problems (larger/less efficient)
> This is discussed fully at 
> https://lists.apache.org/thread.html/29617096819824d5c12247a246d316b763d9e583a21fff2f1c430077@%3Cdev.beam.apache.org%3E
>  
> We may need to implement an AutoValueCoder.
> cc [~pabloem] [~bchambers]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-2283) Consider using actual URIs instead of Strings/ResourceIds in relation to FileSystems

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-2283:
--
Component/s: (was: sdk-java-extensions)
 io-ideas

> Consider using actual URIs instead of Strings/ResourceIds in relation to 
> FileSystems
> 
>
> Key: BEAM-2283
> URL: https://issues.apache.org/jira/browse/BEAM-2283
> Project: Beam
>  Issue Type: Improvement
>  Components: io-ideas, sdk-java-core, sdk-java-gcp, sdk-py-core
>Reporter: Luke Cwik
>Priority: Major
>
> We treat things like URIs because we expect them to have a scheme component 
> and to be able to resolve a parent/child but fail to treat them as URIs in 
> the internal implementation since our string versions don't go through URI 
> normalization. This brings up a few issues:
> * The cost of implementing and maintaining ResourceIds instead of having 
> users use a standard URI implementation. This would just require FileSystems 
> to be able to take a string and give back a URI (to enable them to have 
> custom implementations in case they extend the concept of URIs with scheme 
> specific extensions).
> * The myriad of bugs that will come up because of improper usage of URI like 
> strings and the assumptions associated with them (like 
> https://issues.apache.org/jira/browse/BEAM-2277)
> Note that swapping to URIs adds complexity because:
> * Resolving URIs with glob expressions needs to be handled carefully
> * FileSystems may need to implement a complicated type instead of ResourceId.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-1891) Java @Autovalue: currently doesn't have a good coder

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-1891:
--
Component/s: (was: sdk-java-extensions)
 sdk-ideas

> Java @Autovalue: currently doesn't have a good coder
> 
>
> Key: BEAM-1891
> URL: https://issues.apache.org/jira/browse/BEAM-1891
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Stephen Sisk
>Priority: Minor
>  Labels: newbie, starter
>
> In Java, @AutoValue classes are something that we would like developers to be 
> able to use in PCollections in Beam.
> However, there doesn't appear to be a good existing Beam Coder for Autovalue 
> generated classes:
> * Avrocoder doesn't work
> * Serializable Coder works, but has other problems (larger/less efficient)
> This is discussed fully at 
> https://lists.apache.org/thread.html/29617096819824d5c12247a246d316b763d9e583a21fff2f1c430077@%3Cdev.beam.apache.org%3E
>  
> We may need to implement an AutoValueCoder.
> cc [~pabloem] [~bchambers]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-1879) PTransform style guide should discuss display data

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-1879:
--
Component/s: (was: sdk-java-extensions)
 website

> PTransform style guide should discuss display data
> --
>
> Key: BEAM-1879
> URL: https://issues.apache.org/jira/browse/BEAM-1879
> Project: Beam
>  Issue Type: Improvement
>  Components: website
>Reporter: Stephen Sisk
>Assignee: Eugene Kirpichov
>Priority: Major
>
> Currently, the PTransform style guide 
> (https://beam.apache.org/contribute/ptransform-style-guide/) does not discuss 
> display data at all.
> We should make sure to discuss testing display data - specifically  that 
> using DisplayDataEvaluator is a best practice for testing since without it, 
> you cannot tell whether or not the display data will actually be displayed.
> cc [~swegner] [~bchambers] [~jkff]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-1861) ORC support

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1861?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-1861:
--
Issue Type: Wish  (was: Bug)

> ORC support
> ---
>
> Key: BEAM-1861
> URL: https://issues.apache.org/jira/browse/BEAM-1861
> Project: Beam
>  Issue Type: Wish
>  Components: io-ideas
>Reporter: Tibor Kiss
>Assignee: Tibor Kiss
>Priority: Major
>
> Add support for Optimized Row Columnar (ORC) file format: 
> https://orc.apache.org/
> As ORC does not have Python bindings (yet) this effort focuses on Java based 
> implementation



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-3138) Stop depending on Test JARs

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3138?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-3138:
--
Component/s: (was: sdk-java-extensions)

> Stop depending on Test JARs
> ---
>
> Key: BEAM-3138
> URL: https://issues.apache.org/jira/browse/BEAM-3138
> Project: Beam
>  Issue Type: Bug
>  Components: runner-core, sdk-java-core, sdk-java-gcp, 
> sdk-java-harness
>Reporter: Thomas Groh
>Priority: Minor
>
> Testing components can be in a testing or otherwise signaled package, but 
> shouldn't really be depended on by depending on a test jar in the test scope.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-2805) Potential arithmetic overflow in Generator#nextAuctionLengthMs()

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2805?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles reassigned BEAM-2805:
-

Assignee: (was: Reuven Lax)

> Potential arithmetic overflow in Generator#nextAuctionLengthMs()
> 
>
> Key: BEAM-2805
> URL: https://issues.apache.org/jira/browse/BEAM-2805
> Project: Beam
>  Issue Type: Bug
>  Components: examples-nexmark
>Reporter: Ted Yu
>Priority: Minor
>
> {code}
> long numEventsForAuctions =
> (config.configuration.numInFlightAuctions * 
> GeneratorConfig.PROPORTION_DENOMINATOR)
> / GeneratorConfig.AUCTION_PROPORTION;
> {code}
> The multiplication is done on 32-bit integers while long is expected 
> (numEventsForAuctions).
> There is possibility for arithmetic overflow



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-2458) Move HashingFn from test -> main to make it more accessible

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2458?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-2458:
--
Component/s: (was: sdk-java-extensions)
 io-java-testing

> Move HashingFn from test -> main to make it more accessible
> ---
>
> Key: BEAM-2458
> URL: https://issues.apache.org/jira/browse/BEAM-2458
> Project: Beam
>  Issue Type: Bug
>  Components: io-java-utilities
>Reporter: Stephen Sisk
>Priority: Major
>
> HashingFn is currently only available as a test dependency. There's no reason 
> for that (it could be generally useful to non-test parts of IOs if 
> necessary), so we should move it over to regular directory.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-1987) extend join-library to support 3+ PCollections

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-1987:
--
Component/s: (was: sdk-java-extensions)
 sdk-java-join-library

> extend join-library to support 3+ PCollections
> --
>
> Key: BEAM-1987
> URL: https://issues.apache.org/jira/browse/BEAM-1987
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-join-library
>Reporter: Xu Mingmin
>Priority: Major
>
> {{join.class}} in join-library can handle two source PCollections, to perform 
> inner/left_outer/right_outer join. 
> Extend the capability to handle more upstream PCollections.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (BEAM-2614) Harness doesn't build with Java7

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles resolved BEAM-2614.
---
   Resolution: Won't Fix
Fix Version/s: Not applicable

We are going to Java 8 now, even though this was a major oversight when it 
occurred.

> Harness doesn't build with Java7
> 
>
> Key: BEAM-2614
> URL: https://issues.apache.org/jira/browse/BEAM-2614
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Affects Versions: 2.0.0, 2.1.0
>Reporter: Jean-Baptiste Onofré
>Assignee: Jean-Baptiste Onofré
>Priority: Major
> Fix For: Not applicable
>
>
> Beam is supposed to fully build with Java7. However, the {{harness}} module 
> doesn't:
> {code}
> [ERROR] Failed to execute goal 
> org.apache.maven.plugins:maven-compiler-plugin:3.6.1:compile 
> (default-compile) on project beam-sdks-java-harness: Fatal error compiling: 
> invalid target release: 1.8 -> [Help 1]
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-352) Add DisplayData to HDFS Sources

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-352?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-352:
-
Component/s: (was: sdk-java-extensions)
 io-java-hadoop

> Add DisplayData to HDFS Sources
> ---
>
> Key: BEAM-352
> URL: https://issues.apache.org/jira/browse/BEAM-352
> Project: Beam
>  Issue Type: Bug
>  Components: io-java-hadoop
>Reporter: Ben Chambers
>Assignee: James Malone
>Priority: Minor
>  Labels: starter
>
> Any interesting parameters of the sources/sinks should be exposed as display 
> data. See any of the sources/sinks that already export this (BigQuery, 
> PubSub, etc.) for examples. Also look at the DisplayData builder and 
> HasDisplayData interface for how to wire these up.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-3681) Amazon S3 write breaks randomly

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-3681:
--
Component/s: (was: sdk-java-extensions)
 io-java-aws

> Amazon S3 write breaks randomly
> ---
>
> Key: BEAM-3681
> URL: https://issues.apache.org/jira/browse/BEAM-3681
> Project: Beam
>  Issue Type: Bug
>  Components: io-java-aws
>Affects Versions: 2.3.0, 2.4.0
>Reporter: Ismaël Mejía
>Assignee: Ismaël Mejía
>Priority: Major
>
> When executing a simple write on S3 with the direct runner. It breaks 
> sometimes when it ends up trying to write 'empty' shards to S3.
> {code:java}
> Pipeline pipeline = Pipeline.create(options);
> pipeline
>  .apply("CreateSomeData", Create.of("1", "2", "3"))
>  .apply("WriteToFS", TextIO.write().to(options.getOutput()));
> pipeline.run();{code}
> The related exception is:
> {code:java}
> Exception in thread "main" 
> org.apache.beam.sdk.Pipeline$PipelineExecutionException: java.io.IOException: 
> com.amazonaws.services.s3.model.AmazonS3Exception: The XML you provided was 
> not well-formed or did not validate against our published schema (Service: 
> Amazon S3; Status Code: 400; Error Code: MalformedXML; Request ID: 
> 402E99C2F602AD09; S3 Extended Request ID: 
> SDdU8AqW2mfZuG1xcKUSNeHiR0IUKcRCpZ1Wjx7sAor1CdYf8f+0dDIcQpvr3GXgqwsyk5PGWVE=),
>  S3 Extended Request ID: 
> SDdU8AqW2mfZuG1xcKUSNeHiR0IUKcRCpZ1Wjx7sAor1CdYf8f+0dDIcQpvr3GXgqwsyk5PGWVE=
>     at 
> org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:342)
>     at 
> org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:312)
>     at org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:206)
>     at org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:62)
>     at org.apache.beam.sdk.Pipeline.run(Pipeline.java:311)
>     at org.apache.beam.sdk.Pipeline.run(Pipeline.java:297)
>     at 
> org.apache.beam.samples.ingest.amazon.IngestToS3.main(IngestToS3.java:82)
> Caused by: java.io.IOException: 
> com.amazonaws.services.s3.model.AmazonS3Exception: The XML you provided was 
> not well-formed or did not validate against our published schema (Service: 
> Amazon S3; Status Code: 400; Error Code: MalformedXML; Request ID: 
> 402E99C2F602AD09; S3 Extended Request ID: 
> SDdU8AqW2mfZuG1xcKUSNeHiR0IUKcRCpZ1Wjx7sAor1CdYf8f+0dDIcQpvr3GXgqwsyk5PGWVE=),
>  S3 Extended Request ID: 
> SDdU8AqW2mfZuG1xcKUSNeHiR0IUKcRCpZ1Wjx7sAor1CdYf8f+0dDIcQpvr3GXgqwsyk5PGWVE=
>     at org.apache.beam.sdk.io.aws.s3.S3FileSystem.copy(S3FileSystem.java:563)
>     at 
> org.apache.beam.sdk.io.aws.s3.S3FileSystem.lambda$copy$4(S3FileSystem.java:495)
>     at 
> org.apache.beam.sdk.io.aws.s3.S3FileSystem.lambda$callTasks$8(S3FileSystem.java:642)
>     at 
> org.apache.beam.sdk.util.MoreFutures.lambda$supplyAsync$0(MoreFutures.java:100)
>     at 
> java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1626)
> Caused by: com.amazonaws.services.s3.model.AmazonS3Exception: The XML you 
> provided was not well-formed or did not validate against our published schema 
> (Service: Amazon S3; Status Code: 400; Error Code: MalformedXML; Request ID: 
> 402E99C2F602AD09; S3 Extended Request ID: 
> SDdU8AqW2mfZuG1xcKUSNeHiR0IUKcRCpZ1Wjx7sAor1CdYf8f+0dDIcQpvr3GXgqwsyk5PGWVE=),
>  S3 Extended Request ID: 
> SDdU8AqW2mfZuG1xcKUSNeHiR0IUKcRCpZ1Wjx7sAor1CdYf8f+0dDIcQpvr3GXgqwsyk5PGWVE=
>     at 
> com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1639)
>     at 
> com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1304)
>     at 
> com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1056)
>     at 
> com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:743)
>     at 
> com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:717)
>     at 
> com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699)
>     at 
> com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667)
>     at 
> com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649)
>     at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513)
>     at 
> com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4325)
>     at 
> com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4272)
>     at 
> com.amazonaws.services.s3.AmazonS3Client.completeMultipartUpload(AmazonS3Client.java:3065)
>     at org.apache.beam.sdk.io.aws.s3.S3FileSystem.copy(S3FileSystem.java:561)
>     at 
> 

[jira] [Updated] (BEAM-3211) Add an integration test for TextIO ReadAll transform and dynamic writes

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3211?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-3211:
--
Component/s: (was: sdk-java-extensions)
 sdk-java-core

> Add an integration test for TextIO ReadAll transform and dynamic writes
> ---
>
> Key: BEAM-3211
> URL: https://issues.apache.org/jira/browse/BEAM-3211
> Project: Beam
>  Issue Type: Test
>  Components: sdk-java-core
>Reporter: Chamikara Jayalath
>Assignee: Łukasz Gajowy
>Priority: Major
>
> We should add a small scale version of performance test available in 
> following file run as a part of 'beam_PostCommit_Java_MavenInstall' and 
> 'beam_PostCommit_Java_ValidatesRunner*' Jenkins test suites.
> https://github.com/apache/beam/blob/master/sdks/java/io/file-based-io-tests/src/test/java/org/apache/beam/sdk/io/text/TextIOIT.java



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-949) Need tests for HDFSFileSink

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-949?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-949:
-
Component/s: (was: sdk-java-extensions)
 io-java-hadoop

> Need tests for HDFSFileSink
> ---
>
> Key: BEAM-949
> URL: https://issues.apache.org/jira/browse/BEAM-949
> Project: Beam
>  Issue Type: Test
>  Components: io-java-hadoop
>Reporter: Pei He
>Assignee: James Malone
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-123) Skip header row from csv

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-123:
-
Component/s: (was: sdk-java-extensions)
 sdk-java-core

> Skip header row from csv 
> -
>
> Key: BEAM-123
> URL: https://issues.apache.org/jira/browse/BEAM-123
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-core
>Reporter: Davin Pidoto
>Priority: Minor
>  Labels: newbie, starter
>
> Add functionality to skip header rows when reading from a csv file.
> http://stackoverflow.com/questions/28450554/skipping-header-rows-is-it-possible-with-cloud-dataflow



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (BEAM-2728) Extension for sketch-based statistics

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles resolved BEAM-2728.
---
   Resolution: Fixed
Fix Version/s: 2.3.0

I'd say this is resolved - we've got the extension and it is going out with 
2.3.0

> Extension for sketch-based statistics
> -
>
> Key: BEAM-2728
> URL: https://issues.apache.org/jira/browse/BEAM-2728
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-extensions
>Reporter: Arnaud Fournier
>Assignee: Arnaud Fournier
>Priority: Minor
> Fix For: 2.3.0
>
>  Time Spent: 6.5h
>  Remaining Estimate: 0h
>
> Goal : Provide an extension library to compute approximate statistics on 
> streams.
> Interest : Probabilistic data structures can create an approximation (sketch) 
> of the current state of a stream without storing every element but rather 
> processing each observation quickly to summarize its current state and find 
> useful statistical insights.
> Implementation is here : 
> https://github.com/ArnaudFnr/beam/tree/sketching/sdks/java/extensions/sketching
> More info : 
> https://docs.google.com/document/d/1Xy6g5RPBYX_HadpIr_2WrUeusiwL0Jo2ACI5PEOP1kc/edit



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-2805) Potential arithmetic overflow in Generator#nextAuctionLengthMs()

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2805?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-2805:
--
Component/s: (was: sdk-java-extensions)
 examples-nexmark

> Potential arithmetic overflow in Generator#nextAuctionLengthMs()
> 
>
> Key: BEAM-2805
> URL: https://issues.apache.org/jira/browse/BEAM-2805
> Project: Beam
>  Issue Type: Bug
>  Components: examples-nexmark
>Reporter: Ted Yu
>Assignee: Reuven Lax
>Priority: Minor
>
> {code}
> long numEventsForAuctions =
> (config.configuration.numInFlightAuctions * 
> GeneratorConfig.PROPORTION_DENOMINATOR)
> / GeneratorConfig.AUCTION_PROPORTION;
> {code}
> The multiplication is done on 32-bit integers while long is expected 
> (numEventsForAuctions).
> There is possibility for arithmetic overflow



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-3038) Add support for Azure Data Lake Storage as a Apache Beam FileSystem

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3038?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-3038:
--
Component/s: (was: sdk-java-extensions)
 io-ideas

> Add support for Azure Data Lake Storage as a Apache Beam FileSystem
> ---
>
> Key: BEAM-3038
> URL: https://issues.apache.org/jira/browse/BEAM-3038
> Project: Beam
>  Issue Type: Improvement
>  Components: io-ideas
>Reporter: Romit Girdhar
>Assignee: Jean-Baptiste Onofré
>Priority: Minor
>  Labels: features
>
> This is for providing direct integration with Azure Data Lake Store as an 
> Apache Beam File system.
> There is already support for Azure Data Lake for using it as HDFS: 
> https://hadoop.apache.org/docs/current/hadoop-azure-datalake/index.html



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-3621) HCatalog failing frequently on fetch of org.pentaho:pentaho-aggdesigner-algorithm:5.1.5-jhyde

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-3621:
--
Component/s: (was: sdk-java-extensions)
 io-java-hcatalog

> HCatalog failing frequently on fetch of 
> org.pentaho:pentaho-aggdesigner-algorithm:5.1.5-jhyde
> -
>
> Key: BEAM-3621
> URL: https://issues.apache.org/jira/browse/BEAM-3621
> Project: Beam
>  Issue Type: Bug
>  Components: io-java-hcatalog
>Reporter: Kenneth Knowles
>Assignee: Reuven Lax
>Priority: Critical
>  Labels: flake
>
> The download seems to fail. There are three aspects to this:
>  - The Gradle build should have a local persistent cache so it isn't 
> fetching. But as I understand it Gradle has a policy of still pinging the 
> repo even when cached.
>  - The Gradle build is probably re-fetching due to {{--rerun-tasks}} so we 
> need to stop doing that.
>  - The artifact is not in "Central" repository but in "Spring Plugins". That 
> is probably why all the failures are on this artifact and no other artifacts 
> cause failures.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-3096) generic api support for Graph Computation like GraphX on Spark

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-3096:
--
Component/s: (was: sdk-java-extensions)
 sdk-ideas

> generic api support for Graph Computation like GraphX on Spark
> --
>
> Key: BEAM-3096
> URL: https://issues.apache.org/jira/browse/BEAM-3096
> Project: Beam
>  Issue Type: Wish
>  Components: sdk-ideas
>Reporter: rayeaster
>Priority: Major
>  Labels: features
>   Original Estimate: 1,344h
>  Remaining Estimate: 1,344h
>
> Is there any plan to add support for graph computation like GraphX on Spark?
> * graph representation in PCollection 
> * basic statistics like vertex/edge count
> * base function like vertex/edge-wise mapreduce task(i.e., count the outgoing 
> degree of a vertex)
> * base function like subgraph combine/join
> * ..



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-1861) ORC support

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1861?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-1861:
--
Component/s: (was: sdk-java-extensions)
 io-ideas

> ORC support
> ---
>
> Key: BEAM-1861
> URL: https://issues.apache.org/jira/browse/BEAM-1861
> Project: Beam
>  Issue Type: Bug
>  Components: io-ideas
>Reporter: Tibor Kiss
>Assignee: Tibor Kiss
>Priority: Major
>
> Add support for Optimized Row Columnar (ORC) file format: 
> https://orc.apache.org/
> As ORC does not have Python bindings (yet) this effort focuses on Java based 
> implementation



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-1289) TextIO (and others) should provide more informative messages when encountering RVP

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-1289:
--
Component/s: (was: sdk-java-extensions)
 sdk-java-core

> TextIO (and others) should provide more informative messages when 
> encountering RVP
> --
>
> Key: BEAM-1289
> URL: https://issues.apache.org/jira/browse/BEAM-1289
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Ben Chambers
>Assignee: Sam McVeety
>Priority: Minor
>  Labels: starter
>
> https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/io/TextIO.java#L301
> When reporting failing to validate because a parameter comes from a 
> RuntimeValueProvider (such as when creating templates) we report:
> "Cannot validate with a RVP"
> We should instead provide a message saying what went wrong and how to fix it. 
> Something like:
> "Unable to validate parameters that aren't available until runtime. Disable 
> validation using .withoutValidation()."



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-14) Add declarative DSLs (XML & JSON)

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-14?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-14:

Component/s: (was: sdk-java-extensions)
 sdk-ideas

> Add declarative DSLs (XML & JSON)
> -
>
> Key: BEAM-14
> URL: https://issues.apache.org/jira/browse/BEAM-14
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-ideas
>Reporter: Jean-Baptiste Onofré
>Assignee: Sajeevan Achuthan
>Priority: Major
>
> Even if users would still be able to use directly the API, it would be great 
> to provide a DSL on top of the API covering batch and streaming data 
> processing but also data integration.
> Instead of designing a pipeline as a chain of apply() wrapping function 
> (DoFn), we can provide a fluent DSL allowing users to directly leverage 
> keyturn functions.
> For instance, an user would be able to design a pipeline like:
> {code}
> .from(“kafka:localhost:9092?topic=foo”).reduce(...).split(...).wiretap(...).map(...).to(“jms:queue:foo….”);
> {code}
> The DSL will allow to use existing pipelines, for instance:
> {code}
> .from("cxf:...").reduce().pipeline("other").map().to("kafka:localhost:9092?topic=foo=all")
> {code}
> So it means that we will have to create a IO Sink that can trigger the 
> execution of a target pipeline: (from("trigger:other") triggering the 
> pipeline execution when another pipeline design starts with 
> pipeline("other")). We can also imagine to mix the runners: the pipeline() 
> can be on one runner, the from("trigger:other") can be on another runner). 
> It's not trivial, but it will give strong flexibility and key value for Beam.
> In a second step, we can provide DSLs in different languages (the first one 
> would be Java, but why not providing XML, akka, scala DSLs).
> We can note in previous examples that the DSL would also provide data 
> integration support to bean in addition of data processing. Data Integration 
> is an extension of Beam API to support some Enterprise Integration Patterns 
> (EIPs). As we would need metadata for data integration (even if metadata can 
> also be interesting in stream/batch data processing pipeline), we can provide 
> a DataxMessage built on top of PCollection. A DataxMessage would contain:
> structured headers
> binary payload
> For instance, the headers can contains an Avro schema to describe the payload.
> The headers can also contains useful information coming from the IO Source 
> (for instance the partition/path where the data comes from, …).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-2382) Introduce PubSubService in the PubSubIO

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-2382:
--
Component/s: (was: sdk-java-extensions)
 sdk-java-gcp

> Introduce PubSubService in the PubSubIO
> ---
>
> Key: BEAM-2382
> URL: https://issues.apache.org/jira/browse/BEAM-2382
> Project: Beam
>  Issue Type: Wish
>  Components: sdk-java-gcp
>Reporter: Jean-Baptiste Onofré
>Assignee: Jean-Baptiste Onofré
>Priority: Major
>
> As we have for BigQuery, Cassandra, and other IOs, it would be great to 
> provide a way to mock PubSub for users (especially for the tests).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-676) Use custom POJO and create JMS ConnectionFactory

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-676?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-676:
-
Component/s: (was: sdk-java-extensions)
 io-java-jms

> Use custom POJO and create JMS ConnectionFactory
> 
>
> Key: BEAM-676
> URL: https://issues.apache.org/jira/browse/BEAM-676
> Project: Beam
>  Issue Type: Improvement
>  Components: io-java-jms
>Reporter: Jean-Baptiste Onofré
>Assignee: Jean-Baptiste Onofré
>Priority: Major
>
> The {{JmsIO}} uses JMS {{ConnectionFactory}} as configuration. Unfortunately, 
> {{ConnectionFactory}} interface doesn't extend {{Serializable}} (even if most 
> of the implementations are {{Serializable}}).
> Instead (as I'm doing for the {{JdbcIO}}), I propose to create a 
> {{ConnectionFactoryConfiguration}} POJO to create the {{ConnectionFactory}} 
> in the {{@Setup}} of the reader and writer.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PerformanceTests_Python #914

[Apache Jenkins Server]

See 


Changes:

[pawel.pk.kaczmarczyk] [BEAM-3605] Use verification with timeout instead of 
Thread.sleep

[cclauss] Exception.message was removed in Python 3

[iemejia] [BEAM-3697] Add Maven profile to run error-prone static analysis

[iemejia] [BEAM-3697] Fix MoreFutures errorprone

[alan] [BEAM-3695] Fix failing validates container test

--
[...truncated 15 B...]
Building remotely on beam3 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 359ace325ce117103718f73a2249c140ec135cbe (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 359ace325ce117103718f73a2249c140ec135cbe
Commit message: "Merge pull request #4614 from 
cclauss/Exception.message-removed-in-Python3"
 > git rev-list 0bc6f574ce9a9bebcf68946456d4ac6c36e81911 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins4000263681124995714.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_Python] $ /bin/bash -xe /tmp/jenkins843544305550741394.sh
+ rm -rf .env
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins8140928070583212402.sh
+ virtualenv .env --system-site-packages
New python executable in .env/bin/python
Installing setuptools, pip...done.
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins2580297316760659790.sh
+ .env/bin/pip install --upgrade setuptools pip
Downloading/unpacking setuptools from 
https://pypi.python.org/packages/43/41/033a273f9a25cb63050a390ee8397acbc7eae2159195d85f06f17e7be45a/setuptools-38.5.1-py2.py3-none-any.whl#md5=908b8b5e50bf429e520b2b5fa1b350e5
Downloading/unpacking pip from 
https://pypi.python.org/packages/b6/ac/7015eb97dc749283ffdec1c3a88ddb8ae03b8fad0f0e611408f196358da3/pip-9.0.1-py2.py3-none-any.whl#md5=297dbd16ef53bcef0447d245815f5144
Installing collected packages: setuptools, pip
  Found existing installation: setuptools 2.2
Uninstalling setuptools:
  Successfully uninstalled setuptools
  Found existing installation: pip 1.5.4
Uninstalling pip:
  Successfully uninstalled pip
Successfully installed setuptools pip
Cleaning up...
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins5779279728739312801.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins1577999761598014586.sh
+ .env/bin/pip install -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied: absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: setuptools in ./.env/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 16))
Requirement already satisfied: colorlog[windows]==2.6.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied: futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied: PyYAML==3.12 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied: pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Requirement already satisfied: numpy==1.13.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 22))
Requirement already satisfied: functools32 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23))
Requirement already satisfied: 

[jira] [Updated] (BEAM-3073) Connect to Apache ignite via JdbcIO sdk

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-3073:
--
Component/s: (was: sdk-java-extensions)
 io-java-jdbc

> Connect to Apache ignite via JdbcIO sdk
> ---
>
> Key: BEAM-3073
> URL: https://issues.apache.org/jira/browse/BEAM-3073
> Project: Beam
>  Issue Type: New Feature
>  Components: io-java-jdbc
>Reporter: Rick Lin
>Assignee: Jean-Baptiste Onofré
>Priority: Minor
>
> Hi all,
> {color:#14892c}I tried to connect Apache Ignite(In-memory) via the beam's 
> sdk:org.apache.beam.sdk.io.jdbc.JdbcIO
> Here, i am not sure if the JdbcIO sdk only is provided for some specific 
> Database: MySQL(disk), postgreSQL(disk)?{color}
> my java test code is as follows:
> import java.sql.PreparedStatement;
> import java.sql.SQLException;
> import java.util.ArrayList;
> import java.util.List;
> import org.apache.beam.sdk.Pipeline;
> import org.apache.beam.sdk.io.jdbc.JdbcIO;
> import org.apache.beam.sdk.options.PipelineOptionsFactory;
> import org.apache.beam.sdk.transforms.Create;
> import org.apache.beam.sdk.values.KV;
> import org.apache.beam.sdk.values.PCollection;
> public class BeamtoJDBC {
>   public static void main(String[] args) {
>   Integer[] value=new Integer[] {1,2,3,4,5};
>   List> dataList = new ArrayList<>();
>   int n=value.length;
>   int count=0;
>   for (int i=0; i   {
>   dataList.add(KV.of(count,value[i]));
>   count=count+1;  
>   }
>   
>   Pipeline p = 
> Pipeline.create(PipelineOptionsFactory.fromArgs(args).withValidation().create());
>   
>   PCollection> data=p.apply("create data 
> with time",Create.of(dataList));
>   data.apply(JdbcIO.>write()
>   
> .withDataSourceConfiguration(JdbcIO.DataSourceConfiguration
>   
> .create("org.apache.ignite.IgniteJdbcDriver", 
> "jdbc:ignite://localhost:11211/")
>   )   
>   .withPreparedStatementSetter(new 
> JdbcIO.PreparedStatementSetter>() {
>   public void setParameters(KV Integer> element, PreparedStatement query)
>   throws SQLException {
>   query.setInt(1, 
> element.getKey());
>   query.setInt(2, 
> element.getValue());
>   }
>   })
>   );
>   p.run();
>   }
> }
> {color:#d04437}my error message is: 
> " InvocationTargetException: org.apache.beam.sdk.util.UserCodeException: 
> java.sql.SQLException: Cannot create PoolableConnectionFactory 
> (Failed to establish connection.): Failed to get future result due to waiting 
> timed out. "{color}
> {color:#14892c}I would like to know whether the connection between beam and 
> ignite is feasible or not?{color}
> Thanks
> Rick



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Jenkins build is back to normal : beam_PerformanceTests_Compressed_TextIOIT #142

[Apache Jenkins Server]

See 

Jenkins build is back to stable : beam_PostCommit_Java_ValidatesRunner_Apex #3477

[Apache Jenkins Server]

See 

[jira] [Updated] (BEAM-3703) java.io.IOException: KafkaWriter : failed to send 1 records (since last report)

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-3703:
--
Component/s: runner-flink

> java.io.IOException: KafkaWriter : failed to send 1 records (since last 
> report)
> ---
>
> Key: BEAM-3703
> URL: https://issues.apache.org/jira/browse/BEAM-3703
> Project: Beam
>  Issue Type: Bug
>  Components: io-java-kafka, runner-flink
>Affects Versions: 2.2.0
>Reporter: jagdeep sihota
>Assignee: Raghu Angadi
>Priority: Major
> Fix For: Not applicable
>
>
> I am trying to read from file and write to Kafka in google cloud kafka and 
> getting following error:
>  
> org.apache.beam.sdk.util.UserCodeException: java.io.IOException: KafkaWriter 
> : failed to send 1 records (since last report)
> at org.apache.beam.sdk.util.UserCodeException.wrap(UserCodeException.java:36)
> at 
> org.apache.beam.sdk.io.kafka.KafkaIO$KafkaWriter$DoFnInvoker.invokeProcessElement(Unknown
>  Source)
> at 
> org.apache.beam.runners.core.SimpleDoFnRunner.invokeProcessElement(SimpleDoFnRunner.java:177)
> at 
> org.apache.beam.runners.core.SimpleDoFnRunner.processElement(SimpleDoFnRunner.java:141)
> at 
> org.apache.beam.runners.flink.metrics.DoFnRunnerWithMetricsUpdate.processElement(DoFnRunnerWithMetricsUpdate.java:65)
> at 
> org.apache.beam.runners.flink.translation.functions.FlinkDoFnFunction.mapPartition(FlinkDoFnFunction.java:120)
> at 
> org.apache.flink.runtime.operators.MapPartitionDriver.run(MapPartitionDriver.java:103)
> at org.apache.flink.runtime.operators.BatchTask.run(BatchTask.java:490)
> at org.apache.flink.runtime.operators.BatchTask.invoke(BatchTask.java:355)
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:702)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.io.IOException: KafkaWriter : failed to send 1 records (since 
> last report)
> at 
> org.apache.beam.sdk.io.kafka.KafkaIO$KafkaWriter.checkForFailures(KafkaIO.java:1639)
> at 
> org.apache.beam.sdk.io.kafka.KafkaIO$KafkaWriter.processElement(KafkaIO.java:1581)
> Caused by: org.apache.kafka.common.errors.TimeoutException: Failed to update 
> metadata after 6 ms.
>  
>  
>  
> .apply(KafkaIO._write_()
> .withBootstrapServers("ip1:9092,ip2:9092")
> .withTopic("feed")
> .withValueSerializer(StringSerializer.class)
> .withKeySerializer(StringSerializer.class)
>         
> //.updateProducerProperties(ImmutableMap.of("security.protocol","PLAINTEXT"))
>         //.updateProducerProperties(ImmutableMap.of("sasl.mechanism","PLAIN"))
>  
> .values() // writes values to Kafka with default key
>  
> Kafka is running on google cloud bitnami and I am using Flink runner
> How do I pass security information to Kafka IO?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-3703) java.io.IOException: KafkaWriter : failed to send 1 records (since last report)

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-3703:
--
Component/s: (was: beam-model)
 io-java-kafka

> java.io.IOException: KafkaWriter : failed to send 1 records (since last 
> report)
> ---
>
> Key: BEAM-3703
> URL: https://issues.apache.org/jira/browse/BEAM-3703
> Project: Beam
>  Issue Type: Bug
>  Components: io-java-kafka
>Affects Versions: 2.2.0
>Reporter: jagdeep sihota
>Assignee: Kenneth Knowles
>Priority: Major
> Fix For: Not applicable
>
>
> I am trying to read from file and write to Kafka in google cloud kafka and 
> getting following error:
>  
> org.apache.beam.sdk.util.UserCodeException: java.io.IOException: KafkaWriter 
> : failed to send 1 records (since last report)
> at org.apache.beam.sdk.util.UserCodeException.wrap(UserCodeException.java:36)
> at 
> org.apache.beam.sdk.io.kafka.KafkaIO$KafkaWriter$DoFnInvoker.invokeProcessElement(Unknown
>  Source)
> at 
> org.apache.beam.runners.core.SimpleDoFnRunner.invokeProcessElement(SimpleDoFnRunner.java:177)
> at 
> org.apache.beam.runners.core.SimpleDoFnRunner.processElement(SimpleDoFnRunner.java:141)
> at 
> org.apache.beam.runners.flink.metrics.DoFnRunnerWithMetricsUpdate.processElement(DoFnRunnerWithMetricsUpdate.java:65)
> at 
> org.apache.beam.runners.flink.translation.functions.FlinkDoFnFunction.mapPartition(FlinkDoFnFunction.java:120)
> at 
> org.apache.flink.runtime.operators.MapPartitionDriver.run(MapPartitionDriver.java:103)
> at org.apache.flink.runtime.operators.BatchTask.run(BatchTask.java:490)
> at org.apache.flink.runtime.operators.BatchTask.invoke(BatchTask.java:355)
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:702)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.io.IOException: KafkaWriter : failed to send 1 records (since 
> last report)
> at 
> org.apache.beam.sdk.io.kafka.KafkaIO$KafkaWriter.checkForFailures(KafkaIO.java:1639)
> at 
> org.apache.beam.sdk.io.kafka.KafkaIO$KafkaWriter.processElement(KafkaIO.java:1581)
> Caused by: org.apache.kafka.common.errors.TimeoutException: Failed to update 
> metadata after 6 ms.
>  
>  
>  
> .apply(KafkaIO._write_()
> .withBootstrapServers("ip1:9092,ip2:9092")
> .withTopic("feed")
> .withValueSerializer(StringSerializer.class)
> .withKeySerializer(StringSerializer.class)
>         
> //.updateProducerProperties(ImmutableMap.of("security.protocol","PLAINTEXT"))
>         //.updateProducerProperties(ImmutableMap.of("sasl.mechanism","PLAIN"))
>  
> .values() // writes values to Kafka with default key
>  
> Kafka is running on google cloud bitnami and I am using Flink runner
> How do I pass security information to Kafka IO?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-3703) java.io.IOException: KafkaWriter : failed to send 1 records (since last report)

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles reassigned BEAM-3703:
-

Assignee: Raghu Angadi  (was: Kenneth Knowles)

> java.io.IOException: KafkaWriter : failed to send 1 records (since last 
> report)
> ---
>
> Key: BEAM-3703
> URL: https://issues.apache.org/jira/browse/BEAM-3703
> Project: Beam
>  Issue Type: Bug
>  Components: io-java-kafka
>Affects Versions: 2.2.0
>Reporter: jagdeep sihota
>Assignee: Raghu Angadi
>Priority: Major
> Fix For: Not applicable
>
>
> I am trying to read from file and write to Kafka in google cloud kafka and 
> getting following error:
>  
> org.apache.beam.sdk.util.UserCodeException: java.io.IOException: KafkaWriter 
> : failed to send 1 records (since last report)
> at org.apache.beam.sdk.util.UserCodeException.wrap(UserCodeException.java:36)
> at 
> org.apache.beam.sdk.io.kafka.KafkaIO$KafkaWriter$DoFnInvoker.invokeProcessElement(Unknown
>  Source)
> at 
> org.apache.beam.runners.core.SimpleDoFnRunner.invokeProcessElement(SimpleDoFnRunner.java:177)
> at 
> org.apache.beam.runners.core.SimpleDoFnRunner.processElement(SimpleDoFnRunner.java:141)
> at 
> org.apache.beam.runners.flink.metrics.DoFnRunnerWithMetricsUpdate.processElement(DoFnRunnerWithMetricsUpdate.java:65)
> at 
> org.apache.beam.runners.flink.translation.functions.FlinkDoFnFunction.mapPartition(FlinkDoFnFunction.java:120)
> at 
> org.apache.flink.runtime.operators.MapPartitionDriver.run(MapPartitionDriver.java:103)
> at org.apache.flink.runtime.operators.BatchTask.run(BatchTask.java:490)
> at org.apache.flink.runtime.operators.BatchTask.invoke(BatchTask.java:355)
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:702)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.io.IOException: KafkaWriter : failed to send 1 records (since 
> last report)
> at 
> org.apache.beam.sdk.io.kafka.KafkaIO$KafkaWriter.checkForFailures(KafkaIO.java:1639)
> at 
> org.apache.beam.sdk.io.kafka.KafkaIO$KafkaWriter.processElement(KafkaIO.java:1581)
> Caused by: org.apache.kafka.common.errors.TimeoutException: Failed to update 
> metadata after 6 ms.
>  
>  
>  
> .apply(KafkaIO._write_()
> .withBootstrapServers("ip1:9092,ip2:9092")
> .withTopic("feed")
> .withValueSerializer(StringSerializer.class)
> .withKeySerializer(StringSerializer.class)
>         
> //.updateProducerProperties(ImmutableMap.of("security.protocol","PLAINTEXT"))
>         //.updateProducerProperties(ImmutableMap.of("sasl.mechanism","PLAIN"))
>  
> .values() // writes values to Kafka with default key
>  
> Kafka is running on google cloud bitnami and I am using Flink runner
> How do I pass security information to Kafka IO?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-3704) DoFns should be torn down as part of an orderly shutdown

[Thomas Groh (JIRA)]

Thomas Groh created BEAM-3704:
-

 Summary: DoFns should be torn down as part of an orderly shutdown
 Key: BEAM-3704
 URL: https://issues.apache.org/jira/browse/BEAM-3704
 Project: Beam
  Issue Type: Bug
  Components: sdk-java-harness
Reporter: Thomas Groh


This lets user fns do some deferred cleanup before a harness is shut down.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-3703) java.io.IOException: KafkaWriter : failed to send 1 records (since last report)

[jagdeep sihota (JIRA)]

jagdeep sihota created BEAM-3703:


 Summary: java.io.IOException: KafkaWriter : failed to send 1 
records (since last report)
 Key: BEAM-3703
 URL: https://issues.apache.org/jira/browse/BEAM-3703
 Project: Beam
  Issue Type: Bug
  Components: beam-model
Affects Versions: 2.2.0
Reporter: jagdeep sihota
Assignee: Kenneth Knowles
 Fix For: Not applicable


I am trying to read from file and write to Kafka in google cloud kafka and 
getting following error:

 

org.apache.beam.sdk.util.UserCodeException: java.io.IOException: KafkaWriter : 
failed to send 1 records (since last report)

at org.apache.beam.sdk.util.UserCodeException.wrap(UserCodeException.java:36)

at 
org.apache.beam.sdk.io.kafka.KafkaIO$KafkaWriter$DoFnInvoker.invokeProcessElement(Unknown
 Source)

at 
org.apache.beam.runners.core.SimpleDoFnRunner.invokeProcessElement(SimpleDoFnRunner.java:177)

at 
org.apache.beam.runners.core.SimpleDoFnRunner.processElement(SimpleDoFnRunner.java:141)

at 
org.apache.beam.runners.flink.metrics.DoFnRunnerWithMetricsUpdate.processElement(DoFnRunnerWithMetricsUpdate.java:65)

at 
org.apache.beam.runners.flink.translation.functions.FlinkDoFnFunction.mapPartition(FlinkDoFnFunction.java:120)

at 
org.apache.flink.runtime.operators.MapPartitionDriver.run(MapPartitionDriver.java:103)

at org.apache.flink.runtime.operators.BatchTask.run(BatchTask.java:490)

at org.apache.flink.runtime.operators.BatchTask.invoke(BatchTask.java:355)

at org.apache.flink.runtime.taskmanager.Task.run(Task.java:702)

at java.lang.Thread.run(Thread.java:748)

Caused by: java.io.IOException: KafkaWriter : failed to send 1 records (since 
last report)

at 
org.apache.beam.sdk.io.kafka.KafkaIO$KafkaWriter.checkForFailures(KafkaIO.java:1639)

at 
org.apache.beam.sdk.io.kafka.KafkaIO$KafkaWriter.processElement(KafkaIO.java:1581)

Caused by: org.apache.kafka.common.errors.TimeoutException: Failed to update 
metadata after 6 ms.

 

 

 

.apply(KafkaIO._write_()

.withBootstrapServers("ip1:9092,ip2:9092")

.withTopic("feed")

.withValueSerializer(StringSerializer.class)

.withKeySerializer(StringSerializer.class)

        
//.updateProducerProperties(ImmutableMap.of("security.protocol","PLAINTEXT"))

        //.updateProducerProperties(ImmutableMap.of("sasl.mechanism","PLAIN"))

 

.values() // writes values to Kafka with default key

 

Kafka is running on google cloud bitnami and I am using Flink runner

How do I pass security information to Kafka IO?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Jenkins build became unstable: beam_PostCommit_Java_ValidatesRunner_Apex #3476

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-3694) [SQL] Update SQL documentation

[Blaye Nicolas (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16364422#comment-16364422
 ] 

Blaye Nicolas commented on BEAM-3694:
-

Hi folks,

We are investigating on using beam sql for some data processing jobs, but we 
did not really understand how to use the JOIN (it only sends back column names).

Do you know when you will publish that doc about joins behaviour?

 

Thank you

> [SQL] Update SQL documentation
> --
>
> Key: BEAM-3694
> URL: https://issues.apache.org/jira/browse/BEAM-3694
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql
>Reporter: Anton Kedin
>Priority: Major
>
> Update / add:
>  * windowing inheritance;
>  * HOP parameters swap;
>  * datetime functions;
>  * joins behavior;



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] branch master updated (22453da -> 359ace3)

[altay]

This is an automated email from the ASF dual-hosted git repository.

altay pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 22453da  Merge pull request #4673: [BEAM-3695] Fix failing validates 
container test
 add 35d5bdc  Exception.message was removed in Python 3
 new 359ace3  Merge pull request #4614 from 
cclauss/Exception.message-removed-in-Python3

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 sdks/python/apache_beam/pipeline_test.py   |  2 +-
 .../runners/dataflow/internal/dependency_test.py   | 16 ++---
 .../apache_beam/testing/pipeline_verifiers_test.py |  2 +-
 sdks/python/apache_beam/testing/test_utils_test.py |  2 +-
 .../apache_beam/transforms/ptransform_test.py  | 80 +++---
 sdks/python/apache_beam/typehints/typecheck.py |  2 +-
 .../python/apache_beam/typehints/typehints_test.py | 70 +--
 7 files changed, 87 insertions(+), 87 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
al...@apache.org.

[beam] 01/01: Merge pull request #4614 from cclauss/Exception.message-removed-in-Python3

[altay]

This is an automated email from the ASF dual-hosted git repository.

altay pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 359ace325ce117103718f73a2249c140ec135cbe
Merge: 22453da 35d5bdc
Author: Ahmet Altay 
AuthorDate: Wed Feb 14 08:59:21 2018 -0800

Merge pull request #4614 from cclauss/Exception.message-removed-in-Python3

Exception.message was removed in Python 3

 sdks/python/apache_beam/pipeline_test.py   |  2 +-
 .../runners/dataflow/internal/dependency_test.py   | 16 ++---
 .../apache_beam/testing/pipeline_verifiers_test.py |  2 +-
 sdks/python/apache_beam/testing/test_utils_test.py |  2 +-
 .../apache_beam/transforms/ptransform_test.py  | 80 +++---
 sdks/python/apache_beam/typehints/typecheck.py |  2 +-
 .../python/apache_beam/typehints/typehints_test.py | 70 +--
 7 files changed, 87 insertions(+), 87 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
al...@apache.org.

[jira] [Commented] (BEAM-3690) Dependency Conflict problems: several conflicting classes exist in different JARs

[Luke Cwik (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16364414#comment-16364414
 ] 

Luke Cwik commented on BEAM-3690:
-

[~PandaMonkey], swapping to use mockito-core, hamcrest-core and 
hamcrest-library is the right thing to do. Feel free to open up a PR against 
the repository. It would also be nice if we banned these dependencies from the 
maven build, see 
[https://maven.apache.org/enforcer/enforcer-rules/bannedDependencies.html] for 
details.

> Dependency Conflict problems: several conflicting classes exist in different 
> JARs
> -
>
> Key: BEAM-3690
> URL: https://issues.apache.org/jira/browse/BEAM-3690
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Affects Versions: 2.2.0
>Reporter: PandaMonkey
>Assignee: PandaMonkey
>Priority: Major
> Fix For: 2.3.0
>
> Attachments: beam_conflicts.txt
>
>
> Hi, we found that there are duplicate classes exist in different JARs, and 
> these classes have different features.
> The conflicting JAR pairs are:
> 1. 
> jar-pair:
> 2. 
> jar-pair:
> Some of method only exist in one version of duplicate classes.
> As the JVM only load the classes present first on the classpath and shadow 
> the other duplicate ones with the same names. The dependency conflict problem 
> brings high risks of "*NoSuchMethodException*" or "*NoSuchMethodError*"  
> issues at runtime. The conflicting details are listed in the attachment. 
> Please notice that. Thanks.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (BEAM-3684) Update well-known coder URNs in Go SDK

[Henning Rohde (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Henning Rohde resolved BEAM-3684.
-
   Resolution: Fixed
Fix Version/s: Not applicable

> Update well-known coder URNs in Go SDK
> --
>
> Key: BEAM-3684
> URL: https://issues.apache.org/jira/browse/BEAM-3684
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-go
>Reporter: Eugene Kirpichov
>Assignee: Henning Rohde
>Priority: Major
>  Labels: portability
> Fix For: Not applicable
>
>  Time Spent: 1h
>  Remaining Estimate: 0h
>
> [https://github.com/apache/beam/blob/master/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/ModelCoderRegistrar.java]
>  has recently changed the URNs of well-known coders. The Go SDK needs to 
> update accordingly 
> https://github.com/apache/beam/blob/go-sdk/sdks/go/pkg/beam/core/runtime/graphx/coder.go#L34



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-3690) Dependency Conflict problems: several conflicting classes exist in different JARs

[Kenneth Knowles (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16364352#comment-16364352
 ] 

Kenneth Knowles commented on BEAM-3690:
---

Related, but not the same: I have tried turning on dependency convergence in 
the maven enforcer plugin. We need the same for gradle to ensure long-term 
health and protect from regressions. Maybe the tool that generated this report 
can also fail the build? That would be nice.

> Dependency Conflict problems: several conflicting classes exist in different 
> JARs
> -
>
> Key: BEAM-3690
> URL: https://issues.apache.org/jira/browse/BEAM-3690
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Affects Versions: 2.2.0
>Reporter: PandaMonkey
>Assignee: PandaMonkey
>Priority: Major
> Fix For: 2.3.0
>
> Attachments: beam_conflicts.txt
>
>
> Hi, we found that there are duplicate classes exist in different JARs, and 
> these classes have different features.
> The conflicting JAR pairs are:
> 1. 
> jar-pair:
> 2. 
> jar-pair:
> Some of method only exist in one version of duplicate classes.
> As the JVM only load the classes present first on the classpath and shadow 
> the other duplicate ones with the same names. The dependency conflict problem 
> brings high risks of "*NoSuchMethodException*" or "*NoSuchMethodError*"  
> issues at runtime. The conflicting details are listed in the attachment. 
> Please notice that. Thanks.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] branch master updated (a65d986 -> 22453da)

[kenn]

This is an automated email from the ASF dual-hosted git repository.

kenn pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from a65d986  Merge pull request #4667: [BEAM-3697] Add errorprone Maven 
config and fix a couple warnings
 add 45aaa76  [BEAM-3695] Fix failing validates container test
 new 22453da  Merge pull request #4673: [BEAM-3695] Fix failing validates 
container test

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 ...ommit_Python_ValidatesContainer_Dataflow.groovy |  2 +-
 sdks/python/container/run_validatescontainer.sh| 35 --
 2 files changed, 21 insertions(+), 16 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
k...@apache.org.

[beam] 01/01: Merge pull request #4673: [BEAM-3695] Fix failing validates container test

[kenn]

This is an automated email from the ASF dual-hosted git repository.

kenn pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 22453dadb61f2565448aa6073e9f97c05a6e823d
Merge: a65d986 45aaa76
Author: Kenn Knowles 
AuthorDate: Wed Feb 14 08:12:40 2018 -0800

Merge pull request #4673: [BEAM-3695] Fix failing validates container test

 ...ommit_Python_ValidatesContainer_Dataflow.groovy |  2 +-
 sdks/python/container/run_validatescontainer.sh| 35 --
 2 files changed, 21 insertions(+), 16 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
k...@apache.org.

[jira] [Assigned] (BEAM-3228) KinesisMockReadTest is flaky

[Alexey Romanenko (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alexey Romanenko reassigned BEAM-3228:
--

Assignee: Alexey Romanenko

> KinesisMockReadTest is flaky
> 
>
> Key: BEAM-3228
> URL: https://issues.apache.org/jira/browse/BEAM-3228
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Reporter: Valentyn Tymofieiev
>Assignee: Alexey Romanenko
>Priority: Critical
>  Labels: flake
> Fix For: Not applicable
>
>
> The test failed a few times recently in Java PostCommit test suite [1, 2]:
> Please refer to 
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Java_MavenInstall/src/sdks/java/io/kinesis/target/surefire-reports
>  for the individual test results.
> Please refer to dump files (if any exist) [date]-jvmRun[N].dump, 
> [date].dumpstream and [date]-jvmRun[N].dumpstream.
> The forked VM terminated without properly saying goodbye. VM crash or 
> System.exit called?
> Command was /bin/sh -c cd 
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Java_MavenInstall/src/sdks/java/io/kinesis
>  && /usr/local/asfpackages/java/jdk1.8.0_152/jre/bin/java 
> org.apache.maven.surefire.booter.ForkedBooter 
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Java_MavenInstall/src/sdks/java/io/kinesis/target/surefire
>  2017-11-17T21-31-27_966-jvmRun1 surefire8885947147781365641tmp 
> surefire_234035066469521508403tmp
> Error occurred in starting fork, check output in log
> Process Exit Code: 1
> Crashed tests:
> org.apache.beam.sdk.io.kinesis.KinesisMockReadTest
> org.apache.maven.surefire.booter.SurefireBooterForkException: The forked VM 
> terminated without properly saying goodbye. VM crash or System.exit called?
> Command was /bin/sh -c cd 
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Java_MavenInstall/src/sdks/java/io/kinesis
>  && /usr/local/asfpackages/java/jdk1.8.0_152/jre/bin/java 
> org.apache.maven.surefire.booter.ForkedBooter 
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Java_MavenInstall/src/sdks/java/io/kinesis/target/surefire
>  2017-11-17T21-31-27_966-jvmRun1 surefire8885947147781365641tmp 
> surefire_234035066469521508403tmp
> Error occurred in starting fork, check output in log
> Process Exit Code: 1
> Crashed tests:
> org.apache.beam.sdk.io.kinesis.KinesisMockReadTest
>   at 
> org.apache.maven.plugin.surefire.booterclient.ForkStarter.fork(ForkStarter.java:679)
>   at 
> org.apache.maven.plugin.surefire.booterclient.ForkStarter.fork(ForkStarter.java:533)
>   at 
> org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:279)
>   at 
> org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:243)
>   at 
> org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeProvider(AbstractSurefireMojo.java:1077)
>   at 
> org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeAfterPreconditionsChecked(AbstractSurefireMojo.java:907)
>   at 
> org.apache.maven.plugin.surefire.AbstractSurefireMojo.execute(AbstractSurefireMojo.java:785)
>   at 
> org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:134)
>   at 
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:208)
>   at 
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
>   at 
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
>   at 
> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:116)
>   at 
> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:80)
>   at 
> org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51)
>   at 
> org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128)
>   at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:307)
>   at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:193)
>   at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:106)
>   at 
> org.jvnet.hudson.maven3.launcher.Maven33Launcher.main(Maven33Launcher.java:129)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.codehaus.plexus.classworlds.launcher.Launcher.launchStandard(Launcher.java:330)
>   at 
> org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:238)
>   at 

Jenkins build is back to normal : beam_PostCommit_Python_ValidatesContainer_Dataflow #39

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PostCommit_Python_ValidatesContainer_Dataflow #38

[Apache Jenkins Server]

copying apache_beam/typehints/trivial_inference.py -> 
apache-beam-2.4.0.dev0/apache_beam/typehints
copying apache_beam/typehints/trivial_inference_test.py -> 
apache-beam-2.4.0.dev0/apache_beam/typehints
copying apache_beam/typehints/typecheck.py -> 
apache-beam-2.4.0.dev0/apache_beam/typehints
copying apache_beam/typehints/typed_pipeline_test.py -> 
apache-beam-2.4.0.dev0/apache_beam/typehints
copying apache_beam/typehints/typehints.py -> 
apache-beam-2.4.0.dev0/apache_beam/typehints
copying apache_beam/typehints/typehints_test.py -> 
apache-beam-2.4.0.dev0/apache_beam/typehints
copying apache_beam/utils/__init__.py -> 
apache-beam-2.4.0.dev0/apache_beam/utils
copying apache_beam/utils/annotations.py -> 
apache-beam-2.4.0.dev0/apache_beam/utils
copying apache_beam/utils/annotations_test.py -> 
apache-beam-2.4.0.dev0/apache_beam/utils
copying apache_beam/utils/counters.pxd -> 
apache-beam-2.4.0.dev0/apache_beam/utils
copying apache_beam/utils/counters.py -> 
apache-beam-2.4.0.dev0/apache_beam/utils
copying apache_beam/utils/counters_test.py -> 
apache-beam-2.4.0.dev0/apache_beam/utils
copying apache_beam/utils/plugin.py -> apache-beam-2.4.0.dev0/apache_beam/utils
copying apache_beam/utils/processes.py -> 
apache-beam-2.4.0.dev0/apache_beam/utils
copying apache_beam/utils/processes_test.py -> 
apache-beam-2.4.0.dev0/apache_beam/utils
copying apache_beam/utils/profiler.py -> 
apache-beam-2.4.0.dev0/apache_beam/utils
copying apache_beam/utils/proto_utils.py -> 
apache-beam-2.4.0.dev0/apache_beam/utils
copying apache_beam/utils/retry.py -> apache-beam-2.4.0.dev0/apache_beam/utils
copying apache_beam/utils/retry_test.py -> 
apache-beam-2.4.0.dev0/apache_beam/utils
copying apache_beam/utils/timestamp.py -> 
apache-beam-2.4.0.dev0/apache_beam/utils
copying apache_beam/utils/timestamp_test.py -> 
apache-beam-2.4.0.dev0/apache_beam/utils
copying apache_beam/utils/urns.py -> apache-beam-2.4.0.dev0/apache_beam/utils
copying apache_beam/utils/windowed_value.pxd -> 
apache-beam-2.4.0.dev0/apache_beam/utils
copying apache_beam/utils/windowed_value.py -> 
apache-beam-2.4.0.dev0/apache_beam/utils
copying apache_beam/utils/windowed_value_test.py -> 
apache-beam-2.4.0.dev0/apache_beam/utils
Writing apache-beam-2.4.0.dev0/setup.cfg
creating dist
Creating tar archive
removing 'apache-beam-2.4.0.dev0' (and everything under it)
SDK_LOCATION=$(find dist/apache-beam-*.tar.gz)
find dist/apache-beam-*.tar.gz

# Run ValidatesRunner tests on Google Cloud Dataflow service
echo ">>> RUNNING DATAFLOW RUNNER VALIDATESCONTAINER TEST"
>>> RUNNING DATAFLOW RUNNER VALIDATESCONTAINER TEST
python setup.py nosetests \
  --attr ValidatesContainer \
  --nocapture \
  --processes=1 \
  --process-timeout=900 \
  --test-pipeline-options=" \
--runner=TestDataflowRunner \
--project=$PROJECT \
--worker_harness_container_image=$CONTAINER:$TAG \
--staging_location=$GCS_LOCATION/staging-validatesrunner-test \
--temp_location=$GCS_LOCATION/temp-validatesrunner-test \
--output=$GCS_LOCATION/output \
--sdk_location=$SDK_LOCATION \
--num_workers=1"
<https://builds.apache.org/job/beam_PostCommit_Python_ValidatesContainer_Dataflow/ws/src/sdks/python/container/local/lib/python2.7/site-packages/setuptools/dist.py>:355:
 UserWarning: Normalizing '2.4.0.dev' to '2.4.0.dev0'
  normalized_version,
running nosetests
running egg_info
writing requirements to apache_beam.egg-info/requires.txt
writing apache_beam.egg-info/PKG-INFO
writing top-level names to apache_beam.egg-info/top_level.txt
writing dependency_links to apache_beam.egg-info/dependency_links.txt
writing entry points to apache_beam.egg-info/entry_points.txt
reading manifest file 'apache_beam.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no files found matching 'README.md'
warning: no files found matching 'NOTICE'
warning: no files found matching 'LICENSE'
writing manifest file 'apache_beam.egg-info/SOURCES.txt'
<https://builds.apache.org/job/beam_PostCommit_Python_ValidatesContainer_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/gcsio.py>:166:
 DeprecationWarning: object() takes no parameters
  super(GcsIO, cls).__new__(cls, storage_client))
test_wordcount_fnapi_it (apache_beam.examples.wordcount_it_test.WordCountIT) 
... ok

--
Ran 1 test in 423.550s

OK

# Delete the container locally and remotely
docker rmi $CONTAINER:$TAG
Untagged: us.gcr.io/apache-beam-testing/jenkins/python:20180214-150054
Untagged: 
us.gcr.io/apache-beam-testing/jenkins/python@sha256:6c6336cb38c3e7bb860d78a6fa221a4755b6f8ba200103306302857861857ba8
Deleted: sha256:f403fa2adf989927bc524c9b773f27809983350622443c4cfec8a2ad2512df6b
Deleted: sha256:201ffe7afa2876ff5bb1136803254c4aef37bfbbf0b010926dfc4825e2f1bb84
Deleted: sha256:7b194f610ef1e3b2d7e5969bdf2c62232363655dab9e1fdf3fc474

Build failed in Jenkins: beam_PostCommit_Python_ValidatesContainer_Dataflow #37

[Apache Jenkins Server]

/trivial_inference.py -> 
apache-beam-2.4.0.dev0/apache_beam/typehints
copying apache_beam/typehints/trivial_inference_test.py -> 
apache-beam-2.4.0.dev0/apache_beam/typehints
copying apache_beam/typehints/typecheck.py -> 
apache-beam-2.4.0.dev0/apache_beam/typehints
copying apache_beam/typehints/typed_pipeline_test.py -> 
apache-beam-2.4.0.dev0/apache_beam/typehints
copying apache_beam/typehints/typehints.py -> 
apache-beam-2.4.0.dev0/apache_beam/typehints
copying apache_beam/typehints/typehints_test.py -> 
apache-beam-2.4.0.dev0/apache_beam/typehints
copying apache_beam/utils/__init__.py -> 
apache-beam-2.4.0.dev0/apache_beam/utils
copying apache_beam/utils/annotations.py -> 
apache-beam-2.4.0.dev0/apache_beam/utils
copying apache_beam/utils/annotations_test.py -> 
apache-beam-2.4.0.dev0/apache_beam/utils
copying apache_beam/utils/counters.pxd -> 
apache-beam-2.4.0.dev0/apache_beam/utils
copying apache_beam/utils/counters.py -> 
apache-beam-2.4.0.dev0/apache_beam/utils
copying apache_beam/utils/counters_test.py -> 
apache-beam-2.4.0.dev0/apache_beam/utils
copying apache_beam/utils/plugin.py -> apache-beam-2.4.0.dev0/apache_beam/utils
copying apache_beam/utils/processes.py -> 
apache-beam-2.4.0.dev0/apache_beam/utils
copying apache_beam/utils/processes_test.py -> 
apache-beam-2.4.0.dev0/apache_beam/utils
copying apache_beam/utils/profiler.py -> 
apache-beam-2.4.0.dev0/apache_beam/utils
copying apache_beam/utils/proto_utils.py -> 
apache-beam-2.4.0.dev0/apache_beam/utils
copying apache_beam/utils/retry.py -> apache-beam-2.4.0.dev0/apache_beam/utils
copying apache_beam/utils/retry_test.py -> 
apache-beam-2.4.0.dev0/apache_beam/utils
copying apache_beam/utils/timestamp.py -> 
apache-beam-2.4.0.dev0/apache_beam/utils
copying apache_beam/utils/timestamp_test.py -> 
apache-beam-2.4.0.dev0/apache_beam/utils
copying apache_beam/utils/urns.py -> apache-beam-2.4.0.dev0/apache_beam/utils
copying apache_beam/utils/windowed_value.pxd -> 
apache-beam-2.4.0.dev0/apache_beam/utils
copying apache_beam/utils/windowed_value.py -> 
apache-beam-2.4.0.dev0/apache_beam/utils
copying apache_beam/utils/windowed_value_test.py -> 
apache-beam-2.4.0.dev0/apache_beam/utils
Writing apache-beam-2.4.0.dev0/setup.cfg
creating dist
Creating tar archive
removing 'apache-beam-2.4.0.dev0' (and everything under it)
SDK_LOCATION=$(find dist/apache-beam-*.tar.gz)
find dist/apache-beam-*.tar.gz

# Run ValidatesRunner tests on Google Cloud Dataflow service
echo ">>> RUNNING DATAFLOW RUNNER VALIDATESCONTAINER TEST"
>>> RUNNING DATAFLOW RUNNER VALIDATESCONTAINER TEST
python setup.py nosetests \
  --attr ValidatesContainer \
  --nocapture \
  --processes=1 \
  --process-timeout=900 \
  --test-pipeline-options=" \
--runner=TestDataflowRunner \
--project=$PROJECT \
--worker_harness_container_image=$CONTAINER:$TAG \
--staging_location=$GCS_LOCATION/staging-validatesrunner-test \
--temp_location=$GCS_LOCATION/temp-validatesrunner-test \
--output=$GCS_LOCATION/output \
--sdk_location=$SDK_LOCATION \
--num_workers=1"
<https://builds.apache.org/job/beam_PostCommit_Python_ValidatesContainer_Dataflow/ws/src/sdks/python/container/local/lib/python2.7/site-packages/setuptools/dist.py>:355:
 UserWarning: Normalizing '2.4.0.dev' to '2.4.0.dev0'
  normalized_version,
running nosetests
running egg_info
writing requirements to apache_beam.egg-info/requires.txt
writing apache_beam.egg-info/PKG-INFO
writing top-level names to apache_beam.egg-info/top_level.txt
writing dependency_links to apache_beam.egg-info/dependency_links.txt
writing entry points to apache_beam.egg-info/entry_points.txt
reading manifest file 'apache_beam.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no files found matching 'README.md'
warning: no files found matching 'NOTICE'
warning: no files found matching 'LICENSE'
writing manifest file 'apache_beam.egg-info/SOURCES.txt'
<https://builds.apache.org/job/beam_PostCommit_Python_ValidatesContainer_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/gcsio.py>:166:
 DeprecationWarning: object() takes no parameters
  super(GcsIO, cls).__new__(cls, storage_client))
test_wordcount_fnapi_it (apache_beam.examples.wordcount_it_test.WordCountIT) 
... ok

--
Ran 1 test in 428.993s

OK

# Delete the container locally and remotely
docker rmi $CONTAINER:$TAG
Untagged: us.gcr.io/apache-beam-testing/jenkins/python:20180214-142543
Untagged: 
us.gcr.io/apache-beam-testing/jenkins/python@sha256:1408346dd87967d6db1292b223a80e2ffd48307c0ab22ece34e96f47751a0d93
Deleted: sha256:e79a1d3d5aeffb3844e96b948020de183cd2f1f0e793dfb5683a49c212ec5fe9
Deleted: sha256:77f52e478303965566b907844f148e0949e04589bdedd76519b63047e5496c5d
Deleted: sha256:558e4d6c29b22b05ceced6b39bed2b2bcc93612f62a8392e80545b61c752e6bc
Deleted: sha256:b50180a500106429bf3f3

[jira] [Commented] (BEAM-3697) Add errorprone to maven and gradle builds

[JIRA]

[ 
https://issues.apache.org/jira/browse/BEAM-3697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16364129#comment-16364129
 ] 

Ismaël Mejía commented on BEAM-3697:


Adding the PR that added this to maven because it was done without the Jira 
ticket.

> Add errorprone to maven and gradle builds
> -
>
> Key: BEAM-3697
> URL: https://issues.apache.org/jira/browse/BEAM-3697
> Project: Beam
>  Issue Type: Bug
>  Components: build-system
>Reporter: Eugene Kirpichov
>Assignee: Kenneth Knowles
>Priority: Major
>
> [http://errorprone.info/] is a good static checker that covers a number of 
> bugs not covered by FindBugs or Checkstyle. We use it internally at Google 
> and, when run on the Beam codebase, it occasionally uncovers issues missed 
> during PR review process.
>  
> It has Maven and Gradle plugins:
> [http://errorprone.info/docs/installation]
> [https://github.com/tbroyer/gradle-errorprone-plugin]
>  
> It would be good to integrate it into our Maven and Gradle builds.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] 03/03: Merge pull request #4667: [BEAM-3697] Add errorprone Maven config and fix a couple warnings

[iemejia]

This is an automated email from the ASF dual-hosted git repository.

iemejia pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit a65d986a8466f7265a6456b94bc5196a1eea5d27
Merge: 510fb16 a270d85
Author: Ismaël Mejía 
AuthorDate: Wed Feb 14 15:24:04 2018 +0100

Merge pull request #4667: [BEAM-3697] Add errorprone Maven config and fix a 
couple warnings

 pom.xml| 34 +
 .../java/org/apache/beam/sdk/util/MoreFutures.java | 35 +-
 .../org/apache/beam/sdk/util/MoreFuturesTest.java  | 81 ++
 3 files changed, 134 insertions(+), 16 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
ieme...@apache.org.

[beam] branch master updated (510fb16 -> a65d986)

[iemejia]

This is an automated email from the ASF dual-hosted git repository.

iemejia pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 510fb16  Merge pull request #4668: [BEAM-3605] Use verification with 
timeout instead of Thread.sleep
 new a340ec1  [BEAM-3697] Add Maven profile to run error-prone static 
analysis
 new a270d85  [BEAM-3697] Fix MoreFutures errorprone
 new a65d986  Merge pull request #4667: [BEAM-3697] Add errorprone Maven 
config and fix a couple warnings

The 3 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 pom.xml| 34 +
 .../java/org/apache/beam/sdk/util/MoreFutures.java | 35 +-
 .../org/apache/beam/sdk/util/MoreFuturesTest.java  | 81 ++
 3 files changed, 134 insertions(+), 16 deletions(-)
 create mode 100644 
sdks/java/core/src/test/java/org/apache/beam/sdk/util/MoreFuturesTest.java

-- 
To stop receiving notification emails like this one, please contact
ieme...@apache.org.

[beam] 01/03: [BEAM-3697] Add Maven profile to run error-prone static analysis

[iemejia]

This is an automated email from the ASF dual-hosted git repository.

iemejia pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit a340ec17d939c1a016f00ef325d98c931c8ab7bf
Author: Kenneth Knowles 
AuthorDate: Mon Feb 12 11:23:29 2018 -0800

[BEAM-3697] Add Maven profile to run error-prone static analysis
---
 pom.xml | 34 ++
 1 file changed, 34 insertions(+)

diff --git a/pom.xml b/pom.xml
index d866c85..9a09513 100644
--- a/pom.xml
+++ b/pom.xml
@@ -175,6 +175,8 @@
 3.1.0
 
0.4
 3.1.0
+
2.8.2
+2.2.0
 
 -Werror
 
-Xpkginfo:always
@@ -354,6 +356,38 @@
 
 
 
+  errorprone
+  
+
+  
+  
+org.apache.maven.plugins
+maven-compiler-plugin
+
+  javac-with-errorprone
+  true
+  true
+
+
+  
+org.codehaus.plexus
+plexus-compiler-javac-errorprone
+   ${plexus-compiler-java-errorprone.version}
+  
+  
+  
+com.google.errorprone
+error_prone_core
+   ${error_prone_core.version}
+  
+
+  
+
+  
+
+
+
   build-containers
   
 

-- 
To stop receiving notification emails like this one, please contact
ieme...@apache.org.

[beam] 02/03: [BEAM-3697] Fix MoreFutures errorprone

[iemejia]

This is an automated email from the ASF dual-hosted git repository.

iemejia pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit a270d857870a1c8dae3eacddbab228bc3e25f260
Author: Kenneth Knowles 
AuthorDate: Mon Feb 12 12:21:18 2018 -0800

[BEAM-3697] Fix MoreFutures errorprone
---
 .../java/org/apache/beam/sdk/util/MoreFutures.java | 35 +-
 .../org/apache/beam/sdk/util/MoreFuturesTest.java  | 81 ++
 2 files changed, 100 insertions(+), 16 deletions(-)

diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/MoreFutures.java 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/MoreFutures.java
index 7b49503..8275fad 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/MoreFutures.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/MoreFutures.java
@@ -94,7 +94,8 @@ public class MoreFutures {
   public static  CompletionStage supplyAsync(
   ThrowingSupplier supplier, ExecutorService executorService) {
 CompletableFuture result = new CompletableFuture<>();
-CompletableFuture.runAsync(
+
+CompletionStage wrapper = CompletableFuture.runAsync(
 () -> {
   try {
 result.complete(supplier.get());
@@ -106,7 +107,7 @@ public class MoreFutures {
   }
 },
 executorService);
-return result;
+return wrapper.thenCompose(nothing -> result);
   }
 
   /**
@@ -125,20 +126,22 @@ public class MoreFutures {
   public static CompletionStage runAsync(
   ThrowingRunnable runnable, ExecutorService executorService) {
 CompletableFuture result = new CompletableFuture<>();
-CompletableFuture.runAsync(
-() -> {
-  try {
-runnable.run();
-result.complete(null);
-  } catch (InterruptedException e) {
-result.completeExceptionally(e);
-Thread.currentThread().interrupt();
-  } catch (Throwable t) {
-result.completeExceptionally(t);
-  }
-},
-executorService);
-return result;
+
+CompletionStage wrapper =
+CompletableFuture.runAsync(
+() -> {
+  try {
+runnable.run();
+result.complete(null);
+  } catch (InterruptedException e) {
+result.completeExceptionally(e);
+Thread.currentThread().interrupt();
+  } catch (Throwable t) {
+result.completeExceptionally(t);
+  }
+},
+executorService);
+return wrapper.thenCompose(nothing -> result);
   }
 
   /**
diff --git 
a/sdks/java/core/src/test/java/org/apache/beam/sdk/util/MoreFuturesTest.java 
b/sdks/java/core/src/test/java/org/apache/beam/sdk/util/MoreFuturesTest.java
new file mode 100644
index 000..22ab4c0
--- /dev/null
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/util/MoreFuturesTest.java
@@ -0,0 +1,81 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.util;
+
+import static org.hamcrest.Matchers.equalTo;
+import static org.hamcrest.Matchers.isA;
+import static org.junit.Assert.assertThat;
+
+import java.util.concurrent.CompletionStage;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.atomic.AtomicInteger;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.ExpectedException;
+import org.junit.runner.RunWith;
+import org.junit.runners.JUnit4;
+
+/** Unit tests for {@link MoreFutures}. */
+@RunWith(JUnit4.class)
+public class MoreFuturesTest {
+
+  @Rule public ExpectedException thrown = ExpectedException.none();
+
+  @Test
+  public void supplyAsyncSuccess() throws Exception {
+CompletionStage future = MoreFutures.supplyAsync(() -> 42);
+assertThat(MoreFutures.get(future), equalTo(42));
+  }
+
+  @Test
+  public void supplyAsyncFailure() throws Exception {
+final String testMessage = "this is just a test";
+CompletionStage future = MoreFutures.supplyAsync(() -> {
+  throw new IllegalStateException(testMessage);
+});
+
+thrown.expect(ExecutionException.class);
+

[jira] [Reopened] (BEAM-3228) KinesisMockReadTest is flaky

[JIRA]

 [ 
https://issues.apache.org/jira/browse/BEAM-3228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismaël Mejía reopened BEAM-3228:


I overlooked this one confusing it with BEAM-3317. This is still alive and 
reproducible by removing the workaround on the pom file.

> KinesisMockReadTest is flaky
> 
>
> Key: BEAM-3228
> URL: https://issues.apache.org/jira/browse/BEAM-3228
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Reporter: Valentyn Tymofieiev
>Priority: Critical
>  Labels: flake
> Fix For: Not applicable
>
>
> The test failed a few times recently in Java PostCommit test suite [1, 2]:
> Please refer to 
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Java_MavenInstall/src/sdks/java/io/kinesis/target/surefire-reports
>  for the individual test results.
> Please refer to dump files (if any exist) [date]-jvmRun[N].dump, 
> [date].dumpstream and [date]-jvmRun[N].dumpstream.
> The forked VM terminated without properly saying goodbye. VM crash or 
> System.exit called?
> Command was /bin/sh -c cd 
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Java_MavenInstall/src/sdks/java/io/kinesis
>  && /usr/local/asfpackages/java/jdk1.8.0_152/jre/bin/java 
> org.apache.maven.surefire.booter.ForkedBooter 
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Java_MavenInstall/src/sdks/java/io/kinesis/target/surefire
>  2017-11-17T21-31-27_966-jvmRun1 surefire8885947147781365641tmp 
> surefire_234035066469521508403tmp
> Error occurred in starting fork, check output in log
> Process Exit Code: 1
> Crashed tests:
> org.apache.beam.sdk.io.kinesis.KinesisMockReadTest
> org.apache.maven.surefire.booter.SurefireBooterForkException: The forked VM 
> terminated without properly saying goodbye. VM crash or System.exit called?
> Command was /bin/sh -c cd 
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Java_MavenInstall/src/sdks/java/io/kinesis
>  && /usr/local/asfpackages/java/jdk1.8.0_152/jre/bin/java 
> org.apache.maven.surefire.booter.ForkedBooter 
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Java_MavenInstall/src/sdks/java/io/kinesis/target/surefire
>  2017-11-17T21-31-27_966-jvmRun1 surefire8885947147781365641tmp 
> surefire_234035066469521508403tmp
> Error occurred in starting fork, check output in log
> Process Exit Code: 1
> Crashed tests:
> org.apache.beam.sdk.io.kinesis.KinesisMockReadTest
>   at 
> org.apache.maven.plugin.surefire.booterclient.ForkStarter.fork(ForkStarter.java:679)
>   at 
> org.apache.maven.plugin.surefire.booterclient.ForkStarter.fork(ForkStarter.java:533)
>   at 
> org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:279)
>   at 
> org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:243)
>   at 
> org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeProvider(AbstractSurefireMojo.java:1077)
>   at 
> org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeAfterPreconditionsChecked(AbstractSurefireMojo.java:907)
>   at 
> org.apache.maven.plugin.surefire.AbstractSurefireMojo.execute(AbstractSurefireMojo.java:785)
>   at 
> org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:134)
>   at 
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:208)
>   at 
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
>   at 
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
>   at 
> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:116)
>   at 
> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:80)
>   at 
> org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51)
>   at 
> org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128)
>   at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:307)
>   at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:193)
>   at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:106)
>   at 
> org.jvnet.hudson.maven3.launcher.Maven33Launcher.main(Maven33Launcher.java:129)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.codehaus.plexus.classworlds.launcher.Launcher.launchStandard(Launcher.java:330)
>   at 
> 

[jira] [Commented] (BEAM-2752) KinesisIO throws ConcurrentModificationException on checkpoint

[JIRA]

[ 
https://issues.apache.org/jira/browse/BEAM-2752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16364077#comment-16364077
 ] 

Ismaël Mejía commented on BEAM-2752:


[~pawelbartoszek] there have been a lot of changes in the last weeks on 
KinesisIO, can you somehow try the master version (or SNAPSHOT) and tell us if 
it is still happening.

> KinesisIO throws ConcurrentModificationException on checkpoint
> --
>
> Key: BEAM-2752
> URL: https://issues.apache.org/jira/browse/BEAM-2752
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Affects Versions: 2.0.0, 2.1.0, 2.2.0
>Reporter: Pawel Bartoszek
>Assignee: Chamikara Jayalath
>Priority: Minor
>
> From time to time Kinesis IO throws ConcurrentModificationException on taking 
> a checkpoint.
> {code:java}
> Caused by: java.util.ConcurrentModificationException
>   at java.util.ArrayDeque$DeqIterator.next(ArrayDeque.java:643)
>   at 
> org.apache.beam.sdks.java.io.kinesis.repackaged.com.google.common.collect.TransformedIterator.next(TransformedIterator.java:47)
>   at 
> org.apache.beam.sdks.java.io.kinesis.repackaged.com.google.common.collect.ImmutableCollection$Builder.addAll(ImmutableCollection.java:409)
>   at 
> org.apache.beam.sdks.java.io.kinesis.repackaged.com.google.common.collect.ImmutableList$Builder.addAll(ImmutableList.java:699)
>   at 
> org.apache.beam.sdks.java.io.kinesis.repackaged.com.google.common.collect.ImmutableList.copyOf(ImmutableList.java:256)
>   at 
> org.apache.beam.sdks.java.io.kinesis.repackaged.com.google.common.collect.ImmutableList.copyOf(ImmutableList.java:209)
>   at 
> org.apache.beam.sdk.io.kinesis.KinesisReaderCheckpoint.(KinesisReaderCheckpoint.java:44)
>   at 
> org.apache.beam.sdk.io.kinesis.KinesisReaderCheckpoint.asCurrentStateOf(KinesisReaderCheckpoint.java:49)
>   at 
> org.apache.beam.sdk.io.kinesis.KinesisReader.getCheckpointMark(KinesisReader.java:137)
>   at 
> org.apache.beam.runners.flink.translation.wrappers.streaming.io.UnboundedSourceWrapper.snapshotState(UnboundedSourceWrapper.java:379)
>   at 
> org.apache.flink.streaming.api.functions.util.StreamingFunctionUtils.trySnapshotFunctionState(StreamingFunctionUtils.java:118)
>   at 
> org.apache.flink.streaming.api.functions.util.StreamingFunctionUtils.snapshotFunctionState(StreamingFunctionUtils.java:99)
>   at 
> org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.snapshotState(AbstractUdfStreamOperator.java:100)
>   at 
> org.apache.flink.streaming.api.operators.AbstractStreamOperator.snapshotState(AbstractStreamOperator.java:357)
>   ... 11 more
> {code}
> *What is the issue*
> org.apache.beam.sdk.io.kinesis.RoundRobin class is using ArrayDeque class 
> which is not thread safe. If ConcurrentLinkedDeque deque is used the problem 
> should be fixed.
> *Beam 2.3 (master branch)*
> Kinesis connector have been heavily refactored in master which makes me 
> thinks how we should go about this fix? The 
> org.apache.beam.sdk.io.kinesis.RoundRobin class doesn't exist in master 
> anymore.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (BEAM-3598) kinesis.ShardReadersPoolTest.shouldStopReadersPoolAlsoWhenExceptionsOccurDuringStopping is flaky

[JIRA]

 [ 
https://issues.apache.org/jira/browse/BEAM-3598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismaël Mejía resolved BEAM-3598.

   Resolution: Fixed
Fix Version/s: 2.4.0

Resolved as part of BEAM-3605

> kinesis.ShardReadersPoolTest.shouldStopReadersPoolAlsoWhenExceptionsOccurDuringStopping
>  is flaky
> 
>
> Key: BEAM-3598
> URL: https://issues.apache.org/jira/browse/BEAM-3598
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Reporter: Thomas Groh
>Assignee: Paweł Kaczmarczyk
>Priority: Critical
>  Labels: flake, sickbay
> Fix For: 2.4.0
>
>  Time Spent: 1h
>  Remaining Estimate: 0h
>
> shouldStopReadersPoolAlsoWhenExceptionsOccurDuringStopping fails due to 
> incomplete interactions with the mock.
> {{Wanted but not invoked: firstIterator.readNextBatch(); -> at 
> org.apache.beam.sdk.io.kinesis.ShardReadersPoolTest.shouldStopReadersPoolAlsoWhenExceptionsOccurDuringStopping(ShardReadersPoolTest.java:244)
>  However, there were other interactions with this mock: -> at 
> org.apache.beam.sdk.io.kinesis.ShardReadersPoolTest.shouldStopReadersPoolAlsoWhenExceptionsOccurDuringStopping(ShardReadersPoolTest.java:241)}}
>  
> [https://builds.apache.org/job/beam_PreCommit_Java_MavenInstall/org.apache.beam$beam-sdks-java-io-kinesis/17390/testReport/junit/org.apache.beam.sdk.io.kinesis/ShardReadersPoolTest/shouldStopReadersPoolAlsoWhenExceptionsOccurDuringStopping/]
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (BEAM-3599) kinesis.ShardReadersPoolTest.shouldInterruptKinesisReadingAndStopShortly is flaky

[JIRA]

 [ 
https://issues.apache.org/jira/browse/BEAM-3599?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismaël Mejía resolved BEAM-3599.

   Resolution: Fixed
Fix Version/s: 2.4.0

Resolved as part of BEAM-3605

> kinesis.ShardReadersPoolTest.shouldInterruptKinesisReadingAndStopShortly is 
> flaky
> -
>
> Key: BEAM-3599
> URL: https://issues.apache.org/jira/browse/BEAM-3599
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Reporter: Thomas Groh
>Assignee: Paweł Kaczmarczyk
>Priority: Critical
>  Labels: flake, sickbay
> Fix For: 2.4.0
>
>
> This appears to be unavoidable due to the construction of the test - the 
> amount of realtime that the thread waits is asserted to be less than one 
> second, but as the class doesn't consume a {{Ticker}} (or related way to 
> track elapsed time), we have no way of deterministically controlling that 
> value within the test.
>  
> Example failure: 
> [https://builds.apache.org/job/beam_PreCommit_Java_GradleBuild/1738/testReport/junit/org.apache.beam.sdk.io.kinesis/ShardReadersPoolTest/shouldInterruptKinesisReadingAndStopShortly/]
>  
> {{java.lang.AssertionError: Expecting: <4169L> to be less than: <1000L> at 
> org.apache.beam.sdk.io.kinesis.ShardReadersPoolTest.shouldInterruptKinesisReadingAndStopShortly(ShardReadersPoolTest.java:159)
>  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)}} 
>  
> The test could also do with a more precise error message (it looks like too 
> many elements were received based on the error message, rather than the 
> actual "took too long" failure.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-3598) kinesis.ShardReadersPoolTest.shouldStopReadersPoolAlsoWhenExceptionsOccurDuringStopping is flaky

[JIRA]

 [ 
https://issues.apache.org/jira/browse/BEAM-3598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismaël Mejía reassigned BEAM-3598:
--

Assignee: Paweł Kaczmarczyk  (was: Alexey Romanenko)

> kinesis.ShardReadersPoolTest.shouldStopReadersPoolAlsoWhenExceptionsOccurDuringStopping
>  is flaky
> 
>
> Key: BEAM-3598
> URL: https://issues.apache.org/jira/browse/BEAM-3598
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Reporter: Thomas Groh
>Assignee: Paweł Kaczmarczyk
>Priority: Critical
>  Labels: flake, sickbay
> Fix For: 2.4.0
>
>  Time Spent: 1h
>  Remaining Estimate: 0h
>
> shouldStopReadersPoolAlsoWhenExceptionsOccurDuringStopping fails due to 
> incomplete interactions with the mock.
> {{Wanted but not invoked: firstIterator.readNextBatch(); -> at 
> org.apache.beam.sdk.io.kinesis.ShardReadersPoolTest.shouldStopReadersPoolAlsoWhenExceptionsOccurDuringStopping(ShardReadersPoolTest.java:244)
>  However, there were other interactions with this mock: -> at 
> org.apache.beam.sdk.io.kinesis.ShardReadersPoolTest.shouldStopReadersPoolAlsoWhenExceptionsOccurDuringStopping(ShardReadersPoolTest.java:241)}}
>  
> [https://builds.apache.org/job/beam_PreCommit_Java_MavenInstall/org.apache.beam$beam-sdks-java-io-kinesis/17390/testReport/junit/org.apache.beam.sdk.io.kinesis/ShardReadersPoolTest/shouldStopReadersPoolAlsoWhenExceptionsOccurDuringStopping/]
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (BEAM-3605) Kinesis ShardReadersPoolTest shouldForgetClosedShardIterator failure

[JIRA]

 [ 
https://issues.apache.org/jira/browse/BEAM-3605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismaël Mejía resolved BEAM-3605.

   Resolution: Fixed
Fix Version/s: 2.4.0

> Kinesis ShardReadersPoolTest shouldForgetClosedShardIterator failure
> 
>
> Key: BEAM-3605
> URL: https://issues.apache.org/jira/browse/BEAM-3605
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Reporter: Kenneth Knowles
>Assignee: Paweł Kaczmarczyk
>Priority: Critical
>  Labels: flake, sickbay
> Fix For: 2.4.0
>
>  Time Spent: 2h 50m
>  Remaining Estimate: 0h
>
> Here's one:
> https://builds.apache.org/job/beam_PreCommit_Java_GradleBuild/1758/testReport/junit/org.apache.beam.sdk.io.kinesis/ShardReadersPoolTest/shouldForgetClosedShardIterator/
> Filing all test failures as "Critical" so we can sickbay or fix.
> The Jenkins build will get GC'd so here is the error:
> {code}
> java.lang.AssertionError: 
> Expecting:
>   <["shard1", "shard2"]>
> to contain only:
>   <["shard2"]>
> but the following elements were unexpected:
>   <["shard1"]>
>   at 
> org.apache.beam.sdk.io.kinesis.ShardReadersPoolTest.shouldForgetClosedShardIterator(ShardReadersPoolTest.java:270)
> {code}
> and stderr
> {code}
> Feb 01, 2018 11:24:16 PM org.apache.beam.sdk.io.kinesis.ShardReadersPool 
> readLoop
> INFO: Shard iterator for shard1 shard is closed, finishing the read loop
> org.apache.beam.sdk.io.kinesis.KinesisShardClosedException
> Feb 01, 2018 11:24:16 PM org.apache.beam.sdk.io.kinesis.ShardReadersPool 
> readLoop
> INFO: Kinesis Shard read loop has finished
> Feb 01, 2018 11:24:16 PM org.apache.beam.sdk.io.kinesis.ShardReadersPool 
> readLoop
> INFO: Shard iterator for shard1 shard is closed, finishing the read loop
> org.apache.beam.sdk.io.kinesis.KinesisShardClosedException
> Feb 01, 2018 11:24:16 PM org.apache.beam.sdk.io.kinesis.ShardReadersPool 
> readLoop
> INFO: Kinesis Shard read loop has finished
> Feb 01, 2018 11:24:19 PM org.apache.beam.sdk.io.kinesis.ShardReadersPool 
> readLoop
> INFO: Shard iterator for shard1 shard is closed, finishing the read loop
> org.apache.beam.sdk.io.kinesis.KinesisShardClosedException
> Feb 01, 2018 11:24:19 PM org.apache.beam.sdk.io.kinesis.ShardReadersPool 
> readLoop
> INFO: Kinesis Shard read loop has finished
> Feb 01, 2018 11:24:23 PM org.apache.beam.sdk.io.kinesis.ShardReadersPool 
> readLoop
> INFO: Shard iterator for shard1 shard is closed, finishing the read loop
> org.apache.beam.sdk.io.kinesis.KinesisShardClosedException: Shard iterator 
> reached end of the shard: streamName=null, shardId=shard1
>   at 
> org.apache.beam.sdk.io.kinesis.ShardRecordsIterator.readNextBatch(ShardRecordsIterator.java:70)
>   at 
> org.apache.beam.sdk.io.kinesis.ShardReadersPool.readLoop(ShardReadersPool.java:121)
>   at 
> org.apache.beam.sdk.io.kinesis.ShardReadersPool.lambda$startReadingShards$0(ShardReadersPool.java:112)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>   at java.lang.Thread.run(Thread.java:748)
> Feb 01, 2018 11:24:23 PM org.apache.beam.sdk.io.kinesis.ShardReadersPool 
> readLoop
> INFO: Kinesis Shard read loop has finished
> Feb 01, 2018 11:24:23 PM org.apache.beam.sdk.io.kinesis.ShardReadersPool 
> readLoop
> INFO: Shard iterator for shard2 shard is closed, finishing the read loop
> org.apache.beam.sdk.io.kinesis.KinesisShardClosedException: Shard iterator 
> reached end of the shard: streamName=null, shardId=shard2
>   at 
> org.apache.beam.sdk.io.kinesis.ShardRecordsIterator.readNextBatch(ShardRecordsIterator.java:70)
>   at 
> org.apache.beam.sdk.io.kinesis.ShardReadersPool.readLoop(ShardReadersPool.java:121)
>   at 
> org.apache.beam.sdk.io.kinesis.ShardReadersPool.lambda$startReadingShards$0(ShardReadersPool.java:112)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>   at java.lang.Thread.run(Thread.java:748)
> Feb 01, 2018 11:24:23 PM org.apache.beam.sdk.io.kinesis.ShardReadersPool stop
> INFO: Closing shard iterators pool
> Feb 01, 2018 11:24:24 PM org.apache.beam.sdk.io.kinesis.ShardReadersPool 
> readLoop
> INFO: Kinesis Shard read loop has finished
> Feb 01, 2018 11:24:24 PM 

[jira] [Assigned] (BEAM-3599) kinesis.ShardReadersPoolTest.shouldInterruptKinesisReadingAndStopShortly is flaky

[JIRA]

 [ 
https://issues.apache.org/jira/browse/BEAM-3599?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismaël Mejía reassigned BEAM-3599:
--

Assignee: Paweł Kaczmarczyk  (was: Alexey Romanenko)

> kinesis.ShardReadersPoolTest.shouldInterruptKinesisReadingAndStopShortly is 
> flaky
> -
>
> Key: BEAM-3599
> URL: https://issues.apache.org/jira/browse/BEAM-3599
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Reporter: Thomas Groh
>Assignee: Paweł Kaczmarczyk
>Priority: Critical
>  Labels: flake, sickbay
>
> This appears to be unavoidable due to the construction of the test - the 
> amount of realtime that the thread waits is asserted to be less than one 
> second, but as the class doesn't consume a {{Ticker}} (or related way to 
> track elapsed time), we have no way of deterministically controlling that 
> value within the test.
>  
> Example failure: 
> [https://builds.apache.org/job/beam_PreCommit_Java_GradleBuild/1738/testReport/junit/org.apache.beam.sdk.io.kinesis/ShardReadersPoolTest/shouldInterruptKinesisReadingAndStopShortly/]
>  
> {{java.lang.AssertionError: Expecting: <4169L> to be less than: <1000L> at 
> org.apache.beam.sdk.io.kinesis.ShardReadersPoolTest.shouldInterruptKinesisReadingAndStopShortly(ShardReadersPoolTest.java:159)
>  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)}} 
>  
> The test could also do with a more precise error message (it looks like too 
> many elements were received based on the error message, rather than the 
> actual "took too long" failure.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-3605) Kinesis ShardReadersPoolTest shouldForgetClosedShardIterator failure

[JIRA]

 [ 
https://issues.apache.org/jira/browse/BEAM-3605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismaël Mejía reassigned BEAM-3605:
--

Assignee: Paweł Kaczmarczyk  (was: Alexey Romanenko)

> Kinesis ShardReadersPoolTest shouldForgetClosedShardIterator failure
> 
>
> Key: BEAM-3605
> URL: https://issues.apache.org/jira/browse/BEAM-3605
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Reporter: Kenneth Knowles
>Assignee: Paweł Kaczmarczyk
>Priority: Critical
>  Labels: flake, sickbay
> Fix For: 2.4.0
>
>  Time Spent: 2h 50m
>  Remaining Estimate: 0h
>
> Here's one:
> https://builds.apache.org/job/beam_PreCommit_Java_GradleBuild/1758/testReport/junit/org.apache.beam.sdk.io.kinesis/ShardReadersPoolTest/shouldForgetClosedShardIterator/
> Filing all test failures as "Critical" so we can sickbay or fix.
> The Jenkins build will get GC'd so here is the error:
> {code}
> java.lang.AssertionError: 
> Expecting:
>   <["shard1", "shard2"]>
> to contain only:
>   <["shard2"]>
> but the following elements were unexpected:
>   <["shard1"]>
>   at 
> org.apache.beam.sdk.io.kinesis.ShardReadersPoolTest.shouldForgetClosedShardIterator(ShardReadersPoolTest.java:270)
> {code}
> and stderr
> {code}
> Feb 01, 2018 11:24:16 PM org.apache.beam.sdk.io.kinesis.ShardReadersPool 
> readLoop
> INFO: Shard iterator for shard1 shard is closed, finishing the read loop
> org.apache.beam.sdk.io.kinesis.KinesisShardClosedException
> Feb 01, 2018 11:24:16 PM org.apache.beam.sdk.io.kinesis.ShardReadersPool 
> readLoop
> INFO: Kinesis Shard read loop has finished
> Feb 01, 2018 11:24:16 PM org.apache.beam.sdk.io.kinesis.ShardReadersPool 
> readLoop
> INFO: Shard iterator for shard1 shard is closed, finishing the read loop
> org.apache.beam.sdk.io.kinesis.KinesisShardClosedException
> Feb 01, 2018 11:24:16 PM org.apache.beam.sdk.io.kinesis.ShardReadersPool 
> readLoop
> INFO: Kinesis Shard read loop has finished
> Feb 01, 2018 11:24:19 PM org.apache.beam.sdk.io.kinesis.ShardReadersPool 
> readLoop
> INFO: Shard iterator for shard1 shard is closed, finishing the read loop
> org.apache.beam.sdk.io.kinesis.KinesisShardClosedException
> Feb 01, 2018 11:24:19 PM org.apache.beam.sdk.io.kinesis.ShardReadersPool 
> readLoop
> INFO: Kinesis Shard read loop has finished
> Feb 01, 2018 11:24:23 PM org.apache.beam.sdk.io.kinesis.ShardReadersPool 
> readLoop
> INFO: Shard iterator for shard1 shard is closed, finishing the read loop
> org.apache.beam.sdk.io.kinesis.KinesisShardClosedException: Shard iterator 
> reached end of the shard: streamName=null, shardId=shard1
>   at 
> org.apache.beam.sdk.io.kinesis.ShardRecordsIterator.readNextBatch(ShardRecordsIterator.java:70)
>   at 
> org.apache.beam.sdk.io.kinesis.ShardReadersPool.readLoop(ShardReadersPool.java:121)
>   at 
> org.apache.beam.sdk.io.kinesis.ShardReadersPool.lambda$startReadingShards$0(ShardReadersPool.java:112)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>   at java.lang.Thread.run(Thread.java:748)
> Feb 01, 2018 11:24:23 PM org.apache.beam.sdk.io.kinesis.ShardReadersPool 
> readLoop
> INFO: Kinesis Shard read loop has finished
> Feb 01, 2018 11:24:23 PM org.apache.beam.sdk.io.kinesis.ShardReadersPool 
> readLoop
> INFO: Shard iterator for shard2 shard is closed, finishing the read loop
> org.apache.beam.sdk.io.kinesis.KinesisShardClosedException: Shard iterator 
> reached end of the shard: streamName=null, shardId=shard2
>   at 
> org.apache.beam.sdk.io.kinesis.ShardRecordsIterator.readNextBatch(ShardRecordsIterator.java:70)
>   at 
> org.apache.beam.sdk.io.kinesis.ShardReadersPool.readLoop(ShardReadersPool.java:121)
>   at 
> org.apache.beam.sdk.io.kinesis.ShardReadersPool.lambda$startReadingShards$0(ShardReadersPool.java:112)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>   at java.lang.Thread.run(Thread.java:748)
> Feb 01, 2018 11:24:23 PM org.apache.beam.sdk.io.kinesis.ShardReadersPool stop
> INFO: Closing shard iterators pool
> Feb 01, 2018 11:24:24 PM org.apache.beam.sdk.io.kinesis.ShardReadersPool 
> readLoop
> INFO: Kinesis Shard read loop has finished
> Feb 01, 2018 11:24:24 PM 

[beam] 01/01: Merge pull request #4668: [BEAM-3605] Use verification with timeout instead of Thread.sleep

[iemejia]

This is an automated email from the ASF dual-hosted git repository.

iemejia pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 510fb1623cc2bd20471b3989997b69acb23a1267
Merge: 0bc6f57 83931c8
Author: Ismaël Mejía 
AuthorDate: Wed Feb 14 14:40:49 2018 +0100

Merge pull request #4668: [BEAM-3605] Use verification with timeout instead 
of Thread.sleep

 .../beam/sdk/io/kinesis/ShardReadersPool.java  |  2 +-
 .../beam/sdk/io/kinesis/ShardReadersPoolTest.java  | 36 +-
 2 files changed, 16 insertions(+), 22 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
ieme...@apache.org.

[beam] branch master updated (0bc6f57 -> 510fb16)

[iemejia]

This is an automated email from the ASF dual-hosted git repository.

iemejia pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 0bc6f57  Merge pull request #4629: Functional interface for Fn API 
instructions
 add 83931c8  [BEAM-3605] Use verification with timeout instead of 
Thread.sleep
 new 510fb16  Merge pull request #4668: [BEAM-3605] Use verification with 
timeout instead of Thread.sleep

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../beam/sdk/io/kinesis/ShardReadersPool.java  |  2 +-
 .../beam/sdk/io/kinesis/ShardReadersPoolTest.java  | 36 +-
 2 files changed, 16 insertions(+), 22 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
ieme...@apache.org.

Jenkins build is back to stable : beam_PostCommit_Java_ValidatesRunner_Apex #3473

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-3690) Dependency Conflict problems: several conflicting classes exist in different JARs

[JIRA]

[ 
https://issues.apache.org/jira/browse/BEAM-3690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16363864#comment-16363864
 ] 

Ismaël Mejía commented on BEAM-3690:


Mmm this seems like a handy report, [~PandaMonkey] is the tool you used to find 
this error public/open source ? I am curious to give it a try (also for other 
stuff).

> Dependency Conflict problems: several conflicting classes exist in different 
> JARs
> -
>
> Key: BEAM-3690
> URL: https://issues.apache.org/jira/browse/BEAM-3690
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Affects Versions: 2.2.0
>Reporter: PandaMonkey
>Assignee: PandaMonkey
>Priority: Major
> Fix For: 2.3.0
>
> Attachments: beam_conflicts.txt
>
>
> Hi, we found that there are duplicate classes exist in different JARs, and 
> these classes have different features.
> The conflicting JAR pairs are:
> 1. 
> jar-pair:
> 2. 
> jar-pair:
> Some of method only exist in one version of duplicate classes.
> As the JVM only load the classes present first on the classpath and shadow 
> the other duplicate ones with the same names. The dependency conflict problem 
> brings high risks of "*NoSuchMethodException*" or "*NoSuchMethodError*"  
> issues at runtime. The conflicting details are listed in the attachment. 
> Please notice that. Thanks.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PerformanceTests_Python #913

[Apache Jenkins Server]

See 


--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam7 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 0bc6f574ce9a9bebcf68946456d4ac6c36e81911 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 0bc6f574ce9a9bebcf68946456d4ac6c36e81911
Commit message: "Merge pull request #4629: Functional interface for Fn API 
instructions"
 > git rev-list 0bc6f574ce9a9bebcf68946456d4ac6c36e81911 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_Python] $ /bin/bash -xe /tmp/jenkins447110406681425257.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins4744966751093562125.sh
+ rm -rf .env
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins1529764473363741242.sh
+ virtualenv .env --system-site-packages
New python executable in 

Installing setuptools, pip, wheel...done.
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins5021230108307192583.sh
+ .env/bin/pip install --upgrade setuptools pip
Requirement already up-to-date: setuptools in ./.env/lib/python2.7/site-packages
Requirement already up-to-date: pip in ./.env/lib/python2.7/site-packages
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins7727214672884317232.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins2206565979567482768.sh
+ .env/bin/pip install -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied: absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: setuptools in ./.env/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 16))
Requirement already satisfied: colorlog[windows]==2.6.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied: futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied: PyYAML==3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied: pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Collecting numpy==1.13.3 (from -r PerfKitBenchmarker/requirements.txt (line 22))
:318:
 SNIMissingWarning: An HTTPS request has been made, but the SNI (Subject Name 
Indication) extension to TLS is not available on this platform. This may cause 
the server to present an incorrect TLS certificate, which can cause validation 
failures. You can upgrade to a newer version of Python to solve this. For more 
information, see 
https://urllib3.readthedocs.io/en/latest/security.html#snimissingwarning.
  SNIMissingWarning
:122:
 InsecurePlatformWarning: A true SSLContext object is not available. This 
prevents urllib3 from configuring SSL appropriately and may cause certain SSL 
connections to fail. You can upgrade to a newer version of Python to solve 
this. For more information, see