Jenkins build is back to stable : beam_PostCommit_Java_ValidatesRunner_Dataflow #4902

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PerformanceTests_Python #884

[Apache Jenkins Server]

See 


Changes:

[coheigea] BEAM-3624 - Remove collapsible if statements

[kenn] Switch FullWindowedValueCoder to bypass validation

[kedin] Refactor BeamRecordType and BeamRecord

[ccy] [BEAM-3635] Infer type hints on PTransformOverrides

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam4 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision d7bee8cf41dcc35f70135bd5baab25a6beaeaaca (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f d7bee8cf41dcc35f70135bd5baab25a6beaeaaca
Commit message: "BEAM-3624 - Remove collapsible if statements"
 > git rev-list 92fd3545244d362f56f8f339d2630088f0d1af25 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins6851501810467580239.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins3234221321383151487.sh
+ rm -rf .env
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins3877608149176422395.sh
+ virtualenv .env --system-site-packages
New python executable in .env/bin/python
Installing setuptools, pip...done.
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins8836285189997510112.sh
+ .env/bin/pip install --upgrade setuptools pip
Downloading/unpacking setuptools from 
https://pypi.python.org/packages/43/41/033a273f9a25cb63050a390ee8397acbc7eae2159195d85f06f17e7be45a/setuptools-38.5.1-py2.py3-none-any.whl#md5=908b8b5e50bf429e520b2b5fa1b350e5
Downloading/unpacking pip from 
https://pypi.python.org/packages/b6/ac/7015eb97dc749283ffdec1c3a88ddb8ae03b8fad0f0e611408f196358da3/pip-9.0.1-py2.py3-none-any.whl#md5=297dbd16ef53bcef0447d245815f5144
Installing collected packages: setuptools, pip
  Found existing installation: setuptools 2.2
Uninstalling setuptools:
  Successfully uninstalled setuptools
  Found existing installation: pip 1.5.4
Uninstalling pip:
  Successfully uninstalled pip
Successfully installed setuptools pip
Cleaning up...
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins1222711684998706117.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins7797147748797238086.sh
+ .env/bin/pip install -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied: absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: setuptools in ./.env/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 16))
Requirement already satisfied: colorlog[windows]==2.6.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied: futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied: PyYAML==3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied: pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Collecting numpy==1.13.3 (from -r PerfKitBenchmarker/requirements.txt (line 22))
:318:
 SNIMissingWarning: An HTTPS request has been made, but the SNI (Subject Name 
Indication) extension to TLS is not available on this platform. This may cause 
the server to 

Jenkins build is back to normal : beam_PerformanceTests_TFRecordIOIT #111

[Apache Jenkins Server]

See 

Jenkins build is back to normal : beam_PostCommit_Python_Verify #4166

[Apache Jenkins Server]

See 

Jenkins build is back to normal : beam_PostCommit_Python_ValidatesRunner_Dataflow #847

[Apache Jenkins Server]

See 

[jira] [Resolved] (BEAM-3624) Remove collapsible if statements

[Luke Cwik (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Cwik resolved BEAM-3624.
-
Resolution: Fixed

> Remove collapsible if statements
> 
>
> Key: BEAM-3624
> URL: https://issues.apache.org/jira/browse/BEAM-3624
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Colm O hEigeartaigh
>Assignee: Colm O hEigeartaigh
>Priority: Trivial
> Fix For: 2.4.0
>
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> There are a few instances in the code where we could collapse if statements 
> to make the code more concise.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] 01/01: BEAM-3624 - Remove collapsible if statements

[lcwik]

This is an automated email from the ASF dual-hosted git repository.

lcwik pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit d7bee8cf41dcc35f70135bd5baab25a6beaeaaca
Merge: 0e60ca4 58a6b80
Author: Lukasz Cwik 
AuthorDate: Tue Feb 6 21:30:00 2018 -0800

BEAM-3624 - Remove collapsible if statements

 .../wrappers/streaming/DoFnOperator.java   | 10 +++---
 .../beam/runners/dataflow/TestDataflowRunner.java  | 10 +++---
 .../spark/translation/EvaluationContext.java   | 12 +++
 .../beam/sdk/runners/TransformHierarchy.java   | 36 +--
 .../java/org/apache/beam/sdk/state/StateSpecs.java | 42 --
 .../java/org/apache/beam/sdk/util/ZipFiles.java| 10 +++---
 .../apache/beam/sdk/util/MovingFunctionTest.java   |  6 ++--
 .../beam/sdk/io/elasticsearch/ElasticsearchIO.java | 15 
 .../io/hadoop/inputformat/HadoopInputFormatIO.java |  8 ++---
 9 files changed, 59 insertions(+), 90 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
lc...@apache.org.

[beam] branch master updated (0e60ca4 -> d7bee8c)

[lcwik]

This is an automated email from the ASF dual-hosted git repository.

lcwik pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 0e60ca4  Merge pull request #4618: [BEAM-3627] Switch 
FullWindowedValueCoder to bypass validation
 add 58a6b80  BEAM-3624 - Remove collapsible if statements
 new d7bee8c  BEAM-3624 - Remove collapsible if statements

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../wrappers/streaming/DoFnOperator.java   | 10 +++---
 .../beam/runners/dataflow/TestDataflowRunner.java  | 10 +++---
 .../spark/translation/EvaluationContext.java   | 12 +++
 .../beam/sdk/runners/TransformHierarchy.java   | 36 +--
 .../java/org/apache/beam/sdk/state/StateSpecs.java | 42 --
 .../java/org/apache/beam/sdk/util/ZipFiles.java| 10 +++---
 .../apache/beam/sdk/util/MovingFunctionTest.java   |  6 ++--
 .../beam/sdk/io/elasticsearch/ElasticsearchIO.java | 15 
 .../io/hadoop/inputformat/HadoopInputFormatIO.java |  8 ++---
 9 files changed, 59 insertions(+), 90 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
lc...@apache.org.

Jenkins build is unstable: beam_PostCommit_Java_ValidatesRunner_Dataflow #4901

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PostCommit_Python_Verify #4165

[Apache Jenkins Server]

See 


--
[...truncated 1.53 MB...]
  "kind": "ParallelDo", 
  "name": "s14", 
  "properties": {
"display_data": [
  {
"key": "fn", 
"label": "Transform Function", 
"namespace": "apache_beam.transforms.core.CallableWrapperDoFn", 
"type": "STRING", 
"value": ""
  }, 
  {
"key": "fn", 
"label": "Transform Function", 
"namespace": "apache_beam.transforms.core.ParDo", 
"shortValue": "CallableWrapperDoFn", 
"type": "STRING", 
"value": "apache_beam.transforms.core.CallableWrapperDoFn"
  }
], 
"non_parallel_inputs": {}, 
"output_info": [
  {
"encoding": {
  "@type": "kind:windowed_value", 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}, 
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}
  ], 
  "is_pair_like": true
}, 
{
  "@type": "kind:global_window"
}
  ], 
  "is_wrapper": true
}, 
"output_name": "out", 
"user_name": "write/Write/WriteImpl/Extract.out"
  }
], 
"parallel_input": {
  "@type": "OutputReference", 
  "output_name": "out", 
  "step_name": "s13"
}, 
"serialized_fn": "", 
"user_name": "write/Write/WriteImpl/Extract"
  }
}, 
{
  "kind": "CollectionToSingleton", 
  "name": "SideInput-s15", 
  "properties": {
"output_info": [
  {
"encoding": {
  "@type": "kind:windowed_value", 
  "component_encodings": [
{
  "@type": "kind:windowed_value", 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}, 
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}
  ], 
  "is_pair_like": true
}, 
{
  "@type": "kind:global_window"
}
  ], 
  "is_wrapper": true
}
  ]
}, 
"output_name": "out", 
"user_name": 
"write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(InitializeWrite.out.0).output"
  }
], 
"parallel_input": {
  "@type": "OutputReference", 
  "output_name": "out", 
  "step_name": "s8"
}, 
"user_name": 
"write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(InitializeWrite.out.0)"
  }
}, 
{
  "kind": "CollectionToSingleton", 
  "name": "SideInput-s16", 
  "properties": {
"output_info": [
  {
"encoding": {
  "@type": "kind:windowed_value", 
  "component_encodings": [
{
  "@type": "kind:windowed_value", 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": [
{
  "@type": 

[jira] [Created] (BEAM-3636) DirectRunner proto roundtrip masks stable name errors

[Kenneth Knowles (JIRA)]

Kenneth Knowles created BEAM-3636:
-

 Summary: DirectRunner proto roundtrip masks stable name errors
 Key: BEAM-3636
 URL: https://issues.apache.org/jira/browse/BEAM-3636
 Project: Beam
  Issue Type: Bug
  Components: runner-direct
Reporter: Kenneth Knowles
Assignee: Kenneth Knowles


Example failure: 
https://builds.apache.org/job/beam_PreCommit_Java_MavenInstall/org.apache.beam$beam-sdks-java-io-hbase/17584/testReport/junit/org.apache.beam.sdk.io.hbase/HBaseIOTest/testReadingWithKeyRange/

{code}
Pipeline update will not be possible because the following transforms do not 
have stable unique names: 
PAssert$2/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey,
 CountTEST-KEY-RANGE-TABLE_ByteKeyRange{startKey=[], 
endKey=[32]}/Combine.perKey(Count)/GroupByKey.

Conflicting instances:
- 
name=PAssert$2/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey:
- GroupByKey
- GroupByKey
- name=CountTEST-KEY-RANGE-TABLE_ByteKeyRange{startKey=[], 
endKey=[32]}/Combine.perKey(Count)/GroupByKey:
- GroupByKey
- GroupByKey

You can fix it adding a name when you call apply(): pipeline.apply(, 
).
{code}

The test clearly does have stable name errors. This was exposed by reverting 
the proto roundtrip in the DirectRunner. Precommit and postcommit are otherwise 
green, so this is being masked.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Python_ValidatesRunner_Dataflow #846

[Apache Jenkins Server]

See 


--
[...truncated 235.04 KB...]
{
  "@type": "kind:stream", 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}, 
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}
  ], 
  "is_pair_like": true
}
  ], 
  "is_stream_like": true
}
  ], 
  "is_pair_like": true
}, 
{
  "@type": "kind:global_window"
}
  ], 
  "is_wrapper": true
}, 
"output_name": "out", 
"user_name": "assert_that/Group/GroupByKey.out"
  }
], 
"parallel_input": {
  "@type": "OutputReference", 
  "output_name": "out", 
  "step_name": "s10"
}, 
"serialized_fn": 
"%0AD%22B%0A%1Dref_Coder_GlobalWindowCoder_1%12%21%0A%1F%0A%1D%0A%1Bbeam%3Acoder%3Aglobal_window%3Av1jT%0A%25%0A%23%0A%21beam%3Awindowfn%3Aglobal_windows%3Av0.1%10%01%1A%1Dref_Coder_GlobalWindowCoder_1%22%02%3A%00%28%010%018%01H%01",
 
"user_name": "assert_that/Group/GroupByKey"
  }
}, 
{
  "kind": "ParallelDo", 
  "name": "s12", 
  "properties": {
"display_data": [
  {
"key": "fn", 
"label": "Transform Function", 
"namespace": "apache_beam.transforms.core.CallableWrapperDoFn", 
"type": "STRING", 
"value": "_merge_tagged_vals_under_key"
  }, 
  {
"key": "fn", 
"label": "Transform Function", 
"namespace": "apache_beam.transforms.core.ParDo", 
"shortValue": "CallableWrapperDoFn", 
"type": "STRING", 
"value": "apache_beam.transforms.core.CallableWrapperDoFn"
  }
], 
"non_parallel_inputs": {}, 
"output_info": [
  {
"encoding": {
  "@type": "kind:windowed_value", 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}, 
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}
  ], 
  "is_pair_like": true
}, 
{
  "@type": "kind:global_window"
}
  ], 
  "is_wrapper": true
}, 
"output_name": "out", 
"user_name": 
"assert_that/Group/Map(_merge_tagged_vals_under_key).out"
  }
], 
"parallel_input": {
  "@type": "OutputReference", 
  "output_name": "out", 
  "step_name": "s11"
}, 
"serialized_fn": "", 
"user_name": "assert_that/Group/Map(_merge_tagged_vals_under_key)"
  }
}, 
{
  "kind": "ParallelDo", 
  "name": "s13", 
  "properties": {
"display_data": [
  {
"key": "fn", 
"label": "Transform Function", 
"namespace": "apache_beam.transforms.core.CallableWrapperDoFn", 
"type": "STRING", 
"value": ""
  }, 
  {
"key": "fn", 
"label": "Transform Function", 
"namespace": "apache_beam.transforms.core.ParDo", 

Jenkins build is back to normal : beam_PostCommit_Python_Verify #4163

[Apache Jenkins Server]

See 

[beam] 01/01: Merge pull request #4618: [BEAM-3627] Switch FullWindowedValueCoder to bypass validation

[kenn]

This is an automated email from the ASF dual-hosted git repository.

kenn pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 0e60ca459acfcfc410b7d9ddc819acc8e814871a
Merge: 1ac8a24 ec69c8c
Author: Kenn Knowles 
AuthorDate: Tue Feb 6 18:08:51 2018 -0800

Merge pull request #4618: [BEAM-3627] Switch FullWindowedValueCoder to 
bypass validation

 .../org/apache/beam/sdk/util/WindowedValue.java| 22 ++
 1 file changed, 18 insertions(+), 4 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
k...@apache.org.

[beam] branch master updated (1ac8a24 -> 0e60ca4)

[kenn]

This is an automated email from the ASF dual-hosted git repository.

kenn pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 1ac8a24  Merge pull request #4623 from 
charlesccychen/fix-python-streaming
 add ec69c8c  Switch FullWindowedValueCoder to bypass validation
 new 0e60ca4  Merge pull request #4618: [BEAM-3627] Switch 
FullWindowedValueCoder to bypass validation

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../org/apache/beam/sdk/util/WindowedValue.java| 22 ++
 1 file changed, 18 insertions(+), 4 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
k...@apache.org.

[beam] branch master updated (06a4a15 -> 1ac8a24)

[robertwb]

This is an automated email from the ASF dual-hosted git repository.

robertwb pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 06a4a15  Merge pull request #4593: [BEAM-2721] Refactor BeamRecordType 
and BeamRecord
 add f887ed0  [BEAM-3635] Infer type hints on PTransformOverrides
 new 1ac8a24  Merge pull request #4623 from 
charlesccychen/fix-python-streaming

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 sdks/python/apache_beam/pipeline.py| 57 +-
 sdks/python/apache_beam/pipeline_test.py   | 40 +++
 .../apache_beam/runners/direct/direct_runner.py| 25 +-
 .../runners/direct/transform_evaluator.py  | 22 +
 4 files changed, 100 insertions(+), 44 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
rober...@apache.org.

[beam] 01/01: Merge pull request #4623 from charlesccychen/fix-python-streaming

[robertwb]

This is an automated email from the ASF dual-hosted git repository.

robertwb pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 1ac8a24bbd952bd8a494ca7de5c89488b82d7a54
Merge: 06a4a15 f887ed0
Author: Robert Bradshaw 
AuthorDate: Tue Feb 6 17:39:16 2018 -0800

Merge pull request #4623 from charlesccychen/fix-python-streaming

[BEAM-3635] Infer type hints on PTransformOverrides

 sdks/python/apache_beam/pipeline.py| 57 +-
 sdks/python/apache_beam/pipeline_test.py   | 40 +++
 .../apache_beam/runners/direct/direct_runner.py| 25 +-
 .../runners/direct/transform_evaluator.py  | 22 +
 4 files changed, 100 insertions(+), 44 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
rober...@apache.org.

[jira] [Resolved] (BEAM-3316) Consider: do not use graph.Graph for runtime bundle representation

[Henning Rohde (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Henning Rohde resolved BEAM-3316.
-
   Resolution: Fixed
Fix Version/s: Not applicable

> Consider: do not use graph.Graph for runtime bundle representation
> --
>
> Key: BEAM-3316
> URL: https://issues.apache.org/jira/browse/BEAM-3316
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-go
>Reporter: Henning Rohde
>Assignee: Henning Rohde
>Priority: Minor
> Fix For: Not applicable
>
>
> The internal graph.Graph representation is used to unify user pipeline and 
> the runtime subgraph. Perhaps we should instead translate the bundle 
> descriptors directly to an execution plan (=
>  []exec.Unit, but a encapsulating type seems useful). Aside: we would likely 
> also need to have them be re-usable across bundles serially.
> Such a change would keep the construction-time and runtime worlds more apart 
> instead of trying to unify them and we could remove DataSource/DataSink from 
> the graph. The direct runner would then only translate a graph.Graph to 
> []Unit and not have an awkward injection point for a DataManager.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (BEAM-3302) Go CoGBK support

[Henning Rohde (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3302?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Henning Rohde resolved BEAM-3302.
-
   Resolution: Fixed
Fix Version/s: Not applicable

> Go CoGBK support
> 
>
> Key: BEAM-3302
> URL: https://issues.apache.org/jira/browse/BEAM-3302
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-go
>Reporter: Henning Rohde
>Assignee: Henning Rohde
>Priority: Major
> Fix For: Not applicable
>
>  Time Spent: 1h 10m
>  Remaining Estimate: 0h
>
> Consider using the java approach and implement as union coder over GBK, given 
> that runners may not have CoGBK support.
> Short design doc: 
> https://docs.google.com/document/d/18SjId_0OQLUX2v5EbNjWfJHLnx96R0V8poLSUKmymig/edit#



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-2855) Implement a python version of the nexmark queries

[Ahmet Altay (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ahmet Altay reassigned BEAM-2855:
-

Assignee: (was: Ahmet Altay)

> Implement a python version of the nexmark queries
> -
>
> Key: BEAM-2855
> URL: https://issues.apache.org/jira/browse/BEAM-2855
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py-core, testing
>Reporter: Ismaël Mejía
>Priority: Minor
>  Labels: newbie, nexmark, starter
>
> Currently we have a Java only implementation of Nexmark, a python based 
> implementation would be nice to have to validate the direct and dataflow 
> runners, but also to validate the new support of multiple SDKs in multiple 
> runners via the runner/fn API.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-2535) Allow explicit output time independent of firing specification for all timers

[Batkhuyag Batsaikhan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Batkhuyag Batsaikhan updated BEAM-2535:
---
Description: 
Today, we have insufficient control over the event time timestamp of elements 
output from a timer callback.

1. For an event time timer, it is the timestamp of the timer itself.
 2. For a processing time timer, it is the current input watermark at the time 
of processing.

But for both of these, we may want to reserve the right to output a particular 
time, aka set a "watermark hold".

A naive implementation of a {{TimerWithWatermarkHold}} would work for making 
sure output is not droppable, but does not fully explain window expiration and 
late data/timer dropping.

In the natural interpretation of a timer as a feedback loop on a transform, 
timers should be viewed as another channel of input, with a watermark, and 
items on that channel _all need event time timestamps even if they are 
delivered according to a different time domain_.

I propose that the specification for when a timer should fire should be 
separated (with nice defaults) from the specification of the event time of 
resulting outputs. These timestamps will determine a side channel with a new 
"timer watermark" that constrains the output watermark.
 - We still need to fire event time timers according to the input watermark, so 
that event time timers fire.
 - Late data dropping and window expiration will be in terms of the minimum of 
the input watermark and the timer watermark. In this way, whenever a timer is 
set, the window is not going to be garbage collected.
 - We will need to make sure we have a way to "wake up" a window once it is 
expired; this may be as simple as exhausting the timer channel as soon as the 
input watermark indicates expiration of a window

This is mostly aimed at end-user timers in a stateful+timely {{DoFn}}. It seems 
reasonable to use timers as an implementation detail (e.g. in runners-core 
utilities) without wanting any of this additional machinery. For example, if 
there is no possibility of output from the timer callback.

  was:
Today, we have insufficient control over the event time timestamp of elements 
output from a timer callback.

1. For an event time timer, it is the timestamp of the timer itself.
2. For a processing time timer, it is the current input watermark at the time 
of processing.

But for both of these, we may want to reserve the right to output a particular 
time, aka set a "watermark hold".

A naive implementation of a {{TimerWithWatermarkHold}} would work for making 
sure output is not droppable, but does not fully explain window expiration and 
late data/timer dropping.

In the natural interpretation of a timer as a feedback loop on a transform, 
timers should be viewed as another channel of input, with a watermark, and 
items on that channel _all need event time timestamps even if they are 
delivered according to a different time domain_.

I propose that the specification for when a timer should fire should be 
separated (with nice defaults) from the specification of the event time of 
resulting outputs. These timestamps will determine a side channel with a new 
"timer watermark" that constrains the output watermark.

 - We still need to fire event time timers according to the input watermark, so 
that event time timers fire.
 - Late data dropping and window expiration will be in terms of the minimum of 
the input watermark and the timer watermark. In this way, whenever a timer is 
set, the window is not going to be garbage collected.
 - We will need to make sure we have a way to "wake up" a window once it is 
expired; this may be as simple as exhausting the timer channel as soon as the 
input watermark indicates expiration of a window

This is mostly aimed at end-user timers in a stateful+timely {{DoFn}}. It seems 
reasonable to use timers as an implementation detail (e.g. in runners-core 
utilities) without wanting any of this additional machinery. For example, if 
there is no possibility of output from the timer callback.


> Allow explicit output time independent of firing specification for all timers
> -
>
> Key: BEAM-2535
> URL: https://issues.apache.org/jira/browse/BEAM-2535
> Project: Beam
>  Issue Type: New Feature
>  Components: beam-model, sdk-java-core
>Reporter: Kenneth Knowles
>Assignee: Batkhuyag Batsaikhan
>Priority: Major
>
> Today, we have insufficient control over the event time timestamp of elements 
> output from a timer callback.
> 1. For an event time timer, it is the timestamp of the timer itself.
>  2. For a processing time timer, it is the current input watermark at the 
> time of processing.
> But for both of these, we may want to reserve the right to output a 
> 

[jira] [Assigned] (BEAM-2855) Implement a python version of the nexmark queries

[Batkhuyag Batsaikhan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Batkhuyag Batsaikhan reassigned BEAM-2855:
--

Assignee: Ahmet Altay  (was: Batkhuyag Batsaikhan)

> Implement a python version of the nexmark queries
> -
>
> Key: BEAM-2855
> URL: https://issues.apache.org/jira/browse/BEAM-2855
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py-core, testing
>Reporter: Ismaël Mejía
>Assignee: Ahmet Altay
>Priority: Minor
>  Labels: newbie, nexmark, starter
>
> Currently we have a Java only implementation of Nexmark, a python based 
> implementation would be nice to have to validate the direct and dataflow 
> runners, but also to validate the new support of multiple SDKs in multiple 
> runners via the runner/fn API.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (BEAM-3354) Timer setRelative() does not reset previously set timer.

[Batkhuyag Batsaikhan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Batkhuyag Batsaikhan resolved BEAM-3354.

   Resolution: Fixed
Fix Version/s: 2.3.0

> Timer setRelative() does not reset previously set timer.
> 
>
> Key: BEAM-3354
> URL: https://issues.apache.org/jira/browse/BEAM-3354
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Batkhuyag Batsaikhan
>Assignee: Batkhuyag Batsaikhan
>Priority: Major
> Fix For: 2.3.0
>
>
> When running a pipeline with Timer on direct runner as mentioned in 
> https://beam.apache.org/blog/2017/08/28/timely-processing.html, multiple 
> calls to setRelative() is starting new timers rather than resetting. (set() 
> method is resetting the timer correctly)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Python_Verify #4162

[Apache Jenkins Server]

See 


Changes:

[kedin] Refactor BeamRecordType and BeamRecord

--
[...truncated 1.53 MB...]
root: ERROR: Giving up after 4 attempts.
root: WARNING: A task failed with exception: 'KeyedWorkItem' object is not 
iterable [while running 'assert_that/Group/GroupByKey/GroupByWindow']
- >> end captured logging << -

==
ERROR: test_gbk_execution (apache_beam.testing.test_stream_test.TestStreamTest)
--
Traceback (most recent call last):
  File 
"
 line 148, in test_gbk_execution
p.run()
  File 
"
 line 104, in run
state = result.wait_until_finish()
  File 
"
 line 318, in wait_until_finish
self._executor.await_completion()
  File 
"
 line 358, in await_completion
self._executor.await_completion()
  File 
"
 line 305, in call
side_input_values)
  File 
"
 line 338, in attempt_call
evaluator.process_element(value)
  File 
"
 line 560, in process_element
self.runner.process(element)
  File "apache_beam/runners/common.py", line 468, in 
apache_beam.runners.common.DoFnRunner.process 
(apache_beam/runners/common.c:13915)
def process(self, windowed_value):
  File "apache_beam/runners/common.py", line 474, in 
apache_beam.runners.common.DoFnRunner.process 
(apache_beam/runners/common.c:13719)
self._reraise_augmented(exn)
  File "apache_beam/runners/common.py", line 515, in 
apache_beam.runners.common.DoFnRunner._reraise_augmented 
(apache_beam/runners/common.c:14934)
raise new_exn, None, original_traceback
  File "apache_beam/runners/common.py", line 472, in 
apache_beam.runners.common.DoFnRunner.process 
(apache_beam/runners/common.c:13632)
self.do_fn_invoker.invoke_process(windowed_value)
  File "apache_beam/runners/common.py", line 270, in 
apache_beam.runners.common.SimpleInvoker.invoke_process 
(apache_beam/runners/common.c:9084)
windowed_value, self.process_method(windowed_value.value))
  File 
"
 line 1453, in process
k, vs = element
TypeError: 'KeyedWorkItem' object is not iterable [while running 
'assert_that/Group/GroupByKey/GroupByWindow']
 >> begin captured logging << 
root: INFO: Missing pipeline option (runner). Executing pipeline using the 
default runner: DirectRunner.
root: INFO: Running pipeline with DirectRunner.
root: ERROR: Exception at bundle 
, 
due to an exception.
 Traceback (most recent call last):
  File 
"
 line 305, in call
side_input_values)
  File 
"
 line 338, in attempt_call
evaluator.process_element(value)
  File 
"
 line 560, in process_element
self.runner.process(element)
  File "apache_beam/runners/common.py", line 468, in 
apache_beam.runners.common.DoFnRunner.process 
(apache_beam/runners/common.c:13915)
def process(self, windowed_value):
  File "apache_beam/runners/common.py", line 474, in 
apache_beam.runners.common.DoFnRunner.process 
(apache_beam/runners/common.c:13719)
self._reraise_augmented(exn)
  File "apache_beam/runners/common.py", line 515, in 
apache_beam.runners.common.DoFnRunner._reraise_augmented 
(apache_beam/runners/common.c:14934)
raise new_exn, None, original_traceback
  File "apache_beam/runners/common.py", line 472, in 
apache_beam.runners.common.DoFnRunner.process 
(apache_beam/runners/common.c:13632)
self.do_fn_invoker.invoke_process(windowed_value)
  File "apache_beam/runners/common.py", line 270, in 

Jenkins build became unstable: beam_PostCommit_Java_ValidatesRunner_Dataflow #4899

[Apache Jenkins Server]

See 

[jira] [Created] (BEAM-3635) Infer type hints on PTransformOverrides

[Charles Chen (JIRA)]

Charles Chen created BEAM-3635:
--

 Summary: Infer type hints on PTransformOverrides
 Key: BEAM-3635
 URL: https://issues.apache.org/jira/browse/BEAM-3635
 Project: Beam
  Issue Type: Improvement
  Components: sdk-py-core
Reporter: Charles Chen
Assignee: Charles Chen


Currently, type hints on transforms created by PTransformOverrides are ignored. 
 We should instead infer the output types for these transforms using any type 
hints associated to the transform.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] branch go-sdk updated (52a598d -> a328e44)

[robertwb]

This is an automated email from the ASF dual-hosted git repository.

robertwb pushed a change to branch go-sdk
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 52a598d  [BEAM-3388] Remove reflection from varint codecs
 add 07e534b  Changed core GBK to CoGBK
 add 66ecd91  Add CoGBK support to direct runner and Join example
 add 79fb7e1  [BEAM-3316] Translate bundle descriptors directly to 
execution plans in Go SDK
 add 1cf50dd  Translate CoGBK into GBK for Dataflow and model pipeline 
runners
 add b7dce98  CR: [BEAM-3302] Support CoGBK in the Go SDK
 new a328e44  Merge pull request #4540 [BEAM-3302] Support CoGBK in the Go 
SDK

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 sdks/go/examples/cookbook/join/join.go | 111 
 sdks/go/pkg/beam/coder.go  |   8 +-
 sdks/go/pkg/beam/combine.go|  32 +-
 sdks/go/pkg/beam/core/graph/bind.go|  51 +-
 sdks/go/pkg/beam/core/graph/bind_test.go   |  18 +-
 sdks/go/pkg/beam/core/graph/coder/coder.go |  47 +-
 sdks/go/pkg/beam/core/graph/edge.go| 160 +++--
 sdks/go/pkg/beam/core/runtime/exec/coder.go|  28 +-
 sdks/go/pkg/beam/core/runtime/exec/cogbk.go| 171 +
 sdks/go/pkg/beam/core/runtime/exec/combine.go  |  54 +-
 sdks/go/pkg/beam/core/runtime/exec/combine_test.go |   4 +-
 sdks/go/pkg/beam/core/runtime/exec/datasink.go |  18 +-
 sdks/go/pkg/beam/core/runtime/exec/datasource.go   |  20 +-
 sdks/go/pkg/beam/core/runtime/exec/fn.go   |   4 +-
 sdks/go/pkg/beam/core/runtime/exec/fullvalue.go|   3 +
 .../pkg/beam/core/runtime/exec/optimized/inputs.go | 695 +++--
 .../beam/core/runtime/exec/optimized/inputs.tmpl   |  33 +
 sdks/go/pkg/beam/core/runtime/exec/pardo.go|  25 +-
 sdks/go/pkg/beam/core/runtime/exec/pardo_test.go   |   2 +-
 sdks/go/pkg/beam/core/runtime/exec/plan.go |   2 +
 sdks/go/pkg/beam/core/runtime/exec/translate.go| 433 +
 sdks/go/pkg/beam/core/runtime/graphx/cobgk.go  | 108 
 sdks/go/pkg/beam/core/runtime/graphx/coder.go  |  68 +-
 sdks/go/pkg/beam/core/runtime/graphx/coder_test.go |   8 +-
 sdks/go/pkg/beam/core/runtime/graphx/serialize.go  |  87 ++-
 sdks/go/pkg/beam/core/runtime/graphx/translate.go  | 493 ---
 .../pkg/beam/core/runtime/graphx/translate_test.go |  74 +--
 sdks/go/pkg/beam/core/runtime/graphx/v1/v1.pb.go   | 194 --
 sdks/go/pkg/beam/core/runtime/graphx/v1/v1.proto   |  19 +-
 sdks/go/pkg/beam/core/runtime/harness/datamgr.go   |   9 +-
 sdks/go/pkg/beam/core/runtime/harness/harness.go   |  47 +-
 sdks/go/pkg/beam/core/typex/class.go   |   4 +-
 sdks/go/pkg/beam/core/typex/class_test.go  |   1 -
 sdks/go/pkg/beam/core/typex/fulltype.go|  42 +-
 sdks/go/pkg/beam/core/typex/fulltype_test.go   |  28 +-
 sdks/go/pkg/beam/core/typex/special.go |  10 +-
 sdks/go/pkg/beam/gbk.go|  43 +-
 sdks/go/pkg/beam/impulse.go|   4 +-
 sdks/go/pkg/beam/io/bigqueryio/bigquery.go |   7 +
 sdks/go/pkg/beam/io/textio/textio.go   |   4 +
 sdks/go/pkg/beam/runners/dataflow/translate.go | 182 +-
 sdks/go/pkg/beam/runners/direct/direct.go  |  48 +-
 sdks/go/pkg/beam/runners/direct/gbk.go |  79 ++-
 sdks/go/pkg/beam/testing/passert/passert.go|   4 +-
 sdks/go/pkg/beam/x/debug/print.go  |   4 +-
 45 files changed, 1967 insertions(+), 1519 deletions(-)
 create mode 100644 sdks/go/examples/cookbook/join/join.go
 create mode 100644 sdks/go/pkg/beam/core/runtime/exec/cogbk.go
 create mode 100644 sdks/go/pkg/beam/core/runtime/exec/translate.go
 create mode 100644 sdks/go/pkg/beam/core/runtime/graphx/cobgk.go

-- 
To stop receiving notification emails like this one, please contact
rober...@apache.org.

[beam] 01/01: Merge pull request #4540 [BEAM-3302] Support CoGBK in the Go SDK

[robertwb]

This is an automated email from the ASF dual-hosted git repository.

robertwb pushed a commit to branch go-sdk
in repository https://gitbox.apache.org/repos/asf/beam.git

commit a328e441dbe28f233929b7b3a4a5408663fe7468
Merge: 52a598d b7dce98
Author: Robert Bradshaw 
AuthorDate: Tue Feb 6 16:33:10 2018 -0800

Merge pull request #4540 [BEAM-3302] Support CoGBK in the Go SDK

 sdks/go/examples/cookbook/join/join.go | 111 
 sdks/go/pkg/beam/coder.go  |   8 +-
 sdks/go/pkg/beam/combine.go|  32 +-
 sdks/go/pkg/beam/core/graph/bind.go|  51 +-
 sdks/go/pkg/beam/core/graph/bind_test.go   |  18 +-
 sdks/go/pkg/beam/core/graph/coder/coder.go |  47 +-
 sdks/go/pkg/beam/core/graph/edge.go| 160 +++--
 sdks/go/pkg/beam/core/runtime/exec/coder.go|  28 +-
 sdks/go/pkg/beam/core/runtime/exec/cogbk.go| 171 +
 sdks/go/pkg/beam/core/runtime/exec/combine.go  |  54 +-
 sdks/go/pkg/beam/core/runtime/exec/combine_test.go |   4 +-
 sdks/go/pkg/beam/core/runtime/exec/datasink.go |  18 +-
 sdks/go/pkg/beam/core/runtime/exec/datasource.go   |  20 +-
 sdks/go/pkg/beam/core/runtime/exec/fn.go   |   4 +-
 sdks/go/pkg/beam/core/runtime/exec/fullvalue.go|   3 +
 .../pkg/beam/core/runtime/exec/optimized/inputs.go | 695 +++--
 .../beam/core/runtime/exec/optimized/inputs.tmpl   |  33 +
 sdks/go/pkg/beam/core/runtime/exec/pardo.go|  25 +-
 sdks/go/pkg/beam/core/runtime/exec/pardo_test.go   |   2 +-
 sdks/go/pkg/beam/core/runtime/exec/plan.go |   2 +
 sdks/go/pkg/beam/core/runtime/exec/translate.go| 433 +
 sdks/go/pkg/beam/core/runtime/graphx/cobgk.go  | 108 
 sdks/go/pkg/beam/core/runtime/graphx/coder.go  |  68 +-
 sdks/go/pkg/beam/core/runtime/graphx/coder_test.go |   8 +-
 sdks/go/pkg/beam/core/runtime/graphx/serialize.go  |  87 ++-
 sdks/go/pkg/beam/core/runtime/graphx/translate.go  | 493 ---
 .../pkg/beam/core/runtime/graphx/translate_test.go |  74 +--
 sdks/go/pkg/beam/core/runtime/graphx/v1/v1.pb.go   | 194 --
 sdks/go/pkg/beam/core/runtime/graphx/v1/v1.proto   |  19 +-
 sdks/go/pkg/beam/core/runtime/harness/datamgr.go   |   9 +-
 sdks/go/pkg/beam/core/runtime/harness/harness.go   |  47 +-
 sdks/go/pkg/beam/core/typex/class.go   |   4 +-
 sdks/go/pkg/beam/core/typex/class_test.go  |   1 -
 sdks/go/pkg/beam/core/typex/fulltype.go|  42 +-
 sdks/go/pkg/beam/core/typex/fulltype_test.go   |  28 +-
 sdks/go/pkg/beam/core/typex/special.go |  10 +-
 sdks/go/pkg/beam/gbk.go|  43 +-
 sdks/go/pkg/beam/impulse.go|   4 +-
 sdks/go/pkg/beam/io/bigqueryio/bigquery.go |   7 +
 sdks/go/pkg/beam/io/textio/textio.go   |   4 +
 sdks/go/pkg/beam/runners/dataflow/translate.go | 182 +-
 sdks/go/pkg/beam/runners/direct/direct.go  |  48 +-
 sdks/go/pkg/beam/runners/direct/gbk.go |  79 ++-
 sdks/go/pkg/beam/testing/passert/passert.go|   4 +-
 sdks/go/pkg/beam/x/debug/print.go  |   4 +-
 45 files changed, 1967 insertions(+), 1519 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
rober...@apache.org.

[jira] [Updated] (BEAM-3634) [SQL] Refactor BeamRelNodes into PTransforms

[Anton Kedin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Anton Kedin updated BEAM-3634:
--
Summary: [SQL] Refactor BeamRelNodes into PTransforms  (was: [SQL] Refactor 
PTransforms)

> [SQL] Refactor BeamRelNodes into PTransforms
> 
>
> Key: BEAM-3634
> URL: https://issues.apache.org/jira/browse/BEAM-3634
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql
>Reporter: Anton Kedin
>Priority: Major
>
> BeamRelNode exposes PCollection buildBeamPipeline() which builds 
> a pipeline when parsing.
> It feels like it should instead implement a 
> PTransform which would 
> receive a prepared PCollection, and apply sub-expressions instead of manually 
> invoking expression evaluation to get the input.
> And maybe consider building it lazily.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-3634) [SQL] Refactor PTransforms

[Anton Kedin (JIRA)]

Anton Kedin created BEAM-3634:
-

 Summary: [SQL] Refactor PTransforms
 Key: BEAM-3634
 URL: https://issues.apache.org/jira/browse/BEAM-3634
 Project: Beam
  Issue Type: Improvement
  Components: dsl-sql
Reporter: Anton Kedin


BeamRelNode exposes PCollection buildBeamPipeline() which builds a 
pipeline when parsing.

It feels like it should instead implement a PTransform which would receive a prepared PCollection, and apply 
sub-expressions instead of manually invoking expression evaluation to get the 
input.

And maybe consider building it lazily.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PerformanceTests_TFRecordIOIT #110

[Apache Jenkins Server]

See 


Changes:

[rangadi] KafkaIO : move source and sink implemenations into own files.

[rangadi] minor

[kenn] Deprecate DoFnTester

[klk] Remove DoFnTester from core SDK tests

[kenn] Sickbay ApexRunner ParDoTranslatorTest.testAssertionFailure

[sidhom] Allow docker tag root to be specified as in Maven image build

[herohde] [BEAM-3457] Upgrade gogradle and fix thrift resolution issue

--
[...truncated 13.85 KB...]
Requirement already satisfied: grpc-google-iam-v1<0.12dev,>=0.11.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
gapic-google-cloud-pubsub-v1<0.16dev,>=0.15.0->google-cloud-pubsub==0.26.0->apache-beam==2.4.0.dev0)
Requirement already satisfied: cachetools>=2.0.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
google-auth<2.0.0dev,>=0.4.0->google-cloud-core<0.26dev,>=0.25.0->google-cloud-pubsub==0.26.0->apache-beam==2.4.0.dev0)
Requirement already satisfied: future<0.17dev,>=0.16.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
google-gax<0.16dev,>=0.15.7->gapic-google-cloud-pubsub-v1<0.16dev,>=0.15.0->google-cloud-pubsub==0.26.0->apache-beam==2.4.0.dev0)
Requirement already satisfied: ply==3.8 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
google-gax<0.16dev,>=0.15.7->gapic-google-cloud-pubsub-v1<0.16dev,>=0.15.0->google-cloud-pubsub==0.26.0->apache-beam==2.4.0.dev0)
Requirement already satisfied: requests<3.0dev,>=2.13.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
google-gax<0.16dev,>=0.15.7->gapic-google-cloud-pubsub-v1<0.16dev,>=0.15.0->google-cloud-pubsub==0.26.0->apache-beam==2.4.0.dev0)
Requirement already satisfied: idna<2.6,>=2.5 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
requests<3.0dev,>=2.13.0->google-gax<0.16dev,>=0.15.7->gapic-google-cloud-pubsub-v1<0.16dev,>=0.15.0->google-cloud-pubsub==0.26.0->apache-beam==2.4.0.dev0)
Requirement already satisfied: urllib3<1.22,>=1.21.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
requests<3.0dev,>=2.13.0->google-gax<0.16dev,>=0.15.7->gapic-google-cloud-pubsub-v1<0.16dev,>=0.15.0->google-cloud-pubsub==0.26.0->apache-beam==2.4.0.dev0)
Requirement already satisfied: chardet<3.1.0,>=3.0.2 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
requests<3.0dev,>=2.13.0->google-gax<0.16dev,>=0.15.7->gapic-google-cloud-pubsub-v1<0.16dev,>=0.15.0->google-cloud-pubsub==0.26.0->apache-beam==2.4.0.dev0)
Requirement already satisfied: certifi>=2017.4.17 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
requests<3.0dev,>=2.13.0->google-gax<0.16dev,>=0.15.7->gapic-google-cloud-pubsub-v1<0.16dev,>=0.15.0->google-cloud-pubsub==0.26.0->apache-beam==2.4.0.dev0)
Installing collected packages: apache-beam
  Found existing installation: apache-beam 2.4.0.dev0
Not uninstalling apache-beam at 

 outside environment 

  Running setup.py develop for apache-beam
Successfully installed apache-beam
[beam_PerformanceTests_TFRecordIOIT] $ /bin/bash -xe 
/tmp/jenkins7388911267286422162.sh
+ .env/bin/python PerfKitBenchmarker/pkb.py --project=apache-beam-testing 
--dpb_log_level=INFO --maven_binary=/home/jenkins/tools/maven/latest/bin/mvn 
--bigquery_table=beam_performance.tfrecordioit_pkb_results 
--temp_dir=
 --official=true --benchmarks=beam_integration_benchmark --beam_it_timeout=1200 
--beam_it_profile=io-it --beam_prebuilt=true --beam_sdk=java 
--beam_it_module=sdks/java/io/file-based-io-tests 
--beam_it_class=org.apache.beam.sdk.io.tfrecord.TFRecordIOIT 
'--beam_it_options=[--project=apache-beam-testing,--tempRoot=gs://temp-storage-for-perf-tests,--numberOfRecords=100,--filenamePrefix=gs://temp-storage-for-perf-tests/beam_PerformanceTests_TFRecordIOIT/110/]'
 '--beam_extra_mvn_properties=[filesystem=gcs]'
2018-02-07 00:00:52,176 e18ad919 MainThread INFO Verbose logging to: 

2018-02-07 00:00:52,176 e18ad919 MainThread INFO PerfKitBenchmarker 
version: v1.12.0-260-ge324e61
2018-02-07 00:00:52,177 e18ad919 MainThread INFO Flag values:
--beam_extra_mvn_properties=[filesystem=gcs]
--beam_it_class=org.apache.beam.sdk.io.tfrecord.TFRecordIOIT
--beam_it_timeout=1200
--beam_sdk=java
--temp_dir=
--maven_binary=/home/jenkins/tools/maven/latest/bin/mvn
--beam_it_options=[--project=apache-beam-testing,--tempRoot=gs://temp-storage-for-perf-tests,--numberOfRecords=100,--filenamePrefix=gs://temp-storage-for-perf-tests/beam_PerformanceTests_TFRecordIOIT/110/]
--beam_prebuilt
--project=apache-beam-testing

Build failed in Jenkins: beam_PerformanceTests_Python #883

[Apache Jenkins Server]

See 


Changes:

[rangadi] KafkaIO : move source and sink implemenations into own files.

[rangadi] minor

[kenn] Deprecate DoFnTester

[klk] Remove DoFnTester from core SDK tests

[kenn] Sickbay ApexRunner ParDoTranslatorTest.testAssertionFailure

[sidhom] Allow docker tag root to be specified as in Maven image build

[herohde] [BEAM-3457] Upgrade gogradle and fix thrift resolution issue

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam3 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 92fd3545244d362f56f8f339d2630088f0d1af25 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 92fd3545244d362f56f8f339d2630088f0d1af25
Commit message: "Merge pull request #4507: [BEAM-3159] Deprecated DoFnTester"
 > git rev-list 282636275f3f91cc8a15aa08d8039bcb81d2f9d6 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins6690768810352157030.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins2260185796931340341.sh
+ rm -rf .env
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins9060364513203332968.sh
+ virtualenv .env --system-site-packages
New python executable in .env/bin/python
Installing setuptools, pip...done.
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins1058027827327092644.sh
+ .env/bin/pip install --upgrade setuptools pip
Downloading/unpacking setuptools from 
https://pypi.python.org/packages/43/41/033a273f9a25cb63050a390ee8397acbc7eae2159195d85f06f17e7be45a/setuptools-38.5.1-py2.py3-none-any.whl#md5=908b8b5e50bf429e520b2b5fa1b350e5
Downloading/unpacking pip from 
https://pypi.python.org/packages/b6/ac/7015eb97dc749283ffdec1c3a88ddb8ae03b8fad0f0e611408f196358da3/pip-9.0.1-py2.py3-none-any.whl#md5=297dbd16ef53bcef0447d245815f5144
Installing collected packages: setuptools, pip
  Found existing installation: setuptools 2.2
Uninstalling setuptools:
  Successfully uninstalled setuptools
  Found existing installation: pip 1.5.4
Uninstalling pip:
  Successfully uninstalled pip
Successfully installed setuptools pip
Cleaning up...
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins3853878879441015496.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins5278826318546341380.sh
+ .env/bin/pip install -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied: absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: setuptools in ./.env/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 16))
Requirement already satisfied: colorlog[windows]==2.6.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied: futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied: PyYAML==3.12 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied: pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Requirement already satisfied: numpy==1.13.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 22))
Requirement already satisfied: functools32 in 
/usr/local/lib/python2.7/dist-packages (from -r 

[jira] [Created] (BEAM-3633) [SQL] Reduce usage of Calcite's SqlTypeName

[Anton Kedin (JIRA)]

Anton Kedin created BEAM-3633:
-

 Summary: [SQL] Reduce usage of Calcite's SqlTypeName
 Key: BEAM-3633
 URL: https://issues.apache.org/jira/browse/BEAM-3633
 Project: Beam
  Issue Type: Improvement
  Components: dsl-sql
Reporter: Anton Kedin


We still use Calcite's SqlTypeName as type of BeamSqlExpression and few other 
places. This leaks implementation details, and complicates logic, we now have 
to keep two-way mapping between Beam coders and Calcite's type names.

Feels like we can noticeably simplify the type conversion logic in few places 
by updating BeamSqlExpression and similar classes to use Beam's SqlTypeCoders 
instead of SqlTypeName.

Good place to start is from usages of CalciteUtils.toCalciteType(Coder).

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (BEAM-2721) Augment BeamRecordType to do slicing and concatenation.

[Anton Kedin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Anton Kedin closed BEAM-2721.
-

Assuming this closed. Summary of things completed:
 * removed BeamRecordSqlType;
 * replaced references to java.sql.Types with corresponding Coders;
 * added @AutoValue and factory methods to BeamRecordType and BeamRecord;
 * added BeamRecordType.Field;
 * added java8 stream collectors to BeamRecordType and BeamRecord;
 * replaced iterative BeamRecordType and BeamRecord construction with fluid 
stream-based construction, or factory method calls;

> Augment BeamRecordType to do slicing and concatenation.
> ---
>
> Key: BEAM-2721
> URL: https://issues.apache.org/jira/browse/BEAM-2721
> Project: Beam
>  Issue Type: Bug
>  Components: dsl-sql
>Reporter: Robert Bradshaw
>Assignee: Anton Kedin
>Priority: Major
> Fix For: Not applicable
>
>  Time Spent: 1.5h
>  Remaining Estimate: 0h
>
> Currently in several places we cast to BeamSqlRecordType, extract the field 
> type ints, do the slicing, and then reconstruct a new BeamSqlRecordType. If 
> BeamRecordType had polymorphic methods to slice/concat this would be cleaner 
> and more flexible. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (BEAM-2721) Augment BeamRecordType to do slicing and concatenation.

[Anton Kedin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Anton Kedin resolved BEAM-2721.
---
   Resolution: Fixed
Fix Version/s: Not applicable

> Augment BeamRecordType to do slicing and concatenation.
> ---
>
> Key: BEAM-2721
> URL: https://issues.apache.org/jira/browse/BEAM-2721
> Project: Beam
>  Issue Type: Bug
>  Components: dsl-sql
>Reporter: Robert Bradshaw
>Assignee: Anton Kedin
>Priority: Major
> Fix For: Not applicable
>
>  Time Spent: 1.5h
>  Remaining Estimate: 0h
>
> Currently in several places we cast to BeamSqlRecordType, extract the field 
> type ints, do the slicing, and then reconstruct a new BeamSqlRecordType. If 
> BeamRecordType had polymorphic methods to slice/concat this would be cleaner 
> and more flexible. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] 01/01: Merge pull request #4593: [BEAM-2721] Refactor BeamRecordType and BeamRecord

[kenn]

This is an automated email from the ASF dual-hosted git repository.

kenn pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 06a4a15cf565cae247782da2d2a5483a00ae8f3c
Merge: 92fd354 1419f56
Author: Kenn Knowles 
AuthorDate: Tue Feb 6 16:00:41 2018 -0800

Merge pull request #4593: [BEAM-2721] Refactor BeamRecordType and BeamRecord

 .../apache/beam/sdk/coders/BeamRecordCoder.java|  10 +-
 .../org/apache/beam/sdk/values/BeamRecord.java | 188 ++--
 .../org/apache/beam/sdk/values/BeamRecordType.java | 123 --
 .../org/apache/beam/sdk/values/BeamRecordTest.java | 113 
 .../apache/beam/sdk/values/BeamRecordTypeTest.java |  85 +
 .../beam/sdk/extensions/sql/BeamRecordSqlType.java |  14 +-
 .../sdk/extensions/sql/example/BeamSqlExample.java |  10 +-
 .../operator/BeamSqlInputRefExpression.java|   2 +-
 .../sql/impl/rel/BeamAggregationRel.java   |  40 ++---
 .../sdk/extensions/sql/impl/rel/BeamJoinRel.java   |  20 ++-
 .../sdk/extensions/sql/impl/rel/BeamSortRel.java   |  10 +-
 .../sdk/extensions/sql/impl/rel/BeamValuesRel.java |  43 +++--
 .../extensions/sql/impl/schema/BeamTableUtils.java |  22 +--
 .../impl/transform/BeamAggregationTransforms.java  |  66 +++
 .../sql/impl/transform/BeamJoinTransforms.java |  97 ++-
 .../impl/transform/BeamSqlOutputToConsoleFn.java   |   2 +-
 .../sql/impl/transform/BeamSqlProjectFn.java   |  34 ++--
 .../extensions/sql/impl/utils/CalciteUtils.java|  27 +--
 .../extensions/sql/meta/provider/MetaUtils.java|  17 +-
 .../extensions/sql/BeamSqlDslAggregationTest.java  | 125 ++
 .../sdk/extensions/sql/BeamSqlDslProjectTest.java  |  59 ---
 .../sdk/extensions/sql/BeamSqlDslUdfUdafTest.java  |   4 +-
 .../apache/beam/sdk/extensions/sql/TestUtils.java  |  36 ++--
 .../interpreter/BeamSqlFnExecutorTestBase.java |  27 +--
 .../rel/BeamJoinRelUnboundedVsBoundedTest.java |   3 +-
 .../sql/impl/schema/BeamSqlRowCoderTest.java   |  55 +++---
 .../transform/BeamAggregationTransformTest.java| 189 -
 .../schema/transform/BeamTransformBaseTest.java| 161 +++---
 ...BeamSqlBuiltinFunctionsIntegrationTestBase.java |  23 +--
 .../meta/provider/kafka/BeamKafkaCSVTableTest.java |  40 ++---
 .../meta/provider/text/BeamTextCSVTableTest.java   |  68 +++-
 .../beam/sdk/nexmark/model/sql/BeamRecordSize.java |   2 +-
 .../beam/sdk/nexmark/model/sql/ToBeamRecord.java   |   5 +-
 .../sdk/nexmark/model/sql/BeamRecordSizeTest.java  |  40 ++---
 .../sdk/nexmark/model/sql/ToBeamRecordTest.java|  27 +--
 .../sdk/nexmark/queries/sql/SqlQuery0Test.java |  10 +-
 .../sdk/nexmark/queries/sql/SqlQuery1Test.java |  10 +-
 .../sdk/nexmark/queries/sql/SqlQuery2Test.java |   7 +-
 .../sdk/nexmark/queries/sql/SqlQuery3Test.java |  15 +-
 39 files changed, 1093 insertions(+), 736 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
k...@apache.org.

[beam] branch master updated (92fd354 -> 06a4a15)

[kenn]

This is an automated email from the ASF dual-hosted git repository.

kenn pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 92fd354  Merge pull request #4507: [BEAM-3159] Deprecated DoFnTester
 add 1419f56  Refactor BeamRecordType and BeamRecord
 new 06a4a15  Merge pull request #4593: [BEAM-2721] Refactor BeamRecordType 
and BeamRecord

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../apache/beam/sdk/coders/BeamRecordCoder.java|  10 +-
 .../org/apache/beam/sdk/values/BeamRecord.java | 188 ++--
 .../org/apache/beam/sdk/values/BeamRecordType.java | 123 --
 .../org/apache/beam/sdk/values/BeamRecordTest.java | 113 
 .../apache/beam/sdk/values/BeamRecordTypeTest.java |  85 +
 .../beam/sdk/extensions/sql/BeamRecordSqlType.java |  14 +-
 .../sdk/extensions/sql/example/BeamSqlExample.java |  10 +-
 .../operator/BeamSqlInputRefExpression.java|   2 +-
 .../sql/impl/rel/BeamAggregationRel.java   |  40 ++---
 .../sdk/extensions/sql/impl/rel/BeamJoinRel.java   |  20 ++-
 .../sdk/extensions/sql/impl/rel/BeamSortRel.java   |  10 +-
 .../sdk/extensions/sql/impl/rel/BeamValuesRel.java |  43 +++--
 .../extensions/sql/impl/schema/BeamTableUtils.java |  22 +--
 .../impl/transform/BeamAggregationTransforms.java  |  66 +++
 .../sql/impl/transform/BeamJoinTransforms.java |  97 ++-
 .../impl/transform/BeamSqlOutputToConsoleFn.java   |   2 +-
 .../sql/impl/transform/BeamSqlProjectFn.java   |  34 ++--
 .../extensions/sql/impl/utils/CalciteUtils.java|  27 +--
 .../extensions/sql/meta/provider/MetaUtils.java|  17 +-
 .../extensions/sql/BeamSqlDslAggregationTest.java  | 125 ++
 .../sdk/extensions/sql/BeamSqlDslProjectTest.java  |  59 ---
 .../sdk/extensions/sql/BeamSqlDslUdfUdafTest.java  |   4 +-
 .../apache/beam/sdk/extensions/sql/TestUtils.java  |  36 ++--
 .../interpreter/BeamSqlFnExecutorTestBase.java |  27 +--
 .../rel/BeamJoinRelUnboundedVsBoundedTest.java |   3 +-
 .../sql/impl/schema/BeamSqlRowCoderTest.java   |  55 +++---
 .../transform/BeamAggregationTransformTest.java| 189 -
 .../schema/transform/BeamTransformBaseTest.java| 161 +++---
 ...BeamSqlBuiltinFunctionsIntegrationTestBase.java |  23 +--
 .../meta/provider/kafka/BeamKafkaCSVTableTest.java |  40 ++---
 .../meta/provider/text/BeamTextCSVTableTest.java   |  68 +++-
 .../beam/sdk/nexmark/model/sql/BeamRecordSize.java |   2 +-
 .../beam/sdk/nexmark/model/sql/ToBeamRecord.java   |   5 +-
 .../sdk/nexmark/model/sql/BeamRecordSizeTest.java  |  40 ++---
 .../sdk/nexmark/model/sql/ToBeamRecordTest.java|  27 +--
 .../sdk/nexmark/queries/sql/SqlQuery0Test.java |  10 +-
 .../sdk/nexmark/queries/sql/SqlQuery1Test.java |  10 +-
 .../sdk/nexmark/queries/sql/SqlQuery2Test.java |   7 +-
 .../sdk/nexmark/queries/sql/SqlQuery3Test.java |  15 +-
 39 files changed, 1093 insertions(+), 736 deletions(-)
 create mode 100644 
sdks/java/core/src/test/java/org/apache/beam/sdk/values/BeamRecordTest.java
 create mode 100644 
sdks/java/core/src/test/java/org/apache/beam/sdk/values/BeamRecordTypeTest.java

-- 
To stop receiving notification emails like this one, please contact
k...@apache.org.

[jira] [Comment Edited] (BEAM-3605) Kinesis ShardReadersPoolTest shouldForgetClosedShardIterator failure

[JIRA]

[ 
https://issues.apache.org/jira/browse/BEAM-3605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16354727#comment-16354727
 ] 

Paweł Kaczmarczyk edited comment on BEAM-3605 at 2/6/18 11:37 PM:
--

Hi, sorry for late reply.

Is this the only test case that is flaky? I'm asking because other cases in 
ShardReadersPoolTest also rely on sleeping some amount of time.

I think calling _shardReadersPool.stop()_ should work in this case. Not sure 
about other test cases yet, I would have to take a second look at them. Would 
you like to fix it? I not, I can do it tomorrow.


was (Author: pawel.kaczmarczyk):
Hi, sorry for late reply.

Is this the only test case that is flaky? I'm asking because other cases in 
ShardReadersPoolTest also rely on sleeping some amount of time.

I think calling ___shardReadersPool.stop()_ should work in this case. Not sure 
about other test cases yet, I would have to take a second look at them. Would 
you like to fix it? I not, I can do it tomorrow.

> Kinesis ShardReadersPoolTest shouldForgetClosedShardIterator failure
> 
>
> Key: BEAM-3605
> URL: https://issues.apache.org/jira/browse/BEAM-3605
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Reporter: Kenneth Knowles
>Assignee: Alexey Romanenko
>Priority: Critical
>  Labels: flake, sickbay
>
> Here's one:
> https://builds.apache.org/job/beam_PreCommit_Java_GradleBuild/1758/testReport/junit/org.apache.beam.sdk.io.kinesis/ShardReadersPoolTest/shouldForgetClosedShardIterator/
> Filing all test failures as "Critical" so we can sickbay or fix.
> The Jenkins build will get GC'd so here is the error:
> {code}
> java.lang.AssertionError: 
> Expecting:
>   <["shard1", "shard2"]>
> to contain only:
>   <["shard2"]>
> but the following elements were unexpected:
>   <["shard1"]>
>   at 
> org.apache.beam.sdk.io.kinesis.ShardReadersPoolTest.shouldForgetClosedShardIterator(ShardReadersPoolTest.java:270)
> {code}
> and stderr
> {code}
> Feb 01, 2018 11:24:16 PM org.apache.beam.sdk.io.kinesis.ShardReadersPool 
> readLoop
> INFO: Shard iterator for shard1 shard is closed, finishing the read loop
> org.apache.beam.sdk.io.kinesis.KinesisShardClosedException
> Feb 01, 2018 11:24:16 PM org.apache.beam.sdk.io.kinesis.ShardReadersPool 
> readLoop
> INFO: Kinesis Shard read loop has finished
> Feb 01, 2018 11:24:16 PM org.apache.beam.sdk.io.kinesis.ShardReadersPool 
> readLoop
> INFO: Shard iterator for shard1 shard is closed, finishing the read loop
> org.apache.beam.sdk.io.kinesis.KinesisShardClosedException
> Feb 01, 2018 11:24:16 PM org.apache.beam.sdk.io.kinesis.ShardReadersPool 
> readLoop
> INFO: Kinesis Shard read loop has finished
> Feb 01, 2018 11:24:19 PM org.apache.beam.sdk.io.kinesis.ShardReadersPool 
> readLoop
> INFO: Shard iterator for shard1 shard is closed, finishing the read loop
> org.apache.beam.sdk.io.kinesis.KinesisShardClosedException
> Feb 01, 2018 11:24:19 PM org.apache.beam.sdk.io.kinesis.ShardReadersPool 
> readLoop
> INFO: Kinesis Shard read loop has finished
> Feb 01, 2018 11:24:23 PM org.apache.beam.sdk.io.kinesis.ShardReadersPool 
> readLoop
> INFO: Shard iterator for shard1 shard is closed, finishing the read loop
> org.apache.beam.sdk.io.kinesis.KinesisShardClosedException: Shard iterator 
> reached end of the shard: streamName=null, shardId=shard1
>   at 
> org.apache.beam.sdk.io.kinesis.ShardRecordsIterator.readNextBatch(ShardRecordsIterator.java:70)
>   at 
> org.apache.beam.sdk.io.kinesis.ShardReadersPool.readLoop(ShardReadersPool.java:121)
>   at 
> org.apache.beam.sdk.io.kinesis.ShardReadersPool.lambda$startReadingShards$0(ShardReadersPool.java:112)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>   at java.lang.Thread.run(Thread.java:748)
> Feb 01, 2018 11:24:23 PM org.apache.beam.sdk.io.kinesis.ShardReadersPool 
> readLoop
> INFO: Kinesis Shard read loop has finished
> Feb 01, 2018 11:24:23 PM org.apache.beam.sdk.io.kinesis.ShardReadersPool 
> readLoop
> INFO: Shard iterator for shard2 shard is closed, finishing the read loop
> org.apache.beam.sdk.io.kinesis.KinesisShardClosedException: Shard iterator 
> reached end of the shard: streamName=null, shardId=shard2
>   at 
> org.apache.beam.sdk.io.kinesis.ShardRecordsIterator.readNextBatch(ShardRecordsIterator.java:70)
>   at 
> org.apache.beam.sdk.io.kinesis.ShardReadersPool.readLoop(ShardReadersPool.java:121)
>   at 
> 

[jira] [Commented] (BEAM-3605) Kinesis ShardReadersPoolTest shouldForgetClosedShardIterator failure

[JIRA]

[ 
https://issues.apache.org/jira/browse/BEAM-3605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16354727#comment-16354727
 ] 

Paweł Kaczmarczyk commented on BEAM-3605:
-

Hi, sorry for late reply.

Is this the only test case that is flaky? I'm asking because other cases in 
ShardReadersPoolTest also rely on sleeping some amount of time.

I think calling ___shardReadersPool.stop()_ should work in this case. Not sure 
about other test cases yet, I would have to take a second look at them. Would 
you like to fix it? I not, I can do it tomorrow.

> Kinesis ShardReadersPoolTest shouldForgetClosedShardIterator failure
> 
>
> Key: BEAM-3605
> URL: https://issues.apache.org/jira/browse/BEAM-3605
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Reporter: Kenneth Knowles
>Assignee: Alexey Romanenko
>Priority: Critical
>  Labels: flake, sickbay
>
> Here's one:
> https://builds.apache.org/job/beam_PreCommit_Java_GradleBuild/1758/testReport/junit/org.apache.beam.sdk.io.kinesis/ShardReadersPoolTest/shouldForgetClosedShardIterator/
> Filing all test failures as "Critical" so we can sickbay or fix.
> The Jenkins build will get GC'd so here is the error:
> {code}
> java.lang.AssertionError: 
> Expecting:
>   <["shard1", "shard2"]>
> to contain only:
>   <["shard2"]>
> but the following elements were unexpected:
>   <["shard1"]>
>   at 
> org.apache.beam.sdk.io.kinesis.ShardReadersPoolTest.shouldForgetClosedShardIterator(ShardReadersPoolTest.java:270)
> {code}
> and stderr
> {code}
> Feb 01, 2018 11:24:16 PM org.apache.beam.sdk.io.kinesis.ShardReadersPool 
> readLoop
> INFO: Shard iterator for shard1 shard is closed, finishing the read loop
> org.apache.beam.sdk.io.kinesis.KinesisShardClosedException
> Feb 01, 2018 11:24:16 PM org.apache.beam.sdk.io.kinesis.ShardReadersPool 
> readLoop
> INFO: Kinesis Shard read loop has finished
> Feb 01, 2018 11:24:16 PM org.apache.beam.sdk.io.kinesis.ShardReadersPool 
> readLoop
> INFO: Shard iterator for shard1 shard is closed, finishing the read loop
> org.apache.beam.sdk.io.kinesis.KinesisShardClosedException
> Feb 01, 2018 11:24:16 PM org.apache.beam.sdk.io.kinesis.ShardReadersPool 
> readLoop
> INFO: Kinesis Shard read loop has finished
> Feb 01, 2018 11:24:19 PM org.apache.beam.sdk.io.kinesis.ShardReadersPool 
> readLoop
> INFO: Shard iterator for shard1 shard is closed, finishing the read loop
> org.apache.beam.sdk.io.kinesis.KinesisShardClosedException
> Feb 01, 2018 11:24:19 PM org.apache.beam.sdk.io.kinesis.ShardReadersPool 
> readLoop
> INFO: Kinesis Shard read loop has finished
> Feb 01, 2018 11:24:23 PM org.apache.beam.sdk.io.kinesis.ShardReadersPool 
> readLoop
> INFO: Shard iterator for shard1 shard is closed, finishing the read loop
> org.apache.beam.sdk.io.kinesis.KinesisShardClosedException: Shard iterator 
> reached end of the shard: streamName=null, shardId=shard1
>   at 
> org.apache.beam.sdk.io.kinesis.ShardRecordsIterator.readNextBatch(ShardRecordsIterator.java:70)
>   at 
> org.apache.beam.sdk.io.kinesis.ShardReadersPool.readLoop(ShardReadersPool.java:121)
>   at 
> org.apache.beam.sdk.io.kinesis.ShardReadersPool.lambda$startReadingShards$0(ShardReadersPool.java:112)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>   at java.lang.Thread.run(Thread.java:748)
> Feb 01, 2018 11:24:23 PM org.apache.beam.sdk.io.kinesis.ShardReadersPool 
> readLoop
> INFO: Kinesis Shard read loop has finished
> Feb 01, 2018 11:24:23 PM org.apache.beam.sdk.io.kinesis.ShardReadersPool 
> readLoop
> INFO: Shard iterator for shard2 shard is closed, finishing the read loop
> org.apache.beam.sdk.io.kinesis.KinesisShardClosedException: Shard iterator 
> reached end of the shard: streamName=null, shardId=shard2
>   at 
> org.apache.beam.sdk.io.kinesis.ShardRecordsIterator.readNextBatch(ShardRecordsIterator.java:70)
>   at 
> org.apache.beam.sdk.io.kinesis.ShardReadersPool.readLoop(ShardReadersPool.java:121)
>   at 
> org.apache.beam.sdk.io.kinesis.ShardReadersPool.lambda$startReadingShards$0(ShardReadersPool.java:112)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>   at java.lang.Thread.run(Thread.java:748)
> Feb 

Jenkins build is back to stable : beam_PostCommit_Java_ValidatesRunner_Dataflow #4898

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PostCommit_Python_Verify #4161

[Apache Jenkins Server]

See 


Changes:

[kenn] Deprecate DoFnTester

[klk] Remove DoFnTester from core SDK tests

[sidhom] Allow docker tag root to be specified as in Maven image build

[herohde] [BEAM-3457] Upgrade gogradle and fix thrift resolution issue

--
[...truncated 1.54 MB...]

root: ERROR: Giving up after 4 attempts.
root: WARNING: A task failed with exception: 'KeyedWorkItem' object is not 
iterable [while running 'assert_that/Group/GroupByKey/GroupByWindow']
- >> end captured logging << -

==
ERROR: test_gbk_execution (apache_beam.testing.test_stream_test.TestStreamTest)
--
Traceback (most recent call last):
  File 
"
 line 148, in test_gbk_execution
p.run()
  File 
"
 line 104, in run
state = result.wait_until_finish()
  File 
"
 line 318, in wait_until_finish
self._executor.await_completion()
  File 
"
 line 358, in await_completion
self._executor.await_completion()
  File 
"
 line 305, in call
side_input_values)
  File 
"
 line 338, in attempt_call
evaluator.process_element(value)
  File 
"
 line 560, in process_element
self.runner.process(element)
  File "apache_beam/runners/common.py", line 468, in 
apache_beam.runners.common.DoFnRunner.process 
(apache_beam/runners/common.c:13915)
def process(self, windowed_value):
  File "apache_beam/runners/common.py", line 474, in 
apache_beam.runners.common.DoFnRunner.process 
(apache_beam/runners/common.c:13719)
self._reraise_augmented(exn)
  File "apache_beam/runners/common.py", line 515, in 
apache_beam.runners.common.DoFnRunner._reraise_augmented 
(apache_beam/runners/common.c:14934)
raise new_exn, None, original_traceback
  File "apache_beam/runners/common.py", line 472, in 
apache_beam.runners.common.DoFnRunner.process 
(apache_beam/runners/common.c:13632)
self.do_fn_invoker.invoke_process(windowed_value)
  File "apache_beam/runners/common.py", line 270, in 
apache_beam.runners.common.SimpleInvoker.invoke_process 
(apache_beam/runners/common.c:9084)
windowed_value, self.process_method(windowed_value.value))
  File 
"
 line 1453, in process
k, vs = element
TypeError: 'KeyedWorkItem' object is not iterable [while running 
'assert_that/Group/GroupByKey/GroupByWindow']
 >> begin captured logging << 
root: INFO: Missing pipeline option (runner). Executing pipeline using the 
default runner: DirectRunner.
root: INFO: Running pipeline with DirectRunner.
root: ERROR: Exception at bundle 
, 
due to an exception.
 Traceback (most recent call last):
  File 
"
 line 305, in call
side_input_values)
  File 
"
 line 338, in attempt_call
evaluator.process_element(value)
  File 
"
 line 560, in process_element
self.runner.process(element)
  File "apache_beam/runners/common.py", line 468, in 
apache_beam.runners.common.DoFnRunner.process 
(apache_beam/runners/common.c:13915)
def process(self, windowed_value):
  File "apache_beam/runners/common.py", line 474, in 
apache_beam.runners.common.DoFnRunner.process 
(apache_beam/runners/common.c:13719)
self._reraise_augmented(exn)
  File "apache_beam/runners/common.py", line 515, in 
apache_beam.runners.common.DoFnRunner._reraise_augmented 
(apache_beam/runners/common.c:14934)
raise new_exn, None, original_traceback
  File "apache_beam/runners/common.py", line 472, in 

[jira] [Commented] (BEAM-3632) Table partioning in DynamicDestination is lost with project is not set in Table Destination

[Blaquiere Guillaume (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16354652#comment-16354652
 ] 

Blaquiere Guillaume commented on BEAM-3632:
---

Pull request: https://github.com/apache/beam/pull/4622

> Table partioning in DynamicDestination is lost with project is not set in 
> Table Destination
> ---
>
> Key: BEAM-3632
> URL: https://issues.apache.org/jira/browse/BEAM-3632
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-gcp
>Affects Versions: 2.2.0, 2.3.0
>Reporter: Blaquiere Guillaume
>Assignee: Chamikara Jayalath
>Priority: Major
>   Original Estimate: 10m
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> In the following class 
> https://github.com/apache/beam/tree/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/*WriteTables.java*
> line 140
> When the project is not set in table name, a new TableDestination is created 
> but all the time partitioning parameter is lost.
> Fix is very easy. Add a get time partitioning on the new creation:
>  tableDestination = new TableDestination(tableReference, 
> tableDestination.getTableDescription()*{color:#ff},tableDestination{color}**{color:#ff}.getTimePartitioning(){color}*);
>   
>  Should be enought :P



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-3632) Table partioning in DynamicDestination is lost with project is not set in Table Destination

[Blaquiere Guillaume (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3632?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Blaquiere Guillaume updated BEAM-3632:
--
Description: 
In the following class 
https://github.com/apache/beam/tree/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/*WriteTables.java*

line 140

When the project is not set in table name, a new TableDestination is created 
but all the time partitioning parameter is lost.

Fix is very easy. Add a get time partitioning on the new creation:
 tableDestination = new TableDestination(tableReference, 
tableDestination.getTableDescription()*{color:#ff},tableDestination{color}**{color:#ff}.getTimePartitioning(){color}*);
  
 Should be enought :P

  was:
In the following class 
[beam|https://github.com/apache/beam]/[sdks|https://github.com/apache/beam/tree/master/sdks]/[java|https://github.com/apache/beam/tree/master/sdks/java]/[io|https://github.com/apache/beam/tree/master/sdks/java/io]/[google-cloud-platform|https://github.com/apache/beam/tree/master/sdks/java/io/google-cloud-platform]/[src|https://github.com/apache/beam/tree/master/sdks/java/io/google-cloud-platform/src]/[main|https://github.com/apache/beam/tree/master/sdks/java/io/google-cloud-platform/src/main]/[java|https://github.com/apache/beam/tree/master/sdks/java/io/google-cloud-platform/src/main/java]/[org|https://github.com/apache/beam/tree/master/sdks/java/io/google-cloud-platform/src/main/java/org]/[apache|https://github.com/apache/beam/tree/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache]/[beam|https://github.com/apache/beam/tree/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam]/[sdk|https://github.com/apache/beam/tree/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk]/[io|https://github.com/apache/beam/tree/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io]/[gcp|https://github.com/apache/beam/tree/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp]/[bigquery|https://github.com/apache/beam/tree/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery]/*WriteTables.java*

line 140

When the project is not set in table name, a new TableDestination is created 
but all the time partitioning parameter is lost.

Fix is very easy. Add a get time partitioning on the new creation:
tableDestination = new TableDestination(tableReference, 
tableDestination.getTableDescription()*{color:#FF},tableDestination{color}**{color:#FF}.getTimePartitioning(){color}*);
 
Should be enought :P


> Table partioning in DynamicDestination is lost with project is not set in 
> Table Destination
> ---
>
> Key: BEAM-3632
> URL: https://issues.apache.org/jira/browse/BEAM-3632
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-gcp
>Affects Versions: 2.2.0, 2.3.0
>Reporter: Blaquiere Guillaume
>Assignee: Chamikara Jayalath
>Priority: Major
>   Original Estimate: 10m
>  Remaining Estimate: 10m
>
> In the following class 
> https://github.com/apache/beam/tree/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/*WriteTables.java*
> line 140
> When the project is not set in table name, a new TableDestination is created 
> but all the time partitioning parameter is lost.
> Fix is very easy. Add a get time partitioning on the new creation:
>  tableDestination = new TableDestination(tableReference, 
> tableDestination.getTableDescription()*{color:#ff},tableDestination{color}**{color:#ff}.getTimePartitioning(){color}*);
>   
>  Should be enought :P



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-3632) Table partioning in DynamicDestination is lost with project is not set in Table Destination

[Blaquiere Guillaume (JIRA)]

Blaquiere Guillaume created BEAM-3632:
-

 Summary: Table partioning in DynamicDestination is lost with 
project is not set in Table Destination
 Key: BEAM-3632
 URL: https://issues.apache.org/jira/browse/BEAM-3632
 Project: Beam
  Issue Type: Bug
  Components: sdk-java-gcp
Affects Versions: 2.2.0, 2.3.0
Reporter: Blaquiere Guillaume
Assignee: Chamikara Jayalath


In the following class 
[beam|https://github.com/apache/beam]/[sdks|https://github.com/apache/beam/tree/master/sdks]/[java|https://github.com/apache/beam/tree/master/sdks/java]/[io|https://github.com/apache/beam/tree/master/sdks/java/io]/[google-cloud-platform|https://github.com/apache/beam/tree/master/sdks/java/io/google-cloud-platform]/[src|https://github.com/apache/beam/tree/master/sdks/java/io/google-cloud-platform/src]/[main|https://github.com/apache/beam/tree/master/sdks/java/io/google-cloud-platform/src/main]/[java|https://github.com/apache/beam/tree/master/sdks/java/io/google-cloud-platform/src/main/java]/[org|https://github.com/apache/beam/tree/master/sdks/java/io/google-cloud-platform/src/main/java/org]/[apache|https://github.com/apache/beam/tree/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache]/[beam|https://github.com/apache/beam/tree/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam]/[sdk|https://github.com/apache/beam/tree/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk]/[io|https://github.com/apache/beam/tree/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io]/[gcp|https://github.com/apache/beam/tree/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp]/[bigquery|https://github.com/apache/beam/tree/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery]/*WriteTables.java*

line 140

When the project is not set in table name, a new TableDestination is created 
but all the time partitioning parameter is lost.

Fix is very easy. Add a get time partitioning on the new creation:
tableDestination = new TableDestination(tableReference, 
tableDestination.getTableDescription()*{color:#FF},tableDestination{color}**{color:#FF}.getTimePartitioning(){color}*);
 
Should be enought :P



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] branch master updated (c0b68ca -> 92fd354)

[kenn]

This is an automated email from the ASF dual-hosted git repository.

kenn pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from c0b68ca  [BEAM-3457] Upgrade gogradle and fix thrift resolution issue
 add 8efffe7  Deprecate DoFnTester
 add 1879812  Remove DoFnTester from core SDK tests
 new 92fd354  Merge pull request #4507: [BEAM-3159] Deprecated DoFnTester

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../java/org/apache/beam/sdk/transforms/DoFn.java  |  37 ++--
 .../org/apache/beam/sdk/transforms/DoFnTester.java | 217 -
 .../beam/sdk/transforms/join/CoGroupByKeyTest.java |  42 ++--
 3 files changed, 122 insertions(+), 174 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
k...@apache.org.

[beam] 01/01: Merge pull request #4507: [BEAM-3159] Deprecated DoFnTester

[kenn]

This is an automated email from the ASF dual-hosted git repository.

kenn pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 92fd3545244d362f56f8f339d2630088f0d1af25
Merge: c0b68ca 1879812
Author: Kenn Knowles 
AuthorDate: Tue Feb 6 14:11:33 2018 -0800

Merge pull request #4507: [BEAM-3159] Deprecated DoFnTester

 .../java/org/apache/beam/sdk/transforms/DoFn.java  |  37 ++--
 .../org/apache/beam/sdk/transforms/DoFnTester.java | 217 -
 .../beam/sdk/transforms/join/CoGroupByKeyTest.java |  42 ++--
 3 files changed, 122 insertions(+), 174 deletions(-)


-- 
To stop receiving notification emails like this one, please contact
k...@apache.org.

[beam] 01/01: [BEAM-3457] Upgrade gogradle and fix thrift resolution issue

[lcwik]

This is an automated email from the ASF dual-hosted git repository.

lcwik pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit c0b68caa6b782688c3db33255c0280367f0e4738
Merge: 0842a4f 230e312
Author: Lukasz Cwik 
AuthorDate: Tue Feb 6 13:58:15 2018 -0800

[BEAM-3457] Upgrade gogradle and fix thrift resolution issue

 build.gradle   | 2 +-
 build_rules.gradle | 8 
 2 files changed, 9 insertions(+), 1 deletion(-)

-- 
To stop receiving notification emails like this one, please contact
lc...@apache.org.

[beam] branch master updated (0842a4f -> c0b68ca)

[lcwik]

This is an automated email from the ASF dual-hosted git repository.

lcwik pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 0842a4f  Gradle docker tag
 add 230e312  [BEAM-3457] Upgrade gogradle and fix thrift resolution issue
 new c0b68ca  [BEAM-3457] Upgrade gogradle and fix thrift resolution issue

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 build.gradle   | 2 +-
 build_rules.gradle | 8 
 2 files changed, 9 insertions(+), 1 deletion(-)

-- 
To stop receiving notification emails like this one, please contact
lc...@apache.org.

[beam] 01/01: Gradle docker tag

[lcwik]

This is an automated email from the ASF dual-hosted git repository.

lcwik pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 0842a4f897eb3b42fb29dd3f64a721ec5a905054
Merge: 80868ac e5316e3
Author: Lukasz Cwik 
AuthorDate: Tue Feb 6 13:57:30 2018 -0800

Gradle docker tag

 sdks/java/container/build.gradle   | 9 +++--
 sdks/python/container/build.gradle | 9 +++--
 2 files changed, 14 insertions(+), 4 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
lc...@apache.org.

[beam] branch master updated (80868ac -> 0842a4f)

[lcwik]

This is an automated email from the ASF dual-hosted git repository.

lcwik pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 80868ac  Merge pull request #4617: [BEAM-3272] Sickbay ApexRunner 
ParDoTranslatorTest.testAssertionFailure
 add e5316e3  Allow docker tag root to be specified as in Maven image build
 new 0842a4f  Gradle docker tag

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 sdks/java/container/build.gradle   | 9 +++--
 sdks/python/container/build.gradle | 9 +++--
 2 files changed, 14 insertions(+), 4 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
lc...@apache.org.

[jira] [Created] (BEAM-3631) Have Dataflow map main input windows to side input windows

[Luke Cwik (JIRA)]

Luke Cwik created BEAM-3631:
---

 Summary: Have Dataflow map main input windows to side input windows
 Key: BEAM-3631
 URL: https://issues.apache.org/jira/browse/BEAM-3631
 Project: Beam
  Issue Type: Sub-task
  Components: runner-dataflow
Reporter: Luke Cwik
Assignee: Luke Cwik






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-3630) Have Dataflow use the windowing strategy within the PCollectionToView/CollectionToSingleton expansion

[Luke Cwik (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Cwik updated BEAM-3630:

Issue Type: Sub-task  (was: Improvement)
Parent: BEAM-2863

> Have Dataflow use the windowing strategy within the 
> PCollectionToView/CollectionToSingleton expansion
> -
>
> Key: BEAM-3630
> URL: https://issues.apache.org/jira/browse/BEAM-3630
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-dataflow
>Reporter: Luke Cwik
>Assignee: Luke Cwik
>Priority: Major
>  Labels: portability
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-3630) Have Dataflow use the windowing strategy within the PCollectionToView/CollectionToSingleton expansion

[Luke Cwik (JIRA)]

Luke Cwik created BEAM-3630:
---

 Summary: Have Dataflow use the windowing strategy within the 
PCollectionToView/CollectionToSingleton expansion
 Key: BEAM-3630
 URL: https://issues.apache.org/jira/browse/BEAM-3630
 Project: Beam
  Issue Type: Improvement
  Components: runner-dataflow
Reporter: Luke Cwik
Assignee: Luke Cwik






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-3629) Make the windowing strategy available within the PCollectionView/CollectionToSingleton that is sent to Dataflow

[Luke Cwik (JIRA)]

Luke Cwik created BEAM-3629:
---

 Summary: Make the windowing strategy available within the 
PCollectionView/CollectionToSingleton that is sent to Dataflow
 Key: BEAM-3629
 URL: https://issues.apache.org/jira/browse/BEAM-3629
 Project: Beam
  Issue Type: Sub-task
  Components: runner-dataflow
Reporter: Luke Cwik
Assignee: Luke Cwik






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Dataflow #4897

[Apache Jenkins Server]

See 

[beam] 01/01: Merge pull request #4617: [BEAM-3272] Sickbay ApexRunner ParDoTranslatorTest.testAssertionFailure

[kenn]

This is an automated email from the ASF dual-hosted git repository.

kenn pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 80868acefba6583e3d9d6a6e059e2944a0452a2c
Merge: 8f7e59c 8c68cf8
Author: Kenn Knowles 
AuthorDate: Tue Feb 6 12:56:25 2018 -0800

Merge pull request #4617: [BEAM-3272] Sickbay ApexRunner 
ParDoTranslatorTest.testAssertionFailure

 .../org/apache/beam/runners/apex/translation/ParDoTranslatorTest.java   | 2 ++
 1 file changed, 2 insertions(+)

-- 
To stop receiving notification emails like this one, please contact
k...@apache.org.

[jira] [Created] (BEAM-3628) Python postcommit broken

[Kenneth Knowles (JIRA)]

Kenneth Knowles created BEAM-3628:
-

 Summary: Python postcommit broken
 Key: BEAM-3628
 URL: https://issues.apache.org/jira/browse/BEAM-3628
 Project: Beam
  Issue Type: Bug
  Components: sdk-py-core
Reporter: Kenneth Knowles
Assignee: Charles Chen


https://builds.apache.org/view/A-D/view/Beam/job/beam_PostCommit_Python_Verify/4152/



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-3627) Dataflow ValidatesRunner failing ViewTest

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-3627:
--
Fix Version/s: 2.4.0

> Dataflow ValidatesRunner failing ViewTest
> -
>
> Key: BEAM-3627
> URL: https://issues.apache.org/jira/browse/BEAM-3627
> Project: Beam
>  Issue Type: Bug
>  Components: runner-dataflow
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>Priority: Blocker
> Fix For: 2.4.0
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-3617) Performance degradation on the direct runner

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-3617:
--
Fix Version/s: 2.3.0

> Performance degradation on the direct runner
> 
>
> Key: BEAM-3617
> URL: https://issues.apache.org/jira/browse/BEAM-3617
> Project: Beam
>  Issue Type: Bug
>  Components: runner-direct
>Reporter: Jean-Baptiste Onofré
>Assignee: Kenneth Knowles
>Priority: Blocker
> Fix For: 2.3.0, 2.4.0
>
>  Time Spent: 1.5h
>  Remaining Estimate: 0h
>
> Running Nexmark queries with the direct runner between Beam 2.2.0 and 2.3.0 
> shows a performance degradation:
> {code}
> 
>  Beam 2.2.0   Beam 2.3.0
>   Query  Runtime(sec) Runtime(sec)
> 
>      6.410.6
>   0001   5.110.2
>   0002   3.0 5.8
>   0003   3.8 6.2
>   0004   0.9 1.4
>   0005   5.811.4
>   0006   0.8 1.4
>   0007 193.8  1249.1
>   0008   3.9 6.9
>   0009   0.9 1.3
>   0010   6.4 8.2
>   0011   5.0 9.4
>   0012   4.7 9.1
> {code}
> We can see especially Query 7 that is 10 times longer.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-3617) Performance degradation on the direct runner

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-3617:
--
Affects Version/s: (was: 2.3.0)

> Performance degradation on the direct runner
> 
>
> Key: BEAM-3617
> URL: https://issues.apache.org/jira/browse/BEAM-3617
> Project: Beam
>  Issue Type: Bug
>  Components: runner-direct
>Reporter: Jean-Baptiste Onofré
>Assignee: Kenneth Knowles
>Priority: Blocker
> Fix For: 2.3.0, 2.4.0
>
>  Time Spent: 1.5h
>  Remaining Estimate: 0h
>
> Running Nexmark queries with the direct runner between Beam 2.2.0 and 2.3.0 
> shows a performance degradation:
> {code}
> 
>  Beam 2.2.0   Beam 2.3.0
>   Query  Runtime(sec) Runtime(sec)
> 
>      6.410.6
>   0001   5.110.2
>   0002   3.0 5.8
>   0003   3.8 6.2
>   0004   0.9 1.4
>   0005   5.811.4
>   0006   0.8 1.4
>   0007 193.8  1249.1
>   0008   3.9 6.9
>   0009   0.9 1.3
>   0010   6.4 8.2
>   0011   5.0 9.4
>   0012   4.7 9.1
> {code}
> We can see especially Query 7 that is 10 times longer.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-3617) Performance degradation on the direct runner

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-3617:
--
Fix Version/s: (was: 2.3.0)
   2.4.0

> Performance degradation on the direct runner
> 
>
> Key: BEAM-3617
> URL: https://issues.apache.org/jira/browse/BEAM-3617
> Project: Beam
>  Issue Type: Bug
>  Components: runner-direct
>Reporter: Jean-Baptiste Onofré
>Assignee: Kenneth Knowles
>Priority: Blocker
> Fix For: 2.3.0, 2.4.0
>
>  Time Spent: 1.5h
>  Remaining Estimate: 0h
>
> Running Nexmark queries with the direct runner between Beam 2.2.0 and 2.3.0 
> shows a performance degradation:
> {code}
> 
>  Beam 2.2.0   Beam 2.3.0
>   Query  Runtime(sec) Runtime(sec)
> 
>      6.410.6
>   0001   5.110.2
>   0002   3.0 5.8
>   0003   3.8 6.2
>   0004   0.9 1.4
>   0005   5.811.4
>   0006   0.8 1.4
>   0007 193.8  1249.1
>   0008   3.9 6.9
>   0009   0.9 1.3
>   0010   6.4 8.2
>   0011   5.0 9.4
>   0012   4.7 9.1
> {code}
> We can see especially Query 7 that is 10 times longer.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Python_Verify #4159

[Apache Jenkins Server]

See 


Changes:

[rangadi] KafkaIO : move source and sink implemenations into own files.

[rangadi] minor

[ekirpichov] Adds PositionT and claim callback to RestrictionTracker

[ekirpichov] Changes OutputAndTimeBounded invoker to start checkpoint timer 
after

[ekirpichov] Compresses encoded GrowthState with Snappy - about 2x-3x more 
compact

[ekirpichov] InMemoryStateInternals.copy clones the values using the coder

[ekirpichov] Final fixups

[ekirpichov] Bump worker to 20180205

--
[...truncated 1.53 MB...]
  File 
"
 line 1453, in process
k, vs = element
TypeError: 'KeyedWorkItem' object is not iterable [while running 
'assert_that/Group/GroupByKey/GroupByWindow']

root: ERROR: Giving up after 4 attempts.
- >> end captured logging << -

==
ERROR: test_gbk_execution (apache_beam.testing.test_stream_test.TestStreamTest)
--
Traceback (most recent call last):
  File 
"
 line 148, in test_gbk_execution
p.run()
  File 
"
 line 104, in run
state = result.wait_until_finish()
  File 
"
 line 318, in wait_until_finish
self._executor.await_completion()
  File 
"
 line 358, in await_completion
self._executor.await_completion()
  File 
"
 line 305, in call
side_input_values)
  File 
"
 line 338, in attempt_call
evaluator.process_element(value)
  File 
"
 line 560, in process_element
self.runner.process(element)
  File "apache_beam/runners/common.py", line 468, in 
apache_beam.runners.common.DoFnRunner.process 
(apache_beam/runners/common.c:13915)
def process(self, windowed_value):
  File "apache_beam/runners/common.py", line 474, in 
apache_beam.runners.common.DoFnRunner.process 
(apache_beam/runners/common.c:13719)
self._reraise_augmented(exn)
  File "apache_beam/runners/common.py", line 515, in 
apache_beam.runners.common.DoFnRunner._reraise_augmented 
(apache_beam/runners/common.c:14934)
raise new_exn, None, original_traceback
  File "apache_beam/runners/common.py", line 472, in 
apache_beam.runners.common.DoFnRunner.process 
(apache_beam/runners/common.c:13632)
self.do_fn_invoker.invoke_process(windowed_value)
  File "apache_beam/runners/common.py", line 270, in 
apache_beam.runners.common.SimpleInvoker.invoke_process 
(apache_beam/runners/common.c:9084)
windowed_value, self.process_method(windowed_value.value))
  File 
"
 line 1453, in process
k, vs = element
TypeError: 'KeyedWorkItem' object is not iterable [while running 
'assert_that/Group/GroupByKey/GroupByWindow']
 >> begin captured logging << 
root: INFO: Missing pipeline option (runner). Executing pipeline using the 
default runner: DirectRunner.
root: INFO: Running pipeline with DirectRunner.
root: ERROR: Exception at bundle 
, 
due to an exception.
 Traceback (most recent call last):
  File 
"
 line 305, in call
side_input_values)
  File 
"
 line 338, in attempt_call
evaluator.process_element(value)
  File 
"
 line 560, in process_element
self.runner.process(element)
  File "apache_beam/runners/common.py", line 468, in 
apache_beam.runners.common.DoFnRunner.process 
(apache_beam/runners/common.c:13915)
def process(self, windowed_value):
  File "apache_beam/runners/common.py", line 474, in 

[jira] [Commented] (BEAM-3627) Dataflow ValidatesRunner failing ViewTest

[Kenneth Knowles (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16354473#comment-16354473
 ] 

Kenneth Knowles commented on BEAM-3627:
---

Yea this is caused by a bad rebase that reverted some of the PR.

> Dataflow ValidatesRunner failing ViewTest
> -
>
> Key: BEAM-3627
> URL: https://issues.apache.org/jira/browse/BEAM-3627
> Project: Beam
>  Issue Type: Bug
>  Components: runner-dataflow
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>Priority: Blocker
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (BEAM-3499) Watch can make no progress if a single poll takes more than checkpoint interval

[Eugene Kirpichov (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Kirpichov closed BEAM-3499.
--
   Resolution: Fixed
Fix Version/s: 2.3.0

> Watch can make no progress if a single poll takes more than checkpoint 
> interval
> ---
>
> Key: BEAM-3499
> URL: https://issues.apache.org/jira/browse/BEAM-3499
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Eugene Kirpichov
>Assignee: Eugene Kirpichov
>Priority: Major
> Fix For: 2.3.0
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> E.g. when using it to poll a filepattern with hundreds of thousands of files, 
> a single poll may take >10 seconds (default checkpoint interval in 
> OutputAndTimeBoundedSplittableProcessElementInvoker). Because of that, the 
> tracker (GrowthTracker) gets checkpointed before anything is added to it, 
> i.e. before 
> [https://github.com/apache/beam/blob/0d918b7cab8c4ccb2b5e050501327912161d40a7/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/Watch.java#L727,]
>  at a moment when it doesn't contain any useful information, so the residual 
> checkpoint state is as empty as the initial one. When we resume from the 
> residual checkpoint, the situation simply repeats - until we get lucky enough 
> to either take <10s to poll, or to not be asked to checkpoint for >10s (e.g. 
> cause the checkpointing thread isn't scheduled).
> One possible fix to this is to change the SDF checkpointing strategy to have 
> a progress guarantee: e.g., start counting time from the moment the first 
> block is claimed, or allow the tracker to refuse checkpointing if nothing is 
> claimed yet, or something like that.
>  
> A workaround for users of this (primarily via FileIO.match().continuously()) 
> is to shard their filepattern into a set of finer-granularity filepatterns 
> matching fewer files, so that each match call takes less than 10 seconds.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-3627) Dataflow ValidatesRunner failing ViewTest

[Kenneth Knowles (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16354438#comment-16354438
 ] 

Kenneth Knowles commented on BEAM-3627:
---

This looks definitely related to the valueInEmptyWindows change. We had 
ValidatesRunner green on the PR but Dataflow likely changed underneath it.

> Dataflow ValidatesRunner failing ViewTest
> -
>
> Key: BEAM-3627
> URL: https://issues.apache.org/jira/browse/BEAM-3627
> Project: Beam
>  Issue Type: Bug
>  Components: runner-dataflow
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>Priority: Blocker
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-3627) Dataflow ValidatesRunner failing ViewTest

[Kenneth Knowles (JIRA)]

Kenneth Knowles created BEAM-3627:
-

 Summary: Dataflow ValidatesRunner failing ViewTest
 Key: BEAM-3627
 URL: https://issues.apache.org/jira/browse/BEAM-3627
 Project: Beam
  Issue Type: Bug
  Components: runner-dataflow
Reporter: Kenneth Knowles
Assignee: Kenneth Knowles






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Dataflow #4896

[Apache Jenkins Server]

See 

[beam] branch master updated (2826362 -> 8f7e59c)

[jkff]

This is an automated email from the ASF dual-hosted git repository.

jkff pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 2826362  Merge pull request #4483: [BEAM-3499, BEAM-2607] Gives the 
runner access to positions of SDF claimed blocks
 add 181323d  KafkaIO : move source and sink implemenations into own files.
 add 2da05a7  minor
 new 8f7e59c  Merge pull request #4586: [BEAM-3611] Split KafkaIO.java in 
to smaller files

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../beam/sdk/io/kafka/KafkaCheckpointMark.java |6 +-
 .../beam/sdk/io/kafka/KafkaExactlyOnceSink.java|  643 +
 .../java/org/apache/beam/sdk/io/kafka/KafkaIO.java | 1431 +---
 .../apache/beam/sdk/io/kafka/KafkaRecordCoder.java |   40 +-
 .../beam/sdk/io/kafka/KafkaUnboundedReader.java|  663 +
 .../beam/sdk/io/kafka/KafkaUnboundedSource.java|  165 +++
 .../org/apache/beam/sdk/io/kafka/KafkaWriter.java  |  132 ++
 .../org/apache/beam/sdk/io/kafka/ProducerSpEL.java |   12 +-
 .../org/apache/beam/sdk/io/kafka/KafkaIOTest.java  |   11 +-
 9 files changed, 1639 insertions(+), 1464 deletions(-)
 create mode 100644 
sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaExactlyOnceSink.java
 create mode 100644 
sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaUnboundedReader.java
 create mode 100644 
sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaUnboundedSource.java
 create mode 100644 
sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaWriter.java

-- 
To stop receiving notification emails like this one, please contact
j...@apache.org.

[beam] 01/01: Merge pull request #4586: [BEAM-3611] Split KafkaIO.java in to smaller files

[jkff]

This is an automated email from the ASF dual-hosted git repository.

jkff pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 8f7e59c3f24dbfb85b92eb314bcdc622ad1f9ba5
Merge: 2826362 2da05a7
Author: Eugene Kirpichov 
AuthorDate: Tue Feb 6 11:01:52 2018 -0800

Merge pull request #4586: [BEAM-3611] Split KafkaIO.java in to smaller files

[BEAM-3611] Split KafkaIO.java in to smaller files

 .../beam/sdk/io/kafka/KafkaCheckpointMark.java |6 +-
 .../beam/sdk/io/kafka/KafkaExactlyOnceSink.java|  643 +
 .../java/org/apache/beam/sdk/io/kafka/KafkaIO.java | 1431 +---
 .../apache/beam/sdk/io/kafka/KafkaRecordCoder.java |   40 +-
 .../beam/sdk/io/kafka/KafkaUnboundedReader.java|  663 +
 .../beam/sdk/io/kafka/KafkaUnboundedSource.java|  165 +++
 .../org/apache/beam/sdk/io/kafka/KafkaWriter.java  |  132 ++
 .../org/apache/beam/sdk/io/kafka/ProducerSpEL.java |   12 +-
 .../org/apache/beam/sdk/io/kafka/KafkaIOTest.java  |   11 +-
 9 files changed, 1639 insertions(+), 1464 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
j...@apache.org.

Build failed in Jenkins: beam_PerformanceTests_Python #882

[Apache Jenkins Server]

See 


Changes:

[Pablo] Renaming the ZERO element of DistributionResult to be IDENTITY_ELEMENT.

[ekirpichov] Adds PositionT and claim callback to RestrictionTracker

[ekirpichov] Changes OutputAndTimeBounded invoker to start checkpoint timer 
after

[ekirpichov] Compresses encoded GrowthState with Snappy - about 2x-3x more 
compact

[ekirpichov] InMemoryStateInternals.copy clones the values using the coder

[ekirpichov] Final fixups

[ekirpichov] Bump worker to 20180205

[klk] Sickbay flaky KinesisIO tests

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam4 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 282636275f3f91cc8a15aa08d8039bcb81d2f9d6 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 282636275f3f91cc8a15aa08d8039bcb81d2f9d6
Commit message: "Merge pull request #4483: [BEAM-3499, BEAM-2607] Gives the 
runner access to positions of SDF claimed blocks"
 > git rev-list f3dba1af43b03e24419fb0f5b726479368959eef # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins6897200577422977505.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins2169439159314244710.sh
+ rm -rf .env
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins5999116873627816030.sh
+ virtualenv .env --system-site-packages
New python executable in .env/bin/python
Installing setuptools, pip...done.
[beam_PerformanceTests_Python] $ /bin/bash -xe /tmp/jenkins853925917496898066.sh
+ .env/bin/pip install --upgrade setuptools pip
Downloading/unpacking setuptools from 
https://pypi.python.org/packages/43/41/033a273f9a25cb63050a390ee8397acbc7eae2159195d85f06f17e7be45a/setuptools-38.5.1-py2.py3-none-any.whl#md5=908b8b5e50bf429e520b2b5fa1b350e5
Downloading/unpacking pip from 
https://pypi.python.org/packages/b6/ac/7015eb97dc749283ffdec1c3a88ddb8ae03b8fad0f0e611408f196358da3/pip-9.0.1-py2.py3-none-any.whl#md5=297dbd16ef53bcef0447d245815f5144
Installing collected packages: setuptools, pip
  Found existing installation: setuptools 2.2
Uninstalling setuptools:
  Successfully uninstalled setuptools
  Found existing installation: pip 1.5.4
Uninstalling pip:
  Successfully uninstalled pip
Successfully installed setuptools pip
Cleaning up...
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins6830635159509522126.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins8264345029084439122.sh
+ .env/bin/pip install -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied: absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: setuptools in ./.env/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 16))
Requirement already satisfied: colorlog[windows]==2.6.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied: futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied: PyYAML==3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied: pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Collecting numpy==1.13.3 (from -r PerfKitBenchmarker/requirements.txt (line 

Build failed in Jenkins: beam_PostCommit_Python_Verify #4158

[Apache Jenkins Server]

See 


Changes:

[Pablo] Renaming the ZERO element of DistributionResult to be IDENTITY_ELEMENT.

[klk] Sickbay flaky KinesisIO tests

--
[...truncated 1.51 MB...]
  File 
"
 line 1453, in process
k, vs = element
TypeError: 'KeyedWorkItem' object is not iterable [while running 
'assert_that/Group/GroupByKey/GroupByWindow']

root: ERROR: Giving up after 4 attempts.
root: WARNING: A task failed with exception: 'KeyedWorkItem' object is not 
iterable [while running 'assert_that/Group/GroupByKey/GroupByWindow']
- >> end captured logging << -

==
ERROR: test_gbk_execution (apache_beam.testing.test_stream_test.TestStreamTest)
--
Traceback (most recent call last):
  File 
"
 line 148, in test_gbk_execution
p.run()
  File 
"
 line 104, in run
state = result.wait_until_finish()
  File 
"
 line 318, in wait_until_finish
self._executor.await_completion()
  File 
"
 line 358, in await_completion
self._executor.await_completion()
  File 
"
 line 305, in call
side_input_values)
  File 
"
 line 338, in attempt_call
evaluator.process_element(value)
  File 
"
 line 560, in process_element
self.runner.process(element)
  File "apache_beam/runners/common.py", line 468, in 
apache_beam.runners.common.DoFnRunner.process 
(apache_beam/runners/common.c:13915)
def process(self, windowed_value):
  File "apache_beam/runners/common.py", line 474, in 
apache_beam.runners.common.DoFnRunner.process 
(apache_beam/runners/common.c:13719)
self._reraise_augmented(exn)
  File "apache_beam/runners/common.py", line 515, in 
apache_beam.runners.common.DoFnRunner._reraise_augmented 
(apache_beam/runners/common.c:14934)
raise new_exn, None, original_traceback
  File "apache_beam/runners/common.py", line 472, in 
apache_beam.runners.common.DoFnRunner.process 
(apache_beam/runners/common.c:13632)
self.do_fn_invoker.invoke_process(windowed_value)
  File "apache_beam/runners/common.py", line 270, in 
apache_beam.runners.common.SimpleInvoker.invoke_process 
(apache_beam/runners/common.c:9084)
windowed_value, self.process_method(windowed_value.value))
  File 
"
 line 1453, in process
k, vs = element
TypeError: 'KeyedWorkItem' object is not iterable [while running 
'assert_that/Group/GroupByKey/GroupByWindow']
 >> begin captured logging << 
root: INFO: Missing pipeline option (runner). Executing pipeline using the 
default runner: DirectRunner.
root: INFO: Running pipeline with DirectRunner.
root: ERROR: Exception at bundle 
, 
due to an exception.
 Traceback (most recent call last):
  File 
"
 line 305, in call
side_input_values)
  File 
"
 line 338, in attempt_call
evaluator.process_element(value)
  File 
"
 line 560, in process_element
self.runner.process(element)
  File "apache_beam/runners/common.py", line 468, in 
apache_beam.runners.common.DoFnRunner.process 
(apache_beam/runners/common.c:13915)
def process(self, windowed_value):
  File "apache_beam/runners/common.py", line 474, in 
apache_beam.runners.common.DoFnRunner.process 
(apache_beam/runners/common.c:13719)
self._reraise_augmented(exn)
  File "apache_beam/runners/common.py", line 515, in 

[jira] [Commented] (BEAM-3272) ParDoTranslatorTest: Error creating local cluster while creating checkpoint file

[Thomas Weise (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16354292#comment-16354292
 ] 

Thomas Weise commented on BEAM-3272:


If parallelism is enabled in gradle, that could be the issue. Currently the 
tests don't create unique directories and cannot be run in parallel. Also, 
there is an assumption that 'target' is the build directory, which may also 
cause issues when running the tests from gradle.

> ParDoTranslatorTest: Error creating local cluster while creating checkpoint 
> file
> 
>
> Key: BEAM-3272
> URL: https://issues.apache.org/jira/browse/BEAM-3272
> Project: Beam
>  Issue Type: Bug
>  Components: runner-apex
>Reporter: Eugene Kirpichov
>Assignee: Kenneth Knowles
>Priority: Critical
>  Labels: flake
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> Failed build: 
> https://builds.apache.org/job/beam_PostCommit_Java_MavenInstall/org.apache.beam$beam-runners-apex/5330/console
> Key output:
> {code}
> 2017-11-29T01:21:26.956 [ERROR] 
> testAssertionFailure(org.apache.beam.runners.apex.translation.ParDoTranslatorTest)
>   Time elapsed: 2.007 s  <<< ERROR!
> java.lang.RuntimeException: Error creating local cluster
>   at 
> org.apache.apex.engine.EmbeddedAppLauncherImpl.getController(EmbeddedAppLauncherImpl.java:122)
>   at 
> org.apache.apex.engine.EmbeddedAppLauncherImpl.launchApp(EmbeddedAppLauncherImpl.java:71)
>   at 
> org.apache.apex.engine.EmbeddedAppLauncherImpl.launchApp(EmbeddedAppLauncherImpl.java:46)
>   at org.apache.beam.runners.apex.ApexRunner.run(ApexRunner.java:197)
>   at 
> org.apache.beam.runners.apex.TestApexRunner.run(TestApexRunner.java:57)
>   at 
> org.apache.beam.runners.apex.TestApexRunner.run(TestApexRunner.java:31)
>   at org.apache.beam.sdk.Pipeline.run(Pipeline.java:304)
>   at org.apache.beam.sdk.Pipeline.run(Pipeline.java:290)
>   at 
> org.apache.beam.runners.apex.translation.ParDoTranslatorTest.runExpectingAssertionFailure(ParDoTranslatorTest.java:156)
> {code}
> ...
> {code}
> Caused by: ExitCodeException exitCode=1: chmod: cannot access 
> ‘/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Java_MavenInstall/src/runners/apex/target/com.datatorrent.stram.StramLocalCluster/checkpoints/2/_tmp’:
>  No such file or directory
>   at org.apache.hadoop.util.Shell.runCommand(Shell.java:582)
>   at org.apache.hadoop.util.Shell.run(Shell.java:479)
>   at 
> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:773)
>   at org.apache.hadoop.util.Shell.execCommand(Shell.java:866)
>   at org.apache.hadoop.util.Shell.execCommand(Shell.java:849)
>   at 
> org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:733)
>   at 
> org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.(RawLocalFileSystem.java:225)
>   at 
> org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.(RawLocalFileSystem.java:209)
>   at 
> org.apache.hadoop.fs.RawLocalFileSystem.createOutputStreamWithMode(RawLocalFileSystem.java:307)
>   at 
> org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:296)
>   at 
> org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:328)
>   at org.apache.hadoop.fs.FileSystem.primitiveCreate(FileSystem.java:1017)
>   at 
> org.apache.hadoop.fs.DelegateToFileSystem.createInternal(DelegateToFileSystem.java:99)
>   at 
> org.apache.hadoop.fs.ChecksumFs$ChecksumFSOutputSummer.(ChecksumFs.java:352)
>   at org.apache.hadoop.fs.ChecksumFs.createInternal(ChecksumFs.java:399)
>   at 
> org.apache.hadoop.fs.AbstractFileSystem.create(AbstractFileSystem.java:584)
>   at org.apache.hadoop.fs.FileContext$3.next(FileContext.java:686)
>   at org.apache.hadoop.fs.FileContext$3.next(FileContext.java:682)
>   at org.apache.hadoop.fs.FSLinkResolver.resolve(FSLinkResolver.java:90)
>   at org.apache.hadoop.fs.FileContext.create(FileContext.java:688)
>   at 
> com.datatorrent.common.util.AsyncFSStorageAgent.copyToHDFS(AsyncFSStorageAgent.java:119)
>   ... 50 more
> {code}
> By inspecting code at the stack frames, seems it's trying to copy an 
> operator's checkpoint "to HDFS" (which in this case is the local disk), but 
> fails while creating the target file of the copy - creation creates the file 
> (successfully) and chmods it writable (unsuccessfully). Barring something 
> subtle (e.g. chmod being not allowed to call immediately after creating a 
> FileOutputStream), this looks like the whole directory was possibly deleted 
> from under the process. I don't know why this would be the case though, or 
> how to debug it.
> Either way, the path being accessed is funky: 
> 

[jira] [Created] (BEAM-3626) Support remapping the main input window to side input window inside the Java SDK harness

[Luke Cwik (JIRA)]

Luke Cwik created BEAM-3626:
---

 Summary: Support remapping the main input window to side input 
window inside the Java SDK harness
 Key: BEAM-3626
 URL: https://issues.apache.org/jira/browse/BEAM-3626
 Project: Beam
  Issue Type: Sub-task
  Components: sdk-java-harness
Reporter: Luke Cwik
Assignee: Luke Cwik






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (BEAM-3605) Kinesis ShardReadersPoolTest shouldForgetClosedShardIterator failure

[Alexey Romanenko (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16354264#comment-16354264
 ] 

Alexey Romanenko edited comment on BEAM-3605 at 2/6/18 6:08 PM:


The first observation. This particular test call ShardReadersPool, which 
handles the Kinesis shard readers, and, in its order, it spawns two threads 
(through ExecutorService) for two shards (one thread per shard). Then it checks 
that one shard was closed. So, here, seems, we have a race condition between 
shard thread  and main thread and  200 ms of Thread.sleep() is not enough 
sometimes to wait until all spawn threads will be finished.

Since _sleep()_ never won't be enough, I think that before calling _assert()_ 
we need to make sure that all spawn threads have finished their job. Perhaps, 
one of the way to do this is to call _shardReadersPool.stop()_ before 
_assert()_ because it shutdowns all spawn threads.

Any thoughts?


was (Author: aromanenko):
The first observation. This particular test call ShardReadersPool, which 
handles the Kinesis shard readers, and, in its order, it spawns two threads 
(through ExecutorService) for two shards (one thread per shard). Then it checks 
that one shard was closed. So, here, seems, we have a race condition between 
shard thread  and main thread and  200 ms of Thread.sleep() is not enough 
sometimes to wait until all spawn threads will be finished.

Since _sleep()_ never won't be enough, I think that before calling _assert()_ 
we need to make sure that all spawn threads finished their job. Perhaps, one of 
the way is to call _shardReadersPool.stop()_ before _assert()_ because it 
shutdowns all spawn threads.

Any thoughts?

> Kinesis ShardReadersPoolTest shouldForgetClosedShardIterator failure
> 
>
> Key: BEAM-3605
> URL: https://issues.apache.org/jira/browse/BEAM-3605
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Reporter: Kenneth Knowles
>Assignee: Alexey Romanenko
>Priority: Critical
>  Labels: flake, sickbay
>
> Here's one:
> https://builds.apache.org/job/beam_PreCommit_Java_GradleBuild/1758/testReport/junit/org.apache.beam.sdk.io.kinesis/ShardReadersPoolTest/shouldForgetClosedShardIterator/
> Filing all test failures as "Critical" so we can sickbay or fix.
> The Jenkins build will get GC'd so here is the error:
> {code}
> java.lang.AssertionError: 
> Expecting:
>   <["shard1", "shard2"]>
> to contain only:
>   <["shard2"]>
> but the following elements were unexpected:
>   <["shard1"]>
>   at 
> org.apache.beam.sdk.io.kinesis.ShardReadersPoolTest.shouldForgetClosedShardIterator(ShardReadersPoolTest.java:270)
> {code}
> and stderr
> {code}
> Feb 01, 2018 11:24:16 PM org.apache.beam.sdk.io.kinesis.ShardReadersPool 
> readLoop
> INFO: Shard iterator for shard1 shard is closed, finishing the read loop
> org.apache.beam.sdk.io.kinesis.KinesisShardClosedException
> Feb 01, 2018 11:24:16 PM org.apache.beam.sdk.io.kinesis.ShardReadersPool 
> readLoop
> INFO: Kinesis Shard read loop has finished
> Feb 01, 2018 11:24:16 PM org.apache.beam.sdk.io.kinesis.ShardReadersPool 
> readLoop
> INFO: Shard iterator for shard1 shard is closed, finishing the read loop
> org.apache.beam.sdk.io.kinesis.KinesisShardClosedException
> Feb 01, 2018 11:24:16 PM org.apache.beam.sdk.io.kinesis.ShardReadersPool 
> readLoop
> INFO: Kinesis Shard read loop has finished
> Feb 01, 2018 11:24:19 PM org.apache.beam.sdk.io.kinesis.ShardReadersPool 
> readLoop
> INFO: Shard iterator for shard1 shard is closed, finishing the read loop
> org.apache.beam.sdk.io.kinesis.KinesisShardClosedException
> Feb 01, 2018 11:24:19 PM org.apache.beam.sdk.io.kinesis.ShardReadersPool 
> readLoop
> INFO: Kinesis Shard read loop has finished
> Feb 01, 2018 11:24:23 PM org.apache.beam.sdk.io.kinesis.ShardReadersPool 
> readLoop
> INFO: Shard iterator for shard1 shard is closed, finishing the read loop
> org.apache.beam.sdk.io.kinesis.KinesisShardClosedException: Shard iterator 
> reached end of the shard: streamName=null, shardId=shard1
>   at 
> org.apache.beam.sdk.io.kinesis.ShardRecordsIterator.readNextBatch(ShardRecordsIterator.java:70)
>   at 
> org.apache.beam.sdk.io.kinesis.ShardReadersPool.readLoop(ShardReadersPool.java:121)
>   at 
> org.apache.beam.sdk.io.kinesis.ShardReadersPool.lambda$startReadingShards$0(ShardReadersPool.java:112)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>   at 

[beam] 01/01: Merge pull request #4483: [BEAM-3499, BEAM-2607] Gives the runner access to positions of SDF claimed blocks

[jkff]

This is an automated email from the ASF dual-hosted git repository.

jkff pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 282636275f3f91cc8a15aa08d8039bcb81d2f9d6
Merge: a67262f 6857cb9
Author: Eugene Kirpichov 
AuthorDate: Tue Feb 6 10:07:11 2018 -0800

Merge pull request #4483: [BEAM-3499, BEAM-2607] Gives the runner access to 
positions of SDF claimed blocks

[BEAM-3499, BEAM-2607] Gives the runner access to positions of SDF claimed 
blocks

 .../runners/apex/translation/ParDoTranslator.java  |   2 +-
 .../translation/operators/ApexParDoOperator.java   |   2 +-
 .../core/construction/PTransformMatchersTest.java  |   2 +-
 .../core/construction/ParDoTranslationTest.java|   4 +-
 .../core/construction/SplittableParDoTest.java |   7 +-
 .../beam/runners/core/InMemoryStateInternals.java  |  77 --
 ...TimeBoundedSplittableProcessElementInvoker.java | 100 +---
 .../apache/beam/runners/core/SimpleDoFnRunner.java |   8 +-
 .../core/SplittableParDoViaKeyedWorkItems.java |   4 +-
 .../core/SplittableProcessElementInvoker.java  |   2 +-
 ...BoundedSplittableProcessElementInvokerTest.java | 111 +++---
 .../runners/core/SplittableParDoProcessFnTest.java |  34 +++--
 .../direct/CopyOnAccessInMemoryStateInternals.java |  10 +-
 .../SplittableProcessElementsEvaluatorFactory.java |   6 +-
 .../flink/FlinkStreamingTransformTranslators.java  |   2 +-
 .../wrappers/streaming/SplittableDoFnOperator.java |   2 +-
 runners/google-cloud-dataflow-java/pom.xml |   2 +-
 .../org/apache/beam/sdk/coders/SnappyCoder.java|  65 
 .../org/apache/beam/sdk/transforms/DoFnTester.java |   2 +-
 .../java/org/apache/beam/sdk/transforms/Watch.java | 167 ++---
 .../beam/sdk/transforms/reflect/DoFnInvoker.java   |   6 +-
 .../sdk/transforms/reflect/DoFnSignatures.java |   6 +-
 .../splittabledofn/HasDefaultTracker.java  |   2 +-
 .../splittabledofn/OffsetRangeTracker.java |  12 +-
 .../splittabledofn/RestrictionTracker.java |  76 +-
 .../beam/sdk/transforms/SplittableDoFnTest.java|   6 +-
 .../org/apache/beam/sdk/transforms/WatchTest.java  | 126 ++--
 .../sdk/transforms/reflect/DoFnInvokersTest.java   |  14 +-
 .../reflect/DoFnSignaturesProcessElementTest.java  |   2 +-
 .../reflect/DoFnSignaturesSplittableDoFnTest.java  |  11 +-
 .../splittabledofn/OffsetRangeTrackerTest.java |  73 +
 .../apache/beam/fn/harness/FnApiDoFnRunner.java|   6 +-
 32 files changed, 654 insertions(+), 295 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
j...@apache.org.

[jira] [Commented] (BEAM-3605) Kinesis ShardReadersPoolTest shouldForgetClosedShardIterator failure

[Alexey Romanenko (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16354264#comment-16354264
 ] 

Alexey Romanenko commented on BEAM-3605:


The first observation. This particular test call ShardReadersPool, which 
handles the Kinesis shard readers, and, in its order, it spawns two threads 
(through ExecutorService) for two shards (one thread per shard). Then it checks 
that one shard was closed. So, here, seems, we have a race condition between 
shard thread  and main thread and  200 ms of Thread.sleep() is not enough 
sometimes to wait until all spawn threads will be finished.

Since _sleep()_ never won't be enough, I think that before calling _assert()_ 
we need to make sure that all spawn threads finished their job. Perhaps, one of 
the way is to call _shardReadersPool.stop()_ before _assert()_ because it 
shutdowns all spawn threads.

Any thoughts?

> Kinesis ShardReadersPoolTest shouldForgetClosedShardIterator failure
> 
>
> Key: BEAM-3605
> URL: https://issues.apache.org/jira/browse/BEAM-3605
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Reporter: Kenneth Knowles
>Assignee: Alexey Romanenko
>Priority: Critical
>  Labels: flake, sickbay
>
> Here's one:
> https://builds.apache.org/job/beam_PreCommit_Java_GradleBuild/1758/testReport/junit/org.apache.beam.sdk.io.kinesis/ShardReadersPoolTest/shouldForgetClosedShardIterator/
> Filing all test failures as "Critical" so we can sickbay or fix.
> The Jenkins build will get GC'd so here is the error:
> {code}
> java.lang.AssertionError: 
> Expecting:
>   <["shard1", "shard2"]>
> to contain only:
>   <["shard2"]>
> but the following elements were unexpected:
>   <["shard1"]>
>   at 
> org.apache.beam.sdk.io.kinesis.ShardReadersPoolTest.shouldForgetClosedShardIterator(ShardReadersPoolTest.java:270)
> {code}
> and stderr
> {code}
> Feb 01, 2018 11:24:16 PM org.apache.beam.sdk.io.kinesis.ShardReadersPool 
> readLoop
> INFO: Shard iterator for shard1 shard is closed, finishing the read loop
> org.apache.beam.sdk.io.kinesis.KinesisShardClosedException
> Feb 01, 2018 11:24:16 PM org.apache.beam.sdk.io.kinesis.ShardReadersPool 
> readLoop
> INFO: Kinesis Shard read loop has finished
> Feb 01, 2018 11:24:16 PM org.apache.beam.sdk.io.kinesis.ShardReadersPool 
> readLoop
> INFO: Shard iterator for shard1 shard is closed, finishing the read loop
> org.apache.beam.sdk.io.kinesis.KinesisShardClosedException
> Feb 01, 2018 11:24:16 PM org.apache.beam.sdk.io.kinesis.ShardReadersPool 
> readLoop
> INFO: Kinesis Shard read loop has finished
> Feb 01, 2018 11:24:19 PM org.apache.beam.sdk.io.kinesis.ShardReadersPool 
> readLoop
> INFO: Shard iterator for shard1 shard is closed, finishing the read loop
> org.apache.beam.sdk.io.kinesis.KinesisShardClosedException
> Feb 01, 2018 11:24:19 PM org.apache.beam.sdk.io.kinesis.ShardReadersPool 
> readLoop
> INFO: Kinesis Shard read loop has finished
> Feb 01, 2018 11:24:23 PM org.apache.beam.sdk.io.kinesis.ShardReadersPool 
> readLoop
> INFO: Shard iterator for shard1 shard is closed, finishing the read loop
> org.apache.beam.sdk.io.kinesis.KinesisShardClosedException: Shard iterator 
> reached end of the shard: streamName=null, shardId=shard1
>   at 
> org.apache.beam.sdk.io.kinesis.ShardRecordsIterator.readNextBatch(ShardRecordsIterator.java:70)
>   at 
> org.apache.beam.sdk.io.kinesis.ShardReadersPool.readLoop(ShardReadersPool.java:121)
>   at 
> org.apache.beam.sdk.io.kinesis.ShardReadersPool.lambda$startReadingShards$0(ShardReadersPool.java:112)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>   at java.lang.Thread.run(Thread.java:748)
> Feb 01, 2018 11:24:23 PM org.apache.beam.sdk.io.kinesis.ShardReadersPool 
> readLoop
> INFO: Kinesis Shard read loop has finished
> Feb 01, 2018 11:24:23 PM org.apache.beam.sdk.io.kinesis.ShardReadersPool 
> readLoop
> INFO: Shard iterator for shard2 shard is closed, finishing the read loop
> org.apache.beam.sdk.io.kinesis.KinesisShardClosedException: Shard iterator 
> reached end of the shard: streamName=null, shardId=shard2
>   at 
> org.apache.beam.sdk.io.kinesis.ShardRecordsIterator.readNextBatch(ShardRecordsIterator.java:70)
>   at 
> org.apache.beam.sdk.io.kinesis.ShardReadersPool.readLoop(ShardReadersPool.java:121)
>   at 
> org.apache.beam.sdk.io.kinesis.ShardReadersPool.lambda$startReadingShards$0(ShardReadersPool.java:112)
>   at 
> 

[jira] [Commented] (BEAM-3272) ParDoTranslatorTest: Error creating local cluster while creating checkpoint file

[Kenneth Knowles (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16354263#comment-16354263
 ] 

Kenneth Knowles commented on BEAM-3272:
---

It is worse in gradle, perhaps due to parallelism and/or tighter management of 
directories that gradle considers that it owns.

> ParDoTranslatorTest: Error creating local cluster while creating checkpoint 
> file
> 
>
> Key: BEAM-3272
> URL: https://issues.apache.org/jira/browse/BEAM-3272
> Project: Beam
>  Issue Type: Bug
>  Components: runner-apex
>Reporter: Eugene Kirpichov
>Assignee: Kenneth Knowles
>Priority: Critical
>  Labels: flake
>
> Failed build: 
> https://builds.apache.org/job/beam_PostCommit_Java_MavenInstall/org.apache.beam$beam-runners-apex/5330/console
> Key output:
> {code}
> 2017-11-29T01:21:26.956 [ERROR] 
> testAssertionFailure(org.apache.beam.runners.apex.translation.ParDoTranslatorTest)
>   Time elapsed: 2.007 s  <<< ERROR!
> java.lang.RuntimeException: Error creating local cluster
>   at 
> org.apache.apex.engine.EmbeddedAppLauncherImpl.getController(EmbeddedAppLauncherImpl.java:122)
>   at 
> org.apache.apex.engine.EmbeddedAppLauncherImpl.launchApp(EmbeddedAppLauncherImpl.java:71)
>   at 
> org.apache.apex.engine.EmbeddedAppLauncherImpl.launchApp(EmbeddedAppLauncherImpl.java:46)
>   at org.apache.beam.runners.apex.ApexRunner.run(ApexRunner.java:197)
>   at 
> org.apache.beam.runners.apex.TestApexRunner.run(TestApexRunner.java:57)
>   at 
> org.apache.beam.runners.apex.TestApexRunner.run(TestApexRunner.java:31)
>   at org.apache.beam.sdk.Pipeline.run(Pipeline.java:304)
>   at org.apache.beam.sdk.Pipeline.run(Pipeline.java:290)
>   at 
> org.apache.beam.runners.apex.translation.ParDoTranslatorTest.runExpectingAssertionFailure(ParDoTranslatorTest.java:156)
> {code}
> ...
> {code}
> Caused by: ExitCodeException exitCode=1: chmod: cannot access 
> ‘/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Java_MavenInstall/src/runners/apex/target/com.datatorrent.stram.StramLocalCluster/checkpoints/2/_tmp’:
>  No such file or directory
>   at org.apache.hadoop.util.Shell.runCommand(Shell.java:582)
>   at org.apache.hadoop.util.Shell.run(Shell.java:479)
>   at 
> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:773)
>   at org.apache.hadoop.util.Shell.execCommand(Shell.java:866)
>   at org.apache.hadoop.util.Shell.execCommand(Shell.java:849)
>   at 
> org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:733)
>   at 
> org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.(RawLocalFileSystem.java:225)
>   at 
> org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.(RawLocalFileSystem.java:209)
>   at 
> org.apache.hadoop.fs.RawLocalFileSystem.createOutputStreamWithMode(RawLocalFileSystem.java:307)
>   at 
> org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:296)
>   at 
> org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:328)
>   at org.apache.hadoop.fs.FileSystem.primitiveCreate(FileSystem.java:1017)
>   at 
> org.apache.hadoop.fs.DelegateToFileSystem.createInternal(DelegateToFileSystem.java:99)
>   at 
> org.apache.hadoop.fs.ChecksumFs$ChecksumFSOutputSummer.(ChecksumFs.java:352)
>   at org.apache.hadoop.fs.ChecksumFs.createInternal(ChecksumFs.java:399)
>   at 
> org.apache.hadoop.fs.AbstractFileSystem.create(AbstractFileSystem.java:584)
>   at org.apache.hadoop.fs.FileContext$3.next(FileContext.java:686)
>   at org.apache.hadoop.fs.FileContext$3.next(FileContext.java:682)
>   at org.apache.hadoop.fs.FSLinkResolver.resolve(FSLinkResolver.java:90)
>   at org.apache.hadoop.fs.FileContext.create(FileContext.java:688)
>   at 
> com.datatorrent.common.util.AsyncFSStorageAgent.copyToHDFS(AsyncFSStorageAgent.java:119)
>   ... 50 more
> {code}
> By inspecting code at the stack frames, seems it's trying to copy an 
> operator's checkpoint "to HDFS" (which in this case is the local disk), but 
> fails while creating the target file of the copy - creation creates the file 
> (successfully) and chmods it writable (unsuccessfully). Barring something 
> subtle (e.g. chmod being not allowed to call immediately after creating a 
> FileOutputStream), this looks like the whole directory was possibly deleted 
> from under the process. I don't know why this would be the case though, or 
> how to debug it.
> Either way, the path being accessed is funky: 
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Java_MavenInstall/src/runners/apex/target/...
>  - I think it'd be better if this test used a "@Rule TemporaryFolder" to 
> store Apex checkpoints. I don't 

[beam] branch master updated (a67262f -> 2826362)

[jkff]

This is an automated email from the ASF dual-hosted git repository.

jkff pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from a67262f  Merge pull request #4538: [BEAM-3527] Adding a comment for 
ZERO element of DistributionResult.
 add e003431  Adds PositionT and claim callback to RestrictionTracker
 add eca41b9  Changes OutputAndTimeBounded invoker to start checkpoint 
timer after first claim, and verifies more invariants
 add 0371848  Compresses encoded GrowthState with Snappy - about 2x-3x more 
compact
 add 32a427c  InMemoryStateInternals.copy clones the values using the coder
 add 8151d82  Final fixups
 add 6857cb9  Bump worker to 20180205
 new 2826362  Merge pull request #4483: [BEAM-3499, BEAM-2607] Gives the 
runner access to positions of SDF claimed blocks

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../runners/apex/translation/ParDoTranslator.java  |   2 +-
 .../translation/operators/ApexParDoOperator.java   |   2 +-
 .../core/construction/PTransformMatchersTest.java  |   2 +-
 .../core/construction/ParDoTranslationTest.java|   4 +-
 .../core/construction/SplittableParDoTest.java |   7 +-
 .../beam/runners/core/InMemoryStateInternals.java  |  77 --
 ...TimeBoundedSplittableProcessElementInvoker.java | 100 +---
 .../apache/beam/runners/core/SimpleDoFnRunner.java |   8 +-
 .../core/SplittableParDoViaKeyedWorkItems.java |   4 +-
 .../core/SplittableProcessElementInvoker.java  |   2 +-
 ...BoundedSplittableProcessElementInvokerTest.java | 111 +++---
 .../runners/core/SplittableParDoProcessFnTest.java |  34 +++--
 .../direct/CopyOnAccessInMemoryStateInternals.java |  10 +-
 .../SplittableProcessElementsEvaluatorFactory.java |   6 +-
 .../flink/FlinkStreamingTransformTranslators.java  |   2 +-
 .../wrappers/streaming/SplittableDoFnOperator.java |   2 +-
 runners/google-cloud-dataflow-java/pom.xml |   2 +-
 .../{ShardedKeyCoder.java => SnappyCoder.java} |  49 +++---
 .../org/apache/beam/sdk/transforms/DoFnTester.java |   2 +-
 .../java/org/apache/beam/sdk/transforms/Watch.java | 167 ++---
 .../beam/sdk/transforms/reflect/DoFnInvoker.java   |   6 +-
 .../sdk/transforms/reflect/DoFnSignatures.java |   6 +-
 .../splittabledofn/HasDefaultTracker.java  |   2 +-
 .../splittabledofn/OffsetRangeTracker.java |  12 +-
 .../splittabledofn/RestrictionTracker.java |  76 +-
 .../beam/sdk/transforms/SplittableDoFnTest.java|   6 +-
 .../org/apache/beam/sdk/transforms/WatchTest.java  | 126 ++--
 .../sdk/transforms/reflect/DoFnInvokersTest.java   |  14 +-
 .../reflect/DoFnSignaturesProcessElementTest.java  |   2 +-
 .../reflect/DoFnSignaturesSplittableDoFnTest.java  |  11 +-
 .../splittabledofn/OffsetRangeTrackerTest.java |  73 +
 .../apache/beam/fn/harness/FnApiDoFnRunner.java|   6 +-
 32 files changed, 613 insertions(+), 320 deletions(-)
 copy 
sdks/java/core/src/main/java/org/apache/beam/sdk/coders/{ShardedKeyCoder.java 
=> SnappyCoder.java} (51%)

-- 
To stop receiving notification emails like this one, please contact
j...@apache.org.

[jira] [Commented] (BEAM-3272) ParDoTranslatorTest: Error creating local cluster while creating checkpoint file

[Kenneth Knowles (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16354261#comment-16354261
 ] 

Kenneth Knowles commented on BEAM-3272:
---

This is happening quite a lot. I am going to sickbay the test for now.

> ParDoTranslatorTest: Error creating local cluster while creating checkpoint 
> file
> 
>
> Key: BEAM-3272
> URL: https://issues.apache.org/jira/browse/BEAM-3272
> Project: Beam
>  Issue Type: Bug
>  Components: runner-apex
>Reporter: Eugene Kirpichov
>Assignee: Kenneth Knowles
>Priority: Critical
>  Labels: flake
>
> Failed build: 
> https://builds.apache.org/job/beam_PostCommit_Java_MavenInstall/org.apache.beam$beam-runners-apex/5330/console
> Key output:
> {code}
> 2017-11-29T01:21:26.956 [ERROR] 
> testAssertionFailure(org.apache.beam.runners.apex.translation.ParDoTranslatorTest)
>   Time elapsed: 2.007 s  <<< ERROR!
> java.lang.RuntimeException: Error creating local cluster
>   at 
> org.apache.apex.engine.EmbeddedAppLauncherImpl.getController(EmbeddedAppLauncherImpl.java:122)
>   at 
> org.apache.apex.engine.EmbeddedAppLauncherImpl.launchApp(EmbeddedAppLauncherImpl.java:71)
>   at 
> org.apache.apex.engine.EmbeddedAppLauncherImpl.launchApp(EmbeddedAppLauncherImpl.java:46)
>   at org.apache.beam.runners.apex.ApexRunner.run(ApexRunner.java:197)
>   at 
> org.apache.beam.runners.apex.TestApexRunner.run(TestApexRunner.java:57)
>   at 
> org.apache.beam.runners.apex.TestApexRunner.run(TestApexRunner.java:31)
>   at org.apache.beam.sdk.Pipeline.run(Pipeline.java:304)
>   at org.apache.beam.sdk.Pipeline.run(Pipeline.java:290)
>   at 
> org.apache.beam.runners.apex.translation.ParDoTranslatorTest.runExpectingAssertionFailure(ParDoTranslatorTest.java:156)
> {code}
> ...
> {code}
> Caused by: ExitCodeException exitCode=1: chmod: cannot access 
> ‘/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Java_MavenInstall/src/runners/apex/target/com.datatorrent.stram.StramLocalCluster/checkpoints/2/_tmp’:
>  No such file or directory
>   at org.apache.hadoop.util.Shell.runCommand(Shell.java:582)
>   at org.apache.hadoop.util.Shell.run(Shell.java:479)
>   at 
> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:773)
>   at org.apache.hadoop.util.Shell.execCommand(Shell.java:866)
>   at org.apache.hadoop.util.Shell.execCommand(Shell.java:849)
>   at 
> org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:733)
>   at 
> org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.(RawLocalFileSystem.java:225)
>   at 
> org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.(RawLocalFileSystem.java:209)
>   at 
> org.apache.hadoop.fs.RawLocalFileSystem.createOutputStreamWithMode(RawLocalFileSystem.java:307)
>   at 
> org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:296)
>   at 
> org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:328)
>   at org.apache.hadoop.fs.FileSystem.primitiveCreate(FileSystem.java:1017)
>   at 
> org.apache.hadoop.fs.DelegateToFileSystem.createInternal(DelegateToFileSystem.java:99)
>   at 
> org.apache.hadoop.fs.ChecksumFs$ChecksumFSOutputSummer.(ChecksumFs.java:352)
>   at org.apache.hadoop.fs.ChecksumFs.createInternal(ChecksumFs.java:399)
>   at 
> org.apache.hadoop.fs.AbstractFileSystem.create(AbstractFileSystem.java:584)
>   at org.apache.hadoop.fs.FileContext$3.next(FileContext.java:686)
>   at org.apache.hadoop.fs.FileContext$3.next(FileContext.java:682)
>   at org.apache.hadoop.fs.FSLinkResolver.resolve(FSLinkResolver.java:90)
>   at org.apache.hadoop.fs.FileContext.create(FileContext.java:688)
>   at 
> com.datatorrent.common.util.AsyncFSStorageAgent.copyToHDFS(AsyncFSStorageAgent.java:119)
>   ... 50 more
> {code}
> By inspecting code at the stack frames, seems it's trying to copy an 
> operator's checkpoint "to HDFS" (which in this case is the local disk), but 
> fails while creating the target file of the copy - creation creates the file 
> (successfully) and chmods it writable (unsuccessfully). Barring something 
> subtle (e.g. chmod being not allowed to call immediately after creating a 
> FileOutputStream), this looks like the whole directory was possibly deleted 
> from under the process. I don't know why this would be the case though, or 
> how to debug it.
> Either way, the path being accessed is funky: 
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Java_MavenInstall/src/runners/apex/target/...
>  - I think it'd be better if this test used a "@Rule TemporaryFolder" to 
> store Apex checkpoints. I don't know whether the Apex runner allows that, but 
> I can see 

[jira] [Assigned] (BEAM-3272) ParDoTranslatorTest: Error creating local cluster while creating checkpoint file

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles reassigned BEAM-3272:
-

Assignee: Kenneth Knowles  (was: Thomas Weise)

> ParDoTranslatorTest: Error creating local cluster while creating checkpoint 
> file
> 
>
> Key: BEAM-3272
> URL: https://issues.apache.org/jira/browse/BEAM-3272
> Project: Beam
>  Issue Type: Bug
>  Components: runner-apex
>Reporter: Eugene Kirpichov
>Assignee: Kenneth Knowles
>Priority: Critical
>  Labels: flake
>
> Failed build: 
> https://builds.apache.org/job/beam_PostCommit_Java_MavenInstall/org.apache.beam$beam-runners-apex/5330/console
> Key output:
> {code}
> 2017-11-29T01:21:26.956 [ERROR] 
> testAssertionFailure(org.apache.beam.runners.apex.translation.ParDoTranslatorTest)
>   Time elapsed: 2.007 s  <<< ERROR!
> java.lang.RuntimeException: Error creating local cluster
>   at 
> org.apache.apex.engine.EmbeddedAppLauncherImpl.getController(EmbeddedAppLauncherImpl.java:122)
>   at 
> org.apache.apex.engine.EmbeddedAppLauncherImpl.launchApp(EmbeddedAppLauncherImpl.java:71)
>   at 
> org.apache.apex.engine.EmbeddedAppLauncherImpl.launchApp(EmbeddedAppLauncherImpl.java:46)
>   at org.apache.beam.runners.apex.ApexRunner.run(ApexRunner.java:197)
>   at 
> org.apache.beam.runners.apex.TestApexRunner.run(TestApexRunner.java:57)
>   at 
> org.apache.beam.runners.apex.TestApexRunner.run(TestApexRunner.java:31)
>   at org.apache.beam.sdk.Pipeline.run(Pipeline.java:304)
>   at org.apache.beam.sdk.Pipeline.run(Pipeline.java:290)
>   at 
> org.apache.beam.runners.apex.translation.ParDoTranslatorTest.runExpectingAssertionFailure(ParDoTranslatorTest.java:156)
> {code}
> ...
> {code}
> Caused by: ExitCodeException exitCode=1: chmod: cannot access 
> ‘/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Java_MavenInstall/src/runners/apex/target/com.datatorrent.stram.StramLocalCluster/checkpoints/2/_tmp’:
>  No such file or directory
>   at org.apache.hadoop.util.Shell.runCommand(Shell.java:582)
>   at org.apache.hadoop.util.Shell.run(Shell.java:479)
>   at 
> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:773)
>   at org.apache.hadoop.util.Shell.execCommand(Shell.java:866)
>   at org.apache.hadoop.util.Shell.execCommand(Shell.java:849)
>   at 
> org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:733)
>   at 
> org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.(RawLocalFileSystem.java:225)
>   at 
> org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.(RawLocalFileSystem.java:209)
>   at 
> org.apache.hadoop.fs.RawLocalFileSystem.createOutputStreamWithMode(RawLocalFileSystem.java:307)
>   at 
> org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:296)
>   at 
> org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:328)
>   at org.apache.hadoop.fs.FileSystem.primitiveCreate(FileSystem.java:1017)
>   at 
> org.apache.hadoop.fs.DelegateToFileSystem.createInternal(DelegateToFileSystem.java:99)
>   at 
> org.apache.hadoop.fs.ChecksumFs$ChecksumFSOutputSummer.(ChecksumFs.java:352)
>   at org.apache.hadoop.fs.ChecksumFs.createInternal(ChecksumFs.java:399)
>   at 
> org.apache.hadoop.fs.AbstractFileSystem.create(AbstractFileSystem.java:584)
>   at org.apache.hadoop.fs.FileContext$3.next(FileContext.java:686)
>   at org.apache.hadoop.fs.FileContext$3.next(FileContext.java:682)
>   at org.apache.hadoop.fs.FSLinkResolver.resolve(FSLinkResolver.java:90)
>   at org.apache.hadoop.fs.FileContext.create(FileContext.java:688)
>   at 
> com.datatorrent.common.util.AsyncFSStorageAgent.copyToHDFS(AsyncFSStorageAgent.java:119)
>   ... 50 more
> {code}
> By inspecting code at the stack frames, seems it's trying to copy an 
> operator's checkpoint "to HDFS" (which in this case is the local disk), but 
> fails while creating the target file of the copy - creation creates the file 
> (successfully) and chmods it writable (unsuccessfully). Barring something 
> subtle (e.g. chmod being not allowed to call immediately after creating a 
> FileOutputStream), this looks like the whole directory was possibly deleted 
> from under the process. I don't know why this would be the case though, or 
> how to debug it.
> Either way, the path being accessed is funky: 
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Java_MavenInstall/src/runners/apex/target/...
>  - I think it'd be better if this test used a "@Rule TemporaryFolder" to 
> store Apex checkpoints. I don't know whether the Apex runner allows that, but 
> I can see how it could help reduce interference between tests and 

[jira] [Updated] (BEAM-3272) ParDoTranslatorTest: Error creating local cluster while creating checkpoint file

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-3272:
--
Priority: Critical  (was: Minor)

> ParDoTranslatorTest: Error creating local cluster while creating checkpoint 
> file
> 
>
> Key: BEAM-3272
> URL: https://issues.apache.org/jira/browse/BEAM-3272
> Project: Beam
>  Issue Type: Bug
>  Components: runner-apex
>Reporter: Eugene Kirpichov
>Assignee: Thomas Weise
>Priority: Critical
>  Labels: flake
>
> Failed build: 
> https://builds.apache.org/job/beam_PostCommit_Java_MavenInstall/org.apache.beam$beam-runners-apex/5330/console
> Key output:
> {code}
> 2017-11-29T01:21:26.956 [ERROR] 
> testAssertionFailure(org.apache.beam.runners.apex.translation.ParDoTranslatorTest)
>   Time elapsed: 2.007 s  <<< ERROR!
> java.lang.RuntimeException: Error creating local cluster
>   at 
> org.apache.apex.engine.EmbeddedAppLauncherImpl.getController(EmbeddedAppLauncherImpl.java:122)
>   at 
> org.apache.apex.engine.EmbeddedAppLauncherImpl.launchApp(EmbeddedAppLauncherImpl.java:71)
>   at 
> org.apache.apex.engine.EmbeddedAppLauncherImpl.launchApp(EmbeddedAppLauncherImpl.java:46)
>   at org.apache.beam.runners.apex.ApexRunner.run(ApexRunner.java:197)
>   at 
> org.apache.beam.runners.apex.TestApexRunner.run(TestApexRunner.java:57)
>   at 
> org.apache.beam.runners.apex.TestApexRunner.run(TestApexRunner.java:31)
>   at org.apache.beam.sdk.Pipeline.run(Pipeline.java:304)
>   at org.apache.beam.sdk.Pipeline.run(Pipeline.java:290)
>   at 
> org.apache.beam.runners.apex.translation.ParDoTranslatorTest.runExpectingAssertionFailure(ParDoTranslatorTest.java:156)
> {code}
> ...
> {code}
> Caused by: ExitCodeException exitCode=1: chmod: cannot access 
> ‘/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Java_MavenInstall/src/runners/apex/target/com.datatorrent.stram.StramLocalCluster/checkpoints/2/_tmp’:
>  No such file or directory
>   at org.apache.hadoop.util.Shell.runCommand(Shell.java:582)
>   at org.apache.hadoop.util.Shell.run(Shell.java:479)
>   at 
> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:773)
>   at org.apache.hadoop.util.Shell.execCommand(Shell.java:866)
>   at org.apache.hadoop.util.Shell.execCommand(Shell.java:849)
>   at 
> org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:733)
>   at 
> org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.(RawLocalFileSystem.java:225)
>   at 
> org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.(RawLocalFileSystem.java:209)
>   at 
> org.apache.hadoop.fs.RawLocalFileSystem.createOutputStreamWithMode(RawLocalFileSystem.java:307)
>   at 
> org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:296)
>   at 
> org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:328)
>   at org.apache.hadoop.fs.FileSystem.primitiveCreate(FileSystem.java:1017)
>   at 
> org.apache.hadoop.fs.DelegateToFileSystem.createInternal(DelegateToFileSystem.java:99)
>   at 
> org.apache.hadoop.fs.ChecksumFs$ChecksumFSOutputSummer.(ChecksumFs.java:352)
>   at org.apache.hadoop.fs.ChecksumFs.createInternal(ChecksumFs.java:399)
>   at 
> org.apache.hadoop.fs.AbstractFileSystem.create(AbstractFileSystem.java:584)
>   at org.apache.hadoop.fs.FileContext$3.next(FileContext.java:686)
>   at org.apache.hadoop.fs.FileContext$3.next(FileContext.java:682)
>   at org.apache.hadoop.fs.FSLinkResolver.resolve(FSLinkResolver.java:90)
>   at org.apache.hadoop.fs.FileContext.create(FileContext.java:688)
>   at 
> com.datatorrent.common.util.AsyncFSStorageAgent.copyToHDFS(AsyncFSStorageAgent.java:119)
>   ... 50 more
> {code}
> By inspecting code at the stack frames, seems it's trying to copy an 
> operator's checkpoint "to HDFS" (which in this case is the local disk), but 
> fails while creating the target file of the copy - creation creates the file 
> (successfully) and chmods it writable (unsuccessfully). Barring something 
> subtle (e.g. chmod being not allowed to call immediately after creating a 
> FileOutputStream), this looks like the whole directory was possibly deleted 
> from under the process. I don't know why this would be the case though, or 
> how to debug it.
> Either way, the path being accessed is funky: 
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Java_MavenInstall/src/runners/apex/target/...
>  - I think it'd be better if this test used a "@Rule TemporaryFolder" to 
> store Apex checkpoints. I don't know whether the Apex runner allows that, but 
> I can see how it could help reduce interference between tests and potentially 
> resolve 

[jira] [Updated] (BEAM-3272) ParDoTranslatorTest: Error creating local cluster while creating checkpoint file

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-3272:
--
Affects Version/s: (was: 2.3.0)

> ParDoTranslatorTest: Error creating local cluster while creating checkpoint 
> file
> 
>
> Key: BEAM-3272
> URL: https://issues.apache.org/jira/browse/BEAM-3272
> Project: Beam
>  Issue Type: Bug
>  Components: runner-apex
>Reporter: Eugene Kirpichov
>Assignee: Thomas Weise
>Priority: Critical
>  Labels: flake
>
> Failed build: 
> https://builds.apache.org/job/beam_PostCommit_Java_MavenInstall/org.apache.beam$beam-runners-apex/5330/console
> Key output:
> {code}
> 2017-11-29T01:21:26.956 [ERROR] 
> testAssertionFailure(org.apache.beam.runners.apex.translation.ParDoTranslatorTest)
>   Time elapsed: 2.007 s  <<< ERROR!
> java.lang.RuntimeException: Error creating local cluster
>   at 
> org.apache.apex.engine.EmbeddedAppLauncherImpl.getController(EmbeddedAppLauncherImpl.java:122)
>   at 
> org.apache.apex.engine.EmbeddedAppLauncherImpl.launchApp(EmbeddedAppLauncherImpl.java:71)
>   at 
> org.apache.apex.engine.EmbeddedAppLauncherImpl.launchApp(EmbeddedAppLauncherImpl.java:46)
>   at org.apache.beam.runners.apex.ApexRunner.run(ApexRunner.java:197)
>   at 
> org.apache.beam.runners.apex.TestApexRunner.run(TestApexRunner.java:57)
>   at 
> org.apache.beam.runners.apex.TestApexRunner.run(TestApexRunner.java:31)
>   at org.apache.beam.sdk.Pipeline.run(Pipeline.java:304)
>   at org.apache.beam.sdk.Pipeline.run(Pipeline.java:290)
>   at 
> org.apache.beam.runners.apex.translation.ParDoTranslatorTest.runExpectingAssertionFailure(ParDoTranslatorTest.java:156)
> {code}
> ...
> {code}
> Caused by: ExitCodeException exitCode=1: chmod: cannot access 
> ‘/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Java_MavenInstall/src/runners/apex/target/com.datatorrent.stram.StramLocalCluster/checkpoints/2/_tmp’:
>  No such file or directory
>   at org.apache.hadoop.util.Shell.runCommand(Shell.java:582)
>   at org.apache.hadoop.util.Shell.run(Shell.java:479)
>   at 
> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:773)
>   at org.apache.hadoop.util.Shell.execCommand(Shell.java:866)
>   at org.apache.hadoop.util.Shell.execCommand(Shell.java:849)
>   at 
> org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:733)
>   at 
> org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.(RawLocalFileSystem.java:225)
>   at 
> org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.(RawLocalFileSystem.java:209)
>   at 
> org.apache.hadoop.fs.RawLocalFileSystem.createOutputStreamWithMode(RawLocalFileSystem.java:307)
>   at 
> org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:296)
>   at 
> org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:328)
>   at org.apache.hadoop.fs.FileSystem.primitiveCreate(FileSystem.java:1017)
>   at 
> org.apache.hadoop.fs.DelegateToFileSystem.createInternal(DelegateToFileSystem.java:99)
>   at 
> org.apache.hadoop.fs.ChecksumFs$ChecksumFSOutputSummer.(ChecksumFs.java:352)
>   at org.apache.hadoop.fs.ChecksumFs.createInternal(ChecksumFs.java:399)
>   at 
> org.apache.hadoop.fs.AbstractFileSystem.create(AbstractFileSystem.java:584)
>   at org.apache.hadoop.fs.FileContext$3.next(FileContext.java:686)
>   at org.apache.hadoop.fs.FileContext$3.next(FileContext.java:682)
>   at org.apache.hadoop.fs.FSLinkResolver.resolve(FSLinkResolver.java:90)
>   at org.apache.hadoop.fs.FileContext.create(FileContext.java:688)
>   at 
> com.datatorrent.common.util.AsyncFSStorageAgent.copyToHDFS(AsyncFSStorageAgent.java:119)
>   ... 50 more
> {code}
> By inspecting code at the stack frames, seems it's trying to copy an 
> operator's checkpoint "to HDFS" (which in this case is the local disk), but 
> fails while creating the target file of the copy - creation creates the file 
> (successfully) and chmods it writable (unsuccessfully). Barring something 
> subtle (e.g. chmod being not allowed to call immediately after creating a 
> FileOutputStream), this looks like the whole directory was possibly deleted 
> from under the process. I don't know why this would be the case though, or 
> how to debug it.
> Either way, the path being accessed is funky: 
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Java_MavenInstall/src/runners/apex/target/...
>  - I think it'd be better if this test used a "@Rule TemporaryFolder" to 
> store Apex checkpoints. I don't know whether the Apex runner allows that, but 
> I can see how it could help reduce interference between tests and potentially 
> 

[jira] [Updated] (BEAM-3272) ParDoTranslatorTest: Error creating local cluster while creating checkpoint file

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-3272:
--
Labels: flake  (was: )

> ParDoTranslatorTest: Error creating local cluster while creating checkpoint 
> file
> 
>
> Key: BEAM-3272
> URL: https://issues.apache.org/jira/browse/BEAM-3272
> Project: Beam
>  Issue Type: Bug
>  Components: runner-apex
>Reporter: Eugene Kirpichov
>Assignee: Thomas Weise
>Priority: Critical
>  Labels: flake
>
> Failed build: 
> https://builds.apache.org/job/beam_PostCommit_Java_MavenInstall/org.apache.beam$beam-runners-apex/5330/console
> Key output:
> {code}
> 2017-11-29T01:21:26.956 [ERROR] 
> testAssertionFailure(org.apache.beam.runners.apex.translation.ParDoTranslatorTest)
>   Time elapsed: 2.007 s  <<< ERROR!
> java.lang.RuntimeException: Error creating local cluster
>   at 
> org.apache.apex.engine.EmbeddedAppLauncherImpl.getController(EmbeddedAppLauncherImpl.java:122)
>   at 
> org.apache.apex.engine.EmbeddedAppLauncherImpl.launchApp(EmbeddedAppLauncherImpl.java:71)
>   at 
> org.apache.apex.engine.EmbeddedAppLauncherImpl.launchApp(EmbeddedAppLauncherImpl.java:46)
>   at org.apache.beam.runners.apex.ApexRunner.run(ApexRunner.java:197)
>   at 
> org.apache.beam.runners.apex.TestApexRunner.run(TestApexRunner.java:57)
>   at 
> org.apache.beam.runners.apex.TestApexRunner.run(TestApexRunner.java:31)
>   at org.apache.beam.sdk.Pipeline.run(Pipeline.java:304)
>   at org.apache.beam.sdk.Pipeline.run(Pipeline.java:290)
>   at 
> org.apache.beam.runners.apex.translation.ParDoTranslatorTest.runExpectingAssertionFailure(ParDoTranslatorTest.java:156)
> {code}
> ...
> {code}
> Caused by: ExitCodeException exitCode=1: chmod: cannot access 
> ‘/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Java_MavenInstall/src/runners/apex/target/com.datatorrent.stram.StramLocalCluster/checkpoints/2/_tmp’:
>  No such file or directory
>   at org.apache.hadoop.util.Shell.runCommand(Shell.java:582)
>   at org.apache.hadoop.util.Shell.run(Shell.java:479)
>   at 
> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:773)
>   at org.apache.hadoop.util.Shell.execCommand(Shell.java:866)
>   at org.apache.hadoop.util.Shell.execCommand(Shell.java:849)
>   at 
> org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:733)
>   at 
> org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.(RawLocalFileSystem.java:225)
>   at 
> org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.(RawLocalFileSystem.java:209)
>   at 
> org.apache.hadoop.fs.RawLocalFileSystem.createOutputStreamWithMode(RawLocalFileSystem.java:307)
>   at 
> org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:296)
>   at 
> org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:328)
>   at org.apache.hadoop.fs.FileSystem.primitiveCreate(FileSystem.java:1017)
>   at 
> org.apache.hadoop.fs.DelegateToFileSystem.createInternal(DelegateToFileSystem.java:99)
>   at 
> org.apache.hadoop.fs.ChecksumFs$ChecksumFSOutputSummer.(ChecksumFs.java:352)
>   at org.apache.hadoop.fs.ChecksumFs.createInternal(ChecksumFs.java:399)
>   at 
> org.apache.hadoop.fs.AbstractFileSystem.create(AbstractFileSystem.java:584)
>   at org.apache.hadoop.fs.FileContext$3.next(FileContext.java:686)
>   at org.apache.hadoop.fs.FileContext$3.next(FileContext.java:682)
>   at org.apache.hadoop.fs.FSLinkResolver.resolve(FSLinkResolver.java:90)
>   at org.apache.hadoop.fs.FileContext.create(FileContext.java:688)
>   at 
> com.datatorrent.common.util.AsyncFSStorageAgent.copyToHDFS(AsyncFSStorageAgent.java:119)
>   ... 50 more
> {code}
> By inspecting code at the stack frames, seems it's trying to copy an 
> operator's checkpoint "to HDFS" (which in this case is the local disk), but 
> fails while creating the target file of the copy - creation creates the file 
> (successfully) and chmods it writable (unsuccessfully). Barring something 
> subtle (e.g. chmod being not allowed to call immediately after creating a 
> FileOutputStream), this looks like the whole directory was possibly deleted 
> from under the process. I don't know why this would be the case though, or 
> how to debug it.
> Either way, the path being accessed is funky: 
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Java_MavenInstall/src/runners/apex/target/...
>  - I think it'd be better if this test used a "@Rule TemporaryFolder" to 
> store Apex checkpoints. I don't know whether the Apex runner allows that, but 
> I can see how it could help reduce interference between tests and potentially 
> resolve this 

[jira] [Commented] (BEAM-3623) Python SDK Fn harness should not log to stderr for portable jobs

[Luke Cwik (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16354256#comment-16354256
 ] 

Luke Cwik commented on BEAM-3623:
-

stdout/stderr should be used for all logging before connecting to the logging 
API.

stdout/stderr should be used if logging API is unreachable.

> Python SDK Fn harness should not log to stderr for portable jobs
> 
>
> Key: BEAM-3623
> URL: https://issues.apache.org/jira/browse/BEAM-3623
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-harness
>Reporter: Henning Rohde
>Assignee: Robert Bradshaw
>Priority: Major
>  Labels: portability
>
> We currently log both to stderr and the portable logging API, but we should 
> do only the latter (and potentially fallback to stderr under failures). 
> Otherwise, we're duplicating the log volume.
> Location where the Fn Logger is added: 
> https://github.com/apache/beam/blob/f3dba1af43b03e24419fb0f5b726479368959eef/sdks/python/apache_beam/runners/worker/sdk_worker_main.py#L95



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] 01/01: Merge pull request #4538: [BEAM-3527] Adding a comment for ZERO element of DistributionResult.

[kenn]

This is an automated email from the ASF dual-hosted git repository.

kenn pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit a67262f5b14c55f96de93615d4e9e56c0e89a389
Merge: 2c01ad5 957dcef
Author: Kenn Knowles 
AuthorDate: Tue Feb 6 09:59:41 2018 -0800

Merge pull request #4538: [BEAM-3527] Adding a comment for ZERO element of 
DistributionResult.

 .../apache/beam/runners/core/metrics/MetricsContainerStepMapTest.java | 3 ++-
 .../test/java/org/apache/beam/runners/direct/DirectMetricsTest.java   | 2 +-
 .../main/java/org/apache/beam/runners/dataflow/DataflowMetrics.java   | 2 +-
 .../src/main/java/org/apache/beam/sdk/metrics/DistributionResult.java | 4 +++-
 4 files changed, 7 insertions(+), 4 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
k...@apache.org.

[beam] branch master updated (2c01ad5 -> a67262f)

[kenn]

This is an automated email from the ASF dual-hosted git repository.

kenn pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 2c01ad5  Merge pull request #4605: [BEAM-3598][BEAM-3599][BEAM-3605] 
Sickbay flaky KinesisIO tests
 add 957dcef  Renaming the ZERO element of DistributionResult to be 
IDENTITY_ELEMENT.
 new a67262f  Merge pull request #4538: [BEAM-3527] Adding a comment for 
ZERO element of DistributionResult.

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../apache/beam/runners/core/metrics/MetricsContainerStepMapTest.java | 3 ++-
 .../test/java/org/apache/beam/runners/direct/DirectMetricsTest.java   | 2 +-
 .../main/java/org/apache/beam/runners/dataflow/DataflowMetrics.java   | 2 +-
 .../src/main/java/org/apache/beam/sdk/metrics/DistributionResult.java | 4 +++-
 4 files changed, 7 insertions(+), 4 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
k...@apache.org.

[jira] [Updated] (BEAM-3599) kinesis.ShardReadersPoolTest.shouldInterruptKinesisReadingAndStopShortly is flaky

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3599?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-3599:
--
Labels: flake sickbay  (was: sickbay)

> kinesis.ShardReadersPoolTest.shouldInterruptKinesisReadingAndStopShortly is 
> flaky
> -
>
> Key: BEAM-3599
> URL: https://issues.apache.org/jira/browse/BEAM-3599
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Reporter: Thomas Groh
>Assignee: Alexey Romanenko
>Priority: Critical
>  Labels: flake, sickbay
>
> This appears to be unavoidable due to the construction of the test - the 
> amount of realtime that the thread waits is asserted to be less than one 
> second, but as the class doesn't consume a {{Ticker}} (or related way to 
> track elapsed time), we have no way of deterministically controlling that 
> value within the test.
>  
> Example failure: 
> [https://builds.apache.org/job/beam_PreCommit_Java_GradleBuild/1738/testReport/junit/org.apache.beam.sdk.io.kinesis/ShardReadersPoolTest/shouldInterruptKinesisReadingAndStopShortly/]
>  
> {{java.lang.AssertionError: Expecting: <4169L> to be less than: <1000L> at 
> org.apache.beam.sdk.io.kinesis.ShardReadersPoolTest.shouldInterruptKinesisReadingAndStopShortly(ShardReadersPoolTest.java:159)
>  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)}} 
>  
> The test could also do with a more precise error message (it looks like too 
> many elements were received based on the error message, rather than the 
> actual "took too long" failure.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-3605) Kinesis ShardReadersPoolTest shouldForgetClosedShardIterator failure

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-3605:
--
Labels: flake sickbay  (was: flake)

> Kinesis ShardReadersPoolTest shouldForgetClosedShardIterator failure
> 
>
> Key: BEAM-3605
> URL: https://issues.apache.org/jira/browse/BEAM-3605
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>Priority: Critical
>  Labels: flake, sickbay
>
> Here's one:
> https://builds.apache.org/job/beam_PreCommit_Java_GradleBuild/1758/testReport/junit/org.apache.beam.sdk.io.kinesis/ShardReadersPoolTest/shouldForgetClosedShardIterator/
> Filing all test failures as "Critical" so we can sickbay or fix.
> The Jenkins build will get GC'd so here is the error:
> {code}
> java.lang.AssertionError: 
> Expecting:
>   <["shard1", "shard2"]>
> to contain only:
>   <["shard2"]>
> but the following elements were unexpected:
>   <["shard1"]>
>   at 
> org.apache.beam.sdk.io.kinesis.ShardReadersPoolTest.shouldForgetClosedShardIterator(ShardReadersPoolTest.java:270)
> {code}
> and stderr
> {code}
> Feb 01, 2018 11:24:16 PM org.apache.beam.sdk.io.kinesis.ShardReadersPool 
> readLoop
> INFO: Shard iterator for shard1 shard is closed, finishing the read loop
> org.apache.beam.sdk.io.kinesis.KinesisShardClosedException
> Feb 01, 2018 11:24:16 PM org.apache.beam.sdk.io.kinesis.ShardReadersPool 
> readLoop
> INFO: Kinesis Shard read loop has finished
> Feb 01, 2018 11:24:16 PM org.apache.beam.sdk.io.kinesis.ShardReadersPool 
> readLoop
> INFO: Shard iterator for shard1 shard is closed, finishing the read loop
> org.apache.beam.sdk.io.kinesis.KinesisShardClosedException
> Feb 01, 2018 11:24:16 PM org.apache.beam.sdk.io.kinesis.ShardReadersPool 
> readLoop
> INFO: Kinesis Shard read loop has finished
> Feb 01, 2018 11:24:19 PM org.apache.beam.sdk.io.kinesis.ShardReadersPool 
> readLoop
> INFO: Shard iterator for shard1 shard is closed, finishing the read loop
> org.apache.beam.sdk.io.kinesis.KinesisShardClosedException
> Feb 01, 2018 11:24:19 PM org.apache.beam.sdk.io.kinesis.ShardReadersPool 
> readLoop
> INFO: Kinesis Shard read loop has finished
> Feb 01, 2018 11:24:23 PM org.apache.beam.sdk.io.kinesis.ShardReadersPool 
> readLoop
> INFO: Shard iterator for shard1 shard is closed, finishing the read loop
> org.apache.beam.sdk.io.kinesis.KinesisShardClosedException: Shard iterator 
> reached end of the shard: streamName=null, shardId=shard1
>   at 
> org.apache.beam.sdk.io.kinesis.ShardRecordsIterator.readNextBatch(ShardRecordsIterator.java:70)
>   at 
> org.apache.beam.sdk.io.kinesis.ShardReadersPool.readLoop(ShardReadersPool.java:121)
>   at 
> org.apache.beam.sdk.io.kinesis.ShardReadersPool.lambda$startReadingShards$0(ShardReadersPool.java:112)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>   at java.lang.Thread.run(Thread.java:748)
> Feb 01, 2018 11:24:23 PM org.apache.beam.sdk.io.kinesis.ShardReadersPool 
> readLoop
> INFO: Kinesis Shard read loop has finished
> Feb 01, 2018 11:24:23 PM org.apache.beam.sdk.io.kinesis.ShardReadersPool 
> readLoop
> INFO: Shard iterator for shard2 shard is closed, finishing the read loop
> org.apache.beam.sdk.io.kinesis.KinesisShardClosedException: Shard iterator 
> reached end of the shard: streamName=null, shardId=shard2
>   at 
> org.apache.beam.sdk.io.kinesis.ShardRecordsIterator.readNextBatch(ShardRecordsIterator.java:70)
>   at 
> org.apache.beam.sdk.io.kinesis.ShardReadersPool.readLoop(ShardReadersPool.java:121)
>   at 
> org.apache.beam.sdk.io.kinesis.ShardReadersPool.lambda$startReadingShards$0(ShardReadersPool.java:112)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>   at java.lang.Thread.run(Thread.java:748)
> Feb 01, 2018 11:24:23 PM org.apache.beam.sdk.io.kinesis.ShardReadersPool stop
> INFO: Closing shard iterators pool
> Feb 01, 2018 11:24:24 PM org.apache.beam.sdk.io.kinesis.ShardReadersPool 
> readLoop
> INFO: Kinesis Shard read loop has finished
> Feb 01, 2018 11:24:24 PM org.apache.beam.sdk.io.kinesis.ShardReadersPool 
> readLoop
> INFO: Shard iterator for shard1 shard is closed, 

[jira] [Assigned] (BEAM-3605) Kinesis ShardReadersPoolTest shouldForgetClosedShardIterator failure

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles reassigned BEAM-3605:
-

Assignee: Alexey Romanenko  (was: Kenneth Knowles)

> Kinesis ShardReadersPoolTest shouldForgetClosedShardIterator failure
> 
>
> Key: BEAM-3605
> URL: https://issues.apache.org/jira/browse/BEAM-3605
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Reporter: Kenneth Knowles
>Assignee: Alexey Romanenko
>Priority: Critical
>  Labels: flake, sickbay
>
> Here's one:
> https://builds.apache.org/job/beam_PreCommit_Java_GradleBuild/1758/testReport/junit/org.apache.beam.sdk.io.kinesis/ShardReadersPoolTest/shouldForgetClosedShardIterator/
> Filing all test failures as "Critical" so we can sickbay or fix.
> The Jenkins build will get GC'd so here is the error:
> {code}
> java.lang.AssertionError: 
> Expecting:
>   <["shard1", "shard2"]>
> to contain only:
>   <["shard2"]>
> but the following elements were unexpected:
>   <["shard1"]>
>   at 
> org.apache.beam.sdk.io.kinesis.ShardReadersPoolTest.shouldForgetClosedShardIterator(ShardReadersPoolTest.java:270)
> {code}
> and stderr
> {code}
> Feb 01, 2018 11:24:16 PM org.apache.beam.sdk.io.kinesis.ShardReadersPool 
> readLoop
> INFO: Shard iterator for shard1 shard is closed, finishing the read loop
> org.apache.beam.sdk.io.kinesis.KinesisShardClosedException
> Feb 01, 2018 11:24:16 PM org.apache.beam.sdk.io.kinesis.ShardReadersPool 
> readLoop
> INFO: Kinesis Shard read loop has finished
> Feb 01, 2018 11:24:16 PM org.apache.beam.sdk.io.kinesis.ShardReadersPool 
> readLoop
> INFO: Shard iterator for shard1 shard is closed, finishing the read loop
> org.apache.beam.sdk.io.kinesis.KinesisShardClosedException
> Feb 01, 2018 11:24:16 PM org.apache.beam.sdk.io.kinesis.ShardReadersPool 
> readLoop
> INFO: Kinesis Shard read loop has finished
> Feb 01, 2018 11:24:19 PM org.apache.beam.sdk.io.kinesis.ShardReadersPool 
> readLoop
> INFO: Shard iterator for shard1 shard is closed, finishing the read loop
> org.apache.beam.sdk.io.kinesis.KinesisShardClosedException
> Feb 01, 2018 11:24:19 PM org.apache.beam.sdk.io.kinesis.ShardReadersPool 
> readLoop
> INFO: Kinesis Shard read loop has finished
> Feb 01, 2018 11:24:23 PM org.apache.beam.sdk.io.kinesis.ShardReadersPool 
> readLoop
> INFO: Shard iterator for shard1 shard is closed, finishing the read loop
> org.apache.beam.sdk.io.kinesis.KinesisShardClosedException: Shard iterator 
> reached end of the shard: streamName=null, shardId=shard1
>   at 
> org.apache.beam.sdk.io.kinesis.ShardRecordsIterator.readNextBatch(ShardRecordsIterator.java:70)
>   at 
> org.apache.beam.sdk.io.kinesis.ShardReadersPool.readLoop(ShardReadersPool.java:121)
>   at 
> org.apache.beam.sdk.io.kinesis.ShardReadersPool.lambda$startReadingShards$0(ShardReadersPool.java:112)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>   at java.lang.Thread.run(Thread.java:748)
> Feb 01, 2018 11:24:23 PM org.apache.beam.sdk.io.kinesis.ShardReadersPool 
> readLoop
> INFO: Kinesis Shard read loop has finished
> Feb 01, 2018 11:24:23 PM org.apache.beam.sdk.io.kinesis.ShardReadersPool 
> readLoop
> INFO: Shard iterator for shard2 shard is closed, finishing the read loop
> org.apache.beam.sdk.io.kinesis.KinesisShardClosedException: Shard iterator 
> reached end of the shard: streamName=null, shardId=shard2
>   at 
> org.apache.beam.sdk.io.kinesis.ShardRecordsIterator.readNextBatch(ShardRecordsIterator.java:70)
>   at 
> org.apache.beam.sdk.io.kinesis.ShardReadersPool.readLoop(ShardReadersPool.java:121)
>   at 
> org.apache.beam.sdk.io.kinesis.ShardReadersPool.lambda$startReadingShards$0(ShardReadersPool.java:112)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>   at java.lang.Thread.run(Thread.java:748)
> Feb 01, 2018 11:24:23 PM org.apache.beam.sdk.io.kinesis.ShardReadersPool stop
> INFO: Closing shard iterators pool
> Feb 01, 2018 11:24:24 PM org.apache.beam.sdk.io.kinesis.ShardReadersPool 
> readLoop
> INFO: Kinesis Shard read loop has finished
> Feb 01, 2018 11:24:24 PM org.apache.beam.sdk.io.kinesis.ShardReadersPool 
> readLoop
> INFO: Shard iterator for 

[jira] [Assigned] (BEAM-3317) KinesisReaderTest is Flaky due to overadvanced watermarks

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3317?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles reassigned BEAM-3317:
-

Assignee: Alexey Romanenko  (was: Kenneth Knowles)

> KinesisReaderTest is Flaky due to overadvanced watermarks
> -
>
> Key: BEAM-3317
> URL: https://issues.apache.org/jira/browse/BEAM-3317
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Reporter: Thomas Groh
>Assignee: Alexey Romanenko
>Priority: Critical
>  Labels: flake, sickbay
>
> https://builds.apache.org/job/beam_PreCommit_Java_GradleBuild/392/testReport/junit/org.apache.beam.sdk.io.kinesis/KinesisReaderTest/watermarkAdvancesWhenEnoughRecordsReadRecently/
> org.junit.ComparisonFailure: expected:<[-290308-12-21T19:59:05.225]Z> but 
> was:<[1970-01-01T00:00:01.000]Z>



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-3598) kinesis.ShardReadersPoolTest.shouldStopReadersPoolAlsoWhenExceptionsOccurDuringStopping is flaky

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles reassigned BEAM-3598:
-

Assignee: Alexey Romanenko  (was: Kenneth Knowles)

> kinesis.ShardReadersPoolTest.shouldStopReadersPoolAlsoWhenExceptionsOccurDuringStopping
>  is flaky
> 
>
> Key: BEAM-3598
> URL: https://issues.apache.org/jira/browse/BEAM-3598
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Reporter: Thomas Groh
>Assignee: Alexey Romanenko
>Priority: Critical
>  Labels: flake, sickbay
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> shouldStopReadersPoolAlsoWhenExceptionsOccurDuringStopping fails due to 
> incomplete interactions with the mock.
> {{Wanted but not invoked: firstIterator.readNextBatch(); -> at 
> org.apache.beam.sdk.io.kinesis.ShardReadersPoolTest.shouldStopReadersPoolAlsoWhenExceptionsOccurDuringStopping(ShardReadersPoolTest.java:244)
>  However, there were other interactions with this mock: -> at 
> org.apache.beam.sdk.io.kinesis.ShardReadersPoolTest.shouldStopReadersPoolAlsoWhenExceptionsOccurDuringStopping(ShardReadersPoolTest.java:241)}}
>  
> [https://builds.apache.org/job/beam_PreCommit_Java_MavenInstall/org.apache.beam$beam-sdks-java-io-kinesis/17390/testReport/junit/org.apache.beam.sdk.io.kinesis/ShardReadersPoolTest/shouldStopReadersPoolAlsoWhenExceptionsOccurDuringStopping/]
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-3599) kinesis.ShardReadersPoolTest.shouldInterruptKinesisReadingAndStopShortly is flaky

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3599?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles reassigned BEAM-3599:
-

Assignee: Alexey Romanenko  (was: Kenneth Knowles)

> kinesis.ShardReadersPoolTest.shouldInterruptKinesisReadingAndStopShortly is 
> flaky
> -
>
> Key: BEAM-3599
> URL: https://issues.apache.org/jira/browse/BEAM-3599
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Reporter: Thomas Groh
>Assignee: Alexey Romanenko
>Priority: Critical
>  Labels: sickbay
>
> This appears to be unavoidable due to the construction of the test - the 
> amount of realtime that the thread waits is asserted to be less than one 
> second, but as the class doesn't consume a {{Ticker}} (or related way to 
> track elapsed time), we have no way of deterministically controlling that 
> value within the test.
>  
> Example failure: 
> [https://builds.apache.org/job/beam_PreCommit_Java_GradleBuild/1738/testReport/junit/org.apache.beam.sdk.io.kinesis/ShardReadersPoolTest/shouldInterruptKinesisReadingAndStopShortly/]
>  
> {{java.lang.AssertionError: Expecting: <4169L> to be less than: <1000L> at 
> org.apache.beam.sdk.io.kinesis.ShardReadersPoolTest.shouldInterruptKinesisReadingAndStopShortly(ShardReadersPoolTest.java:159)
>  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)}} 
>  
> The test could also do with a more precise error message (it looks like too 
> many elements were received based on the error message, rather than the 
> actual "took too long" failure.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-3599) kinesis.ShardReadersPoolTest.shouldInterruptKinesisReadingAndStopShortly is flaky

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3599?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-3599:
--
Labels: sickbay  (was: flake)

> kinesis.ShardReadersPoolTest.shouldInterruptKinesisReadingAndStopShortly is 
> flaky
> -
>
> Key: BEAM-3599
> URL: https://issues.apache.org/jira/browse/BEAM-3599
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Reporter: Thomas Groh
>Assignee: Alexey Romanenko
>Priority: Critical
>  Labels: sickbay
>
> This appears to be unavoidable due to the construction of the test - the 
> amount of realtime that the thread waits is asserted to be less than one 
> second, but as the class doesn't consume a {{Ticker}} (or related way to 
> track elapsed time), we have no way of deterministically controlling that 
> value within the test.
>  
> Example failure: 
> [https://builds.apache.org/job/beam_PreCommit_Java_GradleBuild/1738/testReport/junit/org.apache.beam.sdk.io.kinesis/ShardReadersPoolTest/shouldInterruptKinesisReadingAndStopShortly/]
>  
> {{java.lang.AssertionError: Expecting: <4169L> to be less than: <1000L> at 
> org.apache.beam.sdk.io.kinesis.ShardReadersPoolTest.shouldInterruptKinesisReadingAndStopShortly(ShardReadersPoolTest.java:159)
>  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)}} 
>  
> The test could also do with a more precise error message (it looks like too 
> many elements were received based on the error message, rather than the 
> actual "took too long" failure.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-3599) kinesis.ShardReadersPoolTest.shouldInterruptKinesisReadingAndStopShortly is flaky

[Kenneth Knowles (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16354237#comment-16354237
 ] 

Kenneth Knowles commented on BEAM-3599:
---

This is now sickbayed

> kinesis.ShardReadersPoolTest.shouldInterruptKinesisReadingAndStopShortly is 
> flaky
> -
>
> Key: BEAM-3599
> URL: https://issues.apache.org/jira/browse/BEAM-3599
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Reporter: Thomas Groh
>Assignee: Kenneth Knowles
>Priority: Critical
>  Labels: flake
>
> This appears to be unavoidable due to the construction of the test - the 
> amount of realtime that the thread waits is asserted to be less than one 
> second, but as the class doesn't consume a {{Ticker}} (or related way to 
> track elapsed time), we have no way of deterministically controlling that 
> value within the test.
>  
> Example failure: 
> [https://builds.apache.org/job/beam_PreCommit_Java_GradleBuild/1738/testReport/junit/org.apache.beam.sdk.io.kinesis/ShardReadersPoolTest/shouldInterruptKinesisReadingAndStopShortly/]
>  
> {{java.lang.AssertionError: Expecting: <4169L> to be less than: <1000L> at 
> org.apache.beam.sdk.io.kinesis.ShardReadersPoolTest.shouldInterruptKinesisReadingAndStopShortly(ShardReadersPoolTest.java:159)
>  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)}} 
>  
> The test could also do with a more precise error message (it looks like too 
> many elements were received based on the error message, rather than the 
> actual "took too long" failure.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-3625) DoFn.XxxParam does not work for Map and FlatMap

[Robert Bradshaw (JIRA)]

Robert Bradshaw created BEAM-3625:
-

 Summary: DoFn.XxxParam does not work for Map and FlatMap
 Key: BEAM-3625
 URL: https://issues.apache.org/jira/browse/BEAM-3625
 Project: Beam
  Issue Type: Bug
  Components: sdk-py-core
Reporter: Robert Bradshaw
Assignee: Robert Bradshaw






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-3624) Remove collapsible if statements

[Colm O hEigeartaigh (JIRA)]

Colm O hEigeartaigh created BEAM-3624:
-

 Summary: Remove collapsible if statements
 Key: BEAM-3624
 URL: https://issues.apache.org/jira/browse/BEAM-3624
 Project: Beam
  Issue Type: Improvement
  Components: sdk-java-core
Reporter: Colm O hEigeartaigh
Assignee: Colm O hEigeartaigh
 Fix For: 2.4.0


There are a few instances in the code where we could collapse if statements to 
make the code more concise.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] branch master updated (f3dba1a -> 2c01ad5)

[kenn]

This is an automated email from the ASF dual-hosted git repository.

kenn pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from f3dba1a  Merge pull request #4558: Fixes streaming mode recognition in 
Flink Runner
 add df95de6  Sickbay flaky KinesisIO tests
 new 2c01ad5  Merge pull request #4605: [BEAM-3598][BEAM-3599][BEAM-3605] 
Sickbay flaky KinesisIO tests

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../java/org/apache/beam/sdk/io/kinesis/ShardReadersPoolTest.java | 4 
 1 file changed, 4 insertions(+)

-- 
To stop receiving notification emails like this one, please contact
k...@apache.org.

[beam] 01/01: Merge pull request #4605: [BEAM-3598][BEAM-3599][BEAM-3605] Sickbay flaky KinesisIO tests

[kenn]

This is an automated email from the ASF dual-hosted git repository.

kenn pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 2c01ad57d27eae7c287f1a05d674d1f615fbe56c
Merge: f3dba1a df95de6
Author: Kenn Knowles 
AuthorDate: Tue Feb 6 09:33:51 2018 -0800

Merge pull request #4605: [BEAM-3598][BEAM-3599][BEAM-3605] Sickbay flaky 
KinesisIO tests

 .../java/org/apache/beam/sdk/io/kinesis/ShardReadersPoolTest.java | 4 
 1 file changed, 4 insertions(+)

-- 
To stop receiving notification emails like this one, please contact
k...@apache.org.

[jira] [Created] (BEAM-3623) Python SDK Fn harness should not log to stderr for portable jobs

[Henning Rohde (JIRA)]

Henning Rohde created BEAM-3623:
---

 Summary: Python SDK Fn harness should not log to stderr for 
portable jobs
 Key: BEAM-3623
 URL: https://issues.apache.org/jira/browse/BEAM-3623
 Project: Beam
  Issue Type: Bug
  Components: sdk-py-harness
Reporter: Henning Rohde
Assignee: Robert Bradshaw


We currently log both to stderr and the portable logging API, but we should do 
only the latter (and potentially fallback to stderr under failures). Otherwise, 
we're duplicating the log volume.

Location where the Fn Logger is added: 

https://github.com/apache/beam/blob/f3dba1af43b03e24419fb0f5b726479368959eef/sdks/python/apache_beam/runners/worker/sdk_worker_main.py#L95




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-3598) kinesis.ShardReadersPoolTest.shouldStopReadersPoolAlsoWhenExceptionsOccurDuringStopping is flaky

[Thomas Groh (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thomas Groh updated BEAM-3598:
--
Labels: flake sickbay  (was: flake)

> kinesis.ShardReadersPoolTest.shouldStopReadersPoolAlsoWhenExceptionsOccurDuringStopping
>  is flaky
> 
>
> Key: BEAM-3598
> URL: https://issues.apache.org/jira/browse/BEAM-3598
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Reporter: Thomas Groh
>Assignee: Kenneth Knowles
>Priority: Critical
>  Labels: flake, sickbay
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> shouldStopReadersPoolAlsoWhenExceptionsOccurDuringStopping fails due to 
> incomplete interactions with the mock.
> {{Wanted but not invoked: firstIterator.readNextBatch(); -> at 
> org.apache.beam.sdk.io.kinesis.ShardReadersPoolTest.shouldStopReadersPoolAlsoWhenExceptionsOccurDuringStopping(ShardReadersPoolTest.java:244)
>  However, there were other interactions with this mock: -> at 
> org.apache.beam.sdk.io.kinesis.ShardReadersPoolTest.shouldStopReadersPoolAlsoWhenExceptionsOccurDuringStopping(ShardReadersPoolTest.java:241)}}
>  
> [https://builds.apache.org/job/beam_PreCommit_Java_MavenInstall/org.apache.beam$beam-sdks-java-io-kinesis/17390/testReport/junit/org.apache.beam.sdk.io.kinesis/ShardReadersPoolTest/shouldStopReadersPoolAlsoWhenExceptionsOccurDuringStopping/]
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)