Jenkins build is back to normal : beam_PostRelease_NightlySnapshot #87

[Apache Jenkins Server]

See 

[jira] [Closed] (BEAM-3768) Compile error for Flink translation

[Aljoscha Krettek (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aljoscha Krettek closed BEAM-3768.
--
Resolution: Fixed
  Assignee: Thomas Groh  (was: Aljoscha Krettek)

This was resolved via https://github.com/apache/beam/pull/4787

> Compile error for Flink translation
> ---
>
> Key: BEAM-3768
> URL: https://issues.apache.org/jira/browse/BEAM-3768
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Reporter: Robert Bradshaw
>Assignee: Thomas Groh
>Priority: Blocker
> Fix For: 2.4.0
>
>
> 2018-03-01T21:22:58.234 [INFO] --- maven-compiler-plugin:3.7.0:compile 
> (default-compile) @ beam-runners-flink_2.11 ---
> 2018-03-01T21:22:58.258 [INFO] Changes detected - recompiling the module!
> 2018-03-01T21:22:58.259 [INFO] Compiling 75 source files to 
> /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Java_MavenInstall@2/src/runners/flink/target/classes
> 2018-03-01T21:22:59.555 [INFO] 
> /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Java_MavenInstall@2/src/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/io/DedupingOperator.java:
>  Some input files use or override a deprecated API.
> 2018-03-01T21:22:59.556 [INFO] 
> /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Java_MavenInstall@2/src/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/io/DedupingOperator.java:
>  Recompile with -Xlint:deprecation for details.
> 2018-03-01T21:22:59.556 [INFO] 
> /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Java_MavenInstall@2/src/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/SideInputInitializer.java:
>  Some input files use unchecked or unsafe operations.
> 2018-03-01T21:22:59.556 [INFO] 
> /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Java_MavenInstall@2/src/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/SideInputInitializer.java:
>  Recompile with -Xlint:unchecked for details.
> 2018-03-01T21:22:59.556 [INFO] 
> -
> 2018-03-01T21:22:59.556 [ERROR] COMPILATION ERROR : 
> 2018-03-01T21:22:59.556 [INFO] 
> -
> 2018-03-01T21:22:59.557 [ERROR] 
> /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Java_MavenInstall@2/src/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/DoFnOperator.java:[359,11]
>  cannot infer type arguments for 
> org.apache.beam.runners.core.ProcessFnRunner<>
>   reason: cannot infer type-variable(s) InputT,OutputT,RestrictionT
> (argument mismatch; org.apache.beam.runners.core.DoFnRunner cannot be 
> converted to 
> org.apache.beam.runners.core.SimpleDoFnRunner>,OutputT>)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PerformanceTests_Spark #1434

[Apache Jenkins Server]

See 


Changes:

[samuel.waggoner] [BEAM-3777] allow UDAFs to be indirect subclasses of CombineFn

--
[...truncated 90.95 KB...]
'apache-beam-testing:bqjob_r26c2eb02c7ec1980_0161fa11dbe3_1': Invalid schema
update. Field timestamp has changed type from TIMESTAMP to FLOAT

STDERR: 
/usr/lib/google-cloud-sdk/platform/bq/third_party/oauth2client/contrib/gce.py:73:
 UserWarning: You have requested explicit scopes to be used with a GCE service 
account.
Using this argument will have no effect on the actual scopes for tokens
requested. These scopes are set at VM instance creation time and
can't be overridden in the request.

  warnings.warn(_SCOPES_WARNING)
Upload complete.Waiting on bqjob_r26c2eb02c7ec1980_0161fa11dbe3_1 ... (0s) 
Current status: RUNNING 
 Waiting on 
bqjob_r26c2eb02c7ec1980_0161fa11dbe3_1 ... (0s) Current status: DONE   
2018-03-06 06:48:51,909 4e821e3a MainThread INFO Retrying exception running 
IssueRetryableCommand: Command returned a non-zero exit code.

2018-03-06 06:49:14,512 4e821e3a MainThread INFO Running: bq load 
--autodetect --source_format=NEWLINE_DELIMITED_JSON 
beam_performance.pkb_results 

2018-03-06 06:49:16,897 4e821e3a MainThread INFO Ran: {bq load --autodetect 
--source_format=NEWLINE_DELIMITED_JSON beam_performance.pkb_results 

  ReturnCode:1,  WallTime:0:02.37s,  CPU:0.30s,  MaxMemory:29068kb 
STDOUT: 

BigQuery error in load operation: Error processing job
'apache-beam-testing:bqjob_r19dbe69bff6b51c3_0161fa123d4a_1': Invalid schema
update. Field timestamp has changed type from TIMESTAMP to FLOAT

STDERR: 
/usr/lib/google-cloud-sdk/platform/bq/third_party/oauth2client/contrib/gce.py:73:
 UserWarning: You have requested explicit scopes to be used with a GCE service 
account.
Using this argument will have no effect on the actual scopes for tokens
requested. These scopes are set at VM instance creation time and
can't be overridden in the request.

  warnings.warn(_SCOPES_WARNING)
Upload complete.Waiting on bqjob_r19dbe69bff6b51c3_0161fa123d4a_1 ... (0s) 
Current status: RUNNING 
 Waiting on 
bqjob_r19dbe69bff6b51c3_0161fa123d4a_1 ... (0s) Current status: DONE   
2018-03-06 06:49:16,898 4e821e3a MainThread INFO Retrying exception running 
IssueRetryableCommand: Command returned a non-zero exit code.

2018-03-06 06:49:39,709 4e821e3a MainThread INFO Running: bq load 
--autodetect --source_format=NEWLINE_DELIMITED_JSON 
beam_performance.pkb_results 

2018-03-06 06:49:42,144 4e821e3a MainThread INFO Ran: {bq load --autodetect 
--source_format=NEWLINE_DELIMITED_JSON beam_performance.pkb_results 

  ReturnCode:1,  WallTime:0:02.42s,  CPU:0.29s,  MaxMemory:28712kb 
STDOUT: 

BigQuery error in load operation: Error processing job
'apache-beam-testing:bqjob_r3ba680b3b0955cc5_0161fa129fb6_1': Invalid schema
update. Field timestamp has changed type from TIMESTAMP to FLOAT

STDERR: 
/usr/lib/google-cloud-sdk/platform/bq/third_party/oauth2client/contrib/gce.py:73:
 UserWarning: You have requested explicit scopes to be used with a GCE service 
account.
Using this argument will have no effect on the actual scopes for tokens
requested. These scopes are set at VM instance creation time and
can't be overridden in the request.

  warnings.warn(_SCOPES_WARNING)
Upload complete.Waiting on bqjob_r3ba680b3b0955cc5_0161fa129fb6_1 ... (0s) 
Current status: RUNNING 
 Waiting on 
bqjob_r3ba680b3b0955cc5_0161fa129fb6_1 ... (0s) Current status: DONE   
2018-03-06 06:49:42,145 4e821e3a MainThread INFO Retrying exception running 
IssueRetryableCommand: Command returned a non-zero exit code.

2018-03-06 06:50:07,441 4e821e3a MainThread INFO Running: bq load 
--autodetect --source_format=NEWLINE_DELIMITED_JSON 
beam_performance.pkb_results 

2018-03-06 06:50:09,890 4e821e3a MainThread INFO Ran: {bq load --autodetect 
--source_format=NEWLINE_DELIMITED_JSON beam_performance.pkb_results 

  ReturnCode:1,  WallTime:0:02.44s,  CPU:0.31s,  MaxMemory:29068kb 

Jenkins build is back to normal : beam_PerformanceTests_JDBC #294

[Apache Jenkins Server]

See 

Jenkins build is back to normal : beam_PerformanceTests_TextIOIT #234

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PostRelease_NightlySnapshot #85

[Apache Jenkins Server]

See 


--
[...truncated 95.53 KB...]

   
Waiting on bqjob_r42712f179b502061_0161f9f25c6a_1 ... (58s) Current status: 
PENDING

   
Waiting on bqjob_r42712f179b502061_0161f9f25c6a_1 ... (59s) Current status: 
PENDING

   
Waiting on bqjob_r42712f179b502061_0161f9f25c6a_1 ... (60s) Current status: 
PENDING

   
Waiting on bqjob_r42712f179b502061_0161f9f25c6a_1 ... (61s) Current status: 
PENDING

   
Waiting on bqjob_r42712f179b502061_0161f9f25c6a_1 ... (62s) Current status: 
PENDING

   
Waiting on bqjob_r42712f179b502061_0161f9f25c6a_1 ... (63s) Current status: 
PENDING

   
Waiting on bqjob_r42712f179b502061_0161f9f25c6a_1 ... (64s) Current status: 
PENDING

   
Waiting on bqjob_r42712f179b502061_0161f9f25c6a_1 ... (65s) Current status: 
PENDING

   
Waiting on bqjob_r42712f179b502061_0161f9f25c6a_1 ... (66s) Current status: 
PENDING

   
Waiting on bqjob_r42712f179b502061_0161f9f25c6a_1 ... (67s) Current status: 
PENDING

   
Waiting on bqjob_r42712f179b502061_0161f9f25c6a_1 ... (68s) Current status: 
PENDING

   
Waiting on bqjob_r42712f179b502061_0161f9f25c6a_1 ... (69s) Current status: 
PENDING

   
Waiting on bqjob_r42712f179b502061_0161f9f25c6a_1 ... (70s) Current status: 
PENDING

   
Waiting on bqjob_r42712f179b502061_0161f9f25c6a_1 ... (71s) Current status: 
PENDING

   
Waiting on bqjob_r42712f179b502061_0161f9f25c6a_1 ... (72s) Current status: 
PENDING

   
Waiting on bqjob_r42712f179b502061_0161f9f25c6a_1 ... (73s) Current status: 
PENDING

   
Waiting on bqjob_r42712f179b502061_0161f9f25c6a_1 ... (74s) Current status: 
PENDING

   
Waiting on bqjob_r42712f179b502061_0161f9f25c6a_1 ... (75s) Current status: 
PENDING

   
Waiting on bqjob_r42712f179b502061_0161f9f25c6a_1 ... (76s) Current status: 
PENDING

   
Waiting on bqjob_r42712f179b502061_0161f9f25c6a_1 ... (77s) Current status: 
PENDING

   
Waiting on bqjob_r42712f179b502061_0161f9f25c6a_1 ... (78s) Current status: 
PENDING

   
Waiting on bqjob_r42712f179b502061_0161f9f25c6a_1 ... (79s) Current status: 
PENDING

   
Waiting on bqjob_r42712f179b502061_0161f9f25c6a_1 ... (80s) Current status: 
PENDING

   
Waiting on bqjob_r42712f179b502061_0161f9f25c6a_1 ... (81s) Current status: 
PENDING

   
Waiting on bqjob_r42712f179b502061_0161f9f25c6a_1 ... (82s) Current status: 
PENDING

   
Waiting on bqjob_r42712f179b502061_0161f9f25c6a_1 ... (83s) Current status: 
PENDING

   
Waiting on bqjob_r42712f179b502061_0161f9f25c6a_1 ... (84s) Current status: 
PENDING

Build failed in Jenkins: beam_PostCommit_Python_Verify #4368

[Apache Jenkins Server]

See 


Changes:

[samuel.waggoner] [BEAM-3777] allow UDAFs to be indirect subclasses of CombineFn

--
[...truncated 1.01 MB...]
copying apache_beam/runners/dataflow/internal/names.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/clients/__init__.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/internal/clients
copying apache_beam/runners/dataflow/internal/clients/dataflow/__init__.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/dataflow_v1b3_client.py 
-> apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/dataflow_v1b3_messages.py
 -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/message_matchers.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/message_matchers_test.py 
-> apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying apache_beam/runners/dataflow/native_io/__init__.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/dataflow/native_io/iobase.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/dataflow/native_io/iobase_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/dataflow/native_io/streaming_create.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/direct/__init__.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/bundle_factory.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/clock.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/consumer_tracking_pipeline_visitor.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/consumer_tracking_pipeline_visitor_test.py 
-> apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_metrics.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_metrics_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_runner.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_runner_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/evaluation_context.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/executor.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/helper_transforms.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/sdf_direct_runner.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/sdf_direct_runner_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/transform_evaluator.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/util.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/watermark_manager.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/experimental/__init__.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/experimental
copying apache_beam/runners/experimental/python_rpc_direct/__init__.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/experimental/python_rpc_direct
copying 
apache_beam/runners/experimental/python_rpc_direct/python_rpc_direct_runner.py 
-> apache-beam-2.5.0.dev0/apache_beam/runners/experimental/python_rpc_direct
copying apache_beam/runners/experimental/python_rpc_direct/server.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/experimental/python_rpc_direct
copying apache_beam/runners/job/__init__.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/job
copying apache_beam/runners/job/manager.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/job
copying apache_beam/runners/job/utils.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/job
copying apache_beam/runners/portability/__init__.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/portability/fn_api_runner.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/portability/fn_api_runner_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/portability
copying 

[jira] [Created] (BEAM-3785) [SQL] Add support for arrays

[Anton Kedin (JIRA)]

Anton Kedin created BEAM-3785:
-

 Summary: [SQL] Add support for arrays
 Key: BEAM-3785
 URL: https://issues.apache.org/jira/browse/BEAM-3785
 Project: Beam
  Issue Type: Improvement
  Components: dsl-sql
Reporter: Anton Kedin
Assignee: Anton Kedin


Support fields of Array type



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Jenkins build is back to normal : beam_PostRelease_NightlySnapshot #84

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PostCommit_Python_ValidatesRunner_Dataflow #1052

[Apache Jenkins Server]

See 


Changes:

[samuel.waggoner] [BEAM-3777] allow UDAFs to be indirect subclasses of CombineFn

--
[...truncated 103.29 KB...]
pipeline.run()
  File 
"
 line 102, in run
result = super(TestPipeline, self).run()
  File 
"
 line 367, in run
if test_runner_api and self._verify_runner_api_compatible():
  File 
"
 line 570, in _verify_runner_api_compatible
self.visit(Visitor())
  File 
"
 line 410, in visit
self._root_transform().visit(visitor, self, visited)
  File 
"
 line 764, in visit
part.visit(visitor, pipeline, visited)
  File 
"
 line 764, in visit
part.visit(visitor, pipeline, visited)
  File 
"
 line 764, in visit
part.visit(visitor, pipeline, visited)
  File 
"
 line 767, in visit
visitor.visit_transform(self)
  File 
"
 line 561, in visit_transform
enable_trace=False),
  File 
"
 line 206, in dumps
c = zlib.compress(s, 9)
  File 
"
 line 276, in signalhandler
raise TimedOutException()
TimedOutException: 'test_empty_singleton_side_input 
(apache_beam.transforms.sideinputs_test.SideInputsTest)'

==
ERROR: test_flattened_side_input 
(apache_beam.transforms.sideinputs_test.SideInputsTest)
--
Traceback (most recent call last):
  File 
"
 line 812, in run
test(orig)
  File 
"
 line 45, in __call__
return self.run(*arg, **kwarg)
  File 
"
 line 133, in run
self.runTest(result)
  File 
"
 line 151, in runTest
test(result)
  File "/usr/lib/python2.7/unittest/case.py", line 395, in __call__
return self.run(*args, **kwds)
  File "/usr/lib/python2.7/unittest/case.py", line 331, in run
testMethod()
  File 
"
 line 306, in test_flattened_side_input
pipeline.run()
  File 
"
 line 102, in run
result = super(TestPipeline, self).run()
  File 
"
 line 367, in run
if test_runner_api and self._verify_runner_api_compatible():
  File 
"
 line 570, in _verify_runner_api_compatible
self.visit(Visitor())
  File 
"
 line 410, in visit
self._root_transform().visit(visitor, self, visited)
  File 

Build failed in Jenkins: beam_PostRelease_NightlySnapshot #83

[Apache Jenkins Server]

See 


--
[...truncated 63.80 KB...]
utils
Verified UserScore.java

*
* Test: Runs the Mobile-Gaming Code with Direct runner
*


*
* Test: Running: UserScore example on DirectRunner
*

mvn compile exec:java -q 
-Dexec.mainClass=org.apache.beam.examples.complete.game.UserScore 
-Dexec.args="--tempLocation=gs://temp-storage-for-release-validation-tests/mobile-gaming/tmp
 --output=java-userscore-result.txt
 
--input=gs://temp-storage-for-release-validation-tests/mobile-gaming/5000_gaming_data.csv"
 -Pdirect-runner
Using maven /home/jenkins/tools/maven/apache-maven-3.5.2
Mar 06, 2018 5:01:36 AM org.apache.beam.sdk.io.FileBasedSource 
getEstimatedSizeBytes
INFO: Filepattern 
gs://temp-storage-for-release-validation-tests/mobile-gaming/5000_gaming_data.csv
 matched 1 files with total size 386400
Mar 06, 2018 5:01:36 AM org.apache.beam.sdk.io.FileBasedSource split
INFO: Splitting filepattern 
gs://temp-storage-for-release-validation-tests/mobile-gaming/5000_gaming_data.csv
 into bundles of size 96600 took 35 ms and produced 1 files and 4 bundles
Mar 06, 2018 5:01:41 AM 
org.apache.beam.sdk.io.WriteFiles$WriteShardsIntoTempFilesFn processElement
INFO: Opening writer 8ff3a290-2166-404a-aa1d-33e6ed680b9b for window 
org.apache.beam.sdk.transforms.windowing.GlobalWindow@406f3077 pane 
PaneInfo{isFirst=true, isLast=true, timing=ON_TIME, index=0, onTimeIndex=0} 
destination null
Mar 06, 2018 5:01:41 AM 
org.apache.beam.sdk.io.WriteFiles$WriteShardsIntoTempFilesFn processElement
INFO: Opening writer 3e502514-c9b9-4a39-8162-5188811ca8e2 for window 
org.apache.beam.sdk.transforms.windowing.GlobalWindow@406f3077 pane 
PaneInfo{isFirst=true, isLast=true, timing=ON_TIME, index=0, onTimeIndex=0} 
destination null
Mar 06, 2018 5:01:41 AM 
org.apache.beam.sdk.io.WriteFiles$WriteShardsIntoTempFilesFn processElement
INFO: Opening writer 2e52cbb5-3dd1-4dd0-a935-2cdb8c933cf0 for window 
org.apache.beam.sdk.transforms.windowing.GlobalWindow@406f3077 pane 
PaneInfo{isFirst=true, isLast=true, timing=ON_TIME, index=0, onTimeIndex=0} 
destination null
Mar 06, 2018 5:01:41 AM 
org.apache.beam.sdk.io.WriteFiles$WriteShardsIntoTempFilesFn processElement
INFO: Opening writer 708238da-6ce7-4aab-a661-571c23916ffc for window 
org.apache.beam.sdk.transforms.windowing.GlobalWindow@406f3077 pane 
PaneInfo{isFirst=true, isLast=true, timing=ON_TIME, index=0, onTimeIndex=0} 
destination null
Mar 06, 2018 5:01:41 AM org.apache.beam.sdk.io.FileBasedSink$Writer close
INFO: Successfully wrote temporary file 
/tmp/groovy-generated-7542997339843426654-tmpdir/word-count-beam/.temp-beam-2018-03-06_05-01-35-1/708238da-6ce7-4aab-a661-571c23916ffc
Mar 06, 2018 5:01:41 AM org.apache.beam.sdk.io.FileBasedSink$Writer close
INFO: Successfully wrote temporary file 
/tmp/groovy-generated-7542997339843426654-tmpdir/word-count-beam/.temp-beam-2018-03-06_05-01-35-1/2e52cbb5-3dd1-4dd0-a935-2cdb8c933cf0
Mar 06, 2018 5:01:41 AM org.apache.beam.sdk.io.FileBasedSink$Writer close
INFO: Successfully wrote temporary file 
/tmp/groovy-generated-7542997339843426654-tmpdir/word-count-beam/.temp-beam-2018-03-06_05-01-35-1/8ff3a290-2166-404a-aa1d-33e6ed680b9b
Mar 06, 2018 5:01:41 AM org.apache.beam.sdk.io.FileBasedSink$Writer close
INFO: Successfully wrote temporary file 
/tmp/groovy-generated-7542997339843426654-tmpdir/word-count-beam/.temp-beam-2018-03-06_05-01-35-1/3e502514-c9b9-4a39-8162-5188811ca8e2
Mar 06, 2018 5:01:41 AM 
org.apache.beam.sdk.io.WriteFiles$WriteShardsIntoTempFilesFn processElement
INFO: Opening writer 75297eb5-9901-4bb1-ac50-dab84bfa1e45 for window 
org.apache.beam.sdk.transforms.windowing.GlobalWindow@406f3077 pane 
PaneInfo{isFirst=true, isLast=true, timing=ON_TIME, index=0, onTimeIndex=0} 
destination null
Mar 06, 2018 5:01:41 AM org.apache.beam.sdk.io.FileBasedSink$Writer close
INFO: Successfully wrote temporary file 
/tmp/groovy-generated-7542997339843426654-tmpdir/word-count-beam/.temp-beam-2018-03-06_05-01-35-1/75297eb5-9901-4bb1-ac50-dab84bfa1e45
Mar 06, 2018 5:01:41 AM 
org.apache.beam.sdk.io.WriteFiles$FinalizeTempFileBundles$FinalizeFn process
INFO: Finalizing 5 file results
Mar 06, 2018 5:01:41 AM org.apache.beam.sdk.io.FileBasedSink$WriteOperation 
createMissingEmptyShards
INFO: Finalizing for destination null num shards 5.
Mar 06, 2018 5:01:41 AM org.apache.beam.sdk.io.FileBasedSink$WriteOperation 
moveToOutputFiles
INFO: Will copy temporary file 
FileResult{tempFilename=/tmp/groovy-generated-7542997339843426654-tmpdir/word-count-beam/.temp-beam-2018-03-06_05-01-35-1/8ff3a290-2166-404a-aa1d-33e6ed680b9b,
 shard=2, 
window=org.apache.beam.sdk.transforms.windowing.GlobalWindow@406f3077, 
paneInfo=PaneInfo{isFirst=true, isLast=true, timing=ON_TIME, index=0, 
onTimeIndex=0}} to final location 
/tmp/groovy-generated-7542997339843426654-tmpdir/word-count-beam/java-userscore-result.txt

[jira] [Commented] (BEAM-2980) BagState.isEmpty needs a tighter spec

[Kenneth Knowles (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16387298#comment-16387298
 ] 

Kenneth Knowles commented on BEAM-2980:
---

Since you also have BEAM-2975, are these resolved adequately 
[~mil...@google.com]?

> BagState.isEmpty needs a tighter spec
> -
>
> Key: BEAM-2980
> URL: https://issues.apache.org/jira/browse/BEAM-2980
> Project: Beam
>  Issue Type: Bug
>  Components: beam-model
>Reporter: Kenneth Knowles
>Assignee: Daniel Mills
>Priority: Major
>
> Consider the following:
> {code}
> BagState myBag = // empty
> ReadableState isMyBagEmpty = myBag.isEmpty();
> myBag.add(bizzle);
> bool empty = isMyBagEmpty.read();
> {code}
> Should {{empty}} be true or false? We need a consistent answer, across all 
> kinds of state, when snapshots are required.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-2980) BagState.isEmpty needs a tighter spec

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles reassigned BEAM-2980:
-

Assignee: Daniel Mills  (was: Kenneth Knowles)

> BagState.isEmpty needs a tighter spec
> -
>
> Key: BEAM-2980
> URL: https://issues.apache.org/jira/browse/BEAM-2980
> Project: Beam
>  Issue Type: Bug
>  Components: beam-model
>Reporter: Kenneth Knowles
>Assignee: Daniel Mills
>Priority: Major
>
> Consider the following:
> {code}
> BagState myBag = // empty
> ReadableState isMyBagEmpty = myBag.isEmpty();
> myBag.add(bizzle);
> bool empty = isMyBagEmpty.read();
> {code}
> Should {{empty}} be true or false? We need a consistent answer, across all 
> kinds of state, when snapshots are required.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-3069) Transform rehydrator registration should be idempotent

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3069?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles reassigned BEAM-3069:
-

Assignee: Thomas Groh  (was: Kenneth Knowles)

> Transform rehydrator registration should be idempotent
> --
>
> Key: BEAM-3069
> URL: https://issues.apache.org/jira/browse/BEAM-3069
> Project: Beam
>  Issue Type: Bug
>  Components: runner-core
>Reporter: Kenneth Knowles
>Assignee: Thomas Groh
>Priority: Major
>
> Currently, if the same URN is registered twice, it results in an exception. 
> Instead, if it is a duplicate registration (for example, caused by 
> double-shading of the service loader config) it should succeed. This requires 
> probably good equals methods for the rehydrators.
> Other minor issues: it will fail at load time and from then throw class not 
> found errors, which is extremely annoying. It should instead load lazily so 
> we can always get a good stack trace.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-2230) Core SDK ApiSurface should be only org.apache.beam.sdk and should be defined outside of the general ApiSurface class

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles reassigned BEAM-2230:
-

Assignee: (was: Kenneth Knowles)

> Core SDK ApiSurface should be only org.apache.beam.sdk and should be defined 
> outside of the general ApiSurface class
> 
>
> Key: BEAM-2230
> URL: https://issues.apache.org/jira/browse/BEAM-2230
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Kenneth Knowles
>Priority: Major
>
> Currenlty, ApiSurface.getSdkApiSurface() is highly specialized and also not 
> correct.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-2232) ApiSurface tests should run on the jar, not the un-shaded code.

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-2232:
--
Summary: ApiSurface tests should run on the jar, not the un-shaded code.  
(was: ApiSurface tests should run on the jar, not the pre-shaded code.)

> ApiSurface tests should run on the jar, not the un-shaded code.
> ---
>
> Key: BEAM-2232
> URL: https://issues.apache.org/jira/browse/BEAM-2232
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Kenneth Knowles
>Priority: Major
>
> Currently, errors in the core SDK ApiSurface definition and loading are 
> caught only by tests of the ApiSurface of a module that depends on it.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-2232) ApiSurface tests should run on the jar, not the pre-shaded code.

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles reassigned BEAM-2232:
-

Assignee: (was: Kenneth Knowles)

> ApiSurface tests should run on the jar, not the pre-shaded code.
> 
>
> Key: BEAM-2232
> URL: https://issues.apache.org/jira/browse/BEAM-2232
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Kenneth Knowles
>Priority: Major
>
> Currently, errors in the core SDK ApiSurface definition and loading are 
> caught only by tests of the ApiSurface of a module that depends on it.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-1189) Add guide for release verifiers in the release guide

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-1189:
--
Fix Version/s: (was: Not applicable)

> Add guide for release verifiers in the release guide
> 
>
> Key: BEAM-1189
> URL: https://issues.apache.org/jira/browse/BEAM-1189
> Project: Beam
>  Issue Type: Improvement
>  Components: website
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>Priority: Major
>
> This came up during the 0.4.0-incubating release discussion.
> There is this checklist: 
> http://incubator.apache.org/guides/releasemanagement.html#check-list
> And we could point to that but make more detailed Beam-specific instructions 
> on 
> http://beam.apache.org/contribute/release-guide/#vote-on-the-release-candidate
> And the template for the vote email should include a link to suggested 
> verification steps.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (BEAM-1189) Add guide for release verifiers in the release guide

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles resolved BEAM-1189.
---
   Resolution: Fixed
Fix Version/s: Not applicable

> Add guide for release verifiers in the release guide
> 
>
> Key: BEAM-1189
> URL: https://issues.apache.org/jira/browse/BEAM-1189
> Project: Beam
>  Issue Type: Improvement
>  Components: website
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>Priority: Major
> Fix For: Not applicable
>
>
> This came up during the 0.4.0-incubating release discussion.
> There is this checklist: 
> http://incubator.apache.org/guides/releasemanagement.html#check-list
> And we could point to that but make more detailed Beam-specific instructions 
> on 
> http://beam.apache.org/contribute/release-guide/#vote-on-the-release-candidate
> And the template for the vote email should include a link to suggested 
> verification steps.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Reopened] (BEAM-1189) Add guide for release verifiers in the release guide

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles reopened BEAM-1189:
---

> Add guide for release verifiers in the release guide
> 
>
> Key: BEAM-1189
> URL: https://issues.apache.org/jira/browse/BEAM-1189
> Project: Beam
>  Issue Type: Improvement
>  Components: website
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>Priority: Major
>
> This came up during the 0.4.0-incubating release discussion.
> There is this checklist: 
> http://incubator.apache.org/guides/releasemanagement.html#check-list
> And we could point to that but make more detailed Beam-specific instructions 
> on 
> http://beam.apache.org/contribute/release-guide/#vote-on-the-release-candidate
> And the template for the vote email should include a link to suggested 
> verification steps.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-3583) Apex Local Cluster Creation fails in TestE2EWordCount

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles reassigned BEAM-3583:
-

Assignee: (was: Kenneth Knowles)

> Apex Local Cluster Creation fails in TestE2EWordCount
> -
>
> Key: BEAM-3583
> URL: https://issues.apache.org/jira/browse/BEAM-3583
> Project: Beam
>  Issue Type: Bug
>  Components: runner-apex
>Reporter: Thomas Groh
>Priority: Major
>  Labels: sickbay
>
> Specifically, in gradle builds creating the local cluster fails with
> {{Caused by: java.lang.RuntimeException: java.io.FileNotFoundException: File 
> file:/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Java_GradleBuild/src/examples/java/target/com.datatorrent.stram.StramLocalCluster
>  does not exist}}
> This seems to happen reliably on almost all gradle builds: 
> https://builds.apache.org/job/beam_PreCommit_Java_GradleBuild/1597/testReport/junit/org.apache.beam.examples/WordCountIT/testE2EWordCount/



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-3595) Normalize URNs across SDKs and runners.

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles reassigned BEAM-3595:
-

Assignee: Eugene Kirpichov  (was: Kenneth Knowles)

> Normalize URNs across SDKs and runners.
> ---
>
> Key: BEAM-3595
> URL: https://issues.apache.org/jira/browse/BEAM-3595
> Project: Beam
>  Issue Type: Bug
>  Components: beam-model
>Reporter: Robert Bradshaw
>Assignee: Eugene Kirpichov
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-3595) Normalize URNs across SDKs and runners.

[Kenneth Knowles (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3595?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16387290#comment-16387290
 ] 

Kenneth Knowles commented on BEAM-3595:
---

Seems this is underway. Is this ticket now a dupe?

> Normalize URNs across SDKs and runners.
> ---
>
> Key: BEAM-3595
> URL: https://issues.apache.org/jira/browse/BEAM-3595
> Project: Beam
>  Issue Type: Bug
>  Components: beam-model
>Reporter: Robert Bradshaw
>Assignee: Eugene Kirpichov
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-3616) Ensure coders are updatable partially in a cluster (aka add versions to coders)

[Kenneth Knowles (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16387288#comment-16387288
 ] 

Kenneth Knowles commented on BEAM-3616:
---

I think this is, indeed, a core need. I expect there is another issue for this 
since [~reuvenlax] has been thinking about pipeline update / migrations for a 
long time and proposed a snapshot mechanism.

> Ensure coders are updatable partially in a cluster (aka add versions to 
> coders)
> ---
>
> Key: BEAM-3616
> URL: https://issues.apache.org/jira/browse/BEAM-3616
> Project: Beam
>  Issue Type: Improvement
>  Components: beam-model
>Reporter: Romain Manni-Bucau
>Priority: Major
> Fix For: 3.0.0
>
>
> Follow up ticket for the following discussion: 
> https://lists.apache.org/thread.html/f3b2daa740075cc39dc04f08d1eaacfcc2d550cecc04e27be024cf52@%3Cdev.beam.apache.org%3E
> Long story short: seems being able to update a live stream is a constraint 
> beam must address. This implies no change in coder serialization. To do it 
> the only way is to put a marker (like a version) otherwise the live update 
> will be broken if part of a cluster is updated only.
> This task is about ensuring we support it.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-3616) Ensure coders are updatable partially in a cluster (aka add versions to coders)

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3616?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-3616:
--
Component/s: (was: sdk-py-core)
 (was: sdk-java-core)
 (was: sdk-go)
 beam-model

> Ensure coders are updatable partially in a cluster (aka add versions to 
> coders)
> ---
>
> Key: BEAM-3616
> URL: https://issues.apache.org/jira/browse/BEAM-3616
> Project: Beam
>  Issue Type: Improvement
>  Components: beam-model
>Reporter: Romain Manni-Bucau
>Priority: Major
> Fix For: 3.0.0
>
>
> Follow up ticket for the following discussion: 
> https://lists.apache.org/thread.html/f3b2daa740075cc39dc04f08d1eaacfcc2d550cecc04e27be024cf52@%3Cdev.beam.apache.org%3E
> Long story short: seems being able to update a live stream is a constraint 
> beam must address. This implies no change in coder serialization. To do it 
> the only way is to put a marker (like a version) otherwise the live update 
> will be broken if part of a cluster is updated only.
> This task is about ensuring we support it.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-3616) Ensure coders are updatable partially in a cluster (aka add versions to coders)

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3616?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles reassigned BEAM-3616:
-

Assignee: (was: Kenneth Knowles)

> Ensure coders are updatable partially in a cluster (aka add versions to 
> coders)
> ---
>
> Key: BEAM-3616
> URL: https://issues.apache.org/jira/browse/BEAM-3616
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-go, sdk-java-core, sdk-py-core
>Reporter: Romain Manni-Bucau
>Priority: Major
> Fix For: 3.0.0
>
>
> Follow up ticket for the following discussion: 
> https://lists.apache.org/thread.html/f3b2daa740075cc39dc04f08d1eaacfcc2d550cecc04e27be024cf52@%3Cdev.beam.apache.org%3E
> Long story short: seems being able to update a live stream is a constraint 
> beam must address. This implies no change in coder serialization. To do it 
> the only way is to put a marker (like a version) otherwise the live update 
> will be broken if part of a cluster is updated only.
> This task is about ensuring we support it.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-3722) UNION should be implemented via flatten & distinct

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3722?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles reassigned BEAM-3722:
-

Assignee: (was: Kenneth Knowles)

> UNION should be implemented via flatten & distinct
> --
>
> Key: BEAM-3722
> URL: https://issues.apache.org/jira/browse/BEAM-3722
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql
>Reporter: Kenneth Knowles
>Priority: Major
>
> Currently, UNION is implemented as a set operator via CoGroupByKey when a 
> flatten + distinct will suffice. This also will be compatible with more types 
> of triggers.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-3721) UNION ALL should be implemented via Flatten

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles reassigned BEAM-3721:
-

Assignee: (was: Kenneth Knowles)

> UNION ALL should be implemented via Flatten
> ---
>
> Key: BEAM-3721
> URL: https://issues.apache.org/jira/browse/BEAM-3721
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql
>Reporter: Kenneth Knowles
>Priority: Major
>
> Currently, UNION ALL is implemented as an abstracted set operation that goes 
> through CoGroupByKey, when a flatten will do. This is also incompatible with 
> triggers that fire more than once.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-3771) Unable to write using AvroIO without schema

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3771?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-3771:
--
Component/s: (was: beam-model)
 io-java-avro

> Unable to write using AvroIO without schema
> ---
>
> Key: BEAM-3771
> URL: https://issues.apache.org/jira/browse/BEAM-3771
> Project: Beam
>  Issue Type: Bug
>  Components: io-java-avro
>Reporter: Darshan Mehta
>Assignee: Chamikara Jayalath
>Priority: Major
>
> I am working on a specific use case where I don't know the schema while 
> writing the GenericRecords' PCollection to File system. Here's how the use 
> case works:
>  * My dataflow listens to Pubsub's subscription and gets the message in this 
> format : 
> {code:java}
> // {"schema" : , "payload" : ""}
> {code}
>  * It then extracts the id, looks up schema registry and gets the schema for 
> a specific elelemt
>  * The payload is then deserialised into GenericRecord
>  * PCollection of these records is forwarded to BigQuery writer and it gets 
> written to BigQuery
>  * It then is passed to Storage writer that writes to file system using AvroIO
> Now, I am struggling with the last step as AvroIO expects a schema whereas I 
> do not know schema at compile time. All I have is a bunch of elements with 
> schema id embedded.
> Is there any way for AvroIO to write the records to FileSystem without 
> schema? If not, do I have any other alternatives (formats) to write to file 
> system?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-3771) Unable to write using AvroIO without schema

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3771?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles reassigned BEAM-3771:
-

Assignee: Chamikara Jayalath  (was: Kenneth Knowles)

> Unable to write using AvroIO without schema
> ---
>
> Key: BEAM-3771
> URL: https://issues.apache.org/jira/browse/BEAM-3771
> Project: Beam
>  Issue Type: Bug
>  Components: beam-model
>Reporter: Darshan Mehta
>Assignee: Chamikara Jayalath
>Priority: Major
>
> I am working on a specific use case where I don't know the schema while 
> writing the GenericRecords' PCollection to File system. Here's how the use 
> case works:
>  * My dataflow listens to Pubsub's subscription and gets the message in this 
> format : 
> {code:java}
> // {"schema" : , "payload" : ""}
> {code}
>  * It then extracts the id, looks up schema registry and gets the schema for 
> a specific elelemt
>  * The payload is then deserialised into GenericRecord
>  * PCollection of these records is forwarded to BigQuery writer and it gets 
> written to BigQuery
>  * It then is passed to Storage writer that writes to file system using AvroIO
> Now, I am struggling with the last step as AvroIO expects a schema whereas I 
> do not know schema at compile time. All I have is a bunch of elements with 
> schema id embedded.
> Is there any way for AvroIO to write the records to FileSystem without 
> schema? If not, do I have any other alternatives (formats) to write to file 
> system?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-3737) Key-aware batching function

[Kenneth Knowles (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16387286#comment-16387286
 ] 

Kenneth Knowles commented on BEAM-3737:
---

Actually I may have misinterpreted this, but I think [~robertwb] has context 
for this. Unassigning for now.

> Key-aware batching function
> ---
>
> Key: BEAM-3737
> URL: https://issues.apache.org/jira/browse/BEAM-3737
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-py-core
>Reporter: Chuan Yu Foo
>Priority: Major
>
> I have a CombineFn for which add_input has very large overhead. I would like 
> to batch the incoming elements into a large batch before each call to 
> add_input to reduce this overhead. In other words, I would like to do 
> something like: 
> {{elements | GroupByKey() | BatchElements() | CombineValues(MyCombineFn())}}
> Unfortunately, BatchElements is not key-aware, and can't be used after a 
> GroupByKey to batch elements per key. I'm working around this by doing the 
> batching within CombineValues, which makes the CombineFn rather messy. It 
> would be nice if there were a key-aware BatchElements transform which could 
> be used in this context.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-3737) Key-aware batching function

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-3737:
--
Component/s: (was: beam-model)
 sdk-py-core

> Key-aware batching function
> ---
>
> Key: BEAM-3737
> URL: https://issues.apache.org/jira/browse/BEAM-3737
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-py-core
>Reporter: Chuan Yu Foo
>Priority: Major
>
> I have a CombineFn for which add_input has very large overhead. I would like 
> to batch the incoming elements into a large batch before each call to 
> add_input to reduce this overhead. In other words, I would like to do 
> something like: 
> {{elements | GroupByKey() | BatchElements() | CombineValues(MyCombineFn())}}
> Unfortunately, BatchElements is not key-aware, and can't be used after a 
> GroupByKey to batch elements per key. I'm working around this by doing the 
> batching within CombineValues, which makes the CombineFn rather messy. It 
> would be nice if there were a key-aware BatchElements transform which could 
> be used in this context.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-3386) Dependency conflict when Calcite is included in a project.

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-3386:
--
Affects Version/s: 2.3.0

> Dependency conflict when Calcite is included in a project.
> --
>
> Key: BEAM-3386
> URL: https://issues.apache.org/jira/browse/BEAM-3386
> Project: Beam
>  Issue Type: Bug
>  Components: dsl-sql
>Affects Versions: 2.2.0, 2.3.0
>Reporter: Austin Haas
>Assignee: Kenneth Knowles
>Priority: Major
>
> When Calcite (v. 1.13.0) is included in a project that also includes Beam and 
> the Beam SQL extension, then the following error is thrown when trying to run 
> Beam code.
> ClassCastException 
> org.apache.beam.sdk.extensions.sql.impl.planner.BeamRelDataTypeSystem cannot 
> be cast to org.apache.calcite.rel.type.RelDataTypeSystem
> org.apache.calcite.jdbc.CalciteConnectionImpl. 
> (CalciteConnectionImpl.java:120)
> 
> org.apache.calcite.jdbc.CalciteJdbc41Factory$CalciteJdbc41Connection. 
> (CalciteJdbc41Factory.java:114)
> org.apache.calcite.jdbc.CalciteJdbc41Factory.newConnection 
> (CalciteJdbc41Factory.java:59)
> org.apache.calcite.jdbc.CalciteJdbc41Factory.newConnection 
> (CalciteJdbc41Factory.java:44)
> org.apache.calcite.jdbc.CalciteFactory.newConnection 
> (CalciteFactory.java:53)
> org.apache.calcite.avatica.UnregisteredDriver.connect 
> (UnregisteredDriver.java:138)
> java.sql.DriverManager.getConnection (DriverManager.java:664)
> java.sql.DriverManager.getConnection (DriverManager.java:208)
> 
> org.apache.beam.sdks.java.extensions.sql.repackaged.org.apache.calcite.tools.Frameworks.withPrepare
>  (Frameworks.java:145)
> 
> org.apache.beam.sdks.java.extensions.sql.repackaged.org.apache.calcite.tools.Frameworks.withPlanner
>  (Frameworks.java:106)
> 
> org.apache.beam.sdks.java.extensions.sql.repackaged.org.apache.calcite.prepare.PlannerImpl.ready
>  (PlannerImpl.java:140)
> 
> org.apache.beam.sdks.java.extensions.sql.repackaged.org.apache.calcite.prepare.PlannerImpl.parse
>  (PlannerImpl.java:170)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-3386) Dependency conflict when Calcite is included in a project.

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-3386:
--
Priority: Critical  (was: Major)

> Dependency conflict when Calcite is included in a project.
> --
>
> Key: BEAM-3386
> URL: https://issues.apache.org/jira/browse/BEAM-3386
> Project: Beam
>  Issue Type: Bug
>  Components: dsl-sql
>Affects Versions: 2.2.0, 2.3.0
>Reporter: Austin Haas
>Priority: Critical
>
> When Calcite (v. 1.13.0) is included in a project that also includes Beam and 
> the Beam SQL extension, then the following error is thrown when trying to run 
> Beam code.
> ClassCastException 
> org.apache.beam.sdk.extensions.sql.impl.planner.BeamRelDataTypeSystem cannot 
> be cast to org.apache.calcite.rel.type.RelDataTypeSystem
> org.apache.calcite.jdbc.CalciteConnectionImpl. 
> (CalciteConnectionImpl.java:120)
> 
> org.apache.calcite.jdbc.CalciteJdbc41Factory$CalciteJdbc41Connection. 
> (CalciteJdbc41Factory.java:114)
> org.apache.calcite.jdbc.CalciteJdbc41Factory.newConnection 
> (CalciteJdbc41Factory.java:59)
> org.apache.calcite.jdbc.CalciteJdbc41Factory.newConnection 
> (CalciteJdbc41Factory.java:44)
> org.apache.calcite.jdbc.CalciteFactory.newConnection 
> (CalciteFactory.java:53)
> org.apache.calcite.avatica.UnregisteredDriver.connect 
> (UnregisteredDriver.java:138)
> java.sql.DriverManager.getConnection (DriverManager.java:664)
> java.sql.DriverManager.getConnection (DriverManager.java:208)
> 
> org.apache.beam.sdks.java.extensions.sql.repackaged.org.apache.calcite.tools.Frameworks.withPrepare
>  (Frameworks.java:145)
> 
> org.apache.beam.sdks.java.extensions.sql.repackaged.org.apache.calcite.tools.Frameworks.withPlanner
>  (Frameworks.java:106)
> 
> org.apache.beam.sdks.java.extensions.sql.repackaged.org.apache.calcite.prepare.PlannerImpl.ready
>  (PlannerImpl.java:140)
> 
> org.apache.beam.sdks.java.extensions.sql.repackaged.org.apache.calcite.prepare.PlannerImpl.parse
>  (PlannerImpl.java:170)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-3386) Dependency conflict when Calcite is included in a project.

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles reassigned BEAM-3386:
-

Assignee: (was: Kenneth Knowles)

> Dependency conflict when Calcite is included in a project.
> --
>
> Key: BEAM-3386
> URL: https://issues.apache.org/jira/browse/BEAM-3386
> Project: Beam
>  Issue Type: Bug
>  Components: dsl-sql
>Affects Versions: 2.2.0, 2.3.0
>Reporter: Austin Haas
>Priority: Major
>
> When Calcite (v. 1.13.0) is included in a project that also includes Beam and 
> the Beam SQL extension, then the following error is thrown when trying to run 
> Beam code.
> ClassCastException 
> org.apache.beam.sdk.extensions.sql.impl.planner.BeamRelDataTypeSystem cannot 
> be cast to org.apache.calcite.rel.type.RelDataTypeSystem
> org.apache.calcite.jdbc.CalciteConnectionImpl. 
> (CalciteConnectionImpl.java:120)
> 
> org.apache.calcite.jdbc.CalciteJdbc41Factory$CalciteJdbc41Connection. 
> (CalciteJdbc41Factory.java:114)
> org.apache.calcite.jdbc.CalciteJdbc41Factory.newConnection 
> (CalciteJdbc41Factory.java:59)
> org.apache.calcite.jdbc.CalciteJdbc41Factory.newConnection 
> (CalciteJdbc41Factory.java:44)
> org.apache.calcite.jdbc.CalciteFactory.newConnection 
> (CalciteFactory.java:53)
> org.apache.calcite.avatica.UnregisteredDriver.connect 
> (UnregisteredDriver.java:138)
> java.sql.DriverManager.getConnection (DriverManager.java:664)
> java.sql.DriverManager.getConnection (DriverManager.java:208)
> 
> org.apache.beam.sdks.java.extensions.sql.repackaged.org.apache.calcite.tools.Frameworks.withPrepare
>  (Frameworks.java:145)
> 
> org.apache.beam.sdks.java.extensions.sql.repackaged.org.apache.calcite.tools.Frameworks.withPlanner
>  (Frameworks.java:106)
> 
> org.apache.beam.sdks.java.extensions.sql.repackaged.org.apache.calcite.prepare.PlannerImpl.ready
>  (PlannerImpl.java:140)
> 
> org.apache.beam.sdks.java.extensions.sql.repackaged.org.apache.calcite.prepare.PlannerImpl.parse
>  (PlannerImpl.java:170)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostRelease_NightlySnapshot #82

[Apache Jenkins Server]

See 


--
[...truncated 90.80 KB...]
team (ApricotPossum, num members: 18, starting at: 1520307038333, expires in: 
28, robot: Robot-14) is too old; replacing.
start time: 1520307038333, end time: 1520308718333, current time:1520308718618
[-(ApricotPossum, num members: 18, starting at: 1520307038333, expires in: 28, 
robot: Robot-14)]
[+(AlmondAntechinus, num members: 8, starting at: 1520308718618, expires in: 
33, robot: null)]
...DELAY(506270,
 1)
{timestamp_ms=1520308346000}
late data for: Robot-10,MagentaWombat,10,1520308346000,2018-03-05 20:00:52.474
Introducing
 a parse error.

team (AquaKangaroo, num members: 7, starting at: 1520307038333, expires in: 33, 
robot: Robot-12) is too old; replacing.
start time: 1520307038333, end time: 1520309018333, current time:1520309018586
[-(AquaKangaroo, num members: 7, starting at: 1520307038333, expires in: 33, 
robot: Robot-12)]
[+(AmethystNumbat, num members: 5, starting at: 1520309018588, expires in: 32, 
robot: Robot-1)]

team (BananaAntechinus, num members: 5, starting at: 1520307038333, expires in: 
33, robot: null) is too old; replacing.
start time: 1520307038333, end time: 1520309018333, current time:1520309018588
[-(BananaAntechinus, num members: 5, starting at: 1520307038333, expires in: 
33, robot: null)]
[+(AntiqueBrassBandicoot, num members: 10, starting at: 1520309018589, expires 
in: 36, robot: null)]
..
team (AppleGreenCassowary, num members: 7, starting at: 1520307038333, expires 
in: 34, robot: null) is too old; replacing.
start time: 1520307038333, end time: 1520309078333, current time:1520309078379
[-(AppleGreenCassowary, num members: 7, starting at: 1520307038333, expires in: 
34, robot: null)]
[+(MagentaPossum, num members: 8, starting at: 1520309078380, expires in: 33, 
robot: Robot-3)]
...Introducing a parse error.
..DELAY(418272,
 1)
{timestamp_ms=1520308738000}
late data for: 
user4_AntiqueBrassAntechinus,AntiqueBrassAntechinus,1,1520308738000,2018-03-05 
20:05:56.478
...
team (MagentaWombat, num members: 11, starting at: 1520307038332, expires in: 
36, robot: Robot-10) is too old; replacing.
start time: 1520307038332, end time: 1520309198332, current time:1520309198599
[-(MagentaWombat, num members: 11, starting at: 1520307038332, expires in: 36, 
robot: Robot-10)]
[+(BarnRedWallaby, num members: 17, starting at: 1520309198600, expires in: 30, 
robot: null)]

team (BisqueBilby, num members: 10, starting at: 1520307038333, expires in: 36, 
robot: Robot-17) is too old; replacing.
start time: 1520307038333, end time: 1520309198333, current time:1520309198601
[-(BisqueBilby, num members: 10, starting at: 1520307038333, expires in: 36, 
robot: Robot-17)]
[+(AntiqueBrassNumbat, num members: 14, starting at: 1520309198602, expires in: 
20, robot: null)]
...
team (MagentaDingo, num members: 13, starting at: 1520307038333, expires in: 
39, robot: null) is too old; replacing.
start time: 1520307038333, end time: 1520309378333, current time:1520309378351
[-(MagentaDingo, num members: 13, starting at: 1520307038333, expires in: 39, 
robot: null)]
[+(BeigeNumbat, num members: 16, starting at: 1520309378351, expires in: 22, 
robot: Robot-17)]

team (AntiqueBrassAntechinus, num members: 8, starting at: 1520307038332, 
expires in: 39, robot: Robot-4) is too old; replacing.
start time: 1520307038332, end time: 1520309378332, current time:1520309378351
[-(AntiqueBrassAntechinus, num members: 8, starting at: 1520307038332, expires 
in: 39, robot: Robot-4)]
[+(AsparagusKookaburra, num members: 10, starting at: 1520309378351, expires 
in: 31, robot: null)]

[jira] [Assigned] (BEAM-3737) Key-aware batching function

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles reassigned BEAM-3737:
-

Assignee: (was: Kenneth Knowles)

> Key-aware batching function
> ---
>
> Key: BEAM-3737
> URL: https://issues.apache.org/jira/browse/BEAM-3737
> Project: Beam
>  Issue Type: New Feature
>  Components: beam-model
>Reporter: Chuan Yu Foo
>Priority: Major
>
> I have a CombineFn for which add_input has very large overhead. I would like 
> to batch the incoming elements into a large batch before each call to 
> add_input to reduce this overhead. In other words, I would like to do 
> something like: 
> {{elements | GroupByKey() | BatchElements() | CombineValues(MyCombineFn())}}
> Unfortunately, BatchElements is not key-aware, and can't be used after a 
> GroupByKey to batch elements per key. I'm working around this by doing the 
> batching within CombineValues, which makes the CombineFn rather messy. It 
> would be nice if there were a key-aware BatchElements transform which could 
> be used in this context.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-3718) ClassNotFoundException on CloudResourceManager$Builder

[Kenneth Knowles (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16387284#comment-16387284
 ] 

Kenneth Knowles commented on BEAM-3718:
---

It looks like it has to do with GCP generally but since you mentioned Pubsub 
I'm handing it to someone who knows a lot about it. [~chamikara] might be 
another good person.

> ClassNotFoundException on CloudResourceManager$Builder
> --
>
> Key: BEAM-3718
> URL: https://issues.apache.org/jira/browse/BEAM-3718
> Project: Beam
>  Issue Type: Bug
>  Components: io-java-gcp
>Reporter: Yunis Arif Said
>Assignee: Raghu Angadi
>Priority: Trivial
>
> In a spring boot application running google cloud dataflow code. The dataflow 
> takes data from google PubSub, transform incoming data and output result to 
> bigquery for storage. The code does not have any syntax errors. The problem 
> is when the application is run, the following exception is thrown. 
>  
> {code:java}
>  Exception in thread "main" java.lang.RuntimeException: Failed to construct 
> instance from factory method DataflowRunner#fromOptions(interface 
> org.apache.beam.sdk.options.PipelineOptions)
>  at 
> org.apache.beam.sdk.util.InstanceBuilder.buildFromMethod(InstanceBuilder.java:233)
>  at org.apache.beam.sdk.util.InstanceBuilder.build(InstanceBuilder.java:162)
>  at org.apache.beam.sdk.PipelineRunner.fromOptions(PipelineRunner.java:52)
>  at org.apache.beam.sdk.Pipeline.create(Pipeline.java:142)
>  at com.trackers.exlon.ExlonApplication.main(ExlonApplication.java:69)
>  
>  Caused by: java.lang.reflect.InvocationTargetException
>  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>  at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>  at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>  at java.lang.reflect.Method.invoke(Method.java:498)
>  at 
> org.apache.beam.sdk.util.InstanceBuilder.buildFromMethod(InstanceBuilder.java:222)
>  ... 4 more
> Caused by: java.lang.NoClassDefFoundError: 
> com/google/api/services/cloudresourcemanager/CloudResourceManager$Builder
>  at 
> org.apache.beam.sdk.extensions.gcp.options.GcpOptions$GcpTempLocationFactory.newCloudResourceManagerClient(GcpOptions.java:369)
>  at 
> org.apache.beam.sdk.extensions.gcp.options.GcpOptions$GcpTempLocationFactory.create(GcpOptions.java:240)
>  at 
> org.apache.beam.sdk.extensions.gcp.options.GcpOptions$GcpTempLocationFactory.create(GcpOptions.java:228)
>  at 
> org.apache.beam.sdk.options.ProxyInvocationHandler.returnDefaultHelper(ProxyInvocationHandler.java:592)
>  at 
> org.apache.beam.sdk.options.ProxyInvocationHandler.getDefault(ProxyInvocationHandler.java:533)
>  at 
> org.apache.beam.sdk.options.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:156)
>  at com.sun.proxy.$Proxy85.getGcpTempLocation(Unknown Source)
>  at 
> org.apache.beam.runners.dataflow.DataflowRunner.fromOptions(DataflowRunner.java:223)
>  ... 9 more
> Caused by: java.lang.ClassNotFoundException: 
> com.google.api.services.cloudresourcemanager.CloudResourceManager$Builder
>  at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
>  at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>  at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
>  at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>  ... 17 more
> {code}
> Maven dependency tree:
> {code:java}
> [INFO] +- 
> org.springframework.boot:spring-boot-starter-webflux:jar:2.0.0.RC1:compile
>  [INFO] | +- 
> org.springframework.boot:spring-boot-starter:jar:2.0.0.RC1:compile
>  [INFO] | | +- org.springframework.boot:spring-boot:jar:2.0.0.RC1:compile
>  [INFO] | | | \- org.springframework:spring-context:jar:5.0.3.RELEASE:compile
>  [INFO] | | | +- org.springframework:spring-aop:jar:5.0.3.RELEASE:compile
>  [INFO] | | | \- 
> org.springframework:spring-expression:jar:5.0.3.RELEASE:compile
>  [INFO] | | +- 
> org.springframework.boot:spring-boot-autoconfigure:jar:2.0.0.RC1:compile
>  [INFO] | | +- 
> org.springframework.boot:spring-boot-starter-logging:jar:2.0.0.RC1:compile
>  [INFO] | | | +- ch.qos.logback:logback-classic:jar:1.2.3:compile
>  [INFO] | | | | \- ch.qos.logback:logback-core:jar:1.2.3:compile
>  [INFO] | | | +- org.apache.logging.log4j:log4j-to-slf4j:jar:2.10.0:compile
>  [INFO] | | | | \- org.apache.logging.log4j:log4j-api:jar:2.10.0:compile
>  [INFO] | | | \- org.slf4j:jul-to-slf4j:jar:1.7.25:compile
>  [INFO] | | +- javax.annotation:javax.annotation-api:jar:1.3.1:compile
>  [INFO] | | \- org.yaml:snakeyaml:jar:1.19:runtime
>  [INFO] | +- 
> org.springframework.boot:spring-boot-starter-json:jar:2.0.0.RC1:compile
>  [INFO] | | +- 
> com.fasterxml.jackson.datatype:jackson-datatype-jdk8:jar:2.9.2:compile
>  [INFO] | | +- 
> 

[jira] [Assigned] (BEAM-3718) ClassNotFoundException on CloudResourceManager$Builder

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles reassigned BEAM-3718:
-

Assignee: Raghu Angadi  (was: Kenneth Knowles)

> ClassNotFoundException on CloudResourceManager$Builder
> --
>
> Key: BEAM-3718
> URL: https://issues.apache.org/jira/browse/BEAM-3718
> Project: Beam
>  Issue Type: Bug
>  Components: io-java-gcp
>Reporter: Yunis Arif Said
>Assignee: Raghu Angadi
>Priority: Trivial
>
> In a spring boot application running google cloud dataflow code. The dataflow 
> takes data from google PubSub, transform incoming data and output result to 
> bigquery for storage. The code does not have any syntax errors. The problem 
> is when the application is run, the following exception is thrown. 
>  
> {code:java}
>  Exception in thread "main" java.lang.RuntimeException: Failed to construct 
> instance from factory method DataflowRunner#fromOptions(interface 
> org.apache.beam.sdk.options.PipelineOptions)
>  at 
> org.apache.beam.sdk.util.InstanceBuilder.buildFromMethod(InstanceBuilder.java:233)
>  at org.apache.beam.sdk.util.InstanceBuilder.build(InstanceBuilder.java:162)
>  at org.apache.beam.sdk.PipelineRunner.fromOptions(PipelineRunner.java:52)
>  at org.apache.beam.sdk.Pipeline.create(Pipeline.java:142)
>  at com.trackers.exlon.ExlonApplication.main(ExlonApplication.java:69)
>  
>  Caused by: java.lang.reflect.InvocationTargetException
>  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>  at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>  at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>  at java.lang.reflect.Method.invoke(Method.java:498)
>  at 
> org.apache.beam.sdk.util.InstanceBuilder.buildFromMethod(InstanceBuilder.java:222)
>  ... 4 more
> Caused by: java.lang.NoClassDefFoundError: 
> com/google/api/services/cloudresourcemanager/CloudResourceManager$Builder
>  at 
> org.apache.beam.sdk.extensions.gcp.options.GcpOptions$GcpTempLocationFactory.newCloudResourceManagerClient(GcpOptions.java:369)
>  at 
> org.apache.beam.sdk.extensions.gcp.options.GcpOptions$GcpTempLocationFactory.create(GcpOptions.java:240)
>  at 
> org.apache.beam.sdk.extensions.gcp.options.GcpOptions$GcpTempLocationFactory.create(GcpOptions.java:228)
>  at 
> org.apache.beam.sdk.options.ProxyInvocationHandler.returnDefaultHelper(ProxyInvocationHandler.java:592)
>  at 
> org.apache.beam.sdk.options.ProxyInvocationHandler.getDefault(ProxyInvocationHandler.java:533)
>  at 
> org.apache.beam.sdk.options.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:156)
>  at com.sun.proxy.$Proxy85.getGcpTempLocation(Unknown Source)
>  at 
> org.apache.beam.runners.dataflow.DataflowRunner.fromOptions(DataflowRunner.java:223)
>  ... 9 more
> Caused by: java.lang.ClassNotFoundException: 
> com.google.api.services.cloudresourcemanager.CloudResourceManager$Builder
>  at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
>  at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>  at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
>  at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>  ... 17 more
> {code}
> Maven dependency tree:
> {code:java}
> [INFO] +- 
> org.springframework.boot:spring-boot-starter-webflux:jar:2.0.0.RC1:compile
>  [INFO] | +- 
> org.springframework.boot:spring-boot-starter:jar:2.0.0.RC1:compile
>  [INFO] | | +- org.springframework.boot:spring-boot:jar:2.0.0.RC1:compile
>  [INFO] | | | \- org.springframework:spring-context:jar:5.0.3.RELEASE:compile
>  [INFO] | | | +- org.springframework:spring-aop:jar:5.0.3.RELEASE:compile
>  [INFO] | | | \- 
> org.springframework:spring-expression:jar:5.0.3.RELEASE:compile
>  [INFO] | | +- 
> org.springframework.boot:spring-boot-autoconfigure:jar:2.0.0.RC1:compile
>  [INFO] | | +- 
> org.springframework.boot:spring-boot-starter-logging:jar:2.0.0.RC1:compile
>  [INFO] | | | +- ch.qos.logback:logback-classic:jar:1.2.3:compile
>  [INFO] | | | | \- ch.qos.logback:logback-core:jar:1.2.3:compile
>  [INFO] | | | +- org.apache.logging.log4j:log4j-to-slf4j:jar:2.10.0:compile
>  [INFO] | | | | \- org.apache.logging.log4j:log4j-api:jar:2.10.0:compile
>  [INFO] | | | \- org.slf4j:jul-to-slf4j:jar:1.7.25:compile
>  [INFO] | | +- javax.annotation:javax.annotation-api:jar:1.3.1:compile
>  [INFO] | | \- org.yaml:snakeyaml:jar:1.19:runtime
>  [INFO] | +- 
> org.springframework.boot:spring-boot-starter-json:jar:2.0.0.RC1:compile
>  [INFO] | | +- 
> com.fasterxml.jackson.datatype:jackson-datatype-jdk8:jar:2.9.2:compile
>  [INFO] | | +- 
> com.fasterxml.jackson.datatype:jackson-datatype-jsr310:jar:2.9.2:compile
>  [INFO] | | \- 
> com.fasterxml.jackson.module:jackson-module-parameter-names:jar:2.9.2:compile
>  

[jira] [Updated] (BEAM-3718) ClassNotFoundException on CloudResourceManager$Builder

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-3718:
--
Component/s: (was: sdk-java-core)
 io-java-gcp

> ClassNotFoundException on CloudResourceManager$Builder
> --
>
> Key: BEAM-3718
> URL: https://issues.apache.org/jira/browse/BEAM-3718
> Project: Beam
>  Issue Type: Bug
>  Components: io-java-gcp
>Reporter: Yunis Arif Said
>Assignee: Raghu Angadi
>Priority: Trivial
>
> In a spring boot application running google cloud dataflow code. The dataflow 
> takes data from google PubSub, transform incoming data and output result to 
> bigquery for storage. The code does not have any syntax errors. The problem 
> is when the application is run, the following exception is thrown. 
>  
> {code:java}
>  Exception in thread "main" java.lang.RuntimeException: Failed to construct 
> instance from factory method DataflowRunner#fromOptions(interface 
> org.apache.beam.sdk.options.PipelineOptions)
>  at 
> org.apache.beam.sdk.util.InstanceBuilder.buildFromMethod(InstanceBuilder.java:233)
>  at org.apache.beam.sdk.util.InstanceBuilder.build(InstanceBuilder.java:162)
>  at org.apache.beam.sdk.PipelineRunner.fromOptions(PipelineRunner.java:52)
>  at org.apache.beam.sdk.Pipeline.create(Pipeline.java:142)
>  at com.trackers.exlon.ExlonApplication.main(ExlonApplication.java:69)
>  
>  Caused by: java.lang.reflect.InvocationTargetException
>  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>  at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>  at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>  at java.lang.reflect.Method.invoke(Method.java:498)
>  at 
> org.apache.beam.sdk.util.InstanceBuilder.buildFromMethod(InstanceBuilder.java:222)
>  ... 4 more
> Caused by: java.lang.NoClassDefFoundError: 
> com/google/api/services/cloudresourcemanager/CloudResourceManager$Builder
>  at 
> org.apache.beam.sdk.extensions.gcp.options.GcpOptions$GcpTempLocationFactory.newCloudResourceManagerClient(GcpOptions.java:369)
>  at 
> org.apache.beam.sdk.extensions.gcp.options.GcpOptions$GcpTempLocationFactory.create(GcpOptions.java:240)
>  at 
> org.apache.beam.sdk.extensions.gcp.options.GcpOptions$GcpTempLocationFactory.create(GcpOptions.java:228)
>  at 
> org.apache.beam.sdk.options.ProxyInvocationHandler.returnDefaultHelper(ProxyInvocationHandler.java:592)
>  at 
> org.apache.beam.sdk.options.ProxyInvocationHandler.getDefault(ProxyInvocationHandler.java:533)
>  at 
> org.apache.beam.sdk.options.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:156)
>  at com.sun.proxy.$Proxy85.getGcpTempLocation(Unknown Source)
>  at 
> org.apache.beam.runners.dataflow.DataflowRunner.fromOptions(DataflowRunner.java:223)
>  ... 9 more
> Caused by: java.lang.ClassNotFoundException: 
> com.google.api.services.cloudresourcemanager.CloudResourceManager$Builder
>  at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
>  at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>  at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
>  at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>  ... 17 more
> {code}
> Maven dependency tree:
> {code:java}
> [INFO] +- 
> org.springframework.boot:spring-boot-starter-webflux:jar:2.0.0.RC1:compile
>  [INFO] | +- 
> org.springframework.boot:spring-boot-starter:jar:2.0.0.RC1:compile
>  [INFO] | | +- org.springframework.boot:spring-boot:jar:2.0.0.RC1:compile
>  [INFO] | | | \- org.springframework:spring-context:jar:5.0.3.RELEASE:compile
>  [INFO] | | | +- org.springframework:spring-aop:jar:5.0.3.RELEASE:compile
>  [INFO] | | | \- 
> org.springframework:spring-expression:jar:5.0.3.RELEASE:compile
>  [INFO] | | +- 
> org.springframework.boot:spring-boot-autoconfigure:jar:2.0.0.RC1:compile
>  [INFO] | | +- 
> org.springframework.boot:spring-boot-starter-logging:jar:2.0.0.RC1:compile
>  [INFO] | | | +- ch.qos.logback:logback-classic:jar:1.2.3:compile
>  [INFO] | | | | \- ch.qos.logback:logback-core:jar:1.2.3:compile
>  [INFO] | | | +- org.apache.logging.log4j:log4j-to-slf4j:jar:2.10.0:compile
>  [INFO] | | | | \- org.apache.logging.log4j:log4j-api:jar:2.10.0:compile
>  [INFO] | | | \- org.slf4j:jul-to-slf4j:jar:1.7.25:compile
>  [INFO] | | +- javax.annotation:javax.annotation-api:jar:1.3.1:compile
>  [INFO] | | \- org.yaml:snakeyaml:jar:1.19:runtime
>  [INFO] | +- 
> org.springframework.boot:spring-boot-starter-json:jar:2.0.0.RC1:compile
>  [INFO] | | +- 
> com.fasterxml.jackson.datatype:jackson-datatype-jdk8:jar:2.9.2:compile
>  [INFO] | | +- 
> com.fasterxml.jackson.datatype:jackson-datatype-jsr310:jar:2.9.2:compile
>  [INFO] | | \- 
> 

[jira] [Resolved] (BEAM-3527) org.apache.beam.sdk.metrics.DistributionResult mixes up min and max constructor args

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles resolved BEAM-3527.
---
   Resolution: Fixed
Fix Version/s: 2.4.0

Fixed by doing a rename and doc to make it clear what is going on.

>  org.apache.beam.sdk.metrics.DistributionResult mixes up min and max 
> constructor args
> -
>
> Key: BEAM-3527
> URL: https://issues.apache.org/jira/browse/BEAM-3527
> Project: Beam
>  Issue Type: Bug
>  Components: runner-core
>Reporter: Claire McGinty
>Assignee: Kenneth Knowles
>Priority: Minor
> Fix For: 2.4.0
>
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> [GitHub link to 
> class|https://github.com/apache/beam/blob/a39960b131674e6c1d120a971b91b37226c88414/sdks/java/core/src/main/java/org/apache/beam/sdk/metrics/DistributionResult.java]
> given the constructor:
>  
> {code:java}
> public static DistributionResult create(long sum, long count, long min, long 
> max){code}
>  
> the ZERO value
>  
> {code:java}
> public static final DistributionResult ZERO = create(0, 0, Long.MAX_VALUE, 
> Long.MIN_VALUE){code}
>  
> should be:
>  
> {code:java}
> public static final DistributionResult ZERO = create(0, 0, Long.MIN_VALUE, 
> Long.MAX_VALUE);{code}
>  
> unless this is intentional...?
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-3749) support customized trigger/accumulationMode in BeamSql

[Kenneth Knowles (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16387279#comment-16387279
 ] 

Kenneth Knowles commented on BEAM-3749:
---

It turns out that the needed feature exists and is this: 
{{Window.configure().triggering(trigger).discardingFiredPanes().withAllowedLateness(...)}}.

> support customized trigger/accumulationMode in BeamSql
> --
>
> Key: BEAM-3749
> URL: https://issues.apache.org/jira/browse/BEAM-3749
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql
>Reporter: Xu Mingmin
>Assignee: Xu Mingmin
>Priority: Major
> Fix For: 2.5.0
>
>  Time Spent: 2h 10m
>  Remaining Estimate: 0h
>
> Currently BeamSql use {{DefaultTrigger}} for aggregation operations. 
> By adding two options {{withTrigger(Trigger)}} and 
> {{withAccumulationMode(AccumulationMode)}}, developers can specify their own 
> aggregation strategies with BeamSql.
> [~xumingming] [~kedin] [~kenn] for any comments.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-3783) Streaming Beam SQL benchmarks on all of our runners

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-3783:
--
Labels: SQL bigdata cloud gsoc2018 java  (was: gsoc2018)

> Streaming Beam SQL benchmarks on all of our runners
> ---
>
> Key: BEAM-3783
> URL: https://issues.apache.org/jira/browse/BEAM-3783
> Project: Beam
>  Issue Type: New Feature
>  Components: examples-nexmark
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>Priority: Major
>  Labels: SQL, bigdata, cloud, gsoc2018, java
>
> Beam has a number of classic streaming SQL benchmarks known as "Nexmark" 
> coded up in both raw Java and also Beam SQL.
> So far, expanding functionality has been the focus of Beam SQL so there is 
> little known about performance - we know only that it is a pretty 
> straightforward mapping from SQL to Beam that should work OK a lot of the 
> time. It would be interesting to see where the bottlenecks are when these SQL 
> benchmarks are translated via Beam SQL into a Beam pipeline and then again 
> translated to the native capabilities of e.g. Spark and Flink.
> This project will require the ability to read, write, and run Java and SQL.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-3784) Enhance Apache Beam interpreter for Apache Zeppelin

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3784?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-3784:
--
Labels: SQL bigdata cloud gsoc2018 java  (was: gsoc2018)

> Enhance Apache Beam interpreter for Apache Zeppelin
> ---
>
> Key: BEAM-3784
> URL: https://issues.apache.org/jira/browse/BEAM-3784
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-ideas
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>Priority: Minor
>  Labels: SQL, bigdata, cloud, gsoc2018, java
>
> Apache Zeppelin includes an integration with Apache Beam: 
> https://zeppelin.apache.org/docs/0.7.0/interpreter/beam.html
> How well does this work for interactive exploration? Can this be enhanced to 
> support Beam SQL? What about unbounded data? Let's find out by exploring the 
> existing interpreter and enhancing it particularly for streaming SQL.
> This project will require the ability to read, write, and run Java and SQL. 
> You will come out of it with familiarity with two Apache big data projects 
> and lots of ideas!



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-3783) Streaming Beam SQL benchmarks on all of our runners

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-3783:
--
Priority: Major  (was: Minor)

> Streaming Beam SQL benchmarks on all of our runners
> ---
>
> Key: BEAM-3783
> URL: https://issues.apache.org/jira/browse/BEAM-3783
> Project: Beam
>  Issue Type: New Feature
>  Components: examples-nexmark
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>Priority: Major
>  Labels: gsoc2018
>
> Beam has a number of classic streaming SQL benchmarks known as "Nexmark" 
> coded up in both raw Java and also Beam SQL.
> So far, expanding functionality has been the focus of Beam SQL so there is 
> little known about performance - we know only that it is a pretty 
> straightforward mapping from SQL to Beam that should work OK a lot of the 
> time. It would be interesting to see where the bottlenecks are when these SQL 
> benchmarks are translated via Beam SQL into a Beam pipeline and then again 
> translated to the native capabilities of e.g. Spark and Flink.
> This project will require the ability to read, write, and run Java and SQL.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-3783) Streaming Beam SQL benchmarks on all of our runners

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-3783:
--
Priority: Minor  (was: Major)

> Streaming Beam SQL benchmarks on all of our runners
> ---
>
> Key: BEAM-3783
> URL: https://issues.apache.org/jira/browse/BEAM-3783
> Project: Beam
>  Issue Type: New Feature
>  Components: examples-nexmark
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>Priority: Minor
>  Labels: gsoc2018
>
> Beam has a number of classic streaming SQL benchmarks known as "Nexmark" 
> coded up in both raw Java and also Beam SQL.
> So far, expanding functionality has been the focus of Beam SQL so there is 
> little known about performance - we know only that it is a pretty 
> straightforward mapping from SQL to Beam that should work OK a lot of the 
> time. It would be interesting to see where the bottlenecks are when these SQL 
> benchmarks are translated via Beam SQL into a Beam pipeline and then again 
> translated to the native capabilities of e.g. Spark and Flink.
> This project will require the ability to read, write, and run Java and SQL.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-3784) Enhance Apache Beam interpreter for Apache Zeppelin

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3784?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles reassigned BEAM-3784:
-

Assignee: Kenneth Knowles

> Enhance Apache Beam interpreter for Apache Zeppelin
> ---
>
> Key: BEAM-3784
> URL: https://issues.apache.org/jira/browse/BEAM-3784
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-ideas
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>Priority: Major
>  Labels: gsoc2018
>
> Apache Zeppelin includes an integration with Apache Beam: 
> https://zeppelin.apache.org/docs/0.7.0/interpreter/beam.html
> How well does this work for interactive exploration? Can this be enhanced to 
> support Beam SQL? What about unbounded data? Let's find out by exploring the 
> existing interpreter and enhancing it particularly for streaming SQL.
> This project will require the ability to read, write, and run Java and SQL. 
> You will come out of it with familiarity with two Apache big data projects 
> and lots of ideas!



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-3783) Streaming Beam SQL benchmarks on all of our runners

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles reassigned BEAM-3783:
-

Assignee: Kenneth Knowles

> Streaming Beam SQL benchmarks on all of our runners
> ---
>
> Key: BEAM-3783
> URL: https://issues.apache.org/jira/browse/BEAM-3783
> Project: Beam
>  Issue Type: New Feature
>  Components: examples-nexmark
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>Priority: Major
>  Labels: gsoc2018
>
> Beam has a number of classic streaming SQL benchmarks known as "Nexmark" 
> coded up in both raw Java and also Beam SQL.
> So far, expanding functionality has been the focus of Beam SQL so there is 
> little known about performance - we know only that it is a pretty 
> straightforward mapping from SQL to Beam that should work OK a lot of the 
> time. It would be interesting to see where the bottlenecks are when these SQL 
> benchmarks are translated via Beam SQL into a Beam pipeline and then again 
> translated to the native capabilities of e.g. Spark and Flink.
> This project will require the ability to read, write, and run Java and SQL.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-3784) Enhance Apache Beam interpreter for Apache Zeppelin

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3784?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-3784:
--
Priority: Minor  (was: Major)

> Enhance Apache Beam interpreter for Apache Zeppelin
> ---
>
> Key: BEAM-3784
> URL: https://issues.apache.org/jira/browse/BEAM-3784
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-ideas
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>Priority: Minor
>  Labels: gsoc2018
>
> Apache Zeppelin includes an integration with Apache Beam: 
> https://zeppelin.apache.org/docs/0.7.0/interpreter/beam.html
> How well does this work for interactive exploration? Can this be enhanced to 
> support Beam SQL? What about unbounded data? Let's find out by exploring the 
> existing interpreter and enhancing it particularly for streaming SQL.
> This project will require the ability to read, write, and run Java and SQL. 
> You will come out of it with familiarity with two Apache big data projects 
> and lots of ideas!



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-3784) Enhance Apache Beam interpreter for Apache Zeppelin

[Kenneth Knowles (JIRA)]

Kenneth Knowles created BEAM-3784:
-

 Summary: Enhance Apache Beam interpreter for Apache Zeppelin
 Key: BEAM-3784
 URL: https://issues.apache.org/jira/browse/BEAM-3784
 Project: Beam
  Issue Type: New Feature
  Components: sdk-ideas
Reporter: Kenneth Knowles


Apache Zeppelin includes an integration with Apache Beam: 
https://zeppelin.apache.org/docs/0.7.0/interpreter/beam.html

How well does this work for interactive exploration? Can this be enhanced to 
support Beam SQL? What about unbounded data? Let's find out by exploring the 
existing interpreter and enhancing it particularly for streaming SQL.

This project will require the ability to read, write, and run Java and SQL. You 
will come out of it with familiarity with two Apache big data projects and lots 
of ideas!



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Python_Verify #4367

[Apache Jenkins Server]

See 


--
[...truncated 1.02 MB...]
copying apache_beam/runners/portability/fn_api_runner_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/portability/maptask_executor_runner.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/portability/maptask_executor_runner_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/portability/universal_local_runner.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/portability/universal_local_runner_main.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/portability/universal_local_runner_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/test/__init__.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/test
copying apache_beam/runners/worker/__init__.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/bundle_processor.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/data_plane.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/data_plane_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/log_handler.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/log_handler_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/logger.pxd -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/logger.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/logger_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/opcounters.pxd -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/opcounters.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/opcounters_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/operation_specs.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/operations.pxd -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/operations.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/sdk_worker.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/sdk_worker_main.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/sdk_worker_main_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/sdk_worker_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/sideinputs.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/sideinputs_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/statesampler.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/statesampler_fast.pyx -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/statesampler_slow.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/statesampler_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/testing/__init__.py -> 
apache-beam-2.5.0.dev0/apache_beam/testing
copying apache_beam/testing/pipeline_verifiers.py -> 
apache-beam-2.5.0.dev0/apache_beam/testing
copying apache_beam/testing/pipeline_verifiers_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/testing
copying apache_beam/testing/test_pipeline.py -> 
apache-beam-2.5.0.dev0/apache_beam/testing
copying apache_beam/testing/test_pipeline_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/testing
copying apache_beam/testing/test_stream.py -> 
apache-beam-2.5.0.dev0/apache_beam/testing
copying apache_beam/testing/test_stream_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/testing
copying apache_beam/testing/test_utils.py -> 
apache-beam-2.5.0.dev0/apache_beam/testing
copying apache_beam/testing/test_utils_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/testing
copying apache_beam/testing/util.py -> 
apache-beam-2.5.0.dev0/apache_beam/testing
copying apache_beam/testing/util_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/testing
copying apache_beam/testing/data/standard_coders.yaml -> 
apache-beam-2.5.0.dev0/apache_beam/testing/data
copying apache_beam/testing/data/trigger_transcripts.yaml -> 
apache-beam-2.5.0.dev0/apache_beam/testing/data
copying apache_beam/transforms/__init__.py -> 
apache-beam-2.5.0.dev0/apache_beam/transforms
copying 

[jira] [Created] (BEAM-3783) Streaming Beam SQL benchmarks on all of our runners

[Kenneth Knowles (JIRA)]

Kenneth Knowles created BEAM-3783:
-

 Summary: Streaming Beam SQL benchmarks on all of our runners
 Key: BEAM-3783
 URL: https://issues.apache.org/jira/browse/BEAM-3783
 Project: Beam
  Issue Type: New Feature
  Components: examples-nexmark
Reporter: Kenneth Knowles


Beam has a number of classic streaming SQL benchmarks known as "Nexmark" coded 
up in both raw Java and also Beam SQL.

So far, expanding functionality has been the focus of Beam SQL so there is 
little known about performance - we know only that it is a pretty 
straightforward mapping from SQL to Beam that should work OK a lot of the time. 
It would be interesting to see where the bottlenecks are when these SQL 
benchmarks are translated via Beam SQL into a Beam pipeline and then again 
translated to the native capabilities of e.g. Spark and Flink.

This project will require the ability to read, write, and run Java and SQL.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Jenkins build became unstable: beam_PostCommit_Java_ValidatesRunner_Flink #5163

[Apache Jenkins Server]

See 

[jira] [Closed] (BEAM-3777) Subclass of subclass of CombineFn does not work as UDAF

[Anton Kedin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3777?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Anton Kedin closed BEAM-3777.
-
   Resolution: Fixed
Fix Version/s: Not applicable

merged

> Subclass of subclass of CombineFn does not work as UDAF
> ---
>
> Key: BEAM-3777
> URL: https://issues.apache.org/jira/browse/BEAM-3777
> Project: Beam
>  Issue Type: Bug
>  Components: dsl-sql
>Reporter: Samuel Waggoner
>Assignee: Xu Mingmin
>Priority: Major
> Fix For: Not applicable
>
>  Time Spent: 5h
>  Remaining Estimate: 0h
>
> Our team has a use case where we want to use a subclass of a subclass of 
> CombineFn as a UDAF. This is not working. I have created a minimal example in 
> BeamSqlExample to demonstrate.
> Given the following class definitions:
> {code:java}
> public static class SquareSumSub extends SquareSum {}{code}
> {code:java}
> public static class SquareSum extends Combine.CombineFn Integer> {
>  @Override
>  public Integer createAccumulator() {
>  return 0;
>  }
>  @Override
>  public Integer addInput(Integer accumulator, Integer input) {
>  return accumulator + input * input;
>  }
>  @Override
>  public Integer mergeAccumulators(Iterable accumulators) {
>  int v = 0;
>  Iterator ite = accumulators.iterator();
>  while (ite.hasNext()) {
>  v += ite.next();
>  }
>  return v;
>  }
>  @Override
>  public Integer extractOutput(Integer accumulator) {
>  return accumulator;
>  }
> }{code}
> I try to use SquareSumSub:
> {code:java}
> //Case 1. run a simple SQL query over input PCollection with 
> BeamSql.simpleQuery;
> PCollection outputStream = inputTable.apply(
>  BeamSql.query("select squaresum(c1) as c1, c2, c3 from PCOLLECTION Group By 
> c2, c3")
>  .registerUdaf("squaresum", new SquareSumSub()));{code}
> I get the following exception:
> {code:java}
> Exception in thread "main" java.lang.IllegalStateException: 
> org.apache.calcite.tools.ValidationException: java.lang.ClassCastException: 
> java.lang.Class cannot be cast to java.lang.reflect.ParameterizedType
>  at 
> org.apache.beam.sdk.extensions.sql.QueryTransform.expand(QueryTransform.java:76)
>  at 
> org.apache.beam.sdk.extensions.sql.QueryTransform.expand(QueryTransform.java:47)
>  at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:537)
>  at org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:472)
>  at org.apache.beam.sdk.values.PCollection.apply(PCollection.java:286)
>  at 
> org.apache.beam.sdk.extensions.sql.example.BeamSqlExample.main(BeamSqlExample.java:71)
> Caused by: org.apache.calcite.tools.ValidationException: 
> java.lang.ClassCastException: java.lang.Class cannot be cast to 
> java.lang.reflect.ParameterizedType
>  at org.apache.calcite.prepare.PlannerImpl.validate(PlannerImpl.java:195)
>  at 
> org.apache.beam.sdk.extensions.sql.impl.planner.BeamQueryPlanner.validateNode(BeamQueryPlanner.java:173)
>  at 
> org.apache.beam.sdk.extensions.sql.impl.planner.BeamQueryPlanner.validateAndConvert(BeamQueryPlanner.java:153)
>  at 
> org.apache.beam.sdk.extensions.sql.impl.planner.BeamQueryPlanner.convertToBeamRel(BeamQueryPlanner.java:144)
>  at 
> org.apache.beam.sdk.extensions.sql.QueryTransform.expand(QueryTransform.java:73)
>  ... 5 more
> Caused by: java.lang.ClassCastException: java.lang.Class cannot be cast to 
> java.lang.reflect.ParameterizedType
>  at 
> org.apache.beam.sdk.extensions.sql.impl.interpreter.operator.UdafImpl$1.getType(UdafImpl.java:63)
>  at 
> org.apache.calcite.prepare.CalciteCatalogReader.toOp(CalciteCatalogReader.java:321)
>  at 
> org.apache.calcite.prepare.CalciteCatalogReader.access$000(CalciteCatalogReader.java:81)
>  at 
> org.apache.calcite.prepare.CalciteCatalogReader$3.apply(CalciteCatalogReader.java:312)
>  at 
> org.apache.calcite.prepare.CalciteCatalogReader$3.apply(CalciteCatalogReader.java:310)
>  at com.google.common.collect.Iterators$7.transform(Iterators.java:750)
>  at 
> com.google.common.collect.TransformedIterator.next(TransformedIterator.java:47)
>  at java.util.AbstractCollection.toArray(AbstractCollection.java:141)
>  at java.util.ArrayList.addAll(ArrayList.java:577)
>  at 
> org.apache.calcite.prepare.CalciteCatalogReader.lookupOperatorOverloads(CalciteCatalogReader.java:308)
>  at 
> org.apache.calcite.sql.util.ChainedSqlOperatorTable.lookupOperatorOverloads(ChainedSqlOperatorTable.java:72)
>  at 
> org.apache.calcite.sql.validate.SqlValidatorImpl.performUnconditionalRewrites(SqlValidatorImpl.java:1132)
>  at 
> org.apache.calcite.sql.validate.SqlValidatorImpl.performUnconditionalRewrites(SqlValidatorImpl.java:1117)
>  at 
> org.apache.calcite.sql.validate.SqlValidatorImpl.performUnconditionalRewrites(SqlValidatorImpl.java:1146)
>  at 
> 

[beam] 01/01: Merge pull request #4799: [BEAM-3777] allow UDAFs to be indirect subclasses of CombineFn

[kenn]

This is an automated email from the ASF dual-hosted git repository.

kenn pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 13361213a5c7ea159b6038327720273e91d96fe8
Merge: 1eb5f09 7086fd0
Author: Kenn Knowles 
AuthorDate: Mon Mar 5 19:48:32 2018 -0800

Merge pull request #4799: [BEAM-3777] allow UDAFs to be indirect subclasses 
of CombineFn

 .../sql/impl/interpreter/operator/UdafImpl.java| 20 -
 .../sdk/extensions/sql/BeamSqlDslUdfUdafTest.java  | 92 +-
 2 files changed, 106 insertions(+), 6 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
k...@apache.org.

[beam] branch master updated (1eb5f09 -> 1336121)

[kenn]

This is an automated email from the ASF dual-hosted git repository.

kenn pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 1eb5f09  Merge pull request #4785: Add Python lint check for calls to 
unittest.main
 add 7086fd0  [BEAM-3777] allow UDAFs to be indirect subclasses of CombineFn
 new 1336121  Merge pull request #4799: [BEAM-3777] allow UDAFs to be 
indirect subclasses of CombineFn

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../sql/impl/interpreter/operator/UdafImpl.java| 20 -
 .../sdk/extensions/sql/BeamSqlDslUdfUdafTest.java  | 92 +-
 2 files changed, 106 insertions(+), 6 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
k...@apache.org.

[jira] [Created] (BEAM-3782) Do not ignore FileSystem errors and document expected behavior

[Udi Meiri (JIRA)]

Udi Meiri created BEAM-3782:
---

 Summary: Do not ignore FileSystem errors and document expected 
behavior
 Key: BEAM-3782
 URL: https://issues.apache.org/jira/browse/BEAM-3782
 Project: Beam
  Issue Type: Bug
  Components: sdk-py-core
Reporter: Udi Meiri
Assignee: Udi Meiri


copy/rename:
 * it should be an error if the dst file exists
 * it should be an error if the src file doesn't exist

delete:
 * it should be an error if the file/dir doesn't exist

FileBasedSink.finalize_write:
 * should check (src, dst) pairs for existence:
 ** src only - regular rename
 ** dst only - skip rename
 ** both src and dst - if files don't match on metadata (checksum or size), 
delete dst then rename, otherwise delete src and skip rename
 ** neither exist - return error: bad state

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-3782) Do not ignore FileSystem errors and document expected behavior

[Udi Meiri (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Udi Meiri updated BEAM-3782:

Component/s: (was: sdk-py-core)
 sdk-java-core

> Do not ignore FileSystem errors and document expected behavior
> --
>
> Key: BEAM-3782
> URL: https://issues.apache.org/jira/browse/BEAM-3782
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Udi Meiri
>Assignee: Udi Meiri
>Priority: Major
>
> copy/rename:
>  * it should be an error if the dst file exists
>  * it should be an error if the src file doesn't exist
> delete:
>  * it should be an error if the file/dir doesn't exist
> FileBasedSink.finalize_write:
>  * should check (src, dst) pairs for existence:
>  ** src only - regular rename
>  ** dst only - skip rename
>  ** both src and dst - if files don't match on metadata (checksum or size), 
> delete dst then rename, otherwise delete src and skip rename
>  ** neither exist - return error: bad state
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostRelease_NightlySnapshot #80

[Apache Jenkins Server]

See 


--
[...truncated 521.16 KB...]
[INFO] Downloading from central: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-core/2.2.1/maven-core-2.2.1.pom
[INFO] Downloaded from central: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-core/2.2.1/maven-core-2.2.1.pom
 (12 kB at 506 kB/s)
[INFO] Downloading from central: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-settings/2.2.1/maven-settings-2.2.1.pom
[INFO] Downloaded from central: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-settings/2.2.1/maven-settings-2.2.1.pom
 (2.2 kB at 95 kB/s)
[INFO] Downloading from central: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-model/2.2.1/maven-model-2.2.1.pom
[INFO] Downloaded from central: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-model/2.2.1/maven-model-2.2.1.pom
 (3.2 kB at 83 kB/s)
[INFO] Downloading from central: 
https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-interpolation/1.11/plexus-interpolation-1.11.pom
[INFO] Downloaded from central: 
https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-interpolation/1.11/plexus-interpolation-1.11.pom
 (889 B at 33 kB/s)
[INFO] Downloading from central: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-plugin-parameter-documenter/2.2.1/maven-plugin-parameter-documenter-2.2.1.pom
[INFO] Downloaded from central: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-plugin-parameter-documenter/2.2.1/maven-plugin-parameter-documenter-2.2.1.pom
 (2.0 kB at 82 kB/s)
[INFO] Downloading from central: 
https://repo.maven.apache.org/maven2/org/slf4j/slf4j-jdk14/1.5.6/slf4j-jdk14-1.5.6.pom
[INFO] Downloaded from central: 
https://repo.maven.apache.org/maven2/org/slf4j/slf4j-jdk14/1.5.6/slf4j-jdk14-1.5.6.pom
 (1.9 kB at 83 kB/s)
[INFO] Downloading from central: 
https://repo.maven.apache.org/maven2/org/slf4j/slf4j-parent/1.5.6/slf4j-parent-1.5.6.pom
[INFO] Downloaded from central: 
https://repo.maven.apache.org/maven2/org/slf4j/slf4j-parent/1.5.6/slf4j-parent-1.5.6.pom
 (7.9 kB at 330 kB/s)
[INFO] Downloading from central: 
https://repo.maven.apache.org/maven2/org/slf4j/slf4j-api/1.5.6/slf4j-api-1.5.6.pom
[INFO] Downloaded from central: 
https://repo.maven.apache.org/maven2/org/slf4j/slf4j-api/1.5.6/slf4j-api-1.5.6.pom
 (3.0 kB at 96 kB/s)
[INFO] Downloading from central: 
https://repo.maven.apache.org/maven2/org/slf4j/jcl-over-slf4j/1.5.6/jcl-over-slf4j-1.5.6.pom
[INFO] Downloaded from central: 
https://repo.maven.apache.org/maven2/org/slf4j/jcl-over-slf4j/1.5.6/jcl-over-slf4j-1.5.6.pom
 (2.2 kB at 94 kB/s)
[INFO] Downloading from central: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-profile/2.2.1/maven-profile-2.2.1.pom
[INFO] Downloaded from central: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-profile/2.2.1/maven-profile-2.2.1.pom
 (2.2 kB at 94 kB/s)
[INFO] Downloading from central: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-artifact/2.2.1/maven-artifact-2.2.1.pom
[INFO] Downloaded from central: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-artifact/2.2.1/maven-artifact-2.2.1.pom
 (1.6 kB at 69 kB/s)
[INFO] Downloading from central: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-repository-metadata/2.2.1/maven-repository-metadata-2.2.1.pom
[INFO] Downloaded from central: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-repository-metadata/2.2.1/maven-repository-metadata-2.2.1.pom
 (1.9 kB at 81 kB/s)
[INFO] Downloading from central: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-error-diagnostics/2.2.1/maven-error-diagnostics-2.2.1.pom
[INFO] Downloaded from central: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-error-diagnostics/2.2.1/maven-error-diagnostics-2.2.1.pom
 (1.7 kB at 74 kB/s)
[INFO] Downloading from central: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-project/2.2.1/maven-project-2.2.1.pom
[INFO] Downloaded from central: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-project/2.2.1/maven-project-2.2.1.pom
 (2.8 kB at 121 kB/s)
[INFO] Downloading from central: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-artifact-manager/2.2.1/maven-artifact-manager-2.2.1.pom
[INFO] Downloaded from central: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-artifact-manager/2.2.1/maven-artifact-manager-2.2.1.pom
 (3.1 kB at 135 kB/s)
[INFO] Downloading from central: 
https://repo.maven.apache.org/maven2/backport-util-concurrent/backport-util-concurrent/3.1/backport-util-concurrent-3.1.pom
[INFO] Downloaded from central: 
https://repo.maven.apache.org/maven2/backport-util-concurrent/backport-util-concurrent/3.1/backport-util-concurrent-3.1.pom
 (880 B at 24 kB/s)
[INFO] Downloading from central: 

Build failed in Jenkins: beam_PostRelease_NightlySnapshot #79

[Apache Jenkins Server]

See 


--
[...truncated 323.30 KB...]
.Introducing
 a parse error.
..DELAY(397504,
 1)
{timestamp_ms=1520299956000}
late data for: Robot-0,AquaQuokka,2,1520299956000,2018-03-05 17:39:13.893
...
team (ApricotKoala, num members: 19, starting at: 1520299449534, expires in: 
20, robot: Robot-8) is too old; replacing.
start time: 1520299449534, end time: 1520300649534, current time:1520300649967
[-(ApricotKoala, num members: 19, starting at: 1520299449534, expires in: 20, 
robot: Robot-8)]
[+(AmethystNumbat, num members: 14, starting at: 1520300649968, expires in: 25, 
robot: null)]

team (AmberKookaburra, num members: 15, starting at: 1520299449535, expires in: 
20, robot: null) is too old; replacing.
start time: 1520299449535, end time: 1520300649535, current time:1520300649968
[-(AmberKookaburra, num members: 15, starting at: 1520299449535, expires in: 
20, robot: null)]
[+(AmethystCaneToad, num members: 7, starting at: 1520300649969, expires in: 
35, robot: Robot-5)]

team (ApricotEchidna, num members: 14, starting at: 1520299449535, expires in: 
20, robot: null) is too old; replacing.
start time: 1520299449535, end time: 1520300649535, current time:1520300649969
[-(ApricotEchidna, num members: 14, starting at: 1520299449535, expires in: 20, 
robot: null)]
[+(AliceBlueBilby, num members: 16, starting at: 1520300649969, expires in: 36, 
robot: null)]
DELAY(434105, 1)
{timestamp_ms=1520300222000}
late data for: 
user3_BisqueKookaburra,BisqueKookaburra,7,1520300222000,2018-03-05 17:44:16.535
.
team (BarnRedCockatoo, num members: 5, starting at: 1520299449535, expires in: 
21, robot: Robot-11) is too old; replacing.
start time: 1520299449535, end time: 1520300709535, current time:1520300709536
[-(BarnRedCockatoo, num members: 5, starting at: 1520299449535, expires in: 21, 
robot: Robot-11)]
[+(AntiqueBrassKangaroo, num members: 14, starting at: 1520300709536, expires 
in: 33, robot: null)]
...
team (AmethystKookaburra, num members: 13, starting at: 1520299449536, expires 
in: 22, robot: null) is too old; replacing.
start time: 1520299449536, end time: 1520300769536, current time:1520300769636
[-(AmethystKookaburra, num members: 13, starting at: 1520299449536, expires in: 
22, robot: null)]
[+(AlmondKangaroo, num members: 18, starting at: 1520300769636, expires in: 39, 
robot: null)]
.Introducing
 a parse error.
..DELAY(473803,
 1)
{timestamp_ms=1520300485000}
late data for: user2_AquaQuokka,AquaQuokka,13,1520300485000,2018-03-05 
17:49:19.301
...
team (ArmyGreenWombat, num members: 13, starting at: 1520299449536, expires in: 
27, robot: Robot-15) is too old; replacing.
start time: 1520299449536, end time: 1520301069536, current time:1520301070007
[-(ArmyGreenWombat, num members: 13, starting at: 1520299449536, expires in: 
27, robot: Robot-15)]
[+(AsparagusBilby, num members: 19, starting at: 1520301070009, expires in: 30, 

Build failed in Jenkins: beam_PostRelease_NightlySnapshot #78

[Apache Jenkins Server]

See 


--
[...truncated 310.29 KB...]

*
* Test: SUCCEED: UserScore successfully run on DirectRunners.
*


*
* Test: Running: HourlyTeamScore example on DirectRunner
*

mvn compile exec:java 
-Dexec.mainClass=org.apache.beam.examples.complete.game.HourlyTeamScore 
-Dexec.args="--tempLocation=gs://temp-storage-for-release-validation-tests/mobile-gaming/tmp
 --output=java-hourlyteamscore-result.txt
 
--input=gs://temp-storage-for-release-validation-tests/mobile-gaming/5000_gaming_data.csv"
 -Pdirect-runner
Using maven /home/jenkins/tools/maven/apache-maven-3.5.2
[INFO] Scanning for projects...
[INFO] 
[INFO] 
[INFO] Building word-count-beam 0.1
[INFO] 
[INFO] 
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ 
word-count-beam ---
[WARNING] Using platform encoding (UTF-8 actually) to copy filtered resources, 
i.e. build is platform dependent!
[INFO] skip non existing resourceDirectory 
/tmp/groovy-generated-4938518670156135371-tmpdir/word-count-beam/src/main/resources
[INFO] 
[INFO] --- maven-compiler-plugin:3.7.0:compile (default-compile) @ 
word-count-beam ---
[INFO] Nothing to compile - all classes are up to date
[INFO] 
[INFO] --- exec-maven-plugin:1.6.0:java (default-cli) @ word-count-beam ---
Mar 06, 2018 1:12:18 AM org.apache.beam.sdk.io.FileBasedSource 
getEstimatedSizeBytes
INFO: Filepattern 
gs://temp-storage-for-release-validation-tests/mobile-gaming/5000_gaming_data.csv
 matched 1 files with total size 386400
Mar 06, 2018 1:12:18 AM org.apache.beam.sdk.io.FileBasedSource split
INFO: Splitting filepattern 
gs://temp-storage-for-release-validation-tests/mobile-gaming/5000_gaming_data.csv
 into bundles of size 96600 took 38 ms and produced 1 files and 4 bundles
Mar 06, 2018 1:12:24 AM 
org.apache.beam.sdk.io.WriteFiles$WriteShardsIntoTempFilesFn processElement
INFO: Opening writer 56b18707-ef53-4e8f-be7b-e83632d54e74 for window 
[2015-11-17T00:00:00.000Z..2015-11-17T01:00:00.000Z) pane 
PaneInfo{isFirst=true, isLast=true, timing=ON_TIME, index=0, onTimeIndex=0} 
destination null
Mar 06, 2018 1:12:24 AM 
org.apache.beam.sdk.io.WriteFiles$WriteShardsIntoTempFilesFn processElement
INFO: Opening writer c5bd2387-a480-4e18-bd6b-0ae1a37e0d20 for window 
[2015-11-17T00:00:00.000Z..2015-11-17T01:00:00.000Z) pane 
PaneInfo{isFirst=true, isLast=true, timing=ON_TIME, index=0, onTimeIndex=0} 
destination null
Mar 06, 2018 1:12:24 AM 
org.apache.beam.sdk.io.WriteFiles$WriteShardsIntoTempFilesFn processElement
INFO: Opening writer d9497aab-3ed7-4298-96f3-a02c262f2ae4 for window 
[2015-11-17T00:00:00.000Z..2015-11-17T01:00:00.000Z) pane 
PaneInfo{isFirst=true, isLast=true, timing=ON_TIME, index=0, onTimeIndex=0} 
destination null
Mar 06, 2018 1:12:24 AM org.apache.beam.sdk.io.FileBasedSink$Writer close
INFO: Successfully wrote temporary file 
/tmp/groovy-generated-4938518670156135371-tmpdir/word-count-beam/.temp-beam-2018-03-06_01-12-14-0/d9497aab-3ed7-4298-96f3-a02c262f2ae4
Mar 06, 2018 1:12:24 AM org.apache.beam.sdk.io.FileBasedSink$Writer close
INFO: Successfully wrote temporary file 
/tmp/groovy-generated-4938518670156135371-tmpdir/word-count-beam/.temp-beam-2018-03-06_01-12-14-0/c5bd2387-a480-4e18-bd6b-0ae1a37e0d20
Mar 06, 2018 1:12:24 AM org.apache.beam.sdk.io.FileBasedSink$Writer close
INFO: Successfully wrote temporary file 
/tmp/groovy-generated-4938518670156135371-tmpdir/word-count-beam/.temp-beam-2018-03-06_01-12-14-0/56b18707-ef53-4e8f-be7b-e83632d54e74
Mar 06, 2018 1:12:24 AM 
org.apache.beam.sdk.io.WriteFiles$FinalizeTempFileBundles$FinalizeFn process
INFO: Finalizing 1 file results
Mar 06, 2018 1:12:24 AM org.apache.beam.sdk.io.FileBasedSink$WriteOperation 
moveToOutputFiles
INFO: Will copy temporary file 
FileResult{tempFilename=/tmp/groovy-generated-4938518670156135371-tmpdir/word-count-beam/.temp-beam-2018-03-06_01-12-14-0/d9497aab-3ed7-4298-96f3-a02c262f2ae4,
 shard=2, window=[2015-11-17T00:00:00.000Z..2015-11-17T01:00:00.000Z), 
paneInfo=PaneInfo{isFirst=true, isLast=true, timing=ON_TIME, index=0, 
onTimeIndex=0}} to final location 
/tmp/groovy-generated-4938518670156135371-tmpdir/word-count-beam/java-hourlyteamscore-result.txt
-2015-11-16 16:00:00.000-2015-11-16 17:00:00.000-2-of-3
Mar 06, 2018 1:12:24 AM 
org.apache.beam.sdk.io.WriteFiles$FinalizeTempFileBundles$FinalizeFn process
INFO: Finalizing 1 file results
Mar 06, 2018 1:12:24 AM org.apache.beam.sdk.io.FileBasedSink$WriteOperation 
moveToOutputFiles
INFO: Will copy temporary file 
FileResult{tempFilename=/tmp/groovy-generated-4938518670156135371-tmpdir/word-count-beam/.temp-beam-2018-03-06_01-12-14-0/56b18707-ef53-4e8f-be7b-e83632d54e74,
 shard=0, window=[2015-11-17T00:00:00.000Z..2015-11-17T01:00:00.000Z), 

[jira] [Commented] (BEAM-3780) Add a utility to instantiate a partially unknown coder

[Luke Cwik (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16387087#comment-16387087
 ] 

Luke Cwik commented on BEAM-3780:
-

It all depends on if the runner wants to do everything with actual types like 
Coder or deal with their portable representations.

It would make sense to try to keep everything in proto format to me until the 
execution step since that is when we will need to rehydrate everything that is 
relevant instead of roundtripping to java object then proto and then back to 
java object.

> Add a utility to instantiate a partially unknown coder
> --
>
> Key: BEAM-3780
> URL: https://issues.apache.org/jira/browse/BEAM-3780
> Project: Beam
>  Issue Type: Bug
>  Components: runner-core
>Reporter: Thomas Groh
>Assignee: Thomas Groh
>Priority: Major
>
> Coders must be understood by the SDK harness that is encoding or decoding the 
> associated elements. However, the pipeline runner is capable of constructing 
> partial coders, where an unknown coder is replaced with a ByteArrayCoder. It 
> then can decompose the portions of elements it is aware of, without having to 
> understand the custom element encodings.
>  
> This should go in CoderTranslation, as an alternative to the full-fidelity 
> rehydration of a coder.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Python_ValidatesRunner_Dataflow #1050

[Apache Jenkins Server]

See 


Changes:

[ehudm] Add Python lint check for calls to unittest.main.

[github] Fixing formatting bug in filebasedsink.py.

[github] Fix lint issue.

--
[...truncated 126.50 KB...]
  File 
"
 line 227, in view_as
view = cls(self._flags)
  File 
"
 line 150, in __init__
parser = _BeamArgumentParser()
  File "/usr/lib/python2.7/argparse.py", line 1587, in __init__
self._optionals = add_group(_('optional arguments'))
  File "/usr/lib/python2.7/gettext.py", line 581, in gettext
return dgettext(_current_domain, message)
  File "/usr/lib/python2.7/gettext.py", line 545, in dgettext
codeset=_localecodesets.get(domain))
  File "/usr/lib/python2.7/gettext.py", line 480, in translation
mofiles = find(domain, localedir, languages, all=1)
  File "/usr/lib/python2.7/gettext.py", line 450, in find
'LC_MESSAGES', '%s.mo' % domain)
  File 
"
 line 75, in join
if b.startswith('/'):
  File 
"
 line 276, in signalhandler
raise TimedOutException()
TimedOutException: 'test_flattened_side_input 
(apache_beam.transforms.sideinputs_test.SideInputsTest)'

==
ERROR: test_iterable_side_input 
(apache_beam.transforms.sideinputs_test.SideInputsTest)
--
Traceback (most recent call last):
  File 
"
 line 812, in run
test(orig)
  File 
"
 line 45, in __call__
return self.run(*arg, **kwarg)
  File 
"
 line 133, in run
self.runTest(result)
  File 
"
 line 151, in runTest
test(result)
  File "/usr/lib/python2.7/unittest/case.py", line 395, in __call__
return self.run(*args, **kwds)
  File "/usr/lib/python2.7/unittest/case.py", line 331, in run
testMethod()
  File 
"
 line 177, in test_iterable_side_input
assert_that(result, equal_to([3, 4, 6, 8]))
  File 
"
 line 152, in assert_that
actual | AssertThat()  # pylint: disable=expression-not-assigned
  File 
"
 line 107, in __or__
return self.pipeline.apply(ptransform, self)
  File 
"
 line 479, in apply
pvalueish_result = self.runner.apply(transform, pvalueish)
  File 
"
 line 174, in apply
return m(transform, input)
  File 
"
 line 180, in apply_PTransform
return transform.expand(input)
  File 
"
 line 147, in expand
| "Match" >> Map(matcher))
  File 
"
 line 820, in __ror__
return self.transform.__ror__(pvalueish, self.label)
  File 
"
 line 488, in __ror__
result = p.apply(self, pvalueish, label)
  File 

[jira] [Commented] (BEAM-3780) Add a utility to instantiate a partially unknown coder

[Thomas Groh (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16387074#comment-16387074
 ] 

Thomas Groh commented on BEAM-3780:
---

The reason I'm not super certain about using {{LengthPrefixUnknownCoders}} is 
due to the point at which the runner is attempting to construct the coder; it 
may have reference to the {{RemoteGrpcPort}} that is being used by the SDK, 
which includes the id of the {{Coder}} in the {{ProcessBundleDescriptor}} - it 
doesn't seem like it should have to re-traverse the components, but could 
instead just instantiate parts of that coder.

 

This is partially related to being able to rehydrate an unknown coder into a 
java coder, even if it's unusable - converting from the {{RawCoder}} to the 
{{ByteArrayCoder}} (for component coders, if required) seems like a valuable 
utility, which removes the need to be able to add the {{ByteArrayCoder}} to the 
{{Components}} 

> Add a utility to instantiate a partially unknown coder
> --
>
> Key: BEAM-3780
> URL: https://issues.apache.org/jira/browse/BEAM-3780
> Project: Beam
>  Issue Type: Bug
>  Components: runner-core
>Reporter: Thomas Groh
>Assignee: Thomas Groh
>Priority: Major
>
> Coders must be understood by the SDK harness that is encoding or decoding the 
> associated elements. However, the pipeline runner is capable of constructing 
> partial coders, where an unknown coder is replaced with a ByteArrayCoder. It 
> then can decompose the portions of elements it is aware of, without having to 
> understand the custom element encodings.
>  
> This should go in CoderTranslation, as an alternative to the full-fidelity 
> rehydration of a coder.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Python_Verify #4366

[Apache Jenkins Server]

See 


Changes:

[ehudm] Don't cache pubsub subscription prematurely.

[ehudm] Add Python lint check for calls to unittest.main.

[github] Fixing formatting bug in filebasedsink.py.

[github] Fix lint issue.

--
[...truncated 1.02 MB...]
copying apache_beam/runners/direct/direct_runner.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_runner_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/evaluation_context.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/executor.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/helper_transforms.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/sdf_direct_runner.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/sdf_direct_runner_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/transform_evaluator.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/util.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/watermark_manager.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/experimental/__init__.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/experimental
copying apache_beam/runners/experimental/python_rpc_direct/__init__.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/experimental/python_rpc_direct
copying 
apache_beam/runners/experimental/python_rpc_direct/python_rpc_direct_runner.py 
-> apache-beam-2.5.0.dev0/apache_beam/runners/experimental/python_rpc_direct
copying apache_beam/runners/experimental/python_rpc_direct/server.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/experimental/python_rpc_direct
copying apache_beam/runners/job/__init__.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/job
copying apache_beam/runners/job/manager.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/job
copying apache_beam/runners/job/utils.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/job
copying apache_beam/runners/portability/__init__.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/portability/fn_api_runner.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/portability/fn_api_runner_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/portability/maptask_executor_runner.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/portability/maptask_executor_runner_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/portability/universal_local_runner.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/portability/universal_local_runner_main.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/portability/universal_local_runner_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/test/__init__.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/test
copying apache_beam/runners/worker/__init__.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/bundle_processor.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/data_plane.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/data_plane_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/log_handler.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/log_handler_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/logger.pxd -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/logger.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/logger_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/opcounters.pxd -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/opcounters.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/opcounters_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/operation_specs.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/operations.pxd -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/operations.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying 

Build failed in Jenkins: beam_PostRelease_NightlySnapshot #77

[Apache Jenkins Server]

See 


--
[...truncated 309.87 KB...]

*
* Test: SUCCEED: UserScore successfully run on DirectRunners.
*


*
* Test: Running: HourlyTeamScore example on DirectRunner
*

mvn compile exec:java 
-Dexec.mainClass=org.apache.beam.examples.complete.game.HourlyTeamScore 
-Dexec.args="--tempLocation=gs://temp-storage-for-release-validation-tests/mobile-gaming/tmp
 --output=java-hourlyteamscore-result.txt
 
--input=gs://temp-storage-for-release-validation-tests/mobile-gaming/5000_gaming_data.csv"
 -Pdirect-runner
Using maven /home/jenkins/tools/maven/apache-maven-3.5.2
[INFO] Scanning for projects...
[INFO] 
[INFO] 
[INFO] Building word-count-beam 0.1
[INFO] 
[INFO] 
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ 
word-count-beam ---
[WARNING] Using platform encoding (UTF-8 actually) to copy filtered resources, 
i.e. build is platform dependent!
[INFO] skip non existing resourceDirectory 
/tmp/groovy-generated-9060743315487134585-tmpdir/word-count-beam/src/main/resources
[INFO] 
[INFO] --- maven-compiler-plugin:3.7.0:compile (default-compile) @ 
word-count-beam ---
[INFO] Nothing to compile - all classes are up to date
[INFO] 
[INFO] --- exec-maven-plugin:1.6.0:java (default-cli) @ word-count-beam ---
Mar 06, 2018 12:32:08 AM org.apache.beam.sdk.io.FileBasedSource 
getEstimatedSizeBytes
INFO: Filepattern 
gs://temp-storage-for-release-validation-tests/mobile-gaming/5000_gaming_data.csv
 matched 1 files with total size 386400
Mar 06, 2018 12:32:08 AM org.apache.beam.sdk.io.FileBasedSource split
INFO: Splitting filepattern 
gs://temp-storage-for-release-validation-tests/mobile-gaming/5000_gaming_data.csv
 into bundles of size 96600 took 61 ms and produced 1 files and 4 bundles
Mar 06, 2018 12:32:14 AM 
org.apache.beam.sdk.io.WriteFiles$WriteShardsIntoTempFilesFn processElement
INFO: Opening writer 6b051a26-ed29-4bc1-914a-95241159f1be for window 
[2015-11-17T00:00:00.000Z..2015-11-17T01:00:00.000Z) pane 
PaneInfo{isFirst=true, isLast=true, timing=ON_TIME, index=0, onTimeIndex=0} 
destination null
Mar 06, 2018 12:32:14 AM 
org.apache.beam.sdk.io.WriteFiles$WriteShardsIntoTempFilesFn processElement
INFO: Opening writer 75e363eb-17d4-4349-8d05-4ae2ae9a40f0 for window 
[2015-11-17T00:00:00.000Z..2015-11-17T01:00:00.000Z) pane 
PaneInfo{isFirst=true, isLast=true, timing=ON_TIME, index=0, onTimeIndex=0} 
destination null
Mar 06, 2018 12:32:14 AM 
org.apache.beam.sdk.io.WriteFiles$WriteShardsIntoTempFilesFn processElement
INFO: Opening writer 10ad1f8d-bd1c-4924-996f-fe8fc7a98f09 for window 
[2015-11-17T00:00:00.000Z..2015-11-17T01:00:00.000Z) pane 
PaneInfo{isFirst=true, isLast=true, timing=ON_TIME, index=0, onTimeIndex=0} 
destination null
Mar 06, 2018 12:32:14 AM org.apache.beam.sdk.io.FileBasedSink$Writer close
INFO: Successfully wrote temporary file 
/tmp/groovy-generated-9060743315487134585-tmpdir/word-count-beam/.temp-beam-2018-03-06_00-32-06-0/75e363eb-17d4-4349-8d05-4ae2ae9a40f0
Mar 06, 2018 12:32:14 AM org.apache.beam.sdk.io.FileBasedSink$Writer close
INFO: Successfully wrote temporary file 
/tmp/groovy-generated-9060743315487134585-tmpdir/word-count-beam/.temp-beam-2018-03-06_00-32-06-0/10ad1f8d-bd1c-4924-996f-fe8fc7a98f09
Mar 06, 2018 12:32:14 AM org.apache.beam.sdk.io.FileBasedSink$Writer close
INFO: Successfully wrote temporary file 
/tmp/groovy-generated-9060743315487134585-tmpdir/word-count-beam/.temp-beam-2018-03-06_00-32-06-0/6b051a26-ed29-4bc1-914a-95241159f1be
Mar 06, 2018 12:32:15 AM 
org.apache.beam.sdk.io.WriteFiles$FinalizeTempFileBundles$FinalizeFn process
INFO: Finalizing 1 file results
Mar 06, 2018 12:32:15 AM org.apache.beam.sdk.io.FileBasedSink$WriteOperation 
moveToOutputFiles
INFO: Will copy temporary file 
FileResult{tempFilename=/tmp/groovy-generated-9060743315487134585-tmpdir/word-count-beam/.temp-beam-2018-03-06_00-32-06-0/75e363eb-17d4-4349-8d05-4ae2ae9a40f0,
 shard=0, window=[2015-11-17T00:00:00.000Z..2015-11-17T01:00:00.000Z), 
paneInfo=PaneInfo{isFirst=true, isLast=true, timing=ON_TIME, index=0, 
onTimeIndex=0}} to final location 
/tmp/groovy-generated-9060743315487134585-tmpdir/word-count-beam/java-hourlyteamscore-result.txt
-2015-11-16 16:00:00.000-2015-11-16 17:00:00.000-0-of-3
Mar 06, 2018 12:32:15 AM 
org.apache.beam.sdk.io.WriteFiles$FinalizeTempFileBundles$FinalizeFn process
INFO: Finalizing 1 file results
Mar 06, 2018 12:32:15 AM org.apache.beam.sdk.io.FileBasedSink$WriteOperation 
moveToOutputFiles
INFO: Will copy temporary file 
FileResult{tempFilename=/tmp/groovy-generated-9060743315487134585-tmpdir/word-count-beam/.temp-beam-2018-03-06_00-32-06-0/6b051a26-ed29-4bc1-914a-95241159f1be,
 shard=2, 

[jira] [Commented] (BEAM-3780) Add a utility to instantiate a partially unknown coder

[Luke Cwik (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16387040#comment-16387040
 ] 

Luke Cwik commented on BEAM-3780:
-

LengthPrefixUnknownCoders has methods to create a coder specification for the 
runner and for the SDK. Is the issue that a singular graph representation is 
being used in the Runner so you can differentiate SDK and Runner specific 
segments?

> Add a utility to instantiate a partially unknown coder
> --
>
> Key: BEAM-3780
> URL: https://issues.apache.org/jira/browse/BEAM-3780
> Project: Beam
>  Issue Type: Bug
>  Components: runner-core
>Reporter: Thomas Groh
>Assignee: Thomas Groh
>Priority: Major
>
> Coders must be understood by the SDK harness that is encoding or decoding the 
> associated elements. However, the pipeline runner is capable of constructing 
> partial coders, where an unknown coder is replaced with a ByteArrayCoder. It 
> then can decompose the portions of elements it is aware of, without having to 
> understand the custom element encodings.
>  
> This should go in CoderTranslation, as an alternative to the full-fidelity 
> rehydration of a coder.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PerformanceTests_Spark #1433

[Apache Jenkins Server]

See 


Changes:

[ehudm] Don't cache pubsub subscription prematurely.

[ehudm] Add Python lint check for calls to unittest.main.

[github] Fixing formatting bug in filebasedsink.py.

[github] Fix lint issue.

[mariagh] Add TestClock to test

[daniel.o.programmer] [BEAM-3126] Fixing incorrect function call in bundle 
processor.

--
[...truncated 93.91 KB...]
'apache-beam-testing:bqjob_rab7127f3782db28_0161f8ad0b83_1': Invalid schema
update. Field timestamp has changed type from TIMESTAMP to FLOAT

STDERR: 
/usr/lib/google-cloud-sdk/platform/bq/third_party/oauth2client/contrib/gce.py:73:
 UserWarning: You have requested explicit scopes to be used with a GCE service 
account.
Using this argument will have no effect on the actual scopes for tokens
requested. These scopes are set at VM instance creation time and
can't be overridden in the request.

  warnings.warn(_SCOPES_WARNING)

2018-03-06 00:19:07,761 aeb68081 MainThread INFO Retrying exception running 
IssueRetryableCommand: Command returned a non-zero exit code.

2018-03-06 00:19:33,308 aeb68081 MainThread INFO Running: bq load 
--autodetect --source_format=NEWLINE_DELIMITED_JSON 
beam_performance.pkb_results 

2018-03-06 00:19:35,690 aeb68081 MainThread INFO Ran: {bq load --autodetect 
--source_format=NEWLINE_DELIMITED_JSON beam_performance.pkb_results 

  ReturnCode:1,  WallTime:0:02.37s,  CPU:0.34s,  MaxMemory:25384kb 
STDOUT: Upload complete.
Waiting on bqjob_r78396358bd24b7e7_0161f8ad7875_1 ... (0s) Current status: 
RUNNING 
 Waiting on bqjob_r78396358bd24b7e7_0161f8ad7875_1 ... (0s) 
Current status: DONE   
BigQuery error in load operation: Error processing job
'apache-beam-testing:bqjob_r78396358bd24b7e7_0161f8ad7875_1': Invalid schema
update. Field timestamp has changed type from TIMESTAMP to FLOAT

STDERR: 
/usr/lib/google-cloud-sdk/platform/bq/third_party/oauth2client/contrib/gce.py:73:
 UserWarning: You have requested explicit scopes to be used with a GCE service 
account.
Using this argument will have no effect on the actual scopes for tokens
requested. These scopes are set at VM instance creation time and
can't be overridden in the request.

  warnings.warn(_SCOPES_WARNING)

2018-03-06 00:19:35,690 aeb68081 MainThread INFO Retrying exception running 
IssueRetryableCommand: Command returned a non-zero exit code.

2018-03-06 00:19:56,650 aeb68081 MainThread INFO Running: bq load 
--autodetect --source_format=NEWLINE_DELIMITED_JSON 
beam_performance.pkb_results 

2018-03-06 00:19:59,378 aeb68081 MainThread INFO Ran: {bq load --autodetect 
--source_format=NEWLINE_DELIMITED_JSON beam_performance.pkb_results 

  ReturnCode:1,  WallTime:0:02.70s,  CPU:0.42s,  MaxMemory:25400kb 
STDOUT: Upload complete.
Waiting on bqjob_r56012b626cfab008_0161f8add43e_1 ... (0s) Current status: 
RUNNING 
 Waiting on bqjob_r56012b626cfab008_0161f8add43e_1 ... (0s) 
Current status: DONE   
BigQuery error in load operation: Error processing job
'apache-beam-testing:bqjob_r56012b626cfab008_0161f8add43e_1': Invalid schema
update. Field timestamp has changed type from TIMESTAMP to FLOAT

STDERR: 
/usr/lib/google-cloud-sdk/platform/bq/third_party/oauth2client/contrib/gce.py:73:
 UserWarning: You have requested explicit scopes to be used with a GCE service 
account.
Using this argument will have no effect on the actual scopes for tokens
requested. These scopes are set at VM instance creation time and
can't be overridden in the request.

  warnings.warn(_SCOPES_WARNING)

2018-03-06 00:19:59,379 aeb68081 MainThread INFO Retrying exception running 
IssueRetryableCommand: Command returned a non-zero exit code.

2018-03-06 00:20:27,150 aeb68081 MainThread INFO Running: bq load 
--autodetect --source_format=NEWLINE_DELIMITED_JSON 
beam_performance.pkb_results 

2018-03-06 00:20:29,526 aeb68081 MainThread INFO Ran: {bq load --autodetect 
--source_format=NEWLINE_DELIMITED_JSON beam_performance.pkb_results 

  ReturnCode:1,  WallTime:0:02.36s,  CPU:0.38s,  MaxMemory:25552kb 
STDOUT: Upload complete.
Waiting on 

Build failed in Jenkins: beam_PerformanceTests_TextIOIT #233

[Apache Jenkins Server]

See 


Changes:

[ehudm] Don't cache pubsub subscription prematurely.

[ehudm] Add Python lint check for calls to unittest.main.

[github] Fixing formatting bug in filebasedsink.py.

[github] Fix lint issue.

[mariagh] Add TestClock to test

[daniel.o.programmer] [BEAM-3126] Fixing incorrect function call in bundle 
processor.

--
[...truncated 26.41 KB...]
[INFO] Excluding 
org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:2.5.0-SNAPSHOT from 
the shaded jar.
[INFO] Excluding 
org.apache.beam:beam-sdks-java-extensions-protobuf:jar:2.5.0-SNAPSHOT from the 
shaded jar.
[INFO] Excluding io.grpc:grpc-core:jar:1.2.0 from the shaded jar.
[INFO] Excluding com.google.errorprone:error_prone_annotations:jar:2.0.15 from 
the shaded jar.
[INFO] Excluding io.grpc:grpc-context:jar:1.2.0 from the shaded jar.
[INFO] Excluding com.google.instrumentation:instrumentation-api:jar:0.3.0 from 
the shaded jar.
[INFO] Excluding 
com.google.apis:google-api-services-bigquery:jar:v2-rev374-1.22.0 from the 
shaded jar.
[INFO] Excluding com.google.api:gax-grpc:jar:0.20.0 from the shaded jar.
[INFO] Excluding io.grpc:grpc-protobuf:jar:1.2.0 from the shaded jar.
[INFO] Excluding com.google.api:api-common:jar:1.0.0-rc2 from the shaded jar.
[INFO] Excluding com.google.auto.value:auto-value:jar:1.5.3 from the shaded jar.
[INFO] Excluding com.google.api:gax:jar:1.3.1 from the shaded jar.
[INFO] Excluding org.threeten:threetenbp:jar:1.3.3 from the shaded jar.
[INFO] Excluding com.google.cloud:google-cloud-core-grpc:jar:1.2.0 from the 
shaded jar.
[INFO] Excluding com.google.protobuf:protobuf-java-util:jar:3.2.0 from the 
shaded jar.
[INFO] Excluding com.google.code.gson:gson:jar:2.7 from the shaded jar.
[INFO] Excluding com.google.apis:google-api-services-pubsub:jar:v1-rev10-1.22.0 
from the shaded jar.
[INFO] Excluding com.google.api.grpc:grpc-google-cloud-pubsub-v1:jar:0.1.18 
from the shaded jar.
[INFO] Excluding com.google.api.grpc:proto-google-cloud-pubsub-v1:jar:0.1.18 
from the shaded jar.
[INFO] Excluding com.google.api.grpc:proto-google-iam-v1:jar:0.1.18 from the 
shaded jar.
[INFO] Excluding com.google.cloud.datastore:datastore-v1-proto-client:jar:1.4.0 
from the shaded jar.
[INFO] Excluding com.google.http-client:google-http-client-protobuf:jar:1.22.0 
from the shaded jar.
[INFO] Excluding com.google.http-client:google-http-client-jackson:jar:1.22.0 
from the shaded jar.
[INFO] Excluding com.google.cloud.datastore:datastore-v1-protos:jar:1.3.0 from 
the shaded jar.
[INFO] Excluding com.google.api.grpc:grpc-google-common-protos:jar:0.1.9 from 
the shaded jar.
[INFO] Excluding io.grpc:grpc-auth:jar:1.2.0 from the shaded jar.
[INFO] Excluding io.grpc:grpc-netty:jar:1.2.0 from the shaded jar.
[INFO] Excluding io.netty:netty-codec-http2:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-codec-http:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-handler-proxy:jar:4.1.8.Final from the shaded 
jar.
[INFO] Excluding io.netty:netty-codec-socks:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-handler:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-buffer:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-common:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-transport:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-resolver:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-codec:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.grpc:grpc-stub:jar:1.2.0 from the shaded jar.
[INFO] Excluding io.grpc:grpc-all:jar:1.2.0 from the shaded jar.
[INFO] Excluding io.grpc:grpc-okhttp:jar:1.2.0 from the shaded jar.
[INFO] Excluding com.squareup.okhttp:okhttp:jar:2.5.0 from the shaded jar.
[INFO] Excluding com.squareup.okio:okio:jar:1.6.0 from the shaded jar.
[INFO] Excluding io.grpc:grpc-protobuf-lite:jar:1.2.0 from the shaded jar.
[INFO] Excluding io.grpc:grpc-protobuf-nano:jar:1.2.0 from the shaded jar.
[INFO] Excluding com.google.protobuf.nano:protobuf-javanano:jar:3.0.0-alpha-5 
from the shaded jar.
[INFO] Excluding com.google.cloud:google-cloud-core:jar:1.0.2 from the shaded 
jar.
[INFO] Excluding org.json:json:jar:20160810 from the shaded jar.
[INFO] Excluding com.google.cloud:google-cloud-spanner:jar:0.20.0b-beta from 
the shaded jar.
[INFO] Excluding com.google.api.grpc:proto-google-cloud-spanner-v1:jar:0.1.11b 
from the shaded jar.
[INFO] Excluding 
com.google.api.grpc:proto-google-cloud-spanner-admin-instance-v1:jar:0.1.11 
from the shaded jar.
[INFO] Excluding com.google.api.grpc:grpc-google-cloud-spanner-v1:jar:0.1.11b 
from the shaded jar.
[INFO] Excluding 
com.google.api.grpc:grpc-google-cloud-spanner-admin-database-v1:jar:0.1.11 from 
the shaded jar.
[INFO] Excluding 

Build failed in Jenkins: beam_PerformanceTests_Python #989

[Apache Jenkins Server]

See 


Changes:

[ehudm] Don't cache pubsub subscription prematurely.

[ehudm] Add Python lint check for calls to unittest.main.

[github] Fixing formatting bug in filebasedsink.py.

[github] Fix lint issue.

[mariagh] Add TestClock to test

[daniel.o.programmer] [BEAM-3126] Fixing incorrect function call in bundle 
processor.

--
[...truncated 1.39 KB...]
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins3379714250200279738.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins8404860035910337652.sh
+ rm -rf .env
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins8322143071687720427.sh
+ virtualenv .env --system-site-packages
New python executable in .env/bin/python
Installing setuptools, pip...done.
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins4336296923922805833.sh
+ .env/bin/pip install --upgrade setuptools pip
Downloading/unpacking setuptools from 
https://pypi.python.org/packages/43/41/033a273f9a25cb63050a390ee8397acbc7eae2159195d85f06f17e7be45a/setuptools-38.5.1-py2.py3-none-any.whl#md5=908b8b5e50bf429e520b2b5fa1b350e5
Downloading/unpacking pip from 
https://pypi.python.org/packages/b6/ac/7015eb97dc749283ffdec1c3a88ddb8ae03b8fad0f0e611408f196358da3/pip-9.0.1-py2.py3-none-any.whl#md5=297dbd16ef53bcef0447d245815f5144
Installing collected packages: setuptools, pip
  Found existing installation: setuptools 2.2
Uninstalling setuptools:
  Successfully uninstalled setuptools
  Found existing installation: pip 1.5.4
Uninstalling pip:
  Successfully uninstalled pip
Successfully installed setuptools pip
Cleaning up...
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins5285450861727145071.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_Python] $ /bin/bash -xe /tmp/jenkins35422447730846756.sh
+ .env/bin/pip install -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied: absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: setuptools in ./.env/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 16))
Requirement already satisfied: colorlog[windows]==2.6.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied: futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied: PyYAML==3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied: pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Collecting numpy==1.13.3 (from -r PerfKitBenchmarker/requirements.txt (line 22))
:318:
 SNIMissingWarning: An HTTPS request has been made, but the SNI (Subject Name 
Indication) extension to TLS is not available on this platform. This may cause 
the server to present an incorrect TLS certificate, which can cause validation 
failures. You can upgrade to a newer version of Python to solve this. For more 
information, see 
https://urllib3.readthedocs.io/en/latest/security.html#snimissingwarning.
  SNIMissingWarning
:122:
 InsecurePlatformWarning: A true SSLContext object is not available. This 
prevents urllib3 from configuring SSL appropriately and may cause certain SSL 
connections to fail. You can upgrade to a newer version of Python to solve 
this. For more information, see 
https://urllib3.readthedocs.io/en/latest/security.html#insecureplatformwarning.
  InsecurePlatformWarning
  Using cached numpy-1.13.3-cp27-cp27mu-manylinux1_x86_64.whl
Requirement already satisfied: functools32 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23))
Requirement already satisfied: 

[jira] [Resolved] (BEAM-3671) Need Python 3.x interpreter in Jenkins machines

[Ahmet Altay (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ahmet Altay resolved BEAM-3671.
---
   Resolution: Fixed
Fix Version/s: Not applicable

> Need Python 3.x interpreter in Jenkins machines
> ---
>
> Key: BEAM-3671
> URL: https://issues.apache.org/jira/browse/BEAM-3671
> Project: Beam
>  Issue Type: New Feature
>  Components: testing
>Reporter: Ahmet Altay
>Assignee: Alan Myrvold
>Priority: Major
> Fix For: Not applicable
>
>  Time Spent: 12h
>  Remaining Estimate: 0h
>
> This is required for being able to test with Py3. If possible, it would be 
> best to have 3.5 and 3.6. If not we can settle on a single version.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PerformanceTests_JDBC #293

[Apache Jenkins Server]

See 


Changes:

[ehudm] Don't cache pubsub subscription prematurely.

[ehudm] Add Python lint check for calls to unittest.main.

[github] Fixing formatting bug in filebasedsink.py.

[github] Fix lint issue.

[mariagh] Add TestClock to test

[daniel.o.programmer] [BEAM-3126] Fixing incorrect function call in bundle 
processor.

--
[...truncated 724.19 KB...]
[INFO] Excluding com.google.http-client:google-http-client-protobuf:jar:1.22.0 
from the shaded jar.
[INFO] Excluding com.google.http-client:google-http-client-jackson:jar:1.22.0 
from the shaded jar.
[INFO] Excluding com.google.cloud.datastore:datastore-v1-protos:jar:1.3.0 from 
the shaded jar.
[INFO] Excluding com.google.api.grpc:grpc-google-common-protos:jar:0.1.9 from 
the shaded jar.
[INFO] Excluding io.grpc:grpc-auth:jar:1.2.0 from the shaded jar.
[INFO] Excluding io.grpc:grpc-netty:jar:1.2.0 from the shaded jar.
[INFO] Excluding io.netty:netty-codec-http2:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-codec-http:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-handler-proxy:jar:4.1.8.Final from the shaded 
jar.
[INFO] Excluding io.netty:netty-codec-socks:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-handler:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-buffer:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-common:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-transport:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-resolver:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-codec:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.grpc:grpc-stub:jar:1.2.0 from the shaded jar.
[INFO] Excluding io.grpc:grpc-all:jar:1.2.0 from the shaded jar.
[INFO] Excluding io.grpc:grpc-okhttp:jar:1.2.0 from the shaded jar.
[INFO] Excluding com.squareup.okhttp:okhttp:jar:2.5.0 from the shaded jar.
[INFO] Excluding com.squareup.okio:okio:jar:1.6.0 from the shaded jar.
[INFO] Excluding io.grpc:grpc-protobuf-lite:jar:1.2.0 from the shaded jar.
[INFO] Excluding io.grpc:grpc-protobuf-nano:jar:1.2.0 from the shaded jar.
[INFO] Excluding com.google.protobuf.nano:protobuf-javanano:jar:3.0.0-alpha-5 
from the shaded jar.
[INFO] Excluding com.google.cloud:google-cloud-core:jar:1.0.2 from the shaded 
jar.
[INFO] Excluding org.json:json:jar:20160810 from the shaded jar.
[INFO] Excluding com.google.cloud:google-cloud-spanner:jar:0.20.0b-beta from 
the shaded jar.
[INFO] Excluding com.google.api.grpc:proto-google-cloud-spanner-v1:jar:0.1.11b 
from the shaded jar.
[INFO] Excluding 
com.google.api.grpc:proto-google-cloud-spanner-admin-instance-v1:jar:0.1.11 
from the shaded jar.
[INFO] Excluding com.google.api.grpc:grpc-google-cloud-spanner-v1:jar:0.1.11b 
from the shaded jar.
[INFO] Excluding 
com.google.api.grpc:grpc-google-cloud-spanner-admin-database-v1:jar:0.1.11 from 
the shaded jar.
[INFO] Excluding 
com.google.api.grpc:grpc-google-cloud-spanner-admin-instance-v1:jar:0.1.11 from 
the shaded jar.
[INFO] Excluding com.google.api.grpc:grpc-google-longrunning-v1:jar:0.1.11 from 
the shaded jar.
[INFO] Excluding com.google.api.grpc:proto-google-longrunning-v1:jar:0.1.11 
from the shaded jar.
[INFO] Excluding com.google.cloud.bigtable:bigtable-protos:jar:1.0.0-pre3 from 
the shaded jar.
[INFO] Excluding com.google.cloud.bigtable:bigtable-client-core:jar:1.0.0 from 
the shaded jar.
[INFO] Excluding com.google.auth:google-auth-library-appengine:jar:0.7.0 from 
the shaded jar.
[INFO] Excluding io.opencensus:opencensus-contrib-grpc-util:jar:0.7.0 from the 
shaded jar.
[INFO] Excluding io.opencensus:opencensus-api:jar:0.7.0 from the shaded jar.
[INFO] Excluding io.dropwizard.metrics:metrics-core:jar:3.1.2 from the shaded 
jar.
[INFO] Excluding com.google.protobuf:protobuf-java:jar:3.2.0 from the shaded 
jar.
[INFO] Excluding io.netty:netty-tcnative-boringssl-static:jar:1.1.33.Fork26 
from the shaded jar.
[INFO] Excluding 
com.google.api.grpc:proto-google-cloud-spanner-admin-database-v1:jar:0.1.9 from 
the shaded jar.
[INFO] Excluding com.google.api.grpc:proto-google-common-protos:jar:0.1.9 from 
the shaded jar.
[INFO] Excluding com.google.api-client:google-api-client:jar:1.22.0 from the 
shaded jar.
[INFO] Excluding com.google.oauth-client:google-oauth-client:jar:1.22.0 from 
the shaded jar.
[INFO] Excluding com.google.http-client:google-http-client:jar:1.22.0 from the 
shaded jar.
[INFO] Excluding org.apache.httpcomponents:httpclient:jar:4.0.1 from the shaded 
jar.
[INFO] Excluding org.apache.httpcomponents:httpcore:jar:4.0.1 from the shaded 
jar.
[INFO] Excluding commons-codec:commons-codec:jar:1.3 from the shaded jar.
[INFO] Excluding com.google.http-client:google-http-client-jackson2:jar:1.22.0 
from the shaded jar.
[INFO] Excluding 

[jira] [Resolved] (BEAM-3705) ApproximateUnique discards accumulated data with multiple firings.

[Raghu Angadi (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3705?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Raghu Angadi resolved BEAM-3705.

   Resolution: Fixed
Fix Version/s: 2.4.0

> ApproximateUnique discards accumulated data with multiple firings. 
> ---
>
> Key: BEAM-3705
> URL: https://issues.apache.org/jira/browse/BEAM-3705
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Raghu Angadi
>Assignee: Raghu Angadi
>Priority: Major
> Fix For: 2.4.0
>
>  Time Spent: 6h
>  Remaining Estimate: 0h
>
> `extractOutput()` on `ApproximateUniqueCombineFn` resets the accumulated 
> value. This discards accumulated state when making subsequent firings 
> incorrect.
>  
> First reported in https://stackoverflow.com/questions/48698028/. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (BEAM-3754) KAFKA - Can't set commitOffsetsInFinalizeEnabled to false with KafkaIO.readBytes()

[Raghu Angadi (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Raghu Angadi resolved BEAM-3754.

   Resolution: Fixed
Fix Version/s: 2.4.0

> KAFKA - Can't set commitOffsetsInFinalizeEnabled to false with 
> KafkaIO.readBytes()
> --
>
> Key: BEAM-3754
> URL: https://issues.apache.org/jira/browse/BEAM-3754
> Project: Beam
>  Issue Type: Bug
>  Components: io-java-kafka
>Affects Versions: 2.3.0
> Environment: Dataflow pipeline using Kafka as a Sink
>Reporter: Benjamin BENOIST
>Assignee: Raghu Angadi
>Priority: Minor
>  Labels: patch
> Fix For: 2.4.0
>
>   Original Estimate: 2h
>  Time Spent: 0.5h
>  Remaining Estimate: 1.5h
>
> Beam v2.3 introduces finalized offsets, in order to reduce the gaps or 
> duplicate processing of records while restarting a pipeline.
> _read()_ sets this parameter to false [by 
> default|https://github.com/apache/beam/blob/release-2.3.0/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaIO.java#L307]
>  but _readBytes()_ 
> [doesn't|https://github.com/apache/beam/blob/release-2.3.0/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaIO.java#L282],
>  thus creating an exception:
> {noformat}
> Exception in thread "main" java.lang.IllegalStateException: Missing required 
> properties: commitOffsetsInFinalizeEnabled
>      at 
> org.apache.beam.sdk.io.kafka.AutoValue_KafkaIO_Read$Builder.build(AutoValue_KafkaIO_Read.java:344)
>      at 
> org.apache.beam.sdk.io.kafka.KafkaIO.readBytes(KafkaIO.java:291){noformat}
> The parameter can be set to true with _commitOffsetsInFinalize()_ but never 
> to false.
> Using _read()_ in the definition of _readBytes()_ could prevent this kind of 
> error in the future:
> {code:java}
> public static Read readBytes() {
>   return read()
> .setKeyDeserializer(ByteArrayDeserializer.class)
> .setValueDeserializer(ByteArrayDeserializer.class)
> .build();
> }{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Jenkins build is back to stable : beam_PostCommit_Java_MavenInstall #6131

[Apache Jenkins Server]

See 

Jenkins build is still unstable: beam_PostCommit_Java_MavenInstall #6130

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PostRelease_NightlySnapshot #76

[Apache Jenkins Server]

See 


--
[...truncated 308.59 KB...]
Mar 05, 2018 11:38:44 PM org.apache.beam.sdk.io.FileBasedSink$WriteOperation 
removeTemporaryFiles
INFO: Will remove known temporary file 
/tmp/groovy-generated-5435141615528769469-tmpdir/word-count-beam/.temp-beam-2018-03-05_23-38-33-1/5e2ba93e-ea06-4eab-b80b-62d62efdc505
Mar 05, 2018 11:38:44 PM org.apache.beam.sdk.io.FileBasedSink$WriteOperation 
removeTemporaryFiles
INFO: Will remove known temporary file 
/tmp/groovy-generated-5435141615528769469-tmpdir/word-count-beam/.temp-beam-2018-03-05_23-38-33-1/90ca0791-456b-4845-b925-382ddb758c20
Mar 05, 2018 11:38:44 PM org.apache.beam.sdk.io.FileBasedSink$WriteOperation 
removeTemporaryFiles
INFO: Will remove known temporary file 
/tmp/groovy-generated-5435141615528769469-tmpdir/word-count-beam/.temp-beam-2018-03-05_23-38-33-1/118e4787-7769-4f0a-808d-b6c1433c5630
[INFO] 
[INFO] BUILD SUCCESS
[INFO] 
[INFO] Total time: 01:17 min
[INFO] Finished at: 2018-03-05T23:38:44Z
[INFO] Final Memory: 44M/966M
[INFO] 
grep user19_BananaWallaby java-userscore-result.txt* 
java-userscore-result.txt
-2-of-3:total_score: 231, user: user19_BananaWallaby
Verified total_score: 231, user: user19_BananaWallaby

*
* Test: SUCCEED: UserScore successfully run on DirectRunners.
*


*
* Test: Running: HourlyTeamScore example on DirectRunner
*

mvn compile exec:java 
-Dexec.mainClass=org.apache.beam.examples.complete.game.HourlyTeamScore 
-Dexec.args="--tempLocation=gs://temp-storage-for-release-validation-tests/mobile-gaming/tmp
 --output=java-hourlyteamscore-result.txt
 
--input=gs://temp-storage-for-release-validation-tests/mobile-gaming/5000_gaming_data.csv"
 -Pdirect-runner
Using maven /home/jenkins/tools/maven/apache-maven-3.5.2
[INFO] Scanning for projects...
[INFO] 
[INFO] 
[INFO] Building word-count-beam 0.1
[INFO] 
[INFO] 
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ 
word-count-beam ---
[WARNING] Using platform encoding (UTF-8 actually) to copy filtered resources, 
i.e. build is platform dependent!
[INFO] skip non existing resourceDirectory 
/tmp/groovy-generated-5435141615528769469-tmpdir/word-count-beam/src/main/resources
[INFO] 
[INFO] --- maven-compiler-plugin:3.7.0:compile (default-compile) @ 
word-count-beam ---
[INFO] Nothing to compile - all classes are up to date
[INFO] 
[INFO] --- exec-maven-plugin:1.6.0:java (default-cli) @ word-count-beam ---
Mar 05, 2018 11:39:00 PM org.apache.beam.sdk.io.FileBasedSource 
getEstimatedSizeBytes
INFO: Filepattern 
gs://temp-storage-for-release-validation-tests/mobile-gaming/5000_gaming_data.csv
 matched 1 files with total size 386400
Mar 05, 2018 11:39:00 PM org.apache.beam.sdk.io.FileBasedSource split
INFO: Splitting filepattern 
gs://temp-storage-for-release-validation-tests/mobile-gaming/5000_gaming_data.csv
 into bundles of size 96600 took 86 ms and produced 1 files and 4 bundles
Mar 05, 2018 11:39:07 PM 
org.apache.beam.sdk.io.WriteFiles$WriteShardsIntoTempFilesFn processElement
INFO: Opening writer e3c5af47-8643-4494-ba1a-fcb5de085505 for window 
[2015-11-17T00:00:00.000Z..2015-11-17T01:00:00.000Z) pane 
PaneInfo{isFirst=true, isLast=true, timing=ON_TIME, index=0, onTimeIndex=0} 
destination null
Mar 05, 2018 11:39:07 PM 
org.apache.beam.sdk.io.WriteFiles$WriteShardsIntoTempFilesFn processElement
INFO: Opening writer 7a8f1282-67a7-4431-8732-2c97af22e990 for window 
[2015-11-17T00:00:00.000Z..2015-11-17T01:00:00.000Z) pane 
PaneInfo{isFirst=true, isLast=true, timing=ON_TIME, index=0, onTimeIndex=0} 
destination null
Mar 05, 2018 11:39:07 PM org.apache.beam.sdk.io.FileBasedSink$Writer close
INFO: Successfully wrote temporary file 
/tmp/groovy-generated-5435141615528769469-tmpdir/word-count-beam/.temp-beam-2018-03-05_23-38-56-0/e3c5af47-8643-4494-ba1a-fcb5de085505
Mar 05, 2018 11:39:07 PM 
org.apache.beam.sdk.io.WriteFiles$WriteShardsIntoTempFilesFn processElement
INFO: Opening writer 19a9a28a-b3d1-40a5-a210-fe932bb257c6 for window 
[2015-11-17T00:00:00.000Z..2015-11-17T01:00:00.000Z) pane 
PaneInfo{isFirst=true, isLast=true, timing=ON_TIME, index=0, onTimeIndex=0} 
destination null
Mar 05, 2018 11:39:07 PM org.apache.beam.sdk.io.FileBasedSink$Writer close
INFO: Successfully wrote temporary file 
/tmp/groovy-generated-5435141615528769469-tmpdir/word-count-beam/.temp-beam-2018-03-05_23-38-56-0/7a8f1282-67a7-4431-8732-2c97af22e990
Mar 05, 2018 11:39:07 PM org.apache.beam.sdk.io.FileBasedSink$Writer close
INFO: Successfully wrote temporary file 

[beam-site] 01/02: Update site to mention the 2 Python lint envs

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a commit to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git

commit 51cf2e66170216d3887c3477cf99f5809d137a8b
Author: Holden Karau 
AuthorDate: Thu Mar 1 18:37:10 2018 -0800

Update site to mention the 2 Python lint envs
---
 content/contribute/contribution-guide/index.html | 2 +-
 src/contribute/contribution-guide.md | 3 ++-
 2 files changed, 3 insertions(+), 2 deletions(-)

diff --git a/content/contribute/contribution-guide/index.html 
b/content/contribute/contribution-guide/index.html
index ec6addf..fa8ebe8 100644
--- a/content/contribute/contribution-guide/index.html
+++ b/content/contribute/contribution-guide/index.html
@@ -435,7 +435,7 @@ environment before testing your code.
 command.
 
 $ pip install tox
-$ tox -e lint
+$ tox -e lint_py2,lint_py3
 
 
 
diff --git a/src/contribute/contribution-guide.md 
b/src/contribute/contribution-guide.md
index 5a7f0b9..fcf7502 100644
--- a/src/contribute/contribution-guide.md
+++ b/src/contribute/contribution-guide.md
@@ -248,7 +248,8 @@ To Check for lint errors locally, install "tox" package and 
run following
 command.
 
 $ pip install tox
-$ tox -e lint
+$ tox -e lint_py2,lint_py3
+
 
 Beam supports running Python SDK tests using Maven. For this, navigate to root
 directory of your Apache Beam clone and execute following command. Currently

-- 
To stop receiving notification emails like this one, please contact
mergebot-r...@apache.org.

[beam-site] 02/02: This closes #396

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a commit to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git

commit fbdc177305cd57a006f49f5fdae743e2c74325c1
Merge: d4609c5 51cf2e6
Author: Mergebot 
AuthorDate: Mon Mar 5 15:08:12 2018 -0800

This closes #396

 content/contribute/contribution-guide/index.html | 2 +-
 src/contribute/contribution-guide.md | 3 ++-
 2 files changed, 3 insertions(+), 2 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
mergebot-r...@apache.org.

[jira] [Commented] (BEAM-3781) Figure out min supported Python 3 version

[Ahmet Altay (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16386921#comment-16386921
 ] 

Ahmet Altay commented on BEAM-3781:
---

Robert's comment from the discussion thread: "Long-term, we should definitely 
support 3.5+, definitely not support 3.1 or below, and probably look at usage 
numbers/customers/what's shipped with different OSes for 3.2, 3.3, 3.4 (my 
guess is that 3.4 is the lowest we'll want to go)."

> Figure out min supported Python 3 version
> -
>
> Key: BEAM-3781
> URL: https://issues.apache.org/jira/browse/BEAM-3781
> Project: Beam
>  Issue Type: Task
>  Components: sdk-py-core
>Reporter: Ahmet Altay
>Assignee: Ahmet Altay
>Priority: Minor
>
> We have 3.4.3 installed on Jenkins workers. We could target that as we add 
> support, but in th long run we will need to figure out the supported version 
> story for python 3.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-3781) Figure out min supported Python 3 version

[Ahmet Altay (JIRA)]

Ahmet Altay created BEAM-3781:
-

 Summary: Figure out min supported Python 3 version
 Key: BEAM-3781
 URL: https://issues.apache.org/jira/browse/BEAM-3781
 Project: Beam
  Issue Type: Task
  Components: sdk-py-core
Reporter: Ahmet Altay
Assignee: Ahmet Altay


We have 3.4.3 installed on Jenkins workers. We could target that as we add 
support, but in th long run we will need to figure out the supported version 
story for python 3.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Jenkins build became unstable: beam_PostCommit_Java_MavenInstall #6129

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-3780) Add a utility to instantiate a partially unknown coder

[Robert Bradshaw (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16386905#comment-16386905
 ] 

Robert Bradshaw commented on BEAM-3780:
---

Isn't this capability already present as part of LengthPrefixUnknownCoders? Or 
is this a proposal to refactor/modify how that works? 

> Add a utility to instantiate a partially unknown coder
> --
>
> Key: BEAM-3780
> URL: https://issues.apache.org/jira/browse/BEAM-3780
> Project: Beam
>  Issue Type: Bug
>  Components: runner-core
>Reporter: Thomas Groh
>Assignee: Thomas Groh
>Priority: Major
>
> Coders must be understood by the SDK harness that is encoding or decoding the 
> associated elements. However, the pipeline runner is capable of constructing 
> partial coders, where an unknown coder is replaced with a ByteArrayCoder. It 
> then can decompose the portions of elements it is aware of, without having to 
> understand the custom element encodings.
>  
> This should go in CoderTranslation, as an alternative to the full-fidelity 
> rehydration of a coder.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostRelease_NightlySnapshot #75

[Apache Jenkins Server]

See 


--
[...truncated 297.33 KB...]
[INFO] Downloaded from central: 
https://repo.maven.apache.org/maven2/org/apache/commons/commons-parent/35/commons-parent-35.pom
 (58 kB at 2.1 MB/s)
[INFO] Downloading from central: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-toolchain/2.2.1/maven-toolchain-2.2.1.jar
[INFO] Downloading from central: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-settings/2.2.1/maven-settings-2.2.1.jar
[INFO] Downloading from central: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-project/2.2.1/maven-project-2.2.1.jar
[INFO] Downloading from central: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-profile/2.2.1/maven-profile-2.2.1.jar
[INFO] Downloading from central: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-plugin-registry/2.2.1/maven-plugin-registry-2.2.1.jar
[INFO] Downloaded from central: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-settings/2.2.1/maven-settings-2.2.1.jar
 (49 kB at 1.8 MB/s)
[INFO] Downloading from central: 
https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-interpolation/1.11/plexus-interpolation-1.11.jar
[INFO] Downloaded from central: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-plugin-registry/2.2.1/maven-plugin-registry-2.2.1.jar
 (30 kB at 1.0 MB/s)
[INFO] Downloading from central: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-model/2.2.1/maven-model-2.2.1.jar
[INFO] Downloaded from central: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-toolchain/2.2.1/maven-toolchain-2.2.1.jar
 (38 kB at 1.1 MB/s)
[INFO] Downloading from central: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-artifact/2.2.1/maven-artifact-2.2.1.jar
[INFO] Downloaded from central: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-project/2.2.1/maven-project-2.2.1.jar
 (156 kB at 4.6 MB/s)
[INFO] Downloading from central: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-artifact-manager/2.2.1/maven-artifact-manager-2.2.1.jar
[INFO] Downloaded from central: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-profile/2.2.1/maven-profile-2.2.1.jar
 (35 kB at 982 kB/s)
[INFO] Downloading from central: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-repository-metadata/2.2.1/maven-repository-metadata-2.2.1.jar
[INFO] Downloaded from central: 
https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-interpolation/1.11/plexus-interpolation-1.11.jar
 (51 kB at 1.0 MB/s)
[INFO] Downloading from central: 
https://repo.maven.apache.org/maven2/backport-util-concurrent/backport-util-concurrent/3.1/backport-util-concurrent-3.1.jar
[INFO] Downloaded from central: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-model/2.2.1/maven-model-2.2.1.jar
 (88 kB at 1.5 MB/s)
[INFO] Downloading from central: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-core/2.2.1/maven-core-2.2.1.jar
[INFO] Downloaded from central: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-artifact-manager/2.2.1/maven-artifact-manager-2.2.1.jar
 (68 kB at 1.1 MB/s)
[INFO] Downloading from central: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-plugin-parameter-documenter/2.2.1/maven-plugin-parameter-documenter-2.2.1.jar
[INFO] Downloaded from central: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-repository-metadata/2.2.1/maven-repository-metadata-2.2.1.jar
 (26 kB at 414 kB/s)
[INFO] Downloading from central: 
https://repo.maven.apache.org/maven2/org/slf4j/slf4j-jdk14/1.5.6/slf4j-jdk14-1.5.6.jar
[INFO] Downloaded from central: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-artifact/2.2.1/maven-artifact-2.2.1.jar
 (80 kB at 1.1 MB/s)
[INFO] Downloading from central: 
https://repo.maven.apache.org/maven2/org/slf4j/slf4j-api/1.5.6/slf4j-api-1.5.6.jar
[INFO] Downloaded from central: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-plugin-parameter-documenter/2.2.1/maven-plugin-parameter-documenter-2.2.1.jar
 (22 kB at 241 kB/s)
[INFO] Downloading from central: 
https://repo.maven.apache.org/maven2/org/slf4j/jcl-over-slf4j/1.5.6/jcl-over-slf4j-1.5.6.jar
[INFO] Downloaded from central: 
https://repo.maven.apache.org/maven2/org/slf4j/slf4j-jdk14/1.5.6/slf4j-jdk14-1.5.6.jar
 (8.8 kB at 82 kB/s)
[INFO] Downloading from central: 
https://repo.maven.apache.org/maven2/org/apache/maven/reporting/maven-reporting-api/2.2.1/maven-reporting-api-2.2.1.jar
[INFO] Downloaded from central: 
https://repo.maven.apache.org/maven2/org/slf4j/jcl-over-slf4j/1.5.6/jcl-over-slf4j-1.5.6.jar
 (17 kB at 143 kB/s)
[INFO] Downloading from central: 
https://repo.maven.apache.org/maven2/org/apache/maven/doxia/doxia-sink-api/1.1/doxia-sink-api-1.1.jar
[INFO] Downloaded from central: 

[jira] [Created] (BEAM-3780) Add a utility to instantiate a partially unknown coder

[Thomas Groh (JIRA)]

Thomas Groh created BEAM-3780:
-

 Summary: Add a utility to instantiate a partially unknown coder
 Key: BEAM-3780
 URL: https://issues.apache.org/jira/browse/BEAM-3780
 Project: Beam
  Issue Type: Bug
  Components: runner-core
Reporter: Thomas Groh
Assignee: Thomas Groh


Coders must be understood by the SDK harness that is encoding or decoding the 
associated elements. However, the pipeline runner is capable of constructing 
partial coders, where an unknown coder is replaced with a ByteArrayCoder. It 
then can decompose the portions of elements it is aware of, without having to 
understand the custom element encodings.

 

This should go in CoderTranslation, as an alternative to the full-fidelity 
rehydration of a coder.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-3779) Enable deserialization of a non-java Pipeline

[Thomas Groh (JIRA)]

Thomas Groh created BEAM-3779:
-

 Summary: Enable deserialization of a non-java Pipeline
 Key: BEAM-3779
 URL: https://issues.apache.org/jira/browse/BEAM-3779
 Project: Beam
  Issue Type: Bug
  Components: runner-core
Reporter: Thomas Groh
Assignee: Thomas Groh


Currently, rehydrating a Pipeline works on the PCollection and PTransform 
levels with the use of RawPTransform, but the runner-core-construction-java 
utilities will throw if the runner attempts to deserialize a WindowingStrategy 
or Coder which contains non-Java custom (or otherwise unknown) Coders or 
WindowFns.

 

Use a strategy like RawPTransform to deserialize WindowFns and Coders, so they 
can be interacted with as intermediate tokens in the java form.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-3778) Very poor performance of side inputs when input is finely sharded

[Eugene Kirpichov (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16386884#comment-16386884
 ] 

Eugene Kirpichov commented on BEAM-3778:


More precisely: 
[https://github.com/apache/beam/blob/master/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/BatchViewOverrides.java#L1083]

"Each bundle maps to one file exactly" is not what we want here. Something 
that's passed to View.asIterable or View.asList is likely not so big as to try 
to save on a reshuffle, so we should just insert a Reshuffle.viaRandomKey() 
before the ParDo.

> Very poor performance of side inputs when input is finely sharded
> -
>
> Key: BEAM-3778
> URL: https://issues.apache.org/jira/browse/BEAM-3778
> Project: Beam
>  Issue Type: Bug
>  Components: runner-dataflow
>Reporter: Eugene Kirpichov
>Assignee: Luke Cwik
>Priority: Major
>
> This thread:
> https://lists.apache.org/thread.html/324a4f86e567e3e1692466e70f44a08276123b467bacb2ecbf00515f@%3Cuser.beam.apache.org%3E
> The user has a job that reads a few hundred thousand files and then writes 
> them to BigQuery. This generates 1 temp file per input file. Then we gather 
> the temp files into a View.asList() side input - and this side input ends up 
> containing a few hundred thousand tiny ISM files, with 1 element per file, 
> which performs horribly (taking hours to read the side input).
> I think we need to reshuffle things onto a reasonable number of shards before 
> writing them to ISM.
> A side issue: this 
> https://github.com/apache/beam/blob/v2.2.0/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/ReifyAsIterable.java#L46
>  triggers also the coder size estimation logic, which falsely thinks that 
> size estimation in this case is cheap, and does double the work, as evidenced 
> by the following stack trace:
> Processing lull for PT30900.015S in state process of 
> WriteTOBigQuery/BatchLoads/ReifyResults/ParDo(Anonymous)
>  java.net.SocketInputStream.socketRead0(Native Method)
>  java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
>  java.net.SocketInputStream.read(SocketInputStream.java:170)
>  java.net.SocketInputStream.read(SocketInputStream.java:141)
>  sun.security.ssl.InputRecord.readFully(InputRecord.java:465)
>  sun.security.ssl.InputRecord.read(InputRecord.java:503)
>  sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:983)
>  sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:940)
>  sun.security.ssl.AppInputStream.read(AppInputStream.java:105)
>  java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
>  java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
>  java.io.BufferedInputStream.read(BufferedInputStream.java:345)
>  sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:704)
>  sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:647)
>  
> sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1536)
>  
> sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1441)
>  java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480)
>  
> sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:338)
>  
> com.google.api.client.http.javanet.NetHttpResponse.(NetHttpResponse.java:37)
>  
> com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:94)
>  com.google.api.client.http.HttpRequest.execute(HttpRequest.java:981)
>  
> com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:419)
>  
> com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:352)
>  
> com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeMedia(AbstractGoogleClientRequest.java:380)
>  
> com.google.api.services.storage.Storage$Objects$Get.executeMedia(Storage.java:4784)
>  
> com.google.cloud.hadoop.gcsio.GoogleCloudStorageReadChannel.openStreamAndSetMetadata(GoogleCloudStorageReadChannel.java:656)
>  
> com.google.cloud.hadoop.gcsio.GoogleCloudStorageReadChannel.performLazySeek(GoogleCloudStorageReadChannel.java:560)
>  
> com.google.cloud.hadoop.gcsio.GoogleCloudStorageReadChannel.read(GoogleCloudStorageReadChannel.java:289)
>  sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:65)
>  sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:109)
>  sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103)
>  java.io.InputStream.read(InputStream.java:101)
>  sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:81)
>  org.apache.beam.sdk.util.VarInt.decodeLong(VarInt.java:79)
>  

[beam] 01/01: Merge pull request #4785: Add Python lint check for calls to unittest.main

[chamikara]

This is an automated email from the ASF dual-hosted git repository.

chamikara pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 1eb5f09d812b3354541edc267df5d05af6cad5d2
Merge: 7aeb585 7d63497
Author: Chamikara Jayalath 
AuthorDate: Mon Mar 5 14:18:59 2018 -0800

Merge pull request #4785: Add Python lint check for calls to unittest.main

 sdks/python/apache_beam/io/filesystem_test.py  |  5 +
 sdks/python/apache_beam/io/filesystemio_test.py|  5 +
 sdks/python/apache_beam/io/filesystems_test.py |  5 +
 sdks/python/apache_beam/io/gcp/gcsfilesystem_test.py   |  5 +
 sdks/python/apache_beam/io/hadoopfilesystem_test.py|  5 +
 sdks/python/apache_beam/io/localfilesystem_test.py |  5 +
 sdks/python/apache_beam/options/value_provider_test.py |  5 +
 sdks/python/apache_beam/transforms/create_test.py  |  5 +
 sdks/python/apache_beam/transforms/util_test.py|  5 +
 sdks/python/run_pylint.sh  | 13 +
 10 files changed, 58 insertions(+)

-- 
To stop receiving notification emails like this one, please contact
chamik...@apache.org.

[jira] [Assigned] (BEAM-2928) ULR support for portable side input

[Luke Cwik (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Cwik reassigned BEAM-2928:
---

Assignee: Luke Cwik

> ULR support for portable side input
> ---
>
> Key: BEAM-2928
> URL: https://issues.apache.org/jira/browse/BEAM-2928
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-core, runner-direct
>Reporter: Henning Rohde
>Assignee: Luke Cwik
>Priority: Major
>  Labels: portability
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostRelease_NightlySnapshot #74

[Apache Jenkins Server]

See 


--
[...truncated 312.79 KB...]
*
* Test: Running: HourlyTeamScore example on DirectRunner
*

mvn compile exec:java 
-Dexec.mainClass=org.apache.beam.examples.complete.game.HourlyTeamScore 
-Dexec.args="--tempLocation=gs://temp-storage-for-release-validation-tests/mobile-gaming/tmp
 --output=java-hourlyteamscore-result.txt
 
--input=gs://temp-storage-for-release-validation-tests/mobile-gaming/5000_gaming_data.csv"
 -Pdirect-runner
Using maven /home/jenkins/tools/maven/apache-maven-3.5.2
[INFO] Scanning for projects...
[INFO] 
[INFO] 
[INFO] Building word-count-beam 0.1
[INFO] 
[INFO] 
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ 
word-count-beam ---
[WARNING] Using platform encoding (UTF-8 actually) to copy filtered resources, 
i.e. build is platform dependent!
[INFO] skip non existing resourceDirectory 
/tmp/groovy-generated-6182009802695043464-tmpdir/word-count-beam/src/main/resources
[INFO] 
[INFO] --- maven-compiler-plugin:3.7.0:compile (default-compile) @ 
word-count-beam ---
[INFO] Nothing to compile - all classes are up to date
[INFO] 
[INFO] --- exec-maven-plugin:1.6.0:java (default-cli) @ word-count-beam ---
Mar 05, 2018 10:09:46 PM org.apache.beam.sdk.io.FileBasedSource 
getEstimatedSizeBytes
INFO: Filepattern 
gs://temp-storage-for-release-validation-tests/mobile-gaming/5000_gaming_data.csv
 matched 1 files with total size 386400
Mar 05, 2018 10:09:46 PM org.apache.beam.sdk.io.FileBasedSource split
INFO: Splitting filepattern 
gs://temp-storage-for-release-validation-tests/mobile-gaming/5000_gaming_data.csv
 into bundles of size 96600 took 55 ms and produced 1 files and 4 bundles
Mar 05, 2018 10:09:52 PM 
org.apache.beam.sdk.io.WriteFiles$WriteShardsIntoTempFilesFn processElement
INFO: Opening writer a0b9659c-2417-42d5-baf9-d4054fd3d399 for window 
[2015-11-17T00:00:00.000Z..2015-11-17T01:00:00.000Z) pane 
PaneInfo{isFirst=true, isLast=true, timing=ON_TIME, index=0, onTimeIndex=0} 
destination null
Mar 05, 2018 10:09:52 PM 
org.apache.beam.sdk.io.WriteFiles$WriteShardsIntoTempFilesFn processElement
INFO: Opening writer 150d8cc6-1125-466e-a801-774fdea492f2 for window 
[2015-11-17T00:00:00.000Z..2015-11-17T01:00:00.000Z) pane 
PaneInfo{isFirst=true, isLast=true, timing=ON_TIME, index=0, onTimeIndex=0} 
destination null
Mar 05, 2018 10:09:52 PM 
org.apache.beam.sdk.io.WriteFiles$WriteShardsIntoTempFilesFn processElement
INFO: Opening writer 66386efb-f0ad-40fe-a4c9-0102f11f8a28 for window 
[2015-11-17T00:00:00.000Z..2015-11-17T01:00:00.000Z) pane 
PaneInfo{isFirst=true, isLast=true, timing=ON_TIME, index=0, onTimeIndex=0} 
destination null
Mar 05, 2018 10:09:52 PM org.apache.beam.sdk.io.FileBasedSink$Writer close
INFO: Successfully wrote temporary file 
/tmp/groovy-generated-6182009802695043464-tmpdir/word-count-beam/.temp-beam-2018-03-05_22-09-43-0/150d8cc6-1125-466e-a801-774fdea492f2
Mar 05, 2018 10:09:52 PM org.apache.beam.sdk.io.FileBasedSink$Writer close
INFO: Successfully wrote temporary file 
/tmp/groovy-generated-6182009802695043464-tmpdir/word-count-beam/.temp-beam-2018-03-05_22-09-43-0/66386efb-f0ad-40fe-a4c9-0102f11f8a28
Mar 05, 2018 10:09:52 PM org.apache.beam.sdk.io.FileBasedSink$Writer close
INFO: Successfully wrote temporary file 
/tmp/groovy-generated-6182009802695043464-tmpdir/word-count-beam/.temp-beam-2018-03-05_22-09-43-0/a0b9659c-2417-42d5-baf9-d4054fd3d399
Mar 05, 2018 10:09:52 PM 
org.apache.beam.sdk.io.WriteFiles$FinalizeTempFileBundles$FinalizeFn process
INFO: Finalizing 1 file results
Mar 05, 2018 10:09:52 PM org.apache.beam.sdk.io.FileBasedSink$WriteOperation 
moveToOutputFiles
INFO: Will copy temporary file 
FileResult{tempFilename=/tmp/groovy-generated-6182009802695043464-tmpdir/word-count-beam/.temp-beam-2018-03-05_22-09-43-0/66386efb-f0ad-40fe-a4c9-0102f11f8a28,
 shard=2, window=[2015-11-17T00:00:00.000Z..2015-11-17T01:00:00.000Z), 
paneInfo=PaneInfo{isFirst=true, isLast=true, timing=ON_TIME, index=0, 
onTimeIndex=0}} to final location 
/tmp/groovy-generated-6182009802695043464-tmpdir/word-count-beam/java-hourlyteamscore-result.txt
-2015-11-16 16:00:00.000-2015-11-16 17:00:00.000-2-of-3
Mar 05, 2018 10:09:52 PM 
org.apache.beam.sdk.io.WriteFiles$FinalizeTempFileBundles$FinalizeFn process
INFO: Finalizing 1 file results
Mar 05, 2018 10:09:52 PM org.apache.beam.sdk.io.FileBasedSink$WriteOperation 
moveToOutputFiles
INFO: Will copy temporary file 
FileResult{tempFilename=/tmp/groovy-generated-6182009802695043464-tmpdir/word-count-beam/.temp-beam-2018-03-05_22-09-43-0/a0b9659c-2417-42d5-baf9-d4054fd3d399,
 shard=1, window=[2015-11-17T00:00:00.000Z..2015-11-17T01:00:00.000Z), 
paneInfo=PaneInfo{isFirst=true, isLast=true, timing=ON_TIME, index=0, 

[jira] [Commented] (BEAM-3672) FlinkRunner: Implement an Artifact service using the Flink DistributedCache

[Axel Magnuson (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16386828#comment-16386828
 ] 

Axel Magnuson commented on BEAM-3672:
-

Although this is mostly implemented, the current artifact service handles 
strings naively and would likely fail when receiving strings that represent 
illegal input for file paths.  We plan on replacing this path resolution with a 
filepath-safe encoding such as base64, but this is a low priority item that has 
not been tackled yet.  Known affected files are:
 # /runners/local-artifact-service-java/src/main/java/*
 # 
/runners/flink/src/main/java/org/apache/beam/runners/flink/execution/FlinkArtifactSource.java
 # 
/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkCachedArtifactPaths.java

> FlinkRunner: Implement an Artifact service using the Flink DistributedCache
> ---
>
> Key: BEAM-3672
> URL: https://issues.apache.org/jira/browse/BEAM-3672
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-flink
>Reporter: Ben Sidhom
>Assignee: Aljoscha Krettek
>Priority: Major
>
> We need to have a DistributedCache-based artifact service to ship with the 
> portable Flink runner. The DistributedCache is a perfect fit for Flink 
> because it comes for free and is the mechanism that Flink already uses to 
> distribute its own artifacts.
>  
> The final artifact service implementation should be pluggable, but using the 
> DistributedCache allows the Flink runner to work without additional external 
> dependencies (beyond perhaps Docker).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] branch master updated (0ecbb15 -> 7aeb585)

[chamikara]

This is an automated email from the ASF dual-hosted git repository.

chamikara pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 0ecbb15  Merge pull request #4732: Don't cache pubsub subscription 
prematurely
 add 8e57f6f  Fixing formatting bug in filebasedsink.py.
 add a038bc3  Fix lint issue.
 new 7aeb585  Merge pull request #4789: Fixing formatting bug in 
filebasedsink.py

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 sdks/python/apache_beam/io/filebasedsink.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
chamik...@apache.org.

[beam] 01/01: Merge pull request #4789: Fixing formatting bug in filebasedsink.py

[chamikara]

This is an automated email from the ASF dual-hosted git repository.

chamikara pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 7aeb5858f5067a5e02f3a135f98ae931183c005c
Merge: 0ecbb15 a038bc3
Author: Chamikara Jayalath 
AuthorDate: Mon Mar 5 14:06:59 2018 -0800

Merge pull request #4789: Fixing formatting bug in filebasedsink.py

 sdks/python/apache_beam/io/filebasedsink.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
chamik...@apache.org.

Build failed in Jenkins: beam_PostCommit_Python_ValidatesRunner_Dataflow #1049

[Apache Jenkins Server]

See 


Changes:

[ehudm] Don't cache pubsub subscription prematurely.

--
[...truncated 117.97 KB...]
self.visit(Visitor())
  File 
"
 line 410, in visit
self._root_transform().visit(visitor, self, visited)
  File 
"
 line 764, in visit
part.visit(visitor, pipeline, visited)
  File 
"
 line 764, in visit
part.visit(visitor, pipeline, visited)
  File 
"
 line 767, in visit
visitor.visit_transform(self)
  File 
"
 line 561, in visit_transform
enable_trace=False),
  File 
"
 line 206, in dumps
c = zlib.compress(s, 9)
  File 
"
 line 276, in signalhandler
raise TimedOutException()
TimedOutException: 'test_flattened_side_input 
(apache_beam.transforms.sideinputs_test.SideInputsTest)'

==
ERROR: test_iterable_side_input 
(apache_beam.transforms.sideinputs_test.SideInputsTest)
--
Traceback (most recent call last):
  File 
"
 line 812, in run
test(orig)
  File 
"
 line 45, in __call__
return self.run(*arg, **kwarg)
  File 
"
 line 133, in run
self.runTest(result)
  File 
"
 line 151, in runTest
test(result)
  File "/usr/lib/python2.7/unittest/case.py", line 395, in __call__
return self.run(*args, **kwds)
  File "/usr/lib/python2.7/unittest/case.py", line 331, in run
testMethod()
  File 
"
 line 178, in test_iterable_side_input
pipeline.run()
  File 
"
 line 102, in run
result = super(TestPipeline, self).run()
  File 
"
 line 369, in run
self.to_runner_api(), self.runner, self._options).run(False)
  File 
"
 line 580, in to_runner_api
root_transform_id = context.transforms.get_id(self._root_transform())
  File 
"
 line 60, in get_id
self._id_to_proto[id] = obj.to_runner_api(self._pipeline_context)
  File 
"
 line 810, in to_runner_api
for part in self.parts],
  File 
"
 line 60, in get_id
self._id_to_proto[id] = obj.to_runner_api(self._pipeline_context)
  File 
"
 line 810, in to_runner_api
for part in self.parts],
  File 
"
 line 60, in get_id
self._id_to_proto[id] = obj.to_runner_api(self._pipeline_context)
  

Build failed in Jenkins: beam_PostCommit_Python_Verify #4365

[Apache Jenkins Server]

See 


Changes:

[mariagh] Add TestClock to test

[daniel.o.programmer] [BEAM-3126] Fixing incorrect function call in bundle 
processor.

--
[...truncated 1.01 MB...]
copying apache_beam/runners/dataflow/internal/dependency.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/dependency_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/names.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/clients/__init__.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/internal/clients
copying apache_beam/runners/dataflow/internal/clients/dataflow/__init__.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/dataflow_v1b3_client.py 
-> apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/dataflow_v1b3_messages.py
 -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/message_matchers.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/message_matchers_test.py 
-> apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying apache_beam/runners/dataflow/native_io/__init__.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/dataflow/native_io/iobase.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/dataflow/native_io/iobase_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/dataflow/native_io/streaming_create.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/direct/__init__.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/bundle_factory.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/clock.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/consumer_tracking_pipeline_visitor.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/consumer_tracking_pipeline_visitor_test.py 
-> apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_metrics.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_metrics_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_runner.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_runner_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/evaluation_context.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/executor.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/helper_transforms.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/sdf_direct_runner.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/sdf_direct_runner_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/transform_evaluator.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/util.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/watermark_manager.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/experimental/__init__.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/experimental
copying apache_beam/runners/experimental/python_rpc_direct/__init__.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/experimental/python_rpc_direct
copying 
apache_beam/runners/experimental/python_rpc_direct/python_rpc_direct_runner.py 
-> apache-beam-2.5.0.dev0/apache_beam/runners/experimental/python_rpc_direct
copying apache_beam/runners/experimental/python_rpc_direct/server.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/experimental/python_rpc_direct
copying apache_beam/runners/job/__init__.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/job
copying apache_beam/runners/job/manager.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/job
copying apache_beam/runners/job/utils.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/job
copying apache_beam/runners/portability/__init__.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/portability

[beam] 01/01: Merge pull request #4732: Don't cache pubsub subscription prematurely

[chamikara]

This is an automated email from the ASF dual-hosted git repository.

chamikara pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 0ecbb1547d0224df4b8c4fa39f9b870fcbfc113c
Merge: dfad01e f8c3ec8
Author: Chamikara Jayalath 
AuthorDate: Mon Mar 5 13:33:03 2018 -0800

Merge pull request #4732: Don't cache pubsub subscription prematurely


-- 
To stop receiving notification emails like this one, please contact
chamik...@apache.org.

[beam] branch master updated (dfad01e -> 0ecbb15)

[chamikara]

This is an automated email from the ASF dual-hosted git repository.

chamikara pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from dfad01e  Merge pull request #4795 from mariapython/missingClock
 add f8c3ec8  Don't cache pubsub subscription prematurely.
 new 0ecbb15  Merge pull request #4732: Don't cache pubsub subscription 
prematurely

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:

-- 
To stop receiving notification emails like this one, please contact
chamik...@apache.org.

Build failed in Jenkins: beam_PostCommit_Python_ValidatesRunner_Dataflow #1048

[Apache Jenkins Server]

See 


Changes:

[mariagh] Add TestClock to test

[daniel.o.programmer] [BEAM-3126] Fixing incorrect function call in bundle 
processor.

--
[...truncated 122.25 KB...]
save(state)
  File "/usr/lib/python2.7/pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
  File 
"
 line 165, in new_save_module_dict
return old_save_module_dict(pickler, obj)
  File 
"
 line 841, in save_module_dict
StockPickler.save_dict(pickler, obj)
  File "/usr/lib/python2.7/pickle.py", line 649, in save_dict
self._batch_setitems(obj.iteritems())
  File "/usr/lib/python2.7/pickle.py", line 681, in _batch_setitems
save(v)
  File "/usr/lib/python2.7/pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
  File 
"
 line 1311, in save_function
obj.__dict__), obj=obj)
  File "/usr/lib/python2.7/pickle.py", line 401, in save_reduce
save(args)
  File "/usr/lib/python2.7/pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
  File "/usr/lib/python2.7/pickle.py", line 562, in save_tuple
save(element)
  File "/usr/lib/python2.7/pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
  File 
"
 line 165, in new_save_module_dict
return old_save_module_dict(pickler, obj)
  File 
"
 line 841, in save_module_dict
StockPickler.save_dict(pickler, obj)
  File "/usr/lib/python2.7/pickle.py", line 649, in save_dict
self._batch_setitems(obj.iteritems())
  File "/usr/lib/python2.7/pickle.py", line 682, in _batch_setitems
write(SETITEMS)
  File 
"
 line 276, in signalhandler
raise TimedOutException()
TimedOutException: 'test_flattened_side_input 
(apache_beam.transforms.sideinputs_test.SideInputsTest)'

==
ERROR: test_iterable_side_input 
(apache_beam.transforms.sideinputs_test.SideInputsTest)
--
Traceback (most recent call last):
  File 
"
 line 812, in run
test(orig)
  File 
"
 line 45, in __call__
return self.run(*arg, **kwarg)
  File 
"
 line 133, in run
self.runTest(result)
  File 
"
 line 151, in runTest
test(result)
  File "/usr/lib/python2.7/unittest/case.py", line 395, in __call__
return self.run(*args, **kwds)
  File "/usr/lib/python2.7/unittest/case.py", line 331, in run
testMethod()
  File 
"
 line 178, in test_iterable_side_input
pipeline.run()
  File 
"
 line 102, in run
result = super(TestPipeline, self).run()
  File 
"
 line 369, in run
self.to_runner_api(), self.runner, self._options).run(False)
  File 
"
 line 382, in run
return self.runner.run_pipeline(self)
  File 

[beam] branch master updated (a629486 -> dfad01e)

[altay]

This is an automated email from the ASF dual-hosted git repository.

altay pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from a629486  Merge pull request #4797 [BEAM-3126] Fixing incorrect 
function call in bundle processor.
 add bda20d9  Add TestClock to test
 new dfad01e  Merge pull request #4795 from mariapython/missingClock

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 sdks/python/apache_beam/transforms/trigger_test.py | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

-- 
To stop receiving notification emails like this one, please contact
al...@apache.org.

[beam] 01/01: Merge pull request #4795 from mariapython/missingClock

[altay]

This is an automated email from the ASF dual-hosted git repository.

altay pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit dfad01eeb967178d880ae704eb9a00120b34ee05
Merge: a629486 bda20d9
Author: Ahmet Altay 
AuthorDate: Mon Mar 5 12:58:53 2018 -0800

Merge pull request #4795 from mariapython/missingClock

[BEAM-3153] Add TestClock to test

 sdks/python/apache_beam/transforms/trigger_test.py | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

-- 
To stop receiving notification emails like this one, please contact
al...@apache.org.