See
<https://builds.apache.org/job/beam_PostCommit_Python_Verify/2174/display/redirect?page=changes>
Changes:
[ccy] Update Apache Beam Python version to 2.1.0.dev
[davor] Register TestSparkPipelineOptions only in src/test to avoid hard
[davor] Add per-runner profile to Java 8 examples
[altay] Remove trailing whitespace
[davor] Shade dependencies in sdks/core
[davor] Fix checkstyle error
[davor] Include 'sun.reflect' in GcpCoreApiSurfaceTest
------------------------------------------
[...truncated 1.75 MB...]
"@type": "kind:windowed_value",
"component_encodings": [
{
"@type":
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
"component_encodings": [
{
"@type":
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
"component_encodings": []
},
{
"@type":
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
"component_encodings": []
}
],
"is_pair_like": true
},
{
"@type": "kind:global_window"
}
],
"is_wrapper": true
},
"output_name": "out",
"user_name": "assert_that/Unkey.out"
}
],
"parallel_input": {
"@type": "OutputReference",
"output_name": "out",
"step_name": "s12"
},
"serialized_fn": "<string of 964 bytes>",
"user_name": "assert_that/Unkey"
}
},
{
"kind": "ParallelDo",
"name": "s14",
"properties": {
"display_data": [
{
"key": "fn",
"label": "Transform Function",
"namespace": "apache_beam.transforms.core.CallableWrapperDoFn",
"type": "STRING",
"value": "_equal"
},
{
"key": "fn",
"label": "Transform Function",
"namespace": "apache_beam.transforms.core.ParDo",
"shortValue": "CallableWrapperDoFn",
"type": "STRING",
"value": "apache_beam.transforms.core.CallableWrapperDoFn"
}
],
"non_parallel_inputs": {},
"output_info": [
{
"encoding": {
"@type": "kind:windowed_value",
"component_encodings": [
{
"@type":
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
"component_encodings": [
{
"@type":
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
"component_encodings": []
},
{
"@type":
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
"component_encodings": []
}
],
"is_pair_like": true
},
{
"@type": "kind:global_window"
}
],
"is_wrapper": true
},
"output_name": "out",
"user_name": "assert_that/Match.out"
}
],
"parallel_input": {
"@type": "OutputReference",
"output_name": "out",
"step_name": "s13"
},
"serialized_fn": "<string of 1120 bytes>",
"user_name": "assert_that/Match"
}
}
],
"type": "JOB_TYPE_BATCH"
}
root: INFO: Create job: <Job
createTime: u'2017-05-10T01:38:20.116590Z'
currentStateTime: u'1970-01-01T00:00:00Z'
id: u'2017-05-09_18_38_19-10569987656720048055'
location: u'global'
name: u'beamapp-jenkins-0510013805-731745'
projectId: u'apache-beam-testing'
stageStates: []
steps: []
tempFiles: []
type: TypeValueValuesEnum(JOB_TYPE_BATCH, 1)>
root: INFO: Created job with id: [2017-05-09_18_38_19-10569987656720048055]
root: INFO: To access the Dataflow monitoring console, please navigate to
https://console.developers.google.com/project/apache-beam-testing/dataflow/job/2017-05-09_18_38_19-10569987656720048055
root: INFO: Job 2017-05-09_18_38_19-10569987656720048055 is in state
JOB_STATE_RUNNING
root: INFO: 2017-05-10T01:38:19.388Z: JOB_MESSAGE_WARNING: (92b023c5870fd5b3):
Setting the number of workers (1) disables autoscaling for this job. If you are
trying to cap autoscaling, consider only setting max_num_workers. If you want
to disable autoscaling altogether, the documented way is to explicitly use
autoscalingAlgorithm=NONE.
root: INFO: 2017-05-10T01:38:21.145Z: JOB_MESSAGE_DETAILED: (28e4844b147a2c77):
Checking required Cloud APIs are enabled.
root: INFO: 2017-05-10T01:38:23.634Z: JOB_MESSAGE_DEBUG: (28e4844b147a2265):
Combiner lifting skipped for step assert_that/Group/GroupByKey: GroupByKey not
followed by a combiner.
root: INFO: 2017-05-10T01:38:23.636Z: JOB_MESSAGE_DETAILED: (28e4844b147a22a7):
Expanding GroupByKey operations into optimizable parts.
root: INFO: 2017-05-10T01:38:23.639Z: JOB_MESSAGE_DETAILED: (28e4844b147a22e9):
Lifting ValueCombiningMappingFns into MergeBucketsMappingFns
root: INFO: 2017-05-10T01:38:23.643Z: JOB_MESSAGE_DEBUG: (28e4844b147a236d):
Annotating graph with Autotuner information.
root: INFO: 2017-05-10T01:38:23.655Z: JOB_MESSAGE_DETAILED: (28e4844b147a23f1):
Fusing adjacent ParDo, Read, Write, and Flatten operations
root: INFO: 2017-05-10T01:38:23.663Z: JOB_MESSAGE_DETAILED: (28e4844b147a24b7):
Unzipping flatten s10 for input s8.out
root: INFO: 2017-05-10T01:38:23.666Z: JOB_MESSAGE_DETAILED: (28e4844b147a24f9):
Fusing unzipped copy of assert_that/Group/GroupByKey/Reify, through flatten ,
into producer assert_that/Group/pair_with_0
root: INFO: 2017-05-10T01:38:23.669Z: JOB_MESSAGE_DETAILED: (28e4844b147a253b):
Fusing consumer assert_that/Group/Map(_merge_tagged_vals_under_key) into
assert_that/Group/GroupByKey/GroupByWindow
root: INFO: 2017-05-10T01:38:23.671Z: JOB_MESSAGE_DETAILED: (28e4844b147a257d):
Fusing consumer assert_that/Match into assert_that/Unkey
root: INFO: 2017-05-10T01:38:23.673Z: JOB_MESSAGE_DETAILED: (28e4844b147a25bf):
Fusing consumer assert_that/Unkey into
assert_that/Group/Map(_merge_tagged_vals_under_key)
root: INFO: 2017-05-10T01:38:23.679Z: JOB_MESSAGE_DETAILED: (28e4844b147a2643):
Fusing consumer assert_that/Group/GroupByKey/GroupByWindow into
assert_that/Group/GroupByKey/Read
root: INFO: 2017-05-10T01:38:23.686Z: JOB_MESSAGE_DETAILED: (28e4844b147a2709):
Unzipping flatten s10-u13 for input s11-reify-value0-c11
root: INFO: 2017-05-10T01:38:23.689Z: JOB_MESSAGE_DETAILED: (28e4844b147a274b):
Fusing unzipped copy of assert_that/Group/GroupByKey/Write, through flatten ,
into producer assert_that/Group/GroupByKey/Reify
root: INFO: 2017-05-10T01:38:23.691Z: JOB_MESSAGE_DETAILED: (28e4844b147a278d):
Fusing consumer assert_that/Group/GroupByKey/Reify into
assert_that/Group/pair_with_1
root: INFO: 2017-05-10T01:38:23.693Z: JOB_MESSAGE_DETAILED: (28e4844b147a27cf):
Fusing consumer assert_that/Group/GroupByKey/Write into
assert_that/Group/GroupByKey/Reify
root: INFO: 2017-05-10T01:38:23.695Z: JOB_MESSAGE_DETAILED: (28e4844b147a2811):
Fusing consumer assert_that/Group/pair_with_0 into assert_that/Create/Read
root: INFO: 2017-05-10T01:38:23.698Z: JOB_MESSAGE_DETAILED: (28e4844b147a2853):
Fusing consumer assert_that/WindowInto(WindowIntoFn) into compute/Do
root: INFO: 2017-05-10T01:38:23.701Z: JOB_MESSAGE_DETAILED: (28e4844b147a2895):
Fusing consumer assert_that/Group/pair_with_1 into assert_that/ToVoidKey
root: INFO: 2017-05-10T01:38:23.703Z: JOB_MESSAGE_DETAILED: (28e4844b147a28d7):
Fusing consumer assert_that/ToVoidKey into assert_that/WindowInto(WindowIntoFn)
root: INFO: 2017-05-10T01:38:23.706Z: JOB_MESSAGE_DETAILED: (28e4844b147a2919):
Fusing consumer compute/Do into start/Read
root: INFO: 2017-05-10T01:38:23.759Z: JOB_MESSAGE_DEBUG: (28e4844b147a2f49):
Workflow config is missing a default resource spec.
root: INFO: 2017-05-10T01:38:23.761Z: JOB_MESSAGE_DETAILED: (28e4844b147a2f8b):
Adding StepResource setup and teardown to workflow graph.
root: INFO: 2017-05-10T01:38:23.763Z: JOB_MESSAGE_DEBUG: (28e4844b147a2fcd):
Adding workflow start and stop steps.
root: INFO: 2017-05-10T01:38:23.765Z: JOB_MESSAGE_DEBUG: (28e4844b147a200f):
Assigning stage ids.
root: INFO: 2017-05-10T01:38:23.795Z: JOB_MESSAGE_DEBUG: (6d11656c1e5f4c23):
Executing wait step start21
root: INFO: 2017-05-10T01:38:23.802Z: JOB_MESSAGE_BASIC: (6d11656c1e5f4694):
Executing operation side/Read
root: INFO: 2017-05-10T01:38:23.811Z: JOB_MESSAGE_DEBUG: (6d11656c1e5f4982):
Value "side/Read.out" materialized.
root: INFO: 2017-05-10T01:38:23.819Z: JOB_MESSAGE_BASIC: (6d11656c1e5f43f3):
Executing operation compute/SideInput-s3
root: INFO: 2017-05-10T01:38:23.830Z: JOB_MESSAGE_DEBUG: (6d11656c1e5f46e1):
Value "compute/SideInput-s3.output" materialized.
root: INFO: 2017-05-10T01:38:23.840Z: JOB_MESSAGE_BASIC: (558b513d87824a38):
Executing operation assert_that/Group/GroupByKey/Create
root: INFO: 2017-05-10T01:38:24.044Z: JOB_MESSAGE_DEBUG: (284ca35a2d616197):
Starting worker pool setup.
root: INFO: 2017-05-10T01:38:24.046Z: JOB_MESSAGE_BASIC: (284ca35a2d616d89):
Starting 1 workers...
root: INFO: 2017-05-10T01:38:24.060Z: JOB_MESSAGE_DEBUG: (558b513d87824f3b):
Value "assert_that/Group/GroupByKey/Session" materialized.
root: INFO: 2017-05-10T01:38:24.069Z: JOB_MESSAGE_BASIC: (558b513d87824c93):
Executing operation
start/Read+compute/Do+assert_that/WindowInto(WindowIntoFn)+assert_that/ToVoidKey+assert_that/Group/pair_with_1+assert_that/Group/GroupByKey/Reify+assert_that/Group/GroupByKey/Write
root: INFO: 2017-05-10T01:38:24.074Z: JOB_MESSAGE_BASIC: (6d11656c1e5f49cf):
Executing operation
assert_that/Create/Read+assert_that/Group/pair_with_0+assert_that/Group/GroupByKey/Reify+assert_that/Group/GroupByKey/Write
root: INFO: 2017-05-10T01:40:52.405Z: JOB_MESSAGE_DETAILED: (f77e14fb3a3d9825):
Workers have started successfully.
root: INFO: 2017-05-10T01:42:36.472Z: JOB_MESSAGE_ERROR: (cb0fcf6ab63672):
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/dataflow_worker/batchworker.py",
line 581, in do_work
work_executor.execute()
File "/usr/local/lib/python2.7/dist-packages/dataflow_worker/executor.py",
line 166, in execute
op.start()
File
"/usr/local/lib/python2.7/dist-packages/dataflow_worker/native_operations.py",
line 49, in start
self._current_progress = get_progress()
File
"/usr/local/lib/python2.7/dist-packages/dataflow_worker/workercustomsources.py",
line 126, in get_progress
return iobase.ReaderProgress(
AttributeError: 'module' object has no attribute 'ReaderProgress'
root: INFO: 2017-05-10T01:42:39.858Z: JOB_MESSAGE_ERROR: (cb0fcf6ab63842):
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/dataflow_worker/batchworker.py",
line 581, in do_work
work_executor.execute()
File "/usr/local/lib/python2.7/dist-packages/dataflow_worker/executor.py",
line 166, in execute
op.start()
File
"/usr/local/lib/python2.7/dist-packages/dataflow_worker/native_operations.py",
line 49, in start
self._current_progress = get_progress()
File
"/usr/local/lib/python2.7/dist-packages/dataflow_worker/workercustomsources.py",
line 126, in get_progress
return iobase.ReaderProgress(
AttributeError: 'module' object has no attribute 'ReaderProgress'
root: INFO: 2017-05-10T01:42:43.230Z: JOB_MESSAGE_ERROR: (cb0fcf6ab632f5):
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/dataflow_worker/batchworker.py",
line 581, in do_work
work_executor.execute()
File "/usr/local/lib/python2.7/dist-packages/dataflow_worker/executor.py",
line 166, in execute
op.start()
File
"/usr/local/lib/python2.7/dist-packages/dataflow_worker/native_operations.py",
line 49, in start
self._current_progress = get_progress()
File
"/usr/local/lib/python2.7/dist-packages/dataflow_worker/workercustomsources.py",
line 126, in get_progress
return iobase.ReaderProgress(
AttributeError: 'module' object has no attribute 'ReaderProgress'
root: INFO: 2017-05-10T01:42:46.599Z: JOB_MESSAGE_ERROR: (cb0fcf6ab63da8):
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/dataflow_worker/batchworker.py",
line 581, in do_work
work_executor.execute()
File "/usr/local/lib/python2.7/dist-packages/dataflow_worker/executor.py",
line 166, in execute
op.start()
File
"/usr/local/lib/python2.7/dist-packages/dataflow_worker/native_operations.py",
line 49, in start
self._current_progress = get_progress()
File
"/usr/local/lib/python2.7/dist-packages/dataflow_worker/workercustomsources.py",
line 126, in get_progress
return iobase.ReaderProgress(
AttributeError: 'module' object has no attribute 'ReaderProgress'
root: INFO: 2017-05-10T01:42:50.008Z: JOB_MESSAGE_ERROR: (cb0fcf6ab6385b):
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/dataflow_worker/batchworker.py",
line 581, in do_work
work_executor.execute()
File "/usr/local/lib/python2.7/dist-packages/dataflow_worker/executor.py",
line 166, in execute
op.start()
File
"/usr/local/lib/python2.7/dist-packages/dataflow_worker/native_operations.py",
line 49, in start
self._current_progress = get_progress()
File
"/usr/local/lib/python2.7/dist-packages/dataflow_worker/workercustomsources.py",
line 126, in get_progress
return iobase.ReaderProgress(
AttributeError: 'module' object has no attribute 'ReaderProgress'
root: INFO: 2017-05-10T01:42:53.406Z: JOB_MESSAGE_ERROR: (cb0fcf6ab6330e):
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/dataflow_worker/batchworker.py",
line 581, in do_work
work_executor.execute()
File "/usr/local/lib/python2.7/dist-packages/dataflow_worker/executor.py",
line 166, in execute
op.start()
File
"/usr/local/lib/python2.7/dist-packages/dataflow_worker/native_operations.py",
line 49, in start
self._current_progress = get_progress()
File
"/usr/local/lib/python2.7/dist-packages/dataflow_worker/workercustomsources.py",
line 126, in get_progress
return iobase.ReaderProgress(
AttributeError: 'module' object has no attribute 'ReaderProgress'
root: INFO: 2017-05-10T01:42:53.551Z: JOB_MESSAGE_DEBUG: (558b513d878240ec):
Executing failure step failure20
root: INFO: 2017-05-10T01:42:53.554Z: JOB_MESSAGE_ERROR: (558b513d87824042):
Workflow failed. Causes: (558b513d87824240):
S05:start/Read+compute/Do+assert_that/WindowInto(WindowIntoFn)+assert_that/ToVoidKey+assert_that/Group/pair_with_1+assert_that/Group/GroupByKey/Reify+assert_that/Group/GroupByKey/Write
failed.
root: INFO: 2017-05-10T01:42:53.623Z: JOB_MESSAGE_DETAILED: (28e4844b147a297a):
Cleaning up.
root: INFO: 2017-05-10T01:42:53.735Z: JOB_MESSAGE_DEBUG: (28e4844b147a29bc):
Starting worker pool teardown.
root: INFO: 2017-05-10T01:42:53.737Z: JOB_MESSAGE_BASIC: (28e4844b147a29fe):
Stopping worker pool...
root: INFO: 2017-05-10T01:44:18.746Z: JOB_MESSAGE_BASIC: (28e4844b147a2261):
Worker pool stopped.
root: INFO: 2017-05-10T01:44:18.788Z: JOB_MESSAGE_DEBUG: (28e4844b147a2c0e):
Tearing down pending resources...
root: INFO: Job 2017-05-09_18_38_19-10569987656720048055 is in state
JOB_STATE_FAILED
--------------------- >> end captured logging << ---------------------
----------------------------------------------------------------------
Ran 15 tests in 1466.598s
FAILED (errors=14)
Found:
https://console.cloud.google.com/dataflow/job/2017-05-09_18_20_22-7671973164766496890?project=apache-beam-testing
Found:
https://console.cloud.google.com/dataflow/job/2017-05-09_18_26_03-17445787645204623552?project=apache-beam-testing
Found:
https://console.cloud.google.com/dataflow/job/2017-05-09_18_32_14-16246056621494600472?project=apache-beam-testing
Found:
https://console.cloud.google.com/dataflow/job/2017-05-09_18_38_19-10569987656720048055?project=apache-beam-testing
Found:
https://console.cloud.google.com/dataflow/job/2017-05-09_18_20_22-4969442310096312321?project=apache-beam-testing
Found:
https://console.cloud.google.com/dataflow/job/2017-05-09_18_26_21-2200428839861560191?project=apache-beam-testing
Found:
https://console.cloud.google.com/dataflow/job/2017-05-09_18_33_24-13851441563056050815?project=apache-beam-testing
Found:
https://console.cloud.google.com/dataflow/job/2017-05-09_18_20_22-8294999073061692102?project=apache-beam-testing
Found:
https://console.cloud.google.com/dataflow/job/2017-05-09_18_26_16-11540265905883083825?project=apache-beam-testing
Found:
https://console.cloud.google.com/dataflow/job/2017-05-09_18_32_17-7192357206309032277?project=apache-beam-testing
Found:
https://console.cloud.google.com/dataflow/job/2017-05-09_18_38_33-1006407180017650573?project=apache-beam-testing
Found:
https://console.cloud.google.com/dataflow/job/2017-05-09_18_20_21-14971386839334709645?project=apache-beam-testing
Found:
https://console.cloud.google.com/dataflow/job/2017-05-09_18_25_46-15991440303187381446?project=apache-beam-testing
Found:
https://console.cloud.google.com/dataflow/job/2017-05-09_18_31_49-14241720216081775588?project=apache-beam-testing
Found:
https://console.cloud.google.com/dataflow/job/2017-05-09_18_37_34-15670525954546806359?project=apache-beam-testing
Build step 'Execute shell' marked build as failure
Not sending mail to unregistered user k...@google.com
Not sending mail to unregistered user kirpic...@google.com
Not sending mail to unregistered user c...@google.com
