See <https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/1347/display/redirect?page=changes>
Changes: [tweise] [BEAM-5092] Prevent hash-lookup of schema on every record (#6268) ------------------------------------------ [...truncated 18.85 MB...] Aug 25, 2018 1:55:51 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding SpannerIO.Write/Write mutations to Cloud Spanner/Write mutations to Spanner as step s40 Aug 25, 2018 1:55:51 AM org.apache.beam.runners.dataflow.DataflowRunner run INFO: Staging pipeline description to gs://temp-storage-for-end-to-end-tests/spannerwriteit0testreportfailures-jenkins-0825015547-afa24c37/output/results/staging/ Aug 25, 2018 1:55:51 AM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading <115993 bytes, hash YDBQwluWX-0BZOZfeFnCmA> to gs://temp-storage-for-end-to-end-tests/spannerwriteit0testreportfailures-jenkins-0825015547-afa24c37/output/results/staging/pipeline-YDBQwluWX-0BZOZfeFnCmA.pb org.apache.beam.sdk.io.gcp.spanner.SpannerWriteIT > testReportFailures STANDARD_OUT Dataflow SDK version: 2.7.0-SNAPSHOT org.apache.beam.sdk.io.gcp.spanner.SpannerWriteIT > testReportFailures STANDARD_ERROR Aug 25, 2018 1:55:53 AM org.apache.beam.runners.dataflow.DataflowRunner run INFO: To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-08-24_18_55_51-14427569544604155108?project=apache-beam-testing org.apache.beam.sdk.io.gcp.spanner.SpannerWriteIT > testReportFailures STANDARD_OUT Submitted job: 2018-08-24_18_55_51-14427569544604155108 org.apache.beam.sdk.io.gcp.spanner.SpannerWriteIT > testReportFailures STANDARD_ERROR Aug 25, 2018 1:55:53 AM org.apache.beam.runners.dataflow.DataflowRunner run INFO: To cancel the job using the 'gcloud' tool, run: > gcloud dataflow jobs --project=apache-beam-testing cancel --region=us-central1 2018-08-24_18_55_51-14427569544604155108 Aug 25, 2018 1:55:53 AM org.apache.beam.runners.dataflow.TestDataflowRunner run INFO: Running Dataflow job 2018-08-24_18_55_51-14427569544604155108 with 0 expected assertions. Aug 25, 2018 1:56:03 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-08-25T01:55:52.021Z: Autoscaling is enabled for job 2018-08-24_18_55_51-14427569544604155108. The number of workers will be between 1 and 1000. Aug 25, 2018 1:56:03 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-08-25T01:55:52.044Z: Autoscaling was automatically enabled for job 2018-08-24_18_55_51-14427569544604155108. Aug 25, 2018 1:56:03 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-08-25T01:55:54.756Z: Checking required Cloud APIs are enabled. Aug 25, 2018 1:56:03 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-08-25T01:55:55.046Z: Checking permissions granted to controller Service Account. Aug 25, 2018 1:56:03 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-08-25T01:55:58.871Z: Worker configuration: n1-standard-1 in us-central1-c. Aug 25, 2018 1:56:03 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-08-25T01:55:59.321Z: Expanding CoGroupByKey operations into optimizable parts. Aug 25, 2018 1:56:03 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-08-25T01:55:59.451Z: Expanding GroupByKey operations into optimizable parts. Aug 25, 2018 1:56:03 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-08-25T01:55:59.476Z: Lifting ValueCombiningMappingFns into MergeBucketsMappingFns Aug 25, 2018 1:56:03 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-08-25T01:55:59.653Z: Fusing adjacent ParDo, Read, Write, and Flatten operations Aug 25, 2018 1:56:03 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-08-25T01:55:59.674Z: Elided trivial flatten Aug 25, 2018 1:56:03 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-08-25T01:55:59.703Z: Fusing consumer SpannerIO.Write/Write mutations to Cloud Spanner/Wait.OnSignal/Wait/Map into SpannerIO.Write/Write mutations to Cloud Spanner/Create seed/Read(CreateSource) Aug 25, 2018 1:56:03 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-08-25T01:55:59.729Z: Fusing consumer SpannerIO.Write/Write mutations to Cloud Spanner/Read information schema into SpannerIO.Write/Write mutations to Cloud Spanner/Wait.OnSignal/Wait/Map Aug 25, 2018 1:56:03 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-08-25T01:55:59.755Z: Fusing consumer SpannerIO.Write/Write mutations to Cloud Spanner/Schema View/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Write into SpannerIO.Write/Write mutations to Cloud Spanner/Schema View/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey) Aug 25, 2018 1:56:03 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-08-25T01:55:59.783Z: Fusing consumer SpannerIO.Write/Write mutations to Cloud Spanner/Schema View/Combine.GloballyAsSingletonView/ParDo(IsmRecordForSingularValuePerWindow) into SpannerIO.Write/Write mutations to Cloud Spanner/Schema View/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Read Aug 25, 2018 1:56:03 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-08-25T01:55:59.806Z: Fusing consumer SpannerIO.Write/Write mutations to Cloud Spanner/Schema View/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/WithKeys/AddKeys/Map into SpannerIO.Write/Write mutations to Cloud Spanner/Read information schema Aug 25, 2018 1:56:03 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-08-25T01:55:59.833Z: Fusing consumer SpannerIO.Write/Write mutations to Cloud Spanner/Schema View/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/Combine.GroupedValues into SpannerIO.Write/Write mutations to Cloud Spanner/Schema View/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey/Read Aug 25, 2018 1:56:03 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-08-25T01:55:59.857Z: Fusing consumer SpannerIO.Write/Write mutations to Cloud Spanner/Schema View/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Values/Values/Map into SpannerIO.Write/Write mutations to Cloud Spanner/Schema View/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/Combine.GroupedValues/Extract Aug 25, 2018 1:56:03 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-08-25T01:55:59.882Z: Fusing consumer SpannerIO.Write/Write mutations to Cloud Spanner/Schema View/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey) into SpannerIO.Write/Write mutations to Cloud Spanner/Schema View/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Values/Values/Map Aug 25, 2018 1:56:03 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-08-25T01:55:59.908Z: Fusing consumer SpannerIO.Write/Write mutations to Cloud Spanner/Schema View/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/Combine.GroupedValues/Extract into SpannerIO.Write/Write mutations to Cloud Spanner/Schema View/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/Combine.GroupedValues Aug 25, 2018 1:56:03 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-08-25T01:55:59.937Z: Fusing consumer SpannerIO.Write/Write mutations to Cloud Spanner/Schema View/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey+SpannerIO.Write/Write mutations to Cloud Spanner/Schema View/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/Combine.GroupedValues/Partial into SpannerIO.Write/Write mutations to Cloud Spanner/Schema View/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/WithKeys/AddKeys/Map Aug 25, 2018 1:56:03 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-08-25T01:55:59.959Z: Fusing consumer SpannerIO.Write/Write mutations to Cloud Spanner/Schema View/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey/Reify into SpannerIO.Write/Write mutations to Cloud Spanner/Schema View/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey+SpannerIO.Write/Write mutations to Cloud Spanner/Schema View/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/Combine.GroupedValues/Partial Aug 25, 2018 1:56:03 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-08-25T01:55:59.987Z: Fusing consumer SpannerIO.Write/Write mutations to Cloud Spanner/Schema View/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey/Write into SpannerIO.Write/Write mutations to Cloud Spanner/Schema View/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey/Reify Aug 25, 2018 1:56:03 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-08-25T01:56:00.010Z: Fusing consumer SpannerIO.Write/Write mutations to Cloud Spanner/Wait.OnSignal/To wait view 0/ParDo(CollectWindows) into SpannerIO.Write/To mutation group Aug 25, 2018 1:56:03 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-08-25T01:56:00.032Z: Fusing consumer SpannerIO.Write/Write mutations to Cloud Spanner/Extract keys into SpannerIO.Write/Write mutations to Cloud Spanner/Serialize mutations Aug 25, 2018 1:56:03 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-08-25T01:56:00.057Z: Fusing consumer ParDo(GenerateMutations) into GenerateSequence/Read(BoundedCountingSource) Aug 25, 2018 1:56:03 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-08-25T01:56:00.080Z: Fusing consumer SpannerIO.Write/Write mutations to Cloud Spanner/Sample keys/Combine.GroupedValues into SpannerIO.Write/Write mutations to Cloud Spanner/Sample keys/GroupByKey/Read Aug 25, 2018 1:56:03 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-08-25T01:56:00.095Z: Fusing consumer SpannerIO.Write/Write mutations to Cloud Spanner/Batch mutations together into SpannerIO.Write/Write mutations to Cloud Spanner/Group by partition/GroupByWindow Aug 25, 2018 1:56:03 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-08-25T01:56:00.123Z: Fusing consumer SpannerIO.Write/Write mutations to Cloud Spanner/Keys sample as view/ParMultiDo(ToIsmRecordForMapLike) into SpannerIO.Write/Write mutations to Cloud Spanner/Keys sample as view/GBKaSVForData/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Read Aug 25, 2018 1:56:03 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-08-25T01:56:00.143Z: Fusing consumer SpannerIO.Write/Write mutations to Cloud Spanner/Keys sample as view/GBKaSVForData/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Write into SpannerIO.Write/Write mutations to Cloud Spanner/Keys sample as view/GBKaSVForData/ParDo(GroupByKeyHashAndSortByKeyAndWindow) Aug 25, 2018 1:56:03 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-08-25T01:56:00.168Z: Fusing consumer SpannerIO.Write/Write mutations to Cloud Spanner/Keys sample as view/GBKaSVForData/ParDo(GroupByKeyHashAndSortByKeyAndWindow) into SpannerIO.Write/Write mutations to Cloud Spanner/Sample keys/Combine.GroupedValues/Extract Aug 25, 2018 1:56:03 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-08-25T01:56:00.191Z: Fusing consumer SpannerIO.Write/Write mutations to Cloud Spanner/Sample keys/GroupByKey/Write into SpannerIO.Write/Write mutations to Cloud Spanner/Sample keys/GroupByKey/Reify Aug 25, 2018 1:56:03 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-08-25T01:56:00.212Z: Fusing consumer SpannerIO.Write/Write mutations to Cloud Spanner/Sample keys/Combine.GroupedValues/Extract into SpannerIO.Write/Write mutations to Cloud Spanner/Sample keys/Combine.GroupedValues Aug 25, 2018 1:56:03 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-08-25T01:56:00.237Z: Fusing consumer SpannerIO.Write/Write mutations to Cloud Spanner/Sample keys/GroupByKey/Reify into SpannerIO.Write/Write mutations to Cloud Spanner/Sample keys/GroupByKey+SpannerIO.Write/Write mutations to Cloud Spanner/Sample keys/Combine.GroupedValues/Partial Aug 25, 2018 1:56:03 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-08-25T01:56:00.262Z: Fusing consumer SpannerIO.Write/Write mutations to Cloud Spanner/Sample keys/GroupByKey+SpannerIO.Write/Write mutations to Cloud Spanner/Sample keys/Combine.GroupedValues/Partial into SpannerIO.Write/Write mutations to Cloud Spanner/Extract keys Aug 25, 2018 1:56:03 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-08-25T01:56:00.286Z: Fusing consumer SpannerIO.Write/Write mutations to Cloud Spanner/Wait.OnSignal/To wait view 0/Sample.Any/Combine.globally(SampleAny)/Combine.perKey(SampleAny)/GroupByKey/Write into SpannerIO.Write/Write mutations to Cloud Spanner/Wait.OnSignal/To wait view 0/Sample.Any/Combine.globally(SampleAny)/Combine.perKey(SampleAny)/GroupByKey/Reify Aug 25, 2018 1:56:03 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-08-25T01:56:00.312Z: Fusing consumer SpannerIO.Write/Write mutations to Cloud Spanner/Keys sample as view/GBKaSVForSize/Write into SpannerIO.Write/Write mutations to Cloud Spanner/Keys sample as view/ParMultiDo(ToIsmRecordForMapLike) Aug 25, 2018 1:56:03 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-08-25T01:56:00.335Z: Fusing consumer SpannerIO.Write/Write mutations to Cloud Spanner/Wait.OnSignal/To wait view 0/View.AsList/ParDo(ToIsmRecordForGlobalWindow) into SpannerIO.Write/Write mutations to Cloud Spanner/Wait.OnSignal/To wait view 0/Sample.Any/Flatten.Iterables/FlattenIterables/FlatMap Aug 25, 2018 1:56:03 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-08-25T01:56:00.359Z: Fusing consumer SpannerIO.Write/Write mutations to Cloud Spanner/Keys sample as view/GBKaSVForKeys/Write into SpannerIO.Write/Write mutations to Cloud Spanner/Keys sample as view/ParMultiDo(ToIsmRecordForMapLike) Aug 25, 2018 1:56:03 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-08-25T01:56:00.385Z: Fusing consumer SpannerIO.Write/Write mutations to Cloud Spanner/Wait.OnSignal/To wait view 0/Sample.Any/Flatten.Iterables/FlattenIterables/FlatMap into SpannerIO.Write/Write mutations to Cloud Spanner/Wait.OnSignal/To wait view 0/Sample.Any/Combine.globally(SampleAny)/Values/Values/Map Aug 25, 2018 1:56:03 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-08-25T01:56:00.409Z: Fusing consumer SpannerIO.Write/Write mutations to Cloud Spanner/Wait.OnSignal/To wait view 0/Sample.Any/Combine.globally(SampleAny)/Values/Values/Map into SpannerIO.Write/Write mutations to Cloud Spanner/Wait.OnSignal/To wait view 0/Sample.Any/Combine.globally(SampleAny)/Combine.perKey(SampleAny)/Combine.GroupedValues/Extract Aug 25, 2018 1:56:03 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-08-25T01:56:00.431Z: Fusing consumer SpannerIO.Write/Write mutations to Cloud Spanner/Keys sample as view/ParDo(ToIsmMetadataRecordForKey) into SpannerIO.Write/Write mutations to Cloud Spanner/Keys sample as view/GBKaSVForKeys/Read Aug 25, 2018 1:56:03 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-08-25T01:56:00.458Z: Fusing consumer SpannerIO.Write/Write mutations to Cloud Spanner/Wait.OnSignal/To wait view 0/Sample.Any/Combine.globally(SampleAny)/Combine.perKey(SampleAny)/Combine.GroupedValues/Extract into SpannerIO.Write/Write mutations to Cloud Spanner/Wait.OnSignal/To wait view 0/Sample.Any/Combine.globally(SampleAny)/Combine.perKey(SampleAny)/Combine.GroupedValues Aug 25, 2018 1:56:03 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-08-25T01:56:00.481Z: Fusing consumer SpannerIO.Write/Write mutations to Cloud Spanner/Wait.OnSignal/To wait view 0/Sample.Any/Combine.globally(SampleAny)/WithKeys/AddKeys/Map into SpannerIO.Write/Write mutations to Cloud Spanner/Wait.OnSignal/To wait view 0/ParDo(CollectWindows) Aug 25, 2018 1:56:03 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-08-25T01:56:00.511Z: Fusing consumer SpannerIO.Write/Write mutations to Cloud Spanner/Wait.OnSignal/To wait view 0/Sample.Any/Combine.globally(SampleAny)/Combine.perKey(SampleAny)/Combine.GroupedValues into SpannerIO.Write/Write mutations to Cloud Spanner/Wait.OnSignal/To wait view 0/Sample.Any/Combine.globally(SampleAny)/Combine.perKey(SampleAny)/GroupByKey/Read Aug 25, 2018 1:56:03 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-08-25T01:56:00.539Z: Fusing consumer SpannerIO.Write/Write mutations to Cloud Spanner/Write mutations to Spanner into SpannerIO.Write/Write mutations to Cloud Spanner/Batch mutations together Aug 25, 2018 1:56:03 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-08-25T01:56:00.563Z: Fusing consumer SpannerIO.Write/Write mutations to Cloud Spanner/Wait.OnSignal/To wait view 0/Sample.Any/Combine.globally(SampleAny)/Combine.perKey(SampleAny)/GroupByKey/Reify into SpannerIO.Write/Write mutations to Cloud Spanner/Wait.OnSignal/To wait view 0/Sample.Any/Combine.globally(SampleAny)/Combine.perKey(SampleAny)/GroupByKey+SpannerIO.Write/Write mutations to Cloud Spanner/Wait.OnSignal/To wait view 0/Sample.Any/Combine.globally(SampleAny)/Combine.perKey(SampleAny)/Combine.GroupedValues/Partial Aug 25, 2018 1:56:03 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-08-25T01:56:00.585Z: Fusing consumer SpannerIO.Write/Write mutations to Cloud Spanner/Wait.OnSignal/To wait view 0/Sample.Any/Combine.globally(SampleAny)/Combine.perKey(SampleAny)/GroupByKey+SpannerIO.Write/Write mutations to Cloud Spanner/Wait.OnSignal/To wait view 0/Sample.Any/Combine.globally(SampleAny)/Combine.perKey(SampleAny)/Combine.GroupedValues/Partial into SpannerIO.Write/Write mutations to Cloud Spanner/Wait.OnSignal/To wait view 0/Sample.Any/Combine.globally(SampleAny)/WithKeys/AddKeys/Map Aug 25, 2018 1:56:03 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-08-25T01:56:00.614Z: Fusing consumer SpannerIO.Write/Write mutations to Cloud Spanner/Group by partition/Reify into SpannerIO.Write/Write mutations to Cloud Spanner/Partition input Aug 25, 2018 1:56:03 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-08-25T01:56:00.634Z: Fusing consumer SpannerIO.Write/To mutation group into ParDo(GenerateMutations) Aug 25, 2018 1:56:03 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-08-25T01:56:00.660Z: Fusing consumer SpannerIO.Write/Write mutations to Cloud Spanner/Group by partition/GroupByWindow into SpannerIO.Write/Write mutations to Cloud Spanner/Group by partition/Read Aug 25, 2018 1:56:03 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-08-25T01:56:00.682Z: Fusing consumer SpannerIO.Write/Write mutations to Cloud Spanner/Group by partition/Write into SpannerIO.Write/Write mutations to Cloud Spanner/Group by partition/Reify Aug 25, 2018 1:56:03 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-08-25T01:56:00.707Z: Fusing consumer SpannerIO.Write/Write mutations to Cloud Spanner/Keys sample as view/ParDo(ToIsmMetadataRecordForSize) into SpannerIO.Write/Write mutations to Cloud Spanner/Keys sample as view/GBKaSVForSize/Read Aug 25, 2018 1:56:03 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-08-25T01:56:01.130Z: Executing operation SpannerIO.Write/Write mutations to Cloud Spanner/Schema View/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey/Create Aug 25, 2018 1:56:03 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-08-25T01:56:01.166Z: Executing operation SpannerIO.Write/Write mutations to Cloud Spanner/Schema View/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Create Aug 25, 2018 1:56:03 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-08-25T01:56:01.208Z: Executing operation SpannerIO.Write/Write mutations to Cloud Spanner/Sample keys/GroupByKey/Create Aug 25, 2018 1:56:03 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-08-25T01:56:01.208Z: Starting 1 workers in us-central1-c... Aug 25, 2018 1:56:03 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-08-25T01:56:01.234Z: Executing operation SpannerIO.Write/Write mutations to Cloud Spanner/Keys sample as view/GBKaSVForData/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Create Aug 25, 2018 1:56:03 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-08-25T01:56:01.260Z: Executing operation SpannerIO.Write/Write mutations to Cloud Spanner/Keys sample as view/GBKaSVForSize/Create Aug 25, 2018 1:56:03 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-08-25T01:56:01.293Z: Executing operation SpannerIO.Write/Write mutations to Cloud Spanner/Keys sample as view/GBKaSVForKeys/Create Aug 25, 2018 1:56:03 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-08-25T01:56:01.322Z: Executing operation SpannerIO.Write/Write mutations to Cloud Spanner/Group by partition/Create Aug 25, 2018 1:56:03 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-08-25T01:56:01.343Z: Executing operation SpannerIO.Write/Write mutations to Cloud Spanner/Wait.OnSignal/To wait view 0/Sample.Any/Combine.globally(SampleAny)/Combine.perKey(SampleAny)/GroupByKey/Create Aug 25, 2018 1:56:03 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-08-25T01:56:01.791Z: Executing operation GenerateSequence/Read(BoundedCountingSource)+ParDo(GenerateMutations)+SpannerIO.Write/To mutation group+SpannerIO.Write/Write mutations to Cloud Spanner/Wait.OnSignal/To wait view 0/ParDo(CollectWindows)+SpannerIO.Write/Write mutations to Cloud Spanner/Wait.OnSignal/To wait view 0/Sample.Any/Combine.globally(SampleAny)/WithKeys/AddKeys/Map+SpannerIO.Write/Write mutations to Cloud Spanner/Wait.OnSignal/To wait view 0/Sample.Any/Combine.globally(SampleAny)/Combine.perKey(SampleAny)/GroupByKey+SpannerIO.Write/Write mutations to Cloud Spanner/Wait.OnSignal/To wait view 0/Sample.Any/Combine.globally(SampleAny)/Combine.perKey(SampleAny)/Combine.GroupedValues/Partial+SpannerIO.Write/Write mutations to Cloud Spanner/Wait.OnSignal/To wait view 0/Sample.Any/Combine.globally(SampleAny)/Combine.perKey(SampleAny)/GroupByKey/Reify+SpannerIO.Write/Write mutations to Cloud Spanner/Wait.OnSignal/To wait view 0/Sample.Any/Combine.globally(SampleAny)/Combine.perKey(SampleAny)/GroupByKey/Write Aug 25, 2018 1:56:14 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-08-25T01:56:12.614Z: Autoscaling: Raised the number of workers to 0 based on the rate of progress in the currently running step(s). Aug 25, 2018 1:56:34 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-08-25T01:56:33.586Z: Autoscaling: Raised the number of workers to 1 based on the rate of progress in the currently running step(s). Aug 25, 2018 1:56:34 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-08-25T01:56:33.634Z: Autoscaling: Would further reduce the number of workers but reached the minimum number allowed for the job. Aug 25, 2018 1:56:54 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-08-25T01:56:52.435Z: Workers have started successfully. Aug 25, 2018 1:56:54 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-08-25T01:56:52.474Z: Workers have started successfully. Aug 25, 2018 1:57:11 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-08-25T01:57:11.375Z: Executing operation SpannerIO.Write/Write mutations to Cloud Spanner/Wait.OnSignal/To wait view 0/Sample.Any/Combine.globally(SampleAny)/Combine.perKey(SampleAny)/GroupByKey/Close Aug 25, 2018 1:57:12 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-08-25T01:57:11.436Z: Executing operation SpannerIO.Write/Write mutations to Cloud Spanner/Wait.OnSignal/To wait view 0/Sample.Any/Combine.globally(SampleAny)/Combine.perKey(SampleAny)/GroupByKey/Read+SpannerIO.Write/Write mutations to Cloud Spanner/Wait.OnSignal/To wait view 0/Sample.Any/Combine.globally(SampleAny)/Combine.perKey(SampleAny)/Combine.GroupedValues+SpannerIO.Write/Write mutations to Cloud Spanner/Wait.OnSignal/To wait view 0/Sample.Any/Combine.globally(SampleAny)/Combine.perKey(SampleAny)/Combine.GroupedValues/Extract+SpannerIO.Write/Write mutations to Cloud Spanner/Wait.OnSignal/To wait view 0/Sample.Any/Combine.globally(SampleAny)/Values/Values/Map+SpannerIO.Write/Write mutations to Cloud Spanner/Wait.OnSignal/To wait view 0/Sample.Any/Flatten.Iterables/FlattenIterables/FlatMap+SpannerIO.Write/Write mutations to Cloud Spanner/Wait.OnSignal/To wait view 0/View.AsList/ParDo(ToIsmRecordForGlobalWindow) Aug 25, 2018 1:57:22 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-08-25T01:57:19.674Z: Executing operation SpannerIO.Write/Write mutations to Cloud Spanner/Wait.OnSignal/To wait view 0/View.AsList/CreateDataflowView Aug 25, 2018 1:57:22 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-08-25T01:57:19.804Z: Executing operation SpannerIO.Write/Write mutations to Cloud Spanner/Create seed/Read(CreateSource)+SpannerIO.Write/Write mutations to Cloud Spanner/Wait.OnSignal/Wait/Map+SpannerIO.Write/Write mutations to Cloud Spanner/Read information schema+SpannerIO.Write/Write mutations to Cloud Spanner/Schema View/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/WithKeys/AddKeys/Map+SpannerIO.Write/Write mutations to Cloud Spanner/Schema View/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey+SpannerIO.Write/Write mutations to Cloud Spanner/Schema View/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/Combine.GroupedValues/Partial+SpannerIO.Write/Write mutations to Cloud Spanner/Schema View/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey/Reify+SpannerIO.Write/Write mutations to Cloud Spanner/Schema View/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey/Write Aug 25, 2018 1:57:25 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-08-25T01:57:25.639Z: Executing operation SpannerIO.Write/Write mutations to Cloud Spanner/Schema View/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey/Close Aug 25, 2018 1:57:27 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-08-25T01:57:25.699Z: Executing operation SpannerIO.Write/Write mutations to Cloud Spanner/Schema View/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey/Read+SpannerIO.Write/Write mutations to Cloud Spanner/Schema View/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/Combine.GroupedValues+SpannerIO.Write/Write mutations to Cloud Spanner/Schema View/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/Combine.GroupedValues/Extract+SpannerIO.Write/Write mutations to Cloud Spanner/Schema View/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Values/Values/Map+SpannerIO.Write/Write mutations to Cloud Spanner/Schema View/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey)+SpannerIO.Write/Write mutations to Cloud Spanner/Schema View/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Write Aug 25, 2018 1:57:32 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-08-25T01:57:31.214Z: Executing operation SpannerIO.Write/Write mutations to Cloud Spanner/Schema View/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Close Aug 25, 2018 1:57:32 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-08-25T01:57:31.271Z: Executing operation SpannerIO.Write/Write mutations to Cloud Spanner/Schema View/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Read+SpannerIO.Write/Write mutations to Cloud Spanner/Schema View/Combine.GloballyAsSingletonView/ParDo(IsmRecordForSingularValuePerWindow) Aug 25, 2018 1:57:39 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-08-25T01:57:37.628Z: Executing operation SpannerIO.Write/Write mutations to Cloud Spanner/Schema View/Combine.GloballyAsSingletonView/CreateDataflowView Aug 25, 2018 1:57:39 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-08-25T01:57:37.743Z: Executing operation SpannerIO.Write/Write mutations to Cloud Spanner/Serialize mutations+SpannerIO.Write/Write mutations to Cloud Spanner/Extract keys+SpannerIO.Write/Write mutations to Cloud Spanner/Sample keys/GroupByKey+SpannerIO.Write/Write mutations to Cloud Spanner/Sample keys/Combine.GroupedValues/Partial+SpannerIO.Write/Write mutations to Cloud Spanner/Sample keys/GroupByKey/Reify+SpannerIO.Write/Write mutations to Cloud Spanner/Sample keys/GroupByKey/Write Aug 25, 2018 1:57:48 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-08-25T01:57:46.430Z: Executing operation SpannerIO.Write/Write mutations to Cloud Spanner/Sample keys/GroupByKey/Close Aug 25, 2018 1:57:48 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-08-25T01:57:46.491Z: Executing operation SpannerIO.Write/Write mutations to Cloud Spanner/Sample keys/GroupByKey/Read+SpannerIO.Write/Write mutations to Cloud Spanner/Sample keys/Combine.GroupedValues+SpannerIO.Write/Write mutations to Cloud Spanner/Sample keys/Combine.GroupedValues/Extract+SpannerIO.Write/Write mutations to Cloud Spanner/Keys sample as view/GBKaSVForData/ParDo(GroupByKeyHashAndSortByKeyAndWindow)+SpannerIO.Write/Write mutations to Cloud Spanner/Keys sample as view/GBKaSVForData/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Write Aug 25, 2018 1:57:49 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-08-25T01:57:48.871Z: Executing operation SpannerIO.Write/Write mutations to Cloud Spanner/Keys sample as view/GBKaSVForData/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Close Aug 25, 2018 1:57:49 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-08-25T01:57:48.941Z: Executing operation SpannerIO.Write/Write mutations to Cloud Spanner/Keys sample as view/GBKaSVForData/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Read+SpannerIO.Write/Write mutations to Cloud Spanner/Keys sample as view/ParMultiDo(ToIsmRecordForMapLike)+SpannerIO.Write/Write mutations to Cloud Spanner/Keys sample as view/GBKaSVForSize/Write+SpannerIO.Write/Write mutations to Cloud Spanner/Keys sample as view/GBKaSVForKeys/Write Aug 25, 2018 1:57:56 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-08-25T01:57:55.387Z: Executing operation SpannerIO.Write/Write mutations to Cloud Spanner/Keys sample as view/GBKaSVForSize/Close Aug 25, 2018 1:57:56 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-08-25T01:57:55.411Z: Executing operation SpannerIO.Write/Write mutations to Cloud Spanner/Keys sample as view/GBKaSVForKeys/Close Aug 25, 2018 1:57:56 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-08-25T01:57:55.459Z: Executing operation SpannerIO.Write/Write mutations to Cloud Spanner/Keys sample as view/GBKaSVForSize/Read+SpannerIO.Write/Write mutations to Cloud Spanner/Keys sample as view/ParDo(ToIsmMetadataRecordForSize) Aug 25, 2018 1:57:56 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-08-25T01:57:55.489Z: Executing operation SpannerIO.Write/Write mutations to Cloud Spanner/Keys sample as view/GBKaSVForKeys/Read+SpannerIO.Write/Write mutations to Cloud Spanner/Keys sample as view/ParDo(ToIsmMetadataRecordForKey) Aug 25, 2018 1:58:04 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-08-25T01:58:04.369Z: Executing operation SpannerIO.Write/Write mutations to Cloud Spanner/Keys sample as view/Flatten.PCollections Aug 25, 2018 1:58:04 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-08-25T01:58:04.545Z: Executing operation SpannerIO.Write/Write mutations to Cloud Spanner/Keys sample as view/CreateDataflowView Aug 25, 2018 1:58:04 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-08-25T01:58:04.677Z: Executing operation SpannerIO.Write/Write mutations to Cloud Spanner/Partition input+SpannerIO.Write/Write mutations to Cloud Spanner/Group by partition/Reify+SpannerIO.Write/Write mutations to Cloud Spanner/Group by partition/Write Aug 25, 2018 1:58:10 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-08-25T01:58:08.872Z: Executing operation SpannerIO.Write/Write mutations to Cloud Spanner/Group by partition/Close Aug 25, 2018 1:58:10 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-08-25T01:58:08.920Z: Executing operation SpannerIO.Write/Write mutations to Cloud Spanner/Group by partition/Read+SpannerIO.Write/Write mutations to Cloud Spanner/Group by partition/GroupByWindow+SpannerIO.Write/Write mutations to Cloud Spanner/Batch mutations together+SpannerIO.Write/Write mutations to Cloud Spanner/Write mutations to Spanner Aug 25, 2018 1:58:15 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-08-25T01:58:13.071Z: Cleaning up. Aug 25, 2018 1:58:15 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-08-25T01:58:13.273Z: Stopping worker pool... Aug 25, 2018 1:59:11 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-08-25T01:59:10.189Z: Autoscaling: Resized worker pool from 1 to 0. Aug 25, 2018 1:59:11 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-08-25T01:59:10.213Z: Autoscaling: Would further reduce the number of workers but reached the minimum number allowed for the job. Aug 25, 2018 1:59:19 AM org.apache.beam.runners.dataflow.DataflowPipelineJob waitUntilFinish INFO: Job 2018-08-24_18_55_51-14427569544604155108 finished with status DONE. Aug 25, 2018 1:59:19 AM org.apache.beam.runners.dataflow.TestDataflowRunner checkForPAssertSuccess INFO: Success result for Dataflow job 2018-08-24_18_55_51-14427569544604155108. Found 0 success, 0 failures out of 0 expected assertions. Aug 25, 2018 1:59:19 AM org.apache.beam.runners.dataflow.DataflowPipelineJob waitUntilFinish INFO: Job 2018-08-24_18_55_51-14427569544604155108 finished with status DONE. Gradle Test Executor 128 finished executing tests. > Task > :beam-runners-google-cloud-dataflow-java:googleCloudPlatformIntegrationTest Finished generating test XML results (0.011 secs) into: <https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/ws/src/runners/google-cloud-dataflow-java/build/test-results/googleCloudPlatformIntegrationTest> Generating HTML test report... Finished generating test html results (0.013 secs) into: <https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/ws/src/runners/google-cloud-dataflow-java/build/reports/tests/googleCloudPlatformIntegrationTest> Packing task ':beam-runners-google-cloud-dataflow-java:googleCloudPlatformIntegrationTest' :beam-runners-google-cloud-dataflow-java:googleCloudPlatformIntegrationTest (Thread[Task worker for ':' Thread 11,5,main]) completed. Took 31 mins 17.099 secs. FAILURE: Build failed with an exception. * What went wrong: Execution failed for task ':beam-examples-java:compileTestJava'. > Compilation failed with exit code 1; see the compiler error output for > details. * Try: Run with --stacktrace option to get the stack trace. Run with --debug option to get more log output. Run with --scan to get full insights. * Get more help at https://help.gradle.org Deprecated Gradle features were used in this build, making it incompatible with Gradle 5.0. See https://docs.gradle.org/4.8/userguide/command_line_interface.html#sec:command_line_warnings BUILD FAILED in 42m 29s 790 actionable tasks: 787 executed, 3 from cache Publishing build scan... https://gradle.com/s/qxt3ioxxiiow4 Build step 'Invoke Gradle script' changed build result to FAILURE Build step 'Invoke Gradle script' marked build as failure