See <https://ci-beam.apache.org/job/beam_PostCommit_Python39/2676/display/redirect?page=changes>
Changes: [noreply] Merge pull request #29687 [YAML] Better IO documentation. ------------------------------------------ [...truncated 12.16 MB...] [gw5] [32mPASSED[0m apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it [gw0] [32mPASSED[0m apache_beam/io/gcp/bigquery_read_it_test.py::ReadUsingStorageApiTests::test_iobase_source_with_very_selective_filters apache_beam/io/gcp/bigquery_read_it_test.py::ReadNewTypesTests::test_iobase_source [gw0] [32mPASSED[0m apache_beam/io/gcp/bigquery_read_it_test.py::ReadNewTypesTests::test_iobase_source apache_beam/io/gcp/bigquery_read_it_test.py::ReadNewTypesTests::test_native_source [gw2] [32mPASSED[0m apache_beam/io/gcp/datastore/v1new/datastore_write_it_test.py::DatastoreWriteIT::test_datastore_write_limit [gw0] [32mPASSED[0m apache_beam/io/gcp/bigquery_read_it_test.py::ReadNewTypesTests::test_native_source apache_beam/io/gcp/bigquery_read_it_test.py::ReadAllBQTests::test_read_queries [gw0] [32mPASSED[0m apache_beam/io/gcp/bigquery_read_it_test.py::ReadAllBQTests::test_read_queries apache_beam/io/gcp/bigquery_read_it_test.py::ReadInteractiveRunnerTests::test_read_in_interactive_runner [gw0] [32mPASSED[0m apache_beam/io/gcp/bigquery_read_it_test.py::ReadInteractiveRunnerTests::test_read_in_interactive_runner =================================== FAILURES =================================== [31m[1m____________________________ TaxirideIT.test_enrich ____________________________[0m [gw5] linux -- Python 3.9.10 <https://ci-beam.apache.org/job/beam_PostCommit_Python39/ws/src/build/gradleenv/-1734967050/bin/python3.9> self = <apache_beam.examples.dataframe.taxiride_it_test.TaxirideIT testMethod=test_enrich> @pytest.mark.it_postcommit def test_enrich(self): # Standard workers OOM with the enrich pipeline self.test_pipeline.get_pipeline_options().view_as( WorkerOptions).machine_type = 'e2-highmem-2' taxiride.run_enrich_pipeline( self.test_pipeline, 'gs://apache-beam-samples/nyc_taxi/2018/*.csv', self.output_path) # Verify expected = pd.read_csv( os.path.join( os.path.dirname(__file__), 'data', 'taxiride_2018_enrich_truth.csv'), comment='#') expected = expected.sort_values('Borough').reset_index(drop=True) def read_csv(path): with FileSystems.open(path) as fp: return pd.read_csv(fp) result = pd.concat( read_csv(metadata.path) for metadata in FileSystems.match( [f'{self.output_path}*'])[0].metadata_list) result = result.sort_values('Borough').reset_index(drop=True) > pd.testing.assert_frame_equal(expected, result) [1m[31mapache_beam/examples/dataframe/taxiride_it_test.py[0m:102: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [1m[31mpandas/_libs/testing.pyx[0m:52: in pandas._libs.testing.assert_almost_equal ??? _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > ??? [1m[31mE AssertionError: DataFrame.iloc[:, 1] (column name="passenger_count") are different[0m [1m[31mE [0m [1m[31mE DataFrame.iloc[:, 1] (column name="passenger_count") values are different (100.0 %)[0m [1m[31mE [index]: [0, 1, 2, 3, 4, 5, 6][0m [1m[31mE [left]: [1074423, 6972533, 322785, 145129096, 8024331, 39659, 2407529][0m [1m[31mE [right]: [1073783, 6967581, 323101, 145122094, 8018561, 39622, 2409682][0m [1m[31mpandas/_libs/testing.pyx[0m:167: AssertionError ------------------------------ Captured log call ------------------------------- [32mINFO [0m root:transforms.py:182 Computing dataframe stage <ComputeStage(PTransform) label=[[ComputedExpression[set_index_DataFrame_140491368428400]]:140491368362384]> for Stage[inputs={PlaceholderExpression[placeholder_DataFrame_140491368286816]}, partitioning=Arbitrary, ops=[ComputedExpression[set_index_DataFrame_140491368428400]], outputs={PlaceholderExpression[placeholder_DataFrame_140491368286816], ComputedExpression[set_index_DataFrame_140491368428400]}] [32mINFO [0m root:transforms.py:182 Computing dataframe stage <ComputeStage(PTransform) label=[[ComputedExpression[set_index_DataFrame_140491368287584], ComputedExpression[get_column_Series_140491368403344]]:140491368364400]> for Stage[inputs={PlaceholderExpression[placeholder_DataFrame_140491368323536]}, partitioning=Arbitrary, ops=[ComputedExpression[set_index_DataFrame_140491368287584], ComputedExpression[get_column_Series_140491368403344]], outputs={PlaceholderExpression[placeholder_DataFrame_140491368323536], ComputedExpression[get_column_Series_140491368403344]}] [32mINFO [0m root:transforms.py:182 Computing dataframe stage <ComputeStage(PTransform) label=[[ComputedExpression[merge_DataFrame_140491368403104], ComputedExpression[set_index_DataFrame_140491368405456], ComputedExpression[pre_combine_sum_Series_140491367964832]]:140491368711072]> for Stage[inputs={ComputedExpression[set_index_DataFrame_140491368428400], ComputedExpression[get_column_Series_140491368403344]}, partitioning=Index, ops=[ComputedExpression[merge_DataFrame_140491368403104], ComputedExpression[set_index_DataFrame_140491368405456], ComputedExpression[pre_combine_sum_Series_140491367964832]], outputs={ComputedExpression[pre_combine_sum_Series_140491367964832]}] [32mINFO [0m root:transforms.py:182 Computing dataframe stage <ComputeStage(PTransform) label=[[ComputedExpression[post_combine_sum_Series_140491367967952]]:140491368086592]> for Stage[inputs={ComputedExpression[pre_combine_sum_Series_140491367964832]}, partitioning=Index, ops=[ComputedExpression[post_combine_sum_Series_140491367967952]], outputs={ComputedExpression[post_combine_sum_Series_140491367967952]}] [32mINFO [0m apache_beam.io.fileio:fileio.py:611 Added temporary directory gs://temp-storage-for-end-to-end-tests/temp-it/.temp1802d389-164b-42ac-83ed-c816f328ff92 [32mINFO [0m apache_beam.runners.portability.stager:stager.py:322 Copying Beam SDK "<https://ci-beam.apache.org/job/beam_PostCommit_Python39/ws/src/sdks/python/build/apache_beam-2.54.0.dev0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl";> to staging location. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:395 Pipeline has additional dependencies to be installed in SDK worker container, consider using the SDK container image pre-building workflow to avoid repetitive installations. Learn more on https://cloud.google.com/dataflow/docs/guides/using-custom-containers#prebuild [32mINFO [0m root:environments.py:314 Using provided Python SDK container image: gcr.io/cloud-dataflow/v1beta3/beam_python3.9_sdk:beam-master-20231215 [32mINFO [0m root:environments.py:321 Python SDK container image set to "gcr.io/cloud-dataflow/v1beta3/beam_python3.9_sdk:beam-master-20231215" for Docker environment [32mINFO [0m apache_beam.runners.portability.fn_api_runner.translations:translations.py:712 ==================== <function pack_combiners at 0x7fc6e57c51f0> ==================== [32mINFO [0m apache_beam.runners.portability.fn_api_runner.translations:translations.py:712 ==================== <function sort_stages at 0x7fc6e57c59d0> ==================== [32mINFO [0m apache_beam.runners.dataflow.internal.apiclient:apiclient.py:677 Starting GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-1220185217-295783-bfj24uz3.1703098337.295973/apache_beam-2.54.0.dev0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl... [32mINFO [0m apache_beam.runners.dataflow.internal.apiclient:apiclient.py:687 Completed GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-1220185217-295783-bfj24uz3.1703098337.295973/apache_beam-2.54.0.dev0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl in 0 seconds. [32mINFO [0m apache_beam.runners.dataflow.internal.apiclient:apiclient.py:677 Starting GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-1220185217-295783-bfj24uz3.1703098337.295973/pipeline.pb... [32mINFO [0m apache_beam.runners.dataflow.internal.apiclient:apiclient.py:687 Completed GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-1220185217-295783-bfj24uz3.1703098337.295973/pipeline.pb in 0 seconds. [33mWARNING [0m apache_beam.options.pipeline_options:pipeline_options.py:339 Unknown pipeline options received: --sleep_secs=20,--kms_key_name=projects/apache-beam-testing/locations/global/keyRings/beam-it/cryptoKeys/test. Ignore if flags are used for internal purposes. [33mWARNING [0m apache_beam.options.pipeline_options:pipeline_options.py:339 Unknown pipeline options received: --sleep_secs=20,--kms_key_name=projects/apache-beam-testing/locations/global/keyRings/beam-it/cryptoKeys/test. Ignore if flags are used for internal purposes. [32mINFO [0m apache_beam.runners.dataflow.internal.apiclient:apiclient.py:858 Create job: <Job clientRequestId: '20231220185217296869-2087' createTime: '2023-12-20T18:52:18.960293Z' currentStateTime: '1970-01-01T00:00:00Z' id: '2023-12-20_10_52_18-14136225618669481975' location: 'us-central1' name: 'beamapp-jenkins-1220185217-295783-bfj24uz3' projectId: 'apache-beam-testing' stageStates: [] startTime: '2023-12-20T18:52:18.960293Z' steps: [] tempFiles: [] type: TypeValueValuesEnum(JOB_TYPE_BATCH, 1)> [32mINFO [0m apache_beam.runners.dataflow.internal.apiclient:apiclient.py:860 Created job with id: [2023-12-20_10_52_18-14136225618669481975] [32mINFO [0m apache_beam.runners.dataflow.internal.apiclient:apiclient.py:861 Submitted job: 2023-12-20_10_52_18-14136225618669481975 [32mINFO [0m apache_beam.runners.dataflow.internal.apiclient:apiclient.py:862 To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobs/us-central1/2023-12-20_10_52_18-14136225618669481975?project=apache-beam-testing [32mINFO [0m apache_beam.runners.dataflow.test_dataflow_runner:test_dataflow_runner.py:58 Console log: [32mINFO [0m apache_beam.runners.dataflow.test_dataflow_runner:test_dataflow_runner.py:59 https://console.cloud.google.com/dataflow/jobs/us-central1/2023-12-20_10_52_18-14136225618669481975?project=apache-beam-testing [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:151 Job 2023-12-20_10_52_18-14136225618669481975 is in state JOB_STATE_RUNNING [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:201 2023-12-20T18:52:22.245Z: JOB_MESSAGE_BASIC: Worker configuration: e2-highmem-2 in us-central1-a. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:201 2023-12-20T18:52:26.912Z: JOB_MESSAGE_BASIC: Executing operation Read taxi rides/Reshuffle/ReshufflePerKey/GroupByKey/Create [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:201 2023-12-20T18:52:26.929Z: JOB_MESSAGE_BASIC: Executing operation Read zone lookup/Reshuffle/ReshufflePerKey/GroupByKey/Create [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:201 2023-12-20T18:52:26.964Z: JOB_MESSAGE_BASIC: Starting 1 workers in us-central1-a... [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:201 2023-12-20T18:52:27.435Z: JOB_MESSAGE_BASIC: Finished operation Read taxi rides/Reshuffle/ReshufflePerKey/GroupByKey/Create [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:201 2023-12-20T18:52:31.590Z: JOB_MESSAGE_BASIC: Your project already contains 100 Dataflow-created metric descriptors, so new user metrics of the form custom.googleapis.com/* will not be created. However, all user metrics are also available in the metric dataflow.googleapis.com/job/user_counter. If you rely on the custom metrics, you can delete old / unused metric descriptors. See https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list and https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.delete [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:201 2023-12-20T18:52:36.165Z: JOB_MESSAGE_BASIC: Executing operation Read taxi rides/Create/Impulse+Read taxi rides/Create/FlatMap(<lambda at core.py:3774>)+Read taxi rides/Create/Map(decode)+Read taxi rides/MatchAll/ParDo(_MatchAllFn)+Read taxi rides/Map(<lambda at io.py:299>)+Read taxi rides/Reshuffle/AddRandomKeys+Read taxi rides/Reshuffle/ReshufflePerKey/Map(reify_timestamps)+Read taxi rides/Reshuffle/ReshufflePerKey/GroupByKey/Reify+Read taxi rides/Reshuffle/ReshufflePerKey/GroupByKey/Write [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:201 2023-12-20T18:52:36.184Z: JOB_MESSAGE_BASIC: Finished operation Read zone lookup/Reshuffle/ReshufflePerKey/GroupByKey/Create [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:201 2023-12-20T18:52:36.318Z: JOB_MESSAGE_BASIC: Executing operation Read zone lookup/Create/Impulse+Read zone lookup/Create/FlatMap(<lambda at core.py:3774>)+Read zone lookup/Create/Map(decode)+Read zone lookup/MatchAll/ParDo(_MatchAllFn)+Read zone lookup/Map(<lambda at io.py:299>)+Read zone lookup/Reshuffle/AddRandomKeys+Read zone lookup/Reshuffle/ReshufflePerKey/Map(reify_timestamps)+Read zone lookup/Reshuffle/ReshufflePerKey/GroupByKey/Reify+Read zone lookup/Reshuffle/ReshufflePerKey/GroupByKey/Write [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:201 2023-12-20T18:55:29.280Z: JOB_MESSAGE_BASIC: All workers have finished the startup processes and began to receive work requests. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:201 2023-12-20T18:55:30.636Z: JOB_MESSAGE_BASIC: Finished operation Read taxi rides/Create/Impulse+Read taxi rides/Create/FlatMap(<lambda at core.py:3774>)+Read taxi rides/Create/Map(decode)+Read taxi rides/MatchAll/ParDo(_MatchAllFn)+Read taxi rides/Map(<lambda at io.py:299>)+Read taxi rides/Reshuffle/AddRandomKeys+Read taxi rides/Reshuffle/ReshufflePerKey/Map(reify_timestamps)+Read taxi rides/Reshuffle/ReshufflePerKey/GroupByKey/Reify+Read taxi rides/Reshuffle/ReshufflePerKey/GroupByKey/Write [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:201 2023-12-20T18:55:30.694Z: JOB_MESSAGE_BASIC: Finished operation Read zone lookup/Create/Impulse+Read zone lookup/Create/FlatMap(<lambda at core.py:3774>)+Read zone lookup/Create/Map(decode)+Read zone lookup/MatchAll/ParDo(_MatchAllFn)+Read zone lookup/Map(<lambda at io.py:299>)+Read zone lookup/Reshuffle/AddRandomKeys+Read zone lookup/Reshuffle/ReshufflePerKey/Map(reify_timestamps)+Read zone lookup/Reshuffle/ReshufflePerKey/GroupByKey/Reify+Read zone lookup/Reshuffle/ReshufflePerKey/GroupByKey/Write [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:201 2023-12-20T18:55:30.720Z: JOB_MESSAGE_BASIC: Executing operation Read taxi rides/Reshuffle/ReshufflePerKey/GroupByKey/Close [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:201 2023-12-20T18:55:30.765Z: JOB_MESSAGE_BASIC: Executing operation Read zone lookup/Reshuffle/ReshufflePerKey/GroupByKey/Close [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:201 2023-12-20T18:55:30.766Z: JOB_MESSAGE_BASIC: Finished operation Read taxi rides/Reshuffle/ReshufflePerKey/GroupByKey/Close [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:201 2023-12-20T18:55:30.787Z: JOB_MESSAGE_BASIC: Executing operation Read taxi rides/Map(<lambda at io.py:295>)/View-python_side_input0 [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:201 2023-12-20T18:55:30.811Z: JOB_MESSAGE_BASIC: Executing operation Read zone lookup/Map(<lambda at io.py:295>)/View-python_side_input0 [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:201 2023-12-20T18:55:30.819Z: JOB_MESSAGE_BASIC: Finished operation Read zone lookup/Reshuffle/ReshufflePerKey/GroupByKey/Close [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:201 2023-12-20T18:55:30.836Z: JOB_MESSAGE_BASIC: Finished operation Read taxi rides/Map(<lambda at io.py:295>)/View-python_side_input0 [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:201 2023-12-20T18:55:30.857Z: JOB_MESSAGE_BASIC: Finished operation Read zone lookup/Map(<lambda at io.py:295>)/View-python_side_input0 [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:201 2023-12-20T18:55:30.935Z: JOB_MESSAGE_BASIC: Executing operation Read taxi rides/DoOnce/Impulse+Read taxi rides/DoOnce/FlatMap(<lambda at core.py:3774>)+Read taxi rides/DoOnce/Map(decode)+Read taxi rides/Map(<lambda at io.py:295>) [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:201 2023-12-20T18:55:30.956Z: JOB_MESSAGE_BASIC: Executing operation Read zone lookup/DoOnce/Impulse+Read zone lookup/DoOnce/FlatMap(<lambda at core.py:3774>)+Read zone lookup/DoOnce/Map(decode)+Read zone lookup/Map(<lambda at io.py:295>) [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:201 2023-12-20T18:55:32.682Z: JOB_MESSAGE_BASIC: Finished operation Read taxi rides/DoOnce/Impulse+Read taxi rides/DoOnce/FlatMap(<lambda at core.py:3774>)+Read taxi rides/DoOnce/Map(decode)+Read taxi rides/Map(<lambda at io.py:295>) [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:201 2023-12-20T18:55:32.774Z: JOB_MESSAGE_BASIC: Executing operation Read taxi rides/ParDo(_ReadFromPandasDoFn)/View-python_side_input0 [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:201 2023-12-20T18:55:32.810Z: JOB_MESSAGE_BASIC: Finished operation Read zone lookup/DoOnce/Impulse+Read zone lookup/DoOnce/FlatMap(<lambda at core.py:3774>)+Read zone lookup/DoOnce/Map(decode)+Read zone lookup/Map(<lambda at io.py:295>) [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:201 2023-12-20T18:55:32.819Z: JOB_MESSAGE_BASIC: Finished operation Read taxi rides/ParDo(_ReadFromPandasDoFn)/View-python_side_input0 [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:201 2023-12-20T18:55:32.911Z: JOB_MESSAGE_BASIC: Executing operation Read zone lookup/ParDo(_ReadFromPandasDoFn)/View-python_side_input0 [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:201 2023-12-20T18:55:32.929Z: JOB_MESSAGE_BASIC: Executing operation Read taxi rides/Reshuffle/ReshufflePerKey/GroupByKey/Read+Read taxi rides/Reshuffle/ReshufflePerKey/GroupByKey/GroupByWindow+Read taxi rides/Reshuffle/ReshufflePerKey/FlatMap(restore_timestamps)+Read taxi rides/Reshuffle/RemoveRandomKeys+Read taxi rides/ReadMatches/ParDo(_ReadMatchesFn)+ref_AppliedPTransform_Read-taxi-rides-ParDo-_ReadFromPandasDoFn-_26/PairWithRestriction+ref_AppliedPTransform_Read-taxi-rides-ParDo-_ReadFromPandasDoFn-_26/SplitWithSizing [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:201 2023-12-20T18:55:33.045Z: JOB_MESSAGE_BASIC: Finished operation Read zone lookup/ParDo(_ReadFromPandasDoFn)/View-python_side_input0 [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:201 2023-12-20T18:55:33.134Z: JOB_MESSAGE_BASIC: Executing operation Read zone lookup/Reshuffle/ReshufflePerKey/GroupByKey/Read+Read zone lookup/Reshuffle/ReshufflePerKey/GroupByKey/GroupByWindow+Read zone lookup/Reshuffle/ReshufflePerKey/FlatMap(restore_timestamps)+Read zone lookup/Reshuffle/RemoveRandomKeys+Read zone lookup/ReadMatches/ParDo(_ReadMatchesFn)+ref_AppliedPTransform_Read-zone-lookup-ParDo-_ReadFromPandasDoFn-_51/PairWithRestriction+ref_AppliedPTransform_Read-zone-lookup-ParDo-_ReadFromPandasDoFn-_51/SplitWithSizing [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:201 2023-12-20T18:55:34.994Z: JOB_MESSAGE_BASIC: Finished operation Read taxi rides/Reshuffle/ReshufflePerKey/GroupByKey/Read+Read taxi rides/Reshuffle/ReshufflePerKey/GroupByKey/GroupByWindow+Read taxi rides/Reshuffle/ReshufflePerKey/FlatMap(restore_timestamps)+Read taxi rides/Reshuffle/RemoveRandomKeys+Read taxi rides/ReadMatches/ParDo(_ReadMatchesFn)+ref_AppliedPTransform_Read-taxi-rides-ParDo-_ReadFromPandasDoFn-_26/PairWithRestriction+ref_AppliedPTransform_Read-taxi-rides-ParDo-_ReadFromPandasDoFn-_26/SplitWithSizing [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:201 2023-12-20T18:55:35.124Z: JOB_MESSAGE_BASIC: Finished operation Read zone lookup/Reshuffle/ReshufflePerKey/GroupByKey/Read+Read zone lookup/Reshuffle/ReshufflePerKey/GroupByKey/GroupByWindow+Read zone lookup/Reshuffle/ReshufflePerKey/FlatMap(restore_timestamps)+Read zone lookup/Reshuffle/RemoveRandomKeys+Read zone lookup/ReadMatches/ParDo(_ReadMatchesFn)+ref_AppliedPTransform_Read-zone-lookup-ParDo-_ReadFromPandasDoFn-_51/PairWithRestriction+ref_AppliedPTransform_Read-zone-lookup-ParDo-_ReadFromPandasDoFn-_51/SplitWithSizing [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:201 2023-12-20T18:55:35.234Z: JOB_MESSAGE_BASIC: Executing operation ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/taxiride_it-093ddde7-b25b-4a55-9735-ebcb3e360370/output.csv/[ComputedExpression[post_combine_sum_Series_140491367967952]]:140491368086592/SumSizes/CombinePerKey/GroupByKey/Create [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:201 2023-12-20T18:55:35.261Z: JOB_MESSAGE_BASIC: Executing operation ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/taxiride_it-093ddde7-b25b-4a55-9735-ebcb3e360370/output.csv/[ComputedExpression[merge_DataFrame_140491368403104], ComputedExpression[set_index_DataFrame_140491368405456], ComputedExpression[pre_combine_sum_Series_140491367964832]]:140491368711072/CoGroupByKey/CoGroupByKeyImpl/GroupByKey/Create [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:201 2023-12-20T18:55:35.986Z: JOB_MESSAGE_BASIC: Finished operation ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/taxiride_it-093ddde7-b25b-4a55-9735-ebcb3e360370/output.csv/[ComputedExpression[post_combine_sum_Series_140491367967952]]:140491368086592/SumSizes/CombinePerKey/GroupByKey/Create [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:201 2023-12-20T18:55:35.986Z: JOB_MESSAGE_BASIC: Finished operation ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/taxiride_it-093ddde7-b25b-4a55-9735-ebcb3e360370/output.csv/[ComputedExpression[merge_DataFrame_140491368403104], ComputedExpression[set_index_DataFrame_140491368405456], ComputedExpression[pre_combine_sum_Series_140491367964832]]:140491368711072/CoGroupByKey/CoGroupByKeyImpl/GroupByKey/Create [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:201 2023-12-20T18:55:36.111Z: JOB_MESSAGE_BASIC: Executing operation ref_AppliedPTransform_Read-zone-lookup-ParDo-_ReadFromPandasDoFn-_51/ProcessElementAndRestrictionWithSizing+ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/taxiride_it-093ddde7-b25b-4a55-9735-ebcb3e360370/output.csv/[ComputedExpression[set_index_DataFrame_140491368287584], ComputedExpression[get_column_Series_140491368403344]]:140491368364400/Map(<lambda at transforms.py:245>)+ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/taxiride_it-093ddde7-b25b-4a55-9735-ebcb3e360370/output.csv/[ComputedExpression[set_index_DataFrame_140491368287584], ComputedExpression[get_column_Series_140491368403344]]:140491368364400/FlatMap(evaluate)/FlatMap(evaluate)+ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/taxiride_it-093ddde7-b25b-4a55-9735-ebcb3e360370/output.csv/[ComputedExpression[merge_DataFrame_140491368403104], ComputedExpression[set_index_DataFrame_140491368405456], ComputedExpression[pre_combine_sum_Series_140491367964832]]:140491368711072/Partition_Index_get_column_Series_140491368403344/ParDo(_PreBatch)+ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/taxiride_it-093ddde7-b25b-4a55-9735-ebcb3e360370/output.csv/[ComputedExpression[post_combine_sum_Series_140491367967952]]:140491368086592/Size[get_column_Series_140491368403344, True]+ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/taxiride_it-093ddde7-b25b-4a55-9735-ebcb3e360370/output.csv/[ComputedExpression[post_combine_sum_Series_140491367967952]]:140491368086592/SumSizes/KeyWithVoid+ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/taxiride_it-093ddde7-b25b-4a55-9735-ebcb3e360370/output.csv/[ComputedExpression[post_combine_sum_Series_140491367967952]]:140491368086592/SumSizes/CombinePerKey/GroupByKey+ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/taxiride_it-093ddde7-b25b-4a55-9735-ebcb3e360370/output.csv/[ComputedExpression[post_combine_sum_Series_140491367967952]]:140491368086592/SumSizes/CombinePerKey/Combine/Partial+ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/taxiride_it-093ddde7-b25b-4a55-9735-ebcb3e360370/output.csv/[ComputedExpression[post_combine_sum_Series_140491367967952]]:140491368086592/SumSizes/CombinePerKey/GroupByKey/Write+ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/taxiride_it-093ddde7-b25b-4a55-9735-ebcb3e360370/output.csv/[ComputedExpression[merge_DataFrame_140491368403104], ComputedExpression[set_index_DataFrame_140491368405456], ComputedExpression[pre_combine_sum_Series_140491367964832]]:140491368711072/Partition_Index_get_column_Series_140491368403344/FlatMap(partition_fn)+ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/taxiride_it-093ddde7-b25b-4a55-9735-ebcb3e360370/output.csv/[ComputedExpression[merge_DataFrame_140491368403104], ComputedExpression[set_index_DataFrame_140491368405456], ComputedExpression[pre_combine_sum_Series_140491367964832]]:140491368711072/Partition_Index_get_column_Series_140491368403344/Filter(<lambda at transforms.py:231>)+ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/taxiride_it-093ddde7-b25b-4a55-9735-ebcb3e360370/output.csv/[ComputedExpression[merge_DataFrame_140491368403104], ComputedExpression[set_index_DataFrame_140491368405456], ComputedExpression[pre_combine_sum_Series_140491367964832]]:140491368711072/CoGroupByKey/CoGroupByKeyImpl/Tag[1]+ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/taxiride_it-093ddde7-b25b-4a55-9735-ebcb3e360370/output.csv/[ComputedExpression[merge_DataFrame_140491368403104], ComputedExpression[set_index_DataFrame_140491368405456], ComputedExpression[pre_combine_sum_Series_140491367964832]]:140491368711072/CoGroupByKey/CoGroupByKeyImpl/Flatten/OutputIdentity+ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/taxiride_it-093ddde7-b25b-4a55-9735-ebcb3e360370/output.csv/[ComputedExpression[merge_DataFrame_140491368403104], ComputedExpression[set_index_DataFrame_140491368405456], ComputedExpression[pre_combine_sum_Series_140491367964832]]:140491368711072/CoGroupByKey/CoGroupByKeyImpl/GroupByKey/Write [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:201 2023-12-20T18:55:36.127Z: JOB_MESSAGE_BASIC: Executing operation ref_AppliedPTransform_Read-taxi-rides-ParDo-_ReadFromPandasDoFn-_26/ProcessElementAndRestrictionWithSizing+ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/taxiride_it-093ddde7-b25b-4a55-9735-ebcb3e360370/output.csv/[ComputedExpression[set_index_DataFrame_140491368428400]]:140491368362384/Map(<lambda at transforms.py:245>)+ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/taxiride_it-093ddde7-b25b-4a55-9735-ebcb3e360370/output.csv/[ComputedExpression[set_index_DataFrame_140491368428400]]:140491368362384/FlatMap(evaluate)/FlatMap(evaluate)+ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/taxiride_it-093ddde7-b25b-4a55-9735-ebcb3e360370/output.csv/[ComputedExpression[merge_DataFrame_140491368403104], ComputedExpression[set_index_DataFrame_140491368405456], ComputedExpression[pre_combine_sum_Series_140491367964832]]:140491368711072/Partition_Index_set_index_DataFrame_140491368428400/ParDo(_PreBatch)+ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/taxiride_it-093ddde7-b25b-4a55-9735-ebcb3e360370/output.csv/[ComputedExpression[post_combine_sum_Series_140491367967952]]:140491368086592/Size[set_index_DataFrame_140491368428400, True]+ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/taxiride_it-093ddde7-b25b-4a55-9735-ebcb3e360370/output.csv/[ComputedExpression[post_combine_sum_Series_140491367967952]]:140491368086592/SumSizes/KeyWithVoid+ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/taxiride_it-093ddde7-b25b-4a55-9735-ebcb3e360370/output.csv/[ComputedExpression[post_combine_sum_Series_140491367967952]]:140491368086592/SumSizes/CombinePerKey/GroupByKey+ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/taxiride_it-093ddde7-b25b-4a55-9735-ebcb3e360370/output.csv/[ComputedExpression[post_combine_sum_Series_140491367967952]]:140491368086592/SumSizes/CombinePerKey/Combine/Partial+ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/taxiride_it-093ddde7-b25b-4a55-9735-ebcb3e360370/output.csv/[ComputedExpression[post_combine_sum_Series_140491367967952]]:140491368086592/SumSizes/CombinePerKey/GroupByKey/Write+ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/taxiride_it-093ddde7-b25b-4a55-9735-ebcb3e360370/output.csv/[ComputedExpression[merge_DataFrame_140491368403104], ComputedExpression[set_index_DataFrame_140491368405456], ComputedExpression[pre_combine_sum_Series_140491367964832]]:140491368711072/Partition_Index_set_index_DataFrame_140491368428400/FlatMap(partition_fn)+ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/taxiride_it-093ddde7-b25b-4a55-9735-ebcb3e360370/output.csv/[ComputedExpression[merge_DataFrame_140491368403104], ComputedExpression[set_index_DataFrame_140491368405456], ComputedExpression[pre_combine_sum_Series_140491367964832]]:140491368711072/Partition_Index_set_index_DataFrame_140491368428400/Filter(<lambda at transforms.py:231>)+ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/taxiride_it-093ddde7-b25b-4a55-9735-ebcb3e360370/output.csv/[ComputedExpression[merge_DataFrame_140491368403104], ComputedExpression[set_index_DataFrame_140491368405456], ComputedExpression[pre_combine_sum_Series_140491367964832]]:140491368711072/CoGroupByKey/CoGroupByKeyImpl/Tag[0]+ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/taxiride_it-093ddde7-b25b-4a55-9735-ebcb3e360370/output.csv/[ComputedExpression[merge_DataFrame_140491368403104], ComputedExpression[set_index_DataFrame_140491368405456], ComputedExpression[pre_combine_sum_Series_140491367964832]]:140491368711072/CoGroupByKey/CoGroupByKeyImpl/Flatten/OutputIdentity+ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/taxiride_it-093ddde7-b25b-4a55-9735-ebcb3e360370/output.csv/[ComputedExpression[merge_DataFrame_140491368403104], ComputedExpression[set_index_DataFrame_140491368405456], ComputedExpression[pre_combine_sum_Series_140491367964832]]:140491368711072/CoGroupByKey/CoGroupByKeyImpl/GroupByKey/Write [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:201 2023-12-20T18:58:23.113Z: JOB_MESSAGE_BASIC: Finished operation ref_AppliedPTransform_Read-zone-lookup-ParDo-_ReadFromPandasDoFn-_51/ProcessElementAndRestrictionWithSizing+ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/taxiride_it-093ddde7-b25b-4a55-9735-ebcb3e360370/output.csv/[ComputedExpression[set_index_DataFrame_140491368287584], ComputedExpression[get_column_Series_140491368403344]]:140491368364400/Map(<lambda at transforms.py:245>)+ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/taxiride_it-093ddde7-b25b-4a55-9735-ebcb3e360370/output.csv/[ComputedExpression[set_index_DataFrame_140491368287584], ComputedExpression[get_column_Series_140491368403344]]:140491368364400/FlatMap(evaluate)/FlatMap(evaluate)+ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/taxiride_it-093ddde7-b25b-4a55-9735-ebcb3e360370/output.csv/[ComputedExpression[merge_DataFrame_140491368403104], ComputedExpression[set_index_DataFrame_140491368405456], ComputedExpression[pre_combine_sum_Series_140491367964832]]:140491368711072/Partition_Index_get_column_Series_140491368403344/ParDo(_PreBatch)+ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/taxiride_it-093ddde7-b25b-4a55-9735-ebcb3e360370/output.csv/[ComputedExpression[post_combine_sum_Series_140491367967952]]:140491368086592/Size[get_column_Series_140491368403344, True]+ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/taxiride_it-093ddde7-b25b-4a55-9735-ebcb3e360370/output.csv/[ComputedExpression[post_combine_sum_Series_140491367967952]]:140491368086592/SumSizes/KeyWithVoid+ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/taxiride_it-093ddde7-b25b-4a55-9735-ebcb3e360370/output.csv/[ComputedExpression[post_combine_sum_Series_140491367967952]]:140491368086592/SumSizes/CombinePerKey/GroupByKey+ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/taxiride_it-093ddde7-b25b-4a55-9735-ebcb3e360370/output.csv/[ComputedExpression[post_combine_sum_Series_140491367967952]]:140491368086592/SumSizes/CombinePerKey/Combine/Partial+ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/taxiride_it-093ddde7-b25b-4a55-9735-ebcb3e360370/output.csv/[ComputedExpression[post_combine_sum_Series_140491367967952]]:140491368086592/SumSizes/CombinePerKey/GroupByKey/Write+ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/taxiride_it-093ddde7-b25b-4a55-9735-ebcb3e360370/output.csv/[ComputedExpression[merge_DataFrame_140491368403104], ComputedExpression[set_index_DataFrame_140491368405456], ComputedExpression[pre_combine_sum_Series_140491367964832]]:140491368711072/Partition_Index_get_column_Series_140491368403344/FlatMap(partition_fn)+ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/taxiride_it-093ddde7-b25b-4a55-9735-ebcb3e360370/output.csv/[ComputedExpression[merge_DataFrame_140491368403104], ComputedExpression[set_index_DataFrame_140491368405456], ComputedExpression[pre_combine_sum_Series_140491367964832]]:140491368711072/Partition_Index_get_column_Series_140491368403344/Filter(<lambda at transforms.py:231>)+ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/taxiride_it-093ddde7-b25b-4a55-9735-ebcb3e360370/output.csv/[ComputedExpression[merge_DataFrame_140491368403104], ComputedExpression[set_index_DataFrame_140491368405456], ComputedExpression[pre_combine_sum_Series_140491367964832]]:140491368711072/CoGroupByKey/CoGroupByKeyImpl/Tag[1]+ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/taxiride_it-093ddde7-b25b-4a55-9735-ebcb3e360370/output.csv/[ComputedExpression[merge_DataFrame_140491368403104], ComputedExpression[set_index_DataFrame_140491368405456], ComputedExpression[pre_combine_sum_Series_140491367964832]]:140491368711072/CoGroupByKey/CoGroupByKeyImpl/Flatten/OutputIdentity+ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/taxiride_it-093ddde7-b25b-4a55-9735-ebcb3e360370/output.csv/[ComputedExpression[merge_DataFrame_140491368403104], ComputedExpression[set_index_DataFrame_140491368405456], ComputedExpression[pre_combine_sum_Series_140491367964832]]:140491368711072/CoGroupByKey/CoGroupByKeyImpl/GroupByKey/Write [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:201 2023-12-20T19:02:33.640Z: JOB_MESSAGE_BASIC: Finished operation ref_AppliedPTransform_Read-taxi-rides-ParDo-_ReadFromPandasDoFn-_26/ProcessElementAndRestrictionWithSizing+ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/taxiride_it-093ddde7-b25b-4a55-9735-ebcb3e360370/output.csv/[ComputedExpression[set_index_DataFrame_140491368428400]]:140491368362384/Map(<lambda at transforms.py:245>)+ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/taxiride_it-093ddde7-b25b-4a55-9735-ebcb3e360370/output.csv/[ComputedExpression[set_index_DataFrame_140491368428400]]:140491368362384/FlatMap(evaluate)/FlatMap(evaluate)+ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/taxiride_it-093ddde7-b25b-4a55-9735-ebcb3e360370/output.csv/[ComputedExpression[merge_DataFrame_140491368403104], ComputedExpression[set_index_DataFrame_140491368405456], ComputedExpression[pre_combine_sum_Series_140491367964832]]:140491368711072/Partition_Index_set_index_DataFrame_140491368428400/ParDo(_PreBatch)+ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/taxiride_it-093ddde7-b25b-4a55-9735-ebcb3e360370/output.csv/[ComputedExpression[post_combine_sum_Series_140491367967952]]:140491368086592/Size[set_index_DataFrame_140491368428400, True]+ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/taxiride_it-093ddde7-b25b-4a55-9735-ebcb3e360370/output.csv/[ComputedExpression[post_combine_sum_Series_140491367967952]]:140491368086592/SumSizes/KeyWithVoid+ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/taxiride_it-093ddde7-b25b-4a55-9735-ebcb3e360370/output.csv/[ComputedExpression[post_combine_sum_Series_140491367967952]]:140491368086592/SumSizes/CombinePerKey/GroupByKey+ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/taxiride_it-093ddde7-b25b-4a55-9735-ebcb3e360370/output.csv/[ComputedExpression[post_combine_sum_Series_140491367967952]]:140491368086592/SumSizes/CombinePerKey/Combine/Partial+ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/taxiride_it-093ddde7-b25b-4a55-9735-ebcb3e360370/output.csv/[ComputedExpression[post_combine_sum_Series_140491367967952]]:140491368086592/SumSizes/CombinePerKey/GroupByKey/Write+ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/taxiride_it-093ddde7-b25b-4a55-9735-ebcb3e360370/output.csv/[ComputedExpression[merge_DataFrame_140491368403104], ComputedExpression[set_index_DataFrame_140491368405456], ComputedExpression[pre_combine_sum_Series_140491367964832]]:140491368711072/Partition_Index_set_index_DataFrame_140491368428400/FlatMap(partition_fn)+ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/taxiride_it-093ddde7-b25b-4a55-9735-ebcb3e360370/output.csv/[ComputedExpression[merge_DataFrame_140491368403104], ComputedExpression[set_index_DataFrame_140491368405456], ComputedExpression[pre_combine_sum_Series_140491367964832]]:140491368711072/Partition_Index_set_index_DataFrame_140491368428400/Filter(<lambda at transforms.py:231>)+ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/taxiride_it-093ddde7-b25b-4a55-9735-ebcb3e360370/output.csv/[ComputedExpression[merge_DataFrame_140491368403104], ComputedExpression[set_index_DataFrame_140491368405456], ComputedExpression[pre_combine_sum_Series_140491367964832]]:140491368711072/CoGroupByKey/CoGroupByKeyImpl/Tag[0]+ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/taxiride_it-093ddde7-b25b-4a55-9735-ebcb3e360370/output.csv/[ComputedExpression[merge_DataFrame_140491368403104], ComputedExpression[set_index_DataFrame_140491368405456], ComputedExpression[pre_combine_sum_Series_140491367964832]]:140491368711072/CoGroupByKey/CoGroupByKeyImpl/Flatten/OutputIdentity+ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/taxiride_it-093ddde7-b25b-4a55-9735-ebcb3e360370/output.csv/[ComputedExpression[merge_DataFrame_140491368403104], ComputedExpression[set_index_DataFrame_140491368405456], ComputedExpression[pre_combine_sum_Series_140491367964832]]:140491368711072/CoGroupByKey/CoGroupByKeyImpl/GroupByKey/Write [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:201 2023-12-20T19:02:33.713Z: JOB_MESSAGE_BASIC: Executing operation ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/taxiride_it-093ddde7-b25b-4a55-9735-ebcb3e360370/output.csv/[ComputedExpression[post_combine_sum_Series_140491367967952]]:140491368086592/SumSizes/CombinePerKey/GroupByKey/Close [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:201 2023-12-20T19:02:33.734Z: JOB_MESSAGE_BASIC: Executing operation ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/taxiride_it-093ddde7-b25b-4a55-9735-ebcb3e360370/output.csv/[ComputedExpression[merge_DataFrame_140491368403104], ComputedExpression[set_index_DataFrame_140491368405456], ComputedExpression[pre_combine_sum_Series_140491367964832]]:140491368711072/CoGroupByKey/CoGroupByKeyImpl/GroupByKey/Close [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:201 2023-12-20T19:02:33.779Z: JOB_MESSAGE_BASIC: Finished operation ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/taxiride_it-093ddde7-b25b-4a55-9735-ebcb3e360370/output.csv/[ComputedExpression[merge_DataFrame_140491368403104], ComputedExpression[set_index_DataFrame_140491368405456], ComputedExpression[pre_combine_sum_Series_140491367964832]]:140491368711072/CoGroupByKey/CoGroupByKeyImpl/GroupByKey/Close [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:201 2023-12-20T19:02:34.271Z: JOB_MESSAGE_BASIC: Finished operation ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/taxiride_it-093ddde7-b25b-4a55-9735-ebcb3e360370/output.csv/[ComputedExpression[post_combine_sum_Series_140491367967952]]:140491368086592/SumSizes/CombinePerKey/GroupByKey/Close [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:201 2023-12-20T19:02:34.312Z: JOB_MESSAGE_BASIC: Executing operation ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/taxiride_it-093ddde7-b25b-4a55-9735-ebcb3e360370/output.csv/[ComputedExpression[post_combine_sum_Series_140491367967952]]:140491368086592/SumSizes/CombinePerKey/GroupByKey/Read+ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/taxiride_it-093ddde7-b25b-4a55-9735-ebcb3e360370/output.csv/[ComputedExpression[post_combine_sum_Series_140491367967952]]:140491368086592/SumSizes/CombinePerKey/Combine+ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/taxiride_it-093ddde7-b25b-4a55-9735-ebcb3e360370/output.csv/[ComputedExpression[post_combine_sum_Series_140491367967952]]:140491368086592/SumSizes/CombinePerKey/Combine/Extract+ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/taxiride_it-093ddde7-b25b-4a55-9735-ebcb3e360370/output.csv/[ComputedExpression[post_combine_sum_Series_140491367967952]]:140491368086592/SumSizes/UnKey [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:201 2023-12-20T19:02:35.839Z: JOB_MESSAGE_BASIC: Finished operation ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/taxiride_it-093ddde7-b25b-4a55-9735-ebcb3e360370/output.csv/[ComputedExpression[post_combine_sum_Series_140491367967952]]:140491368086592/SumSizes/CombinePerKey/GroupByKey/Read+ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/taxiride_it-093ddde7-b25b-4a55-9735-ebcb3e360370/output.csv/[ComputedExpression[post_combine_sum_Series_140491367967952]]:140491368086592/SumSizes/CombinePerKey/Combine+ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/taxiride_it-093ddde7-b25b-4a55-9735-ebcb3e360370/output.csv/[ComputedExpression[post_combine_sum_Series_140491367967952]]:140491368086592/SumSizes/CombinePerKey/Combine/Extract+ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/taxiride_it-093ddde7-b25b-4a55-9735-ebcb3e360370/output.csv/[ComputedExpression[post_combine_sum_Series_140491367967952]]:140491368086592/SumSizes/UnKey [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:201 2023-12-20T19:02:35.942Z: JOB_MESSAGE_BASIC: Executing operation ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/taxiride_it-093ddde7-b25b-4a55-9735-ebcb3e360370/output.csv/[ComputedExpression[post_combine_sum_Series_140491367967952]]:140491368086592/SumSizes/InjectDefault/View-python_side_input0 [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:201 2023-12-20T19:02:35.995Z: JOB_MESSAGE_BASIC: Finished operation ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/taxiride_it-093ddde7-b25b-4a55-9735-ebcb3e360370/output.csv/[ComputedExpression[post_combine_sum_Series_140491367967952]]:140491368086592/SumSizes/InjectDefault/View-python_side_input0 [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:201 2023-12-20T19:02:36.081Z: JOB_MESSAGE_BASIC: Executing operation ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/taxiride_it-093ddde7-b25b-4a55-9735-ebcb3e360370/output.csv/[ComputedExpression[post_combine_sum_Series_140491367967952]]:140491368086592/SumSizes/DoOnce/Impulse+ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/taxiride_it-093ddde7-b25b-4a55-9735-ebcb3e360370/output.csv/[ComputedExpression[post_combine_sum_Series_140491367967952]]:140491368086592/SumSizes/DoOnce/FlatMap(<lambda at core.py:3774>)+ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/taxiride_it-093ddde7-b25b-4a55-9735-ebcb3e360370/output.csv/[ComputedExpression[post_combine_sum_Series_140491367967952]]:140491368086592/SumSizes/DoOnce/Map(decode)+ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/taxiride_it-093ddde7-b25b-4a55-9735-ebcb3e360370/output.csv/[ComputedExpression[post_combine_sum_Series_140491367967952]]:140491368086592/SumSizes/InjectDefault+ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/taxiride_it-093ddde7-b25b-4a55-9735-ebcb3e360370/output.csv/[ComputedExpression[post_combine_sum_Series_140491367967952]]:140491368086592/NumPartitions [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:201 2023-12-20T19:02:37.753Z: JOB_MESSAGE_BASIC: Finished operation ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/taxiride_it-093ddde7-b25b-4a55-9735-ebcb3e360370/output.csv/[ComputedExpression[post_combine_sum_Series_140491367967952]]:140491368086592/SumSizes/DoOnce/Impulse+ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/taxiride_it-093ddde7-b25b-4a55-9735-ebcb3e360370/output.csv/[ComputedExpression[post_combine_sum_Series_140491367967952]]:140491368086592/SumSizes/DoOnce/FlatMap(<lambda at core.py:3774>)+ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/taxiride_it-093ddde7-b25b-4a55-9735-ebcb3e360370/output.csv/[ComputedExpression[post_combine_sum_Series_140491367967952]]:140491368086592/SumSizes/DoOnce/Map(decode)+ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/taxiride_it-093ddde7-b25b-4a55-9735-ebcb3e360370/output.csv/[ComputedExpression[post_combine_sum_Series_140491367967952]]:140491368086592/SumSizes/InjectDefault+ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/taxiride_it-093ddde7-b25b-4a55-9735-ebcb3e360370/output.csv/[ComputedExpression[post_combine_sum_Series_140491367967952]]:140491368086592/NumPartitions [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:201 2023-12-20T19:02:37.864Z: JOB_MESSAGE_BASIC: Executing operation ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/taxiride_it-093ddde7-b25b-4a55-9735-ebcb3e360370/output.csv/[ComputedExpression[post_combine_sum_Series_140491367967952]]:140491368086592/Partition_Index_pre_combine_sum_Series_140491367964832/FlatMap(partition_fn)/View-python_side_input0 [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:201 2023-12-20T19:02:37.915Z: JOB_MESSAGE_BASIC: Finished operation ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/taxiride_it-093ddde7-b25b-4a55-9735-ebcb3e360370/output.csv/[ComputedExpression[post_combine_sum_Series_140491367967952]]:140491368086592/Partition_Index_pre_combine_sum_Series_140491367964832/FlatMap(partition_fn)/View-python_side_input0 [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:201 2023-12-20T19:02:38.030Z: JOB_MESSAGE_BASIC: Executing operation ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/taxiride_it-093ddde7-b25b-4a55-9735-ebcb3e360370/output.csv/[ComputedExpression[post_combine_sum_Series_140491367967952]]:140491368086592/CoGroupByKey/CoGroupByKeyImpl/GroupByKey/Create [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:201 2023-12-20T19:02:38.175Z: JOB_MESSAGE_BASIC: Finished operation ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/taxiride_it-093ddde7-b25b-4a55-9735-ebcb3e360370/output.csv/[ComputedExpression[post_combine_sum_Series_140491367967952]]:140491368086592/CoGroupByKey/CoGroupByKeyImpl/GroupByKey/Create [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:201 2023-12-20T19:02:38.298Z: JOB_MESSAGE_BASIC: Executing operation ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/taxiride_it-093ddde7-b25b-4a55-9735-ebcb3e360370/output.csv/[ComputedExpression[merge_DataFrame_140491368403104], ComputedExpression[set_index_DataFrame_140491368405456], ComputedExpression[pre_combine_sum_Series_140491367964832]]:140491368711072/CoGroupByKey/CoGroupByKeyImpl/GroupByKey/Read+ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/taxiride_it-093ddde7-b25b-4a55-9735-ebcb3e360370/output.csv/[ComputedExpression[merge_DataFrame_140491368403104], ComputedExpression[set_index_DataFrame_140491368405456], ComputedExpression[pre_combine_sum_Series_140491367964832]]:140491368711072/CoGroupByKey/CoGroupByKeyImpl/MapTuple(collect_values)+ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/taxiride_it-093ddde7-b25b-4a55-9735-ebcb3e360370/output.csv/[ComputedExpression[merge_DataFrame_140491368403104], ComputedExpression[set_index_DataFrame_140491368405456], ComputedExpression[pre_combine_sum_Series_140491367964832]]:140491368711072/CoGroupByKey/RestoreTags+ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/taxiride_it-093ddde7-b25b-4a55-9735-ebcb3e360370/output.csv/[ComputedExpression[merge_DataFrame_140491368403104], ComputedExpression[set_index_DataFrame_140491368405456], ComputedExpression[pre_combine_sum_Series_140491367964832]]:140491368711072/ParDo(_ReBatch)+ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/taxiride_it-093ddde7-b25b-4a55-9735-ebcb3e360370/output.csv/[ComputedExpression[merge_DataFrame_140491368403104], ComputedExpression[set_index_DataFrame_140491368405456], ComputedExpression[pre_combine_sum_Series_140491367964832]]:140491368711072/FlatMap(evaluate)/FlatMap(evaluate)+ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/taxiride_it-093ddde7-b25b-4a55-9735-ebcb3e360370/output.csv/[ComputedExpression[post_combine_sum_Series_140491367967952]]:140491368086592/Partition_Index_pre_combine_sum_Series_140491367964832/ParDo(_PreBatch)+ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/taxiride_it-093ddde7-b25b-4a55-9735-ebcb3e360370/output.csv/[ComputedExpression[post_combine_sum_Series_140491367967952]]:140491368086592/Partition_Index_pre_combine_sum_Series_140491367964832/FlatMap(partition_fn)+ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/taxiride_it-093ddde7-b25b-4a55-9735-ebcb3e360370/output.csv/[ComputedExpression[post_combine_sum_Series_140491367967952]]:140491368086592/Partition_Index_pre_combine_sum_Series_140491367964832/Filter(<lambda at transforms.py:231>)+ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/taxiride_it-093ddde7-b25b-4a55-9735-ebcb3e360370/output.csv/[ComputedExpression[post_combine_sum_Series_140491367967952]]:140491368086592/CoGroupByKey/CoGroupByKeyImpl/Tag[0]+ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/taxiride_it-093ddde7-b25b-4a55-9735-ebcb3e360370/output.csv/[ComputedExpression[post_combine_sum_Series_140491367967952]]:140491368086592/CoGroupByKey/CoGroupByKeyImpl/Flatten/OutputIdentity+ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/taxiride_it-093ddde7-b25b-4a55-9735-ebcb3e360370/output.csv/[ComputedExpression[post_combine_sum_Series_140491367967952]]:140491368086592/CoGroupByKey/CoGroupByKeyImpl/GroupByKey/Write [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:201 2023-12-20T19:12:05.857Z: JOB_MESSAGE_BASIC: Finished operation ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/taxiride_it-093ddde7-b25b-4a55-9735-ebcb3e360370/output.csv/[ComputedExpression[merge_DataFrame_140491368403104], ComputedExpression[set_index_DataFrame_140491368405456], ComputedExpression[pre_combine_sum_Series_140491367964832]]:140491368711072/CoGroupByKey/CoGroupByKeyImpl/GroupByKey/Read+ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/taxiride_it-093ddde7-b25b-4a55-9735-ebcb3e360370/output.csv/[ComputedExpression[merge_DataFrame_140491368403104], ComputedExpression[set_index_DataFrame_140491368405456], ComputedExpression[pre_combine_sum_Series_140491367964832]]:140491368711072/CoGroupByKey/CoGroupByKeyImpl/MapTuple(collect_values)+ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/taxiride_it-093ddde7-b25b-4a55-9735-ebcb3e360370/output.csv/[ComputedExpression[merge_DataFrame_140491368403104], ComputedExpression[set_index_DataFrame_140491368405456], ComputedExpression[pre_combine_sum_Series_140491367964832]]:140491368711072/CoGroupByKey/RestoreTags+ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/taxiride_it-093ddde7-b25b-4a55-9735-ebcb3e360370/output.csv/[ComputedExpression[merge_DataFrame_140491368403104], ComputedExpression[set_index_DataFrame_140491368405456], ComputedExpression[pre_combine_sum_Series_140491367964832]]:140491368711072/ParDo(_ReBatch)+ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/taxiride_it-093ddde7-b25b-4a55-9735-ebcb3e360370/output.csv/[ComputedExpression[merge_DataFrame_140491368403104], ComputedExpression[set_index_DataFrame_140491368405456], ComputedExpression[pre_combine_sum_Series_140491367964832]]:140491368711072/FlatMap(evaluate)/FlatMap(evaluate)+ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/taxiride_it-093ddde7-b25b-4a55-9735-ebcb3e360370/output.csv/[ComputedExpression[post_combine_sum_Series_140491367967952]]:140491368086592/Partition_Index_pre_combine_sum_Series_140491367964832/ParDo(_PreBatch)+ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/taxiride_it-093ddde7-b25b-4a55-9735-ebcb3e360370/output.csv/[ComputedExpression[post_combine_sum_Series_140491367967952]]:140491368086592/Partition_Index_pre_combine_sum_Series_140491367964832/FlatMap(partition_fn)+ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/taxiride_it-093ddde7-b25b-4a55-9735-ebcb3e360370/output.csv/[ComputedExpression[post_combine_sum_Series_140491367967952]]:140491368086592/Partition_Index_pre_combine_sum_Series_140491367964832/Filter(<lambda at transforms.py:231>)+ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/taxiride_it-093ddde7-b25b-4a55-9735-ebcb3e360370/output.csv/[ComputedExpression[post_combine_sum_Series_140491367967952]]:140491368086592/CoGroupByKey/CoGroupByKeyImpl/Tag[0]+ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/taxiride_it-093ddde7-b25b-4a55-9735-ebcb3e360370/output.csv/[ComputedExpression[post_combine_sum_Series_140491367967952]]:140491368086592/CoGroupByKey/CoGroupByKeyImpl/Flatten/OutputIdentity+ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/taxiride_it-093ddde7-b25b-4a55-9735-ebcb3e360370/output.csv/[ComputedExpression[post_combine_sum_Series_140491367967952]]:140491368086592/CoGroupByKey/CoGroupByKeyImpl/GroupByKey/Write [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:201 2023-12-20T19:12:05.919Z: JOB_MESSAGE_BASIC: Executing operation ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/taxiride_it-093ddde7-b25b-4a55-9735-ebcb3e360370/output.csv/[ComputedExpression[post_combine_sum_Series_140491367967952]]:140491368086592/CoGroupByKey/CoGroupByKeyImpl/GroupByKey/Close [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:201 2023-12-20T19:12:06.176Z: JOB_MESSAGE_BASIC: Finished operation ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/taxiride_it-093ddde7-b25b-4a55-9735-ebcb3e360370/output.csv/[ComputedExpression[post_combine_sum_Series_140491367967952]]:140491368086592/CoGroupByKey/CoGroupByKeyImpl/GroupByKey/Close [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:201 2023-12-20T19:12:06.251Z: JOB_MESSAGE_BASIC: Executing operation WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/taxiride_it-093ddde7-b25b-4a55-9735-ebcb3e360370/output.csv/WriteToFiles/GroupRecordsByDestinationAndShard/Create [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:201 2023-12-20T19:12:06.387Z: JOB_MESSAGE_BASIC: Finished operation WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/taxiride_it-093ddde7-b25b-4a55-9735-ebcb3e360370/output.csv/WriteToFiles/GroupRecordsByDestinationAndShard/Create [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:201 2023-12-20T19:12:06.470Z: JOB_MESSAGE_BASIC: Executing operation WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/taxiride_it-093ddde7-b25b-4a55-9735-ebcb3e360370/output.csv/WriteToFiles/GroupTempFilesByDestination/Create [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:201 2023-12-20T19:12:06.595Z: JOB_MESSAGE_BASIC: Finished operation WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/taxiride_it-093ddde7-b25b-4a55-9735-ebcb3e360370/output.csv/WriteToFiles/GroupTempFilesByDestination/Create [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:201 2023-12-20T19:12:06.687Z: JOB_MESSAGE_BASIC: Executing operation ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/taxiride_it-093ddde7-b25b-4a55-9735-ebcb3e360370/output.csv/[ComputedExpression[post_combine_sum_Series_140491367967952]]:140491368086592/CoGroupByKey/CoGroupByKeyImpl/GroupByKey/Read+ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/taxiride_it-093ddde7-b25b-4a55-9735-ebcb3e360370/output.csv/[ComputedExpression[post_combine_sum_Series_140491367967952]]:140491368086592/CoGroupByKey/CoGroupByKeyImpl/MapTuple(collect_values)+ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/taxiride_it-093ddde7-b25b-4a55-9735-ebcb3e360370/output.csv/[ComputedExpression[post_combine_sum_Series_140491367967952]]:140491368086592/CoGroupByKey/RestoreTags+ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/taxiride_it-093ddde7-b25b-4a55-9735-ebcb3e360370/output.csv/[ComputedExpression[post_combine_sum_Series_140491367967952]]:140491368086592/ParDo(_ReBatch)+ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/taxiride_it-093ddde7-b25b-4a55-9735-ebcb3e360370/output.csv/[ComputedExpression[post_combine_sum_Series_140491367967952]]:140491368086592/FlatMap(evaluate)/FlatMap(evaluate)+WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/taxiride_it-093ddde7-b25b-4a55-9735-ebcb3e360370/output.csv/WriteToFiles/ParDo(_WriteUnshardedRecordsFn)/ParDo(_WriteUnshardedRecordsFn)+WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/taxiride_it-093ddde7-b25b-4a55-9735-ebcb3e360370/output.csv/WriteToFiles/Map(<lambda at fileio.py:640>)+WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/taxiride_it-093ddde7-b25b-4a55-9735-ebcb3e360370/output.csv/WriteToFiles/GroupTempFilesByDestination/Write+WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/taxiride_it-093ddde7-b25b-4a55-9735-ebcb3e360370/output.csv/WriteToFiles/ParDo(_AppendShardedDestination)+WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/taxiride_it-093ddde7-b25b-4a55-9735-ebcb3e360370/output.csv/WriteToFiles/GroupRecordsByDestinationAndShard/Write [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:201 2023-12-20T19:12:07.488Z: JOB_MESSAGE_BASIC: Finished operation ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/taxiride_it-093ddde7-b25b-4a55-9735-ebcb3e360370/output.csv/[ComputedExpression[post_combine_sum_Series_140491367967952]]:140491368086592/CoGroupByKey/CoGroupByKeyImpl/GroupByKey/Read+ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/taxiride_it-093ddde7-b25b-4a55-9735-ebcb3e360370/output.csv/[ComputedExpression[post_combine_sum_Series_140491367967952]]:140491368086592/CoGroupByKey/CoGroupByKeyImpl/MapTuple(collect_values)+ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/taxiride_it-093ddde7-b25b-4a55-9735-ebcb3e360370/output.csv/[ComputedExpression[post_combine_sum_Series_140491367967952]]:140491368086592/CoGroupByKey/RestoreTags+ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/taxiride_it-093ddde7-b25b-4a55-9735-ebcb3e360370/output.csv/[ComputedExpression[post_combine_sum_Series_140491367967952]]:140491368086592/ParDo(_ReBatch)+ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/taxiride_it-093ddde7-b25b-4a55-9735-ebcb3e360370/output.csv/[ComputedExpression[post_combine_sum_Series_140491367967952]]:140491368086592/FlatMap(evaluate)/FlatMap(evaluate)+WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/taxiride_it-093ddde7-b25b-4a55-9735-ebcb3e360370/output.csv/WriteToFiles/ParDo(_WriteUnshardedRecordsFn)/ParDo(_WriteUnshardedRecordsFn)+WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/taxiride_it-093ddde7-b25b-4a55-9735-ebcb3e360370/output.csv/WriteToFiles/Map(<lambda at fileio.py:640>)+WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/taxiride_it-093ddde7-b25b-4a55-9735-ebcb3e360370/output.csv/WriteToFiles/GroupTempFilesByDestination/Write+WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/taxiride_it-093ddde7-b25b-4a55-9735-ebcb3e360370/output.csv/WriteToFiles/ParDo(_AppendShardedDestination)+WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/taxiride_it-093ddde7-b25b-4a55-9735-ebcb3e360370/output.csv/WriteToFiles/GroupRecordsByDestinationAndShard/Write [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:201 2023-12-20T19:12:07.528Z: JOB_MESSAGE_BASIC: Executing operation WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/taxiride_it-093ddde7-b25b-4a55-9735-ebcb3e360370/output.csv/WriteToFiles/GroupRecordsByDestinationAndShard/Close [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:201 2023-12-20T19:12:07.567Z: JOB_MESSAGE_BASIC: Finished operation WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/taxiride_it-093ddde7-b25b-4a55-9735-ebcb3e360370/output.csv/WriteToFiles/GroupRecordsByDestinationAndShard/Close [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:201 2023-12-20T19:12:07.608Z: JOB_MESSAGE_BASIC: Executing operation WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/taxiride_it-093ddde7-b25b-4a55-9735-ebcb3e360370/output.csv/WriteToFiles/GroupRecordsByDestinationAndShard/Read+WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/taxiride_it-093ddde7-b25b-4a55-9735-ebcb3e360370/output.csv/WriteToFiles/ParDo(_WriteShardedRecordsFn)+WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/taxiride_it-093ddde7-b25b-4a55-9735-ebcb3e360370/output.csv/WriteToFiles/Map(<lambda at fileio.py:640>)+WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/taxiride_it-093ddde7-b25b-4a55-9735-ebcb3e360370/output.csv/WriteToFiles/GroupTempFilesByDestination/Write [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:201 2023-12-20T19:12:07.788Z: JOB_MESSAGE_BASIC: Finished operation WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/taxiride_it-093ddde7-b25b-4a55-9735-ebcb3e360370/output.csv/WriteToFiles/GroupRecordsByDestinationAndShard/Read+WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/taxiride_it-093ddde7-b25b-4a55-9735-ebcb3e360370/output.csv/WriteToFiles/ParDo(_WriteShardedRecordsFn)+WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/taxiride_it-093ddde7-b25b-4a55-9735-ebcb3e360370/output.csv/WriteToFiles/Map(<lambda at fileio.py:640>)+WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/taxiride_it-093ddde7-b25b-4a55-9735-ebcb3e360370/output.csv/WriteToFiles/GroupTempFilesByDestination/Write [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:201 2023-12-20T19:12:07.830Z: JOB_MESSAGE_BASIC: Executing operation WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/taxiride_it-093ddde7-b25b-4a55-9735-ebcb3e360370/output.csv/WriteToFiles/GroupTempFilesByDestination/Close [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:201 2023-12-20T19:12:07.869Z: JOB_MESSAGE_BASIC: Finished operation WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/taxiride_it-093ddde7-b25b-4a55-9735-ebcb3e360370/output.csv/WriteToFiles/GroupTempFilesByDestination/Close [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:201 2023-12-20T19:12:07.906Z: JOB_MESSAGE_BASIC: Executing operation WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/taxiride_it-093ddde7-b25b-4a55-9735-ebcb3e360370/output.csv/WriteToFiles/GroupTempFilesByDestination/Read+WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/taxiride_it-093ddde7-b25b-4a55-9735-ebcb3e360370/output.csv/WriteToFiles/ParDo(_MoveTempFilesIntoFinalDestinationFn) [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:201 2023-12-20T19:12:08.734Z: JOB_MESSAGE_BASIC: Finished operation WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/taxiride_it-093ddde7-b25b-4a55-9735-ebcb3e360370/output.csv/WriteToFiles/GroupTempFilesByDestination/Read+WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/taxiride_it-093ddde7-b25b-4a55-9735-ebcb3e360370/output.csv/WriteToFiles/ParDo(_MoveTempFilesIntoFinalDestinationFn) [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:201 2023-12-20T19:12:08.887Z: JOB_MESSAGE_BASIC: Stopping worker pool... [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:201 2023-12-20T19:14:52.216Z: JOB_MESSAGE_BASIC: Worker pool stopped. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:151 Job 2023-12-20_10_52_18-14136225618669481975 is in state JOB_STATE_DONE [32mINFO [0m apache_beam.io.gcp.gcsio:gcsio.py:491 Finished listing 1 files in 0.0766444206237793 seconds. [32mINFO [0m apache_beam.io.gcp.gcsio:gcsio.py:491 Finished listing 1 files in 0.07679343223571777 seconds. [33m=============================== warnings summary ===============================[0m apache_beam/io/gcp/bigquery.py:2603 <https://ci-beam.apache.org/job/beam_PostCommit_Python39/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:2603: DeprecationWarning: invalid escape sequence \# """Read data from BigQuery. apache_beam/io/gcp/bigquery_read_it_test.py::ReadTests::test_native_source <https://ci-beam.apache.org/job/beam_PostCommit_Python39/ws/src/sdks/python/apache_beam/io/gcp/bigquery_read_it_test.py>:170: BeamDeprecationWarning: BigQuerySource is deprecated since 2.25.0. Use ReadFromBigQuery instead. beam.io.BigQuerySource(query=self.query, use_standard_sql=True))) apache_beam/io/gcp/bigquery_test.py::PubSubBigQueryIT::test_file_loads apache_beam/io/gcp/bigquery_test.py::PubSubBigQueryIT::test_streaming_inserts apache_beam/examples/complete/game/game_stats_it_test.py::GameStatsIT::test_game_stats_it apache_beam/examples/complete/game/hourly_team_score_it_test.py::HourlyTeamScoreIT::test_hourly_team_score_it apache_beam/examples/complete/game/leader_board_it_test.py::LeaderBoardIT::test_leader_board_it <https://ci-beam.apache.org/job/beam_PostCommit_Python39/ws/src/sdks/python/apache_beam/io/gcp/tests/utils.py>:63: PendingDeprecationWarning: Client.dataset is deprecated and will be removed in a future version. Use a string like 'my_project.my_dataset' or a cloud.google.bigquery.DatasetReference object, instead. dataset_ref = client.dataset(unique_dataset_name, project=project) apache_beam/examples/dataframe/flight_delays_it_test.py::FlightDelaysTest::test_flight_delays <https://ci-beam.apache.org/job/beam_PostCommit_Python39/ws/src/sdks/python/apache_beam/examples/dataframe/flight_delays.py>:47: FutureWarning: The default value of numeric_only in DataFrame.mean is deprecated. In a future version, it will default to False. In addition, specifying 'numeric_only=None' is deprecated. Select only valid columns or specify the value of numeric_only to silence this warning. return airline_df[at_top_airports].mean() apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it <https://ci-beam.apache.org/job/beam_PostCommit_Python39/ws/src/sdks/python/apache_beam/io/gcp/tests/utils.py>:100: PendingDeprecationWarning: Client.dataset is deprecated and will be removed in a future version. Use a string like 'my_project.my_dataset' or a cloud.google.bigquery.DatasetReference object, instead. table_ref = client.dataset(dataset_id).table(table_id) apache_beam/io/gcp/bigquery_read_it_test.py::ReadNewTypesTests::test_native_source <https://ci-beam.apache.org/job/beam_PostCommit_Python39/ws/src/sdks/python/apache_beam/io/gcp/bigquery_read_it_test.py>:706: BeamDeprecationWarning: BigQuerySource is deprecated since 2.25.0. Use ReadFromBigQuery instead. beam.io.BigQuerySource(query=self.query, use_standard_sql=True))) -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html - generated xml file: <https://ci-beam.apache.org/job/beam_PostCommit_Python39/ws/src/sdks/python/pytest_postCommitIT-df-py39.xml> - [36m[1m=========================== short test summary info ============================[0m [31mFAILED[0m apache_beam/examples/dataframe/taxiride_it_test.py::[1mTaxirideIT::test_enrich[0m - AssertionError: DataFrame.iloc[:, 1] (column name="passenger_count") are different DataFrame.iloc[:, 1] (column name="passenger_count") values are different (100.0 %) [index]: [0, 1, 2, 3, 4, 5, 6] [left]: [1074423, 6972533, 322785, 145129096, 8024331, 39659, 2407529] [right]: [1073783, 6967581, 323101, 145122094, 8018561, 39622, 2409682] [31m====== [31m[1m1 failed[0m, [32m82 passed[0m, [33m49 skipped[0m, [33m10 warnings[0m[31m in 5790.53s (1:36:30)[0m[31m ======[0m > Task :sdks:python:test-suites:dataflow:py39:postCommitIT FAILED FAILURE: Build failed with an exception. * Where: Script '<https://ci-beam.apache.org/job/beam_PostCommit_Python39/ws/src/sdks/python/test-suites/dataflow/common.gradle'> line: 139 * What went wrong: Execution failed for task ':sdks:python:test-suites:dataflow:py39:postCommitIT'. > Process 'command 'sh'' finished with non-zero exit value 1 * Try: > Run with --stacktrace option to get the stack trace. > Run with --info or --debug option to get more log output. > Get more help at https://help.gradle.org. Deprecated Gradle features were used in this build, making it incompatible with Gradle 9.0. You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins. For more on this, please refer to https://docs.gradle.org/8.4/userguide/command_line_interface.html#sec:command_line_warnings in the Gradle documentation. BUILD FAILED in 1h 43m 14s 222 actionable tasks: 156 executed, 62 from cache, 4 up-to-date Publishing build scan... The request was rejected. Gradle Enterprise plugin version 3.16 is newer than the newest version supported by Gradle Enterprise 2023.3.4 which is 3.15. Please update to a newer version of Gradle Enterprise. Build step 'Invoke Gradle script' changed build result to FAILURE Build step 'Invoke Gradle script' marked build as failure --------------------------------------------------------------------- To unsubscribe, e-mail: builds-unsubscr...@beam.apache.org For additional commands, e-mail: builds-h...@beam.apache.org
