Hi everyone, I ran the Integration Tests - BigQueryStreamingInsertTransformIntegrationTests[1] and BigQueryFileLoadsIT[2] on the master branch locally, with the following command: ./scripts/run_integration_test.sh --test_opts --tests=apache_beam.io.gcp.bigquery_test:BigQueryStreamingInsertTransformIntegrationTests The Dataflow jobs for the tests failed with the following error: root: INFO: 2019-06-03T18:36:53.021Z: JOB_MESSAGE_ERROR: Traceback (most recent call last): File "/usr/local/lib/python2.7/dist-packages/dataflow_worker/batchworker.py", line 649, in do_work work_executor.execute() File "/usr/local/lib/python2.7/dist-packages/dataflow_worker/executor.py", line 150, in execute test_shuffle_sink=self._test_shuffle_sink) File "/usr/local/lib/python2.7/dist-packages/dataflow_worker/executor.py", line 116, in create_operation is_streaming=False) File "apache_beam/runners/worker/operations.py", line 962, in apache_beam.runners.worker.operations.create_operation op = BatchGroupAlsoByWindowsOperation( File "dataflow_worker/shuffle_operations.py", line 219, in dataflow_worker.shuffle_operations.BatchGroupAlsoByWindowsOperation.__init__ self.windowing = deserialize_windowing_strategy(self.spec.window_fn) File "dataflow_worker/shuffle_operations.py", line 207, in dataflow_worker.shuffle_operations.deserialize_windowing_strategy return pickler.loads(serialized_data) File "/usr/local/lib/python2.7/dist-packages/apache_beam/internal/pickler.py", line 248, in loads c = base64.b64decode(encoded) File "/usr/lib/python2.7/base64.py", line 78, in b64decode raise TypeError(msg) TypeError: Incorrect padding
I tested the same tests on the 2.13.0-RC#2 branch as well and they passed. These tests also don't fail in the most recent Python post-commit tests[3-5]. Keeping in mind the recent b64 changes in BQ, none of the tests in the test classes mentioned above makes use of a "BYTES" type field. Would love to get pointers to possible reasons. Thank You - TT [1] https://github.com/apache/beam/blob/master/sdks/python/apache_beam/io/gcp/bigquery_test.py#L479-L630 [2] https://github.com/apache/beam/blob/master/sdks/python/apache_beam/io/gcp/bigquery_file_loads_test.py#L358-L528 [3] https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/ [4] https://builds.apache.org/job/beam_PostCommit_Python3_Verify/lastCompletedBuild/ [5] https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/
