chunyang commented on pull request #17153: URL: https://github.com/apache/beam/pull/17153#issuecomment-1075818775
Just to explore some other options other than converting schema to JSON, prior to #15610, there was no generator or TableSchema to cause pickling errors. Instead of storing the generator as an instance attribute, a list of [TextSource](https://github.com/apache/beam/blob/v2.34.0/sdks/python/apache_beam/io/gcp/bigquery.py#L793-L798) and notably its `coder` attribute were stored instead (we can assume `use_json_exports=True` for this discussion). The default `coder`, `_JsonToDictCoder`, had a method [`_convert_to_tuple`](https://github.com/apache/beam/blob/v2.34.0/sdks/python/apache_beam/io/gcp/bigquery_read_internal.py#L401-L413) to marshal the TableSchema into an object more amenable to pickling. Perhaps I can use the same `_convert_to_tuple` method to create a picklable version of TableSchema and store that as an attribute rather than going the JSON route? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
