stankiewicz commented on PR #36502: URL: https://github.com/apache/beam/pull/36502#issuecomment-3422973691
sure, @ahmedabu98 , "Convert dict to Beam Row" transform is Map per per element transform that invokes bigquery_tools.beam_row_from_dict(element, schema). if schema call is string or dict, it invokes conversion dict -> json string -> bigquery.TableSchema. Conversion happens for every element and it's expensive. Change here is to add to Map transform the SetupContextParam parameter that will invoke conversion once. Previous attempt (https://github.com/apache/beam/pull/36393) was reverted due to proto serialization issues. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
