rworley-monster commented on pull request #12389: URL: https://github.com/apache/beam/pull/12389#issuecomment-666280160
> This needs tests! Indeed, I will add appropriate tests in `BigQueryUtilsTest`. > BigQuery doesn't fundamentally support maps. This change modifies the IO to implicitly run unnest() on maps when writing to BigQuery. It seems like this could create confusion for users. The motivation for this change is to allow Avro maps (or anything that can become a Beam Schema Map type) to flow directly into BigQuery. I agree that a map becoming an array of key/value records is not completely intuitive, but it seemed the best solution that I was able to figure out within the current features of BigQuery schemas. > The BigQuery IO is bidirectional, users will expect to be able to read their maps back. Is that something you are considering adding? Yes, that makes sense. It appears this would be done in `fromTableFieldSchemaType`, we would detect if a record has only 'key' and 'value' fields, and in that case return a Map FieldType. A possible concern is that this would intercept key/value records that did not necessarily originate as Beam Maps. Perhaps this would be a desired feature in many cases, but it can be debated. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
