enzomaruffa opened a new pull request, #37253:
URL: https://github.com/apache/beam/pull/37253
## Summary
This PR adds support for `DATE`, `DATETIME`, and `JSON` BigQuery types in
the Python SDK's BigQuery I/O type mappings.
### Background
We were migrating a production Dataflow pipeline from Java to the Python
implementation, using the Storage Write API for better performance and cost
efficiency. Our pipeline processes events and writes to multiple BigQuery
tables, some of which contain `DATE`, `DATETIME`, and `JSON` columns.
When switching to Storage Write API with `use_beam_io_types=True` (BEAM_ROW
format), the pipeline failed immediately:
```python
ValueError: Encountered an unsupported type: 'DATETIME'
```
We created a fork with the type mappings added, and **the pipeline now runs
successfully in production** using the Storage Write API with tables containing
all three types. This PR upstreams those changes.
### Solution
Following the same pattern established in #36121 (GEOGRAPHY type support),
this PR adds the missing type mappings in two files:
- `bigquery_tools.py`: `BIGQUERY_TYPE_TO_PYTHON_TYPE` dict
- `bigquery_schema_tools.py`: `BIG_QUERY_TO_PYTHON_TYPES` dict
All three types are mapped to `str`:
- `DATE` → `str` (format: `YYYY-MM-DD`)
- `DATETIME` → `str` (format: `YYYY-MM-DDTHH:MM:SS`)
- `JSON` → `str` (JSON string)
### A note on type mappings
Mapping these types to `str` is a pragmatic solution that unblocks us. A
more "proper" implementation might use Python's native types (`datetime.date`,
`datetime.datetime`) or dedicated JSON types, but this would require deeper
changes to the serialization layer and cross-language schema handling. The
`str` mapping is consistent with how `GEOGRAPHY` is handled and works correctly
with the Storage Write API's expectations.
If there's a preferred approach the Beam team would like us to take instead,
we're happy to adjust the implementation - but might need some guidance on how
to given Beam's complexity.
### Testing
Following the test pattern from #36121, each type includes:
- Schema conversion tests with all modes (REQUIRED, NULLABLE, REPEATED)
- Type mapping dictionary verification
- `bq_field_to_type` function tests for all modes including edge cases
(None, empty string)
- `convert_to_usertype` function tests
- `BeamSchemaConversionDoFn` processing tests
------------------------
Thank you for your contribution! Follow this checklist to help us
incorporate your contribution quickly and easily:
- [ ] Mention the appropriate issue in your description (for example:
`addresses #123`), if applicable. This will automatically add a link to the
pull request in the issue. If you would like the issue to automatically close
on merging the pull request, comment `fixes #<ISSUE NUMBER>` instead.
- [x] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more
tips on [how to make review process
smoother](https://github.com/apache/beam/blob/master/CONTRIBUTING.md#make-the-reviewers-job-easier).
To check the build health, please visit
[https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md](https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md)
GitHub Actions Tests Status (on master branch)
------------------------------------------------------------------------------------------------
[](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Go+tests%22+branch%3Amaster+event%3Aschedule)
See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more
information about GitHub Actions CI or the [workflows
README](https://github.com/apache/beam/blob/master/.github/workflows/README.md)
to see a list of phrases to trigger workflows.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
