raams95 opened a new pull request, #40680:
URL: https://github.com/apache/beam/pull/40680
Fixes #20899
Type inference gives a slice of a fixed-size tuple the type of the whole
tuple. For an element of type `Tuple[str, str, int]`, `row[1:]` is inferred as
`Tuple[str, str, int]` instead of `Tuple[str, int]`. This breaks pipelines, not
only the reported `element_type`. Dropping the first field before combining
fails at construction:
```python
rows = [('day1', 'a', 1), ('day2', 'a', 2), ('day1', 'b', 3)]
with beam.Pipeline() as p:
_ = (
p
| beam.Create(rows).with_output_types(Tuple[str, str, int])
| 'DropDate' >> beam.Map(lambda row: row[1:])
| beam.CombinePerKey(sum))
```
```
TypeCheckError: The transform 'CombinePerKey(sum)' requires PCollections of
type 'Tuple[TypeVariable[K], Any]' but was applied to a PCollection of type
'Tuple[<class 'str'>, <class 'str'>, <class 'int'>]' (produced by the transform
'DropDate').
```
When the sliced PCollection is encoded, for example before a `Reshuffle`,
the coder for the whole tuple is used and the pipeline fails with `TypeError:
'<' not supported between instances of 'str' and 'int'` for `row[1:]`, or
`IndexError: tuple index out of range` for `row[:2]`.
With this change, a slice of a fixed-size `Tuple` with constant bounds is
inferred as the tuple of the selected element types (`Tuple[str, int]` above,
`Tuple[float, str, int]` for `[::-1]` on `Tuple[int, str, float]`). A slice
with bounds that aren't constants, such as `row[i:]`, is inferred as
`Tuple[Union[...], ...]` of the element types, in line with a variable index
(`row[i]` is inferred as `Union[...]`) and tuple unpacking (`(*a, *b)`). Slices
of lists, `Tuple[T, ...]` and `str` keep the type of the sliced value, as
before.
Slices compile to different bytecode depending on the Python version:
`BUILD_SLICE` followed by a subscript on 3.10 and 3.11 and for slices with a
step, `BINARY_SLICE` on 3.12 and later, and a constant `slice` object on 3.14.
`build_slice` now keeps constant bounds as a `Const` slice, and `binary_subscr`
and `binary_slice` share one helper, so all versions infer the same types.
Tests:
- `testGetItemSlice` in `trivial_inference_test.py` has new cases for
fixed-size tuples.
- `test_inferred_type_of_tuple_slice` in `typed_pipeline_test.py` runs the
pipeline above.
Both fail before the change and pass after it on Python 3.10, 3.11, 3.12,
3.13 and 3.14.
Testing:
- `pytest apache_beam/typehints/trivial_inference_test.py
apache_beam/typehints/typed_pipeline_test.py` on Python 3.10, 3.11, 3.12, 3.13
and 3.14
- `pytest apache_beam/typehints apache_beam/transforms apache_beam/coders
apache_beam/pipeline_test.py apache_beam/dataframe/transforms_test.py
apache_beam/yaml/yaml_mapping_test.py apache_beam/yaml/yaml_udf_test.py` on
Python 3.12
- `yapf --diff` and `ruff check` on the changed files
------------------------
Thank you for your contribution! Follow this checklist to help us
incorporate your contribution quickly and easily:
- [x] Mention the appropriate issue in your description (for example:
`addresses #123` or `fixes #456`).
- [x] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). (Not
needed, small change.)
See the [Contributor Guide](https://beam.apache.org/contribute) for more
tips on [how to make review process
smoother](https://github.com/apache/beam/blob/master/CONTRIBUTING.md#make-the-reviewers-job-easier).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
