LibenHailu opened a new issue, #40902:
URL: https://github.com/apache/beam/issues/40902
### What happened?
There are two bugs in Beam YAML windowing transforms:
#### 1. `YamlProviders.WindowInto._parse_duration` defines `'d'` (days) as
12 hours instead of 24 hours
In `sdks/python/apache_beam/yaml/yaml_provider.py`
(`YamlProviders.WindowInto._parse_duration`), the time unit multiplier for
`'d'` is defined as:
```python
time_units = {
'ms': 0.001, 's': 1, 'm': 60, 'h': 60 * 60, 'd': 60 * 60 * 12
}
```
Because `60 * 60 * 12` is `43,200` seconds (12 hours) rather than `86,400`
seconds (24 hours), any `WindowInto` or `windowing` configuration using the
`'d'` suffix (e.g., `size: 1d`) creates 12-hour windows instead of 24-hour
windows.
**Steps to Reproduce:**
```python
from apache_beam.yaml import yaml_provider
# Returns 43200.0 instead of 86400.0
print(yaml_provider.YamlProviders.WindowInto._parse_duration('1d', 'size'))
```
---
#### 2. `_ExtractWindowingInfo` fails to detect duplicate field names in the
input schema
In `sdks/python/apache_beam/yaml/yaml_mapping.py` (`_ExtractWindowingInfo`),
`existing_fields` is obtained via
`named_fields_from_element_type(pcoll.element_type)`, which returns a list of
`(field_name, field_type)` tuples (`list[tuple[str, type]]`).
However, the duplicate field check does:
```python
elif field in existing_fields:
raise ValueError(f'Input schema already has a field named {field}.')
```
Because `field` is a `str` and `existing_fields` contains `(str, type)`
tuples, `field in existing_fields` always evaluates to `False`. Instead of
raising `ValueError`, `_ExtractWindowingInfo` appends the duplicate field to
the output `RowTypeConstraint` (`existing_fields + new_fields`).
**Steps to Reproduce:**
```python
import apache_beam as beam
from apache_beam.yaml.yaml_transform import YamlTransform
with beam.Pipeline() as p:
elements = p | beam.Create([beam.Row(timestamp=1, value='a')])
# Should raise ValueError("Input schema already has a field named
timestamp.")
# Instead, produces an invalid RowTypeConstraint with duplicate
'timestamp' fields
_ = elements | YamlTransform(
'''
type: ExtractWindowingInfo
config:
fields: [timestamp]
''')
```
### Issue Priority
Priority: 2 (default / most bugs should be filed as P2)
### Issue Components
- [x] Component: Python SDK
- [ ] Component: Java SDK
- [ ] Component: Go SDK
- [ ] Component: Typescript SDK
- [ ] Component: IO connector
- [x] Component: Beam YAML
- [ ] Component: Beam examples
- [ ] Component: Beam playground
- [ ] Component: Beam katas
- [ ] Component: Website
- [ ] Component: Infrastructure
- [ ] Component: Spark Runner
- [ ] Component: Flink Runner
- [ ] Component: Prism Runner
- [ ] Component: Hazelcast Jet Runner
- [ ] Component: Google Cloud Dataflow Runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
