Chenzo1001 opened a new issue, #35780:
URL: https://github.com/apache/beam/issues/35780
### What would you like to happen?
## Background
Currently, when using `apache_beam.yaml` to parse and validate YAML
pipelines, transforms such as `ReadFromCsv` or other IO-related transforms may
attempt to access real files or data sources during pipeline construction or
parameter validation. This leads to errors like `FileNotFoundError`, even if
the user only wants to check the syntax, structure, and transform parameter
correctness of the YAML file, not actually run any data processing.
## Problem
This behavior prevents users and tools from performing lightweight
validation ("validate-only" or "dry-run") of YAML pipelines. It is especially
problematic for CI, code review, or plugin scenarios where users want to:
- Check for YAML syntax correctness
- Validate required fields and config types for transforms
- Ensure wiring of inputs/outputs is correct
- But do **not** want to require the presence of real data files, databases,
or external services
Currently, there is no official API or option to trigger a validate-only
mode that avoids all IO/data access.
## Proposal
- Add an official `validate_only` or `dry_run` mode/flag to YAML pipeline
parsing and transform construction.
- In this mode, all IO/data transforms should **not** attempt to access or
glob real files, open network/database connections, or perform any side effects.
- Only the transform schema, parameters, input/output structure, and wiring
should be checked.
- Ideally, this should be available both via the Python API and the CLI.
## Impact
- This would greatly improve the developer experience for YAML/Beam users.
- It would enable safer CI/linting, better code review automation, and more
robust plugin development.
- It would prevent false negatives and usability issues for users who only
want to check pipeline structure.
## Example
```python
# Current (undesired):
# Raises FileNotFoundError if /fake/path.csv doesn't exist
options, constructor = build_pipeline_components_from_yaml(yaml_str, [],
validate_schema='per_transform')
# Desired:
options, constructor = build_pipeline_components_from_yaml(yaml_str, [],
validate_schema='per_transform', validate_only=True) # No IO attempted
```
Please consider this feature to improve YAML pipeline validation usability!
### Issue Priority
Priority: 2 (default / most feature requests should be filed as P2)
### Issue Components
- [ ] Component: Python SDK
- [ ] Component: Java SDK
- [ ] Component: Go SDK
- [ ] Component: Typescript SDK
- [ ] Component: IO connector
- [x] Component: Beam YAML
- [ ] Component: Beam examples
- [ ] Component: Beam playground
- [ ] Component: Beam katas
- [ ] Component: Website
- [ ] Component: Infrastructure
- [ ] Component: Spark Runner
- [ ] Component: Flink Runner
- [ ] Component: Samza Runner
- [ ] Component: Twister2 Runner
- [ ] Component: Hazelcast Jet Runner
- [ ] Component: Google Cloud Dataflow Runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
