gemini-code-assist[bot] commented on code in PR #35931: URL: https://github.com/apache/beam/pull/35931#discussion_r2304055817
########## website/www/site/content/en/documentation/sdks/yaml.md: ########## @@ -742,6 +742,74 @@ or writing dated sources and sinks, e.g. would write to files like `gs://path/to/2016/08/04/dated-output*.json`. +A user can also use the `% include` directive to pull in other common templates: + +<PATH_TO_YOUR_REPO>/pipeline.yaml +```sh Review Comment:  The language for this code block should be `yaml` instead of `sh`, as it contains YAML content. ```suggestion ```yaml ``` ########## website/www/site/content/en/documentation/sdks/yaml.md: ########## @@ -742,6 +742,74 @@ would write to files like `gs://path/to/2016/08/04/dated-output*.json`. +A user can also use the `% include` directive to pull in other common templates: + +<PATH_TO_YOUR_REPO>/pipeline.yaml +```sh +pipeline: + transforms: + - name: Read from GCS + type: ReadFromText + config: +{% include '<PATH_TO_YOUR_REPO>/submodules/readFromText.yaml' %} + - name: Write to GCS + type: WriteToText + input: Read from GCS + config: + path: "gs://MY-BUCKET/wordCounts/" +``` + +<PATH_TO_YOUR_REPO>/submodules/readFromText.yaml +```sh Review Comment:  The language for this code block should be `yaml` instead of `sh`. ```suggestion ```yaml ``` ########## website/www/site/content/en/documentation/sdks/yaml.md: ########## @@ -742,6 +742,74 @@ would write to files like `gs://path/to/2016/08/04/dated-output*.json`. +A user can also use the `% include` directive to pull in other common templates: + +<PATH_TO_YOUR_REPO>/pipeline.yaml +```sh +pipeline: + transforms: + - name: Read from GCS + type: ReadFromText + config: +{% include '<PATH_TO_YOUR_REPO>/submodules/readFromText.yaml' %} + - name: Write to GCS Review Comment:  The indentation of this transform is incorrect. It should have 4 spaces to align with the previous list item (`- name: Read from GCS`) for the YAML to be valid. ```suggestion - name: Write to GCS ``` ########## website/www/site/content/en/documentation/sdks/yaml.md: ########## @@ -742,6 +742,74 @@ would write to files like `gs://path/to/2016/08/04/dated-output*.json`. +A user can also use the `% include` directive to pull in other common templates: + +<PATH_TO_YOUR_REPO>/pipeline.yaml +```sh +pipeline: + transforms: + - name: Read from GCS + type: ReadFromText + config: +{% include '<PATH_TO_YOUR_REPO>/submodules/readFromText.yaml' %} + - name: Write to GCS + type: WriteToText + input: Read from GCS + config: + path: "gs://MY-BUCKET/wordCounts/" +``` + +<PATH_TO_YOUR_REPO>/submodules/readFromText.yaml +```sh + path: {{readFromText.path}} +``` + +This pipeline can be ran like this: + +```sh +python -m apache_beam.yaml.main \ + --yaml_pipeline_file=pipeline.yaml \ + --jinja_variables='{"readFromText": {"path": "gs://dataflow-samples/shakespeare/kinglear.txt"}}' +``` + +The `% import` jinja directive can also be used to pull in macros: + +<PATH_TO_YOUR_REPO>/pipeline.yaml +```sh +{% import '<PATH_TO_YOUR_REPO>/macros.yaml' as macros %} + +pipeline: + type: chain + transforms: + +# Read in text file +{{ macros.readFromText(readFromText) | indent(4, true) }} + +# Write to text file on GCS, locally, etc + - name: Write to GCS + type: WriteToText + input: Read from GCS + config: + path: "gs://MY-BUCKET/wordCounts/" +``` Review Comment:  This example has a few issues that make the resulting YAML invalid: 1. The code block language should be `yaml` or `jinja`, not `sh`. 2. The comments are not indented and will cause a YAML parsing error. They should be indented to be part of the `transforms` list. 3. The `Write to GCS` transform is incorrectly indented with 3 spaces instead of 4. ```suggestion {% import '<PATH_TO_YOUR_REPO>/macros.yaml' as macros %} pipeline: type: chain transforms: # Read in text file {{ macros.readFromText(readFromText) | indent(4, true) }} # Write to text file on GCS, locally, etc - name: Write to GCS type: WriteToText input: Read from GCS config: path: "gs://MY-BUCKET/wordCounts/" ``` ########## website/www/site/content/en/documentation/sdks/yaml.md: ########## @@ -742,6 +742,74 @@ would write to files like `gs://path/to/2016/08/04/dated-output*.json`. +A user can also use the `% include` directive to pull in other common templates: + +<PATH_TO_YOUR_REPO>/pipeline.yaml +```sh +pipeline: + transforms: + - name: Read from GCS + type: ReadFromText + config: +{% include '<PATH_TO_YOUR_REPO>/submodules/readFromText.yaml' %} + - name: Write to GCS + type: WriteToText + input: Read from GCS + config: + path: "gs://MY-BUCKET/wordCounts/" +``` + +<PATH_TO_YOUR_REPO>/submodules/readFromText.yaml +```sh + path: {{readFromText.path}} +``` + +This pipeline can be ran like this: + +```sh +python -m apache_beam.yaml.main \ + --yaml_pipeline_file=pipeline.yaml \ + --jinja_variables='{"readFromText": {"path": "gs://dataflow-samples/shakespeare/kinglear.txt"}}' +``` + +The `% import` jinja directive can also be used to pull in macros: + +<PATH_TO_YOUR_REPO>/pipeline.yaml +```sh +{% import '<PATH_TO_YOUR_REPO>/macros.yaml' as macros %} + +pipeline: + type: chain + transforms: + +# Read in text file +{{ macros.readFromText(readFromText) | indent(4, true) }} + +# Write to text file on GCS, locally, etc + - name: Write to GCS + type: WriteToText + input: Read from GCS + config: + path: "gs://MY-BUCKET/wordCounts/" +``` + +<PATH_TO_YOUR_REPO>/macros.yaml +```sh Review Comment:  For clarity, it's better to use `jinja` or `yaml` for this code block's language instead of `sh`, as it contains a Jinja macro. ```suggestion ```jinja ``` ########## website/www/site/content/en/documentation/sdks/yaml.md: ########## @@ -742,6 +742,74 @@ would write to files like `gs://path/to/2016/08/04/dated-output*.json`. +A user can also use the `% include` directive to pull in other common templates: + +<PATH_TO_YOUR_REPO>/pipeline.yaml +```sh +pipeline: + transforms: + - name: Read from GCS + type: ReadFromText + config: +{% include '<PATH_TO_YOUR_REPO>/submodules/readFromText.yaml' %} + - name: Write to GCS + type: WriteToText + input: Read from GCS + config: + path: "gs://MY-BUCKET/wordCounts/" +``` + +<PATH_TO_YOUR_REPO>/submodules/readFromText.yaml +```sh + path: {{readFromText.path}} +``` + +This pipeline can be ran like this: + +```sh +python -m apache_beam.yaml.main \ + --yaml_pipeline_file=pipeline.yaml \ + --jinja_variables='{"readFromText": {"path": "gs://dataflow-samples/shakespeare/kinglear.txt"}}' +``` + +The `% import` jinja directive can also be used to pull in macros: + +<PATH_TO_YOUR_REPO>/pipeline.yaml +```sh +{% import '<PATH_TO_YOUR_REPO>/macros.yaml' as macros %} + +pipeline: + type: chain + transforms: + +# Read in text file +{{ macros.readFromText(readFromText) | indent(4, true) }} + +# Write to text file on GCS, locally, etc + - name: Write to GCS + type: WriteToText + input: Read from GCS + config: + path: "gs://MY-BUCKET/wordCounts/" +``` + +<PATH_TO_YOUR_REPO>/macros.yaml +```sh +{%- macro readFromText(params) -%} +- name: Read from GCS + type: ReadFromText + config: + path: "{{ params.path }}" +{%- endmacro -%} +``` + +This pipeline can be ran with the same command as in the `% include` example +above. Review Comment:  Minor grammatical correction: "ran" should be "run". ```suggestion This pipeline can be run with the same command as in the `% include` example above. ``` ########## website/www/site/content/en/documentation/sdks/yaml.md: ########## @@ -742,6 +742,74 @@ would write to files like `gs://path/to/2016/08/04/dated-output*.json`. +A user can also use the `% include` directive to pull in other common templates: + +<PATH_TO_YOUR_REPO>/pipeline.yaml +```sh +pipeline: + transforms: + - name: Read from GCS + type: ReadFromText + config: +{% include '<PATH_TO_YOUR_REPO>/submodules/readFromText.yaml' %} + - name: Write to GCS + type: WriteToText + input: Read from GCS + config: + path: "gs://MY-BUCKET/wordCounts/" +``` + +<PATH_TO_YOUR_REPO>/submodules/readFromText.yaml +```sh + path: {{readFromText.path}} +``` + +This pipeline can be ran like this: Review Comment:  Minor grammatical correction: "ran" should be "run". ```suggestion This pipeline can be run like this: ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
