I validated python quick starts (direct, dataflow) X (batch, streaming). I
ran into an issue with the dataflow batch case, running the wordcount with
the standard:
python -m apache_beam.examples.wordcount \
--output <GCS> \
--staging_location <GCS> \
--temp_location <GCS>\
--runner DataflowRunner \
--job_name wordcount-$USER \
--project <PROJECT> \
--num_workers 1 \
--region us-central1 \
--sdk_location apache-beam-2.44.0.zip
results in:
"/usr/local/lib/python3.10/site-packages/dataflow_worker/shuffle.py", line
589, in __enter__ raise RuntimeError(_PYTHON_310_SHUFFLE_ERROR_MESSAGE)
RuntimeError: This pipeline requires Dataflow Runner v2 in order to run
with currently used version of Apache Beam on Python 3.10+. Please verify
that the Dataflow Runner v2 is not disabled in the pipeline options or
enable it explicitly via: --dataflow_service_option=use_runner_v2.
Alternatively, downgrade to Python 3.9 to use Dataflow Runner v1.
Questions:
- I am not explicitly opting out of runner v2, and this is a standard
wordcount example, I expected it to just work.
Then I tried to add --dataflow_service_option=use_runner_v2 to the above
wordcount command, which results in the following error:
"message": "Dataflow Runner v2 requires a valid FnApi job, Please
resubmit your job with a valid configuration. Note that if using Templates,
you may need to regenerate your template with the '--use_runner_v2'."
Maybe I am doing something wrong and it is an error on my end. It would be
good for someone else with python experience to check this.
/cc @Valentyn Tymofieiev <valen...@google.com>
Ahmet
On Tue, Jan 10, 2023 at 10:54 AM Kenneth Knowles <k...@apache.org> wrote:
> I have published a new maven staging repository:
> https://repository.apache.org/content/repositories/orgapachebeam-1290/
>
> It looks like it has everything, though I did not automate a check. At
> least there were no errors during publish which I ran with --no-parallel
> overnight, and some specific things that were missing from
> orgapachebeam-1289 are present.
>
> I will restart the 72 hour waiting period, since the RC is only now usable.
>
> Kenn
>
> On Mon, Jan 9, 2023 at 6:51 PM Kenneth Knowles <k...@apache.org> wrote:
>
>> I have discovered that many pom files are missing from the nexus
>> repository. I should be able to re-publish a new one. It will take some
>> time as this is one of the longest-running processes.
>>
>> On Mon, Jan 9, 2023 at 1:42 PM Kenneth Knowles <k...@apache.org> wrote:
>>
>>> Correction: this is release candidate #1.
>>>
>>> On Mon, Jan 9, 2023 at 1:25 PM Kenneth Knowles <k...@apache.org> wrote:
>>>
>>>> Hi everyone,
>>>>
>>>> Please review and vote on the release candidate #3 for the version
>>>> 2.44.0, as follows:
>>>> [ ] +1, Approve the release
>>>> [ ] -1, Do not approve the release (please provide specific comments)
>>>>
>>>> Reviewers are encouraged to test their own use cases with the release
>>>> candidate, and vote +1 if
>>>> no issues are found.
>>>>
>>>> The complete staging area is available for your review, which includes:
>>>> * GitHub Release notes [1],
>>>> * the official Apache source release to be deployed to dist.apache.org
>>>> [2], which is signed with the key with fingerprint 6ED551A8AE02461C [3],
>>>> * all artifacts to be deployed to the Maven Central Repository [4],
>>>> * source code tag "v2.44.0-RC1" [5],
>>>> * website pull request listing the release [6], the blog post [6], and
>>>> publishing the API reference manual [7].
>>>> * Java artifacts were built with Gradle 7.5.1 and OpenJDK 1.8.0_232.
>>>> * Python artifacts are deployed along with the source release to the
>>>> dist.apache.org [2] and PyPI [8].
>>>> * Go artifacts and documentation are available at pkg.go.dev [9]
>>>> (waiting on these to appear)
>>>> * Validation sheet with a tab for 2.44.0 release to help with
>>>> validation [10].
>>>> * Docker images published to Docker Hub [11].
>>>>
>>>> The vote will be open for at least 72 hours. It is adopted by majority
>>>> approval, with at least 3 PMC affirmative votes.
>>>>
>>>> For guidelines on how to try the release in your projects, check out
>>>> our blog post at /blog/validate-beam-release/.
>>>>
>>>> Thanks,
>>>> Kenn
>>>>
>>>> [1] https://github.com/apache/beam/milestone/7
>>>> [2] https://dist.apache.org/repos/dist/dev/beam/2.44.0/
>>>> [3] https://dist.apache.org/repos/dist/release/beam/KEYS
>>>> [4]
>>>> https://repository.apache.org/content/repositories/orgapachebeam-1289/
>>>> [5] https://github.com/apache/beam/tree/v2.44.0-RC1
>>>> [6] https://github.com/apache/beam/pull/24951
>>>> [7] https://github.com/apache/beam-site/pull/638
>>>> [8] https://pypi.org/project/apache-beam/2.44.0rc1/
>>>> [9]
>>>> https://pkg.go.dev/github.com/apache/beam/sdks/v2@v2.44.0-RC1/go/pkg/beam
>>>> [10]
>>>> https://docs.google.com/spreadsheets/d/1qk-N5vjXvbcEk68GjbkSZTR8AGqyNUM-oLFo_ZXBpJw/edit#gid=622267910
>>>> [11] https://hub.docker.com/search?q=apache%2Fbeam&type=image
>>>>
>>>
