Hi All, I'm working trying to re-enable some tests like
LoadTests_Combine_Flink_Python.groovy and fix some vms leaked in those
tests. https://issues.apache.org/jira/browse/BEAM-12898
The version of dataproc used before was 1.2 and now it's 1.5.
The problem is that dataproc 1.5 flink version is 1.9 and actually we use
flink 1.13. Causing a mismatch and error running the tests.
In dataproc 1.2 a init script was passed with all the info related with
flink version, but now in optional components only told the component to
install
This was the way to create a cluster in dataproc 1.2
gcloud dataproc clusters create $CLUSTER_NAME --region=global
--num-workers=$num_dataproc_workers --initialization-actions
$DOCKER_INIT,$BEAM_INIT,$FLINK_INIT --metadata "${metadata}",
--image-version=$image_version --zone=$GCLOUD_ZONE --quiet
And this is the way to do it in dataproc 1.5
gcloud dataproc clusters create $CLUSTER_NAME --region=global
--num-workers=$num_dataproc_workers --metadata "${metadata}",
--image-version=$image_version --zone=$GCLOUD_ZONE
--optional-components=FLINK,DOCKER --quiet--
There is a way to force the flink version in dataproc ? I tried to use
Flink_init with initialization action but it didn't work.
Any help would be appreciated.
Thank you!
Andoni Guzman | WIZELINE
Software Engineer II
[email protected]
Amado Nervo 2200, Esfera P6, Col. Ciudad del Sol, 45050 Zapopan, Jal.
--
*This email and its contents (including any attachments) are being sent to
you on the condition of confidentiality and may be protected by legal
privilege. Access to this email by anyone other than the intended recipient
is unauthorized. If you are not the intended recipient, please immediately
notify the sender by replying to this message and delete the material
immediately from your system. Any further use, dissemination, distribution
or reproduction of this email is strictly prohibited. Further, no
representation is made with respect to any content contained in this email.*
