Hey everyone, I've got an awesome looking Flink cluster set up with web.submit.enable=true, and plenty of bash for handling jar upload and then submission to a JobManager - all good so far.
Unfortunately, when I try to submit the classic WordCount example, I get a massive error with the jist of it being:
"Job was submitted in detached mode. Results of job execution, such as accumulators, runtime, etc. are not available. Please make sure your program doesn't call an eager execution function [collect, print, printToErr, count]."
So, how do I run it *not* in detached mode using curl please?
I'm intentionally not using the Flink CLI because I am using an nginx with auth proxy set up - so I'm doing everything with curl, in a bash script - (so, two requests - one to upload the jar, then I get the ID from the response, and then submit the job with that ID).
At https://ci.apache.org/projects/flink/flink-docs-release-1.12/ops/rest_api.html if you ctrl-f for /run, there's nothing obvious that indicates how I can run in blocking mode - the biggest clue I've got is `programArg`. So I'm wondering if I can provide that somehow.
For those who prefer code:
curl ${auth_options} ${self_signed_flag} ${ca_cert_flag} -X POST -H "Content-Type: application/json" https://${JOBMANAGER}/jars/${uploaded_jar_string}/run$programArgsToUse
Whereby programArgsToUse is user args, and I'm cool with them being query parameters for now - I think.
I'm passing them on the end with:
if [[ ! -z $program_args ]] ; then
programArgsToUse="?programArg=$program_args"
fi
if [[ ! -z $program_args ]] ; then
programArgsToUse="?programArg=$program_args"
fi
curl --cacert /etc/ssl/tester/certs/ca.crt -X POST -H 'Content-Type: application/json' 'https://tester-minimal-tls-sample-jobmanager:8081/jars/fdc7684f-323d-49fa-a60a-96683d953be8_WordCount.jar/run?programArg=detached=false'
(obviously, what's at the end looks really wrong, but IDK what to use)
The only mention of "detach" I see documented is at https://ci.apache.org/projects/flink/flink-docs-stable/deployment/config.html, and that says by default it's not in detached mode for execution. If there are any better docs or examples, please send them my way - or if you've spotted me being just plain silly with my bash, that would be fantastic to point out.
Thanks a lot in advance, cheers, happy to share any more code/background if need be
IBM United Kingdom Limited - Registered in England and Wales with number 741598.
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
