Hi,
I use Flink 1.0.0. I have a persistent yarn container set (a persistent flink
job manager) that I use for streaming jobs ; and I use the “yarn-cluster” mode
to launch my batches.
I’ve just switched “HA” mode on for my streaming persistent job manager and it
seems to works ; however my batches are not working any longer because they now
execute themselves inside the persistent container (and fail because it lacks
slots) and not in a separate standalone job manager.
My batch launch options:
CONTAINER_OPTIONS="-m yarn-cluster -yn $FLINK_NBCONTAINERS -ytm $FLINK_MEMORY
-yqu $FLINK_QUEUE -ys $FLINK_NBSLOTS -yD
yarn.heap-cutoff-ratio=$FLINK_HEAP_CUTOFF_RATIO -yD akka.ask.timeout=300s"
JVM_ARGS="${JVM_ARGS} -Drecovery.mode=standalone
-Dyarn.properties-file.location=/tmp/flink/batch"
$FLINK_DIR/flink run $CONTAINER_OPTIONS --class $MAIN_CLASS_KUBERA $JAR_SUPP
$listArgs $ACTION
My persistent cluster launch option :
export FLINK_HA_OPTIONS="-Dyarn.application-attempts=10
-Drecovery.mode=zookeeper
-Drecovery.zookeeper.quorum=${FLINK_HA_ZOOKEEPER_SERVERS}
-Drecovery.zookeeper.path.root=${FLINK_HA_ZOOKEEPER_PATH}
-Dstate.backend=filesystem
-Dstate.backend.fs.checkpointdir=hdfs:///tmp/${FLINK_HA_ZOOKEEPER_PATH}/checkpoints
-Drecovery.zookeeper.storageDir=hdfs:///tmp/${FLINK_HA_ZOOKEEPER_PATH}/recovery/"
$FLINK_DIR/yarn-session.sh -Dyarn.heap-cutoff-ratio=$FLINK_HEAP_CUTOFF_RATIO
$FLINK_HA_OPTIONS -st -d -n $FLINK_NBCONTAINERS -s $FLINK_NBSLOTS -tm
$FLINK_MEMORY -qu $FLINK_QUEUE -nm ${GANESH_TYPE_PF}_KuberaFlink
I’ve switched back to the FLINK_HA_OPTIONS="" way of launching the container
for now, but I lack HA.
Is it a (un)known bug or am I missing a magic option?
Best regards,
Arnaud
________________________________
L'intégrité de ce message n'étant pas assurée sur internet, la société
expéditrice ne peut être tenue responsable de son contenu ni de ses pièces
jointes. Toute utilisation ou diffusion non autorisée est interdite. Si vous
n'êtes pas destinataire de ce message, merci de le détruire et d'avertir
l'expéditeur.
The integrity of this message cannot be guaranteed on the Internet. The company
that sent this message cannot therefore be held liable for its content nor
attachments. Any unauthorized use or dissemination is prohibited. If you are
not the intended recipient of this message, then please delete it and notify
the sender.
