Hi Robert, indeed my docker-compose does work only if I add also Hadoop and yarn home while I was expecting that those two variables were generated automatically just setting env.xxx variables in FLINK_PROPERTIES variable..
I just want to understand what to expect, if I really need to specify Hadoop and yarn home as env variables or not Il gio 15 apr 2021, 20:39 Robert Metzger <rmetz...@apache.org> ha scritto: > Hi, > > I'm not aware of any known issues with Hadoop and Flink on Docker. > > I also tried what you are doing locally, and it seems to work: > > flink-jobmanager | 2021-04-15 18:37:48,300 INFO > org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Starting > StandaloneSessionClusterEntrypoint. > flink-jobmanager | 2021-04-15 18:37:48,338 INFO > org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Install > default filesystem. > flink-jobmanager | 2021-04-15 18:37:48,375 INFO > org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Install > security context. > flink-jobmanager | 2021-04-15 18:37:48,404 INFO > org.apache.flink.runtime.security.modules.HadoopModule [] - Hadoop > user set to flink (auth:SIMPLE) > flink-jobmanager | 2021-04-15 18:37:48,408 INFO > org.apache.flink.runtime.security.modules.JaasModule [] - Jaas > file will be created as /tmp/jaas-811306162058602256.conf. > flink-jobmanager | 2021-04-15 18:37:48,415 INFO > org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - > Initializing cluster services. > > Here's my code: > > https://gist.github.com/rmetzger/0cf4ba081d685d26478525bf69c7bd39 > > Hope this helps! > > On Wed, Apr 14, 2021 at 5:37 PM Flavio Pompermaier <pomperma...@okkam.it> > wrote: > >> Hi everybody, >> I'm trying to set up reading from HDFS using docker-compose and Flink >> 1.11.3. >> If I pass 'env.hadoop.conf.dir' and 'env.yarn.conf.dir' >> using FLINK_PROPERTIES (under environment section of the docker-compose >> service) I see in the logs the following line: >> >> "Could not find Hadoop configuration via any of the supported method" >> >> If I'm not wrong, this means that the HADOOP_CONF_DIR is actually not >> generated by the run scripts. >> Indeed, If I add HADOOP_CONF_DIR and YARN_CONF_DIR (always under >> environment section of the docker-compose service) I don't see that line. >> >> Is this the expected behavior? >> >> Below the relevant docker-compose service I use (I've removed the content >> of HADOOP_CLASSPATH content because is too long and I didn't report the >> taskmanager that is similar): >> >> flink-jobmanager: >> container_name: flink-jobmanager >> build: >> context: . >> dockerfile: Dockerfile.flink >> args: >> FLINK_VERSION: 1.11.3-scala_2.12-java11 >> image: 'flink-test:1.11.3-scala_2.12-java11' >> ports: >> - "8091:8081" >> - "8092:8082" >> command: jobmanager >> environment: >> - | >> FLINK_PROPERTIES= >> jobmanager.rpc.address: flink-jobmanager >> rest.port: 8081 >> historyserver.web.port: 8082 >> web.upload.dir: /opt/flink >> env.hadoop.conf.dir: /opt/hadoop/conf >> env.yarn.conf.dir: /opt/hadoop/conf >> - | >> HADOOP_CLASSPATH=... >> - HADOOP_CONF_DIR=/opt/hadoop/conf >> - YARN_CONF_DIR=/opt/hadoop/conf >> volumes: >> - 'flink_shared_folder:/tmp/test' >> - 'flink_uploads:/opt/flink/flink-web-upload' >> - 'flink_hadoop_conf:/opt/hadoop/conf' >> - 'flink_hadoop_libs:/opt/hadoop-3.2.1/share' >> >> >> Thanks in advance for any support, >> Flavio >> >
