I have a very simple program that just reads all the files in the path. However, flink is not working as expected.
Everytime I execute this job I only see flink reading 2 files, even though there are more in that directory. On closer look it appears that it might be related to: [flink-akka.actor.default-dispatcher-3] INFO org.apache.flink.runtime.taskmanager.TaskManager - TaskManager has 2 task slot(s). My question is, isn't flink supposed to iterate over the directory after those 2 slots become free again? I am assuming this problem is caused because there are only 2 slots. Code --- PDFFileInputFormat format = new PDFFileInputFormat(); format.setFilePath(args[0]); format.setNestedFileEnumeration(true); logger.info("Number of splits " + format.getNumSplits()); // logger.info(Paths.get(".").toAbsolutePath().normalize().toString()); env.createInput(format, TypeInformation.of(StringValue.class)).print();