Hi Till, Thanks for the clarification and suggestions
Regards Sidhant Gupta On Wed, Dec 2, 2020, 10:10 PM Till Rohrmann <trohrm...@apache.org> wrote: > Hi Sidhant, > > Have you seen this discussion [1]? If you want to use S3, then you need to > make sure that you start your Flink processes with the appropriate > FileSystemProvider for S3 [2]. So the problem you are seeing is most likely > caused by the JVM not knowing a S3 file system implementation. > > Be aware that the filesystem used by the FileUploadHandler > is java.nio.file.FileSystem and not > Flink's org.apache.flink.core.fs.FileSystem for which we provide different > FileSystem implementations. > > My best recommendation would be to not use the web submission for Flink > jobs when running Flink behind a load balancer. Instead use the CLI which > will retrieve the leader address from ZooKeeper. > > [1] > https://lists.apache.org/thread.html/1dbaf68bc86ad3a1acc87e4bbc2dd5fa8c7560b3f59eb256f81bc855%40%3Cuser.flink.apache.org%3E > [2] > https://stackoverflow.com/questions/41113119/java-nio-file-implementation-for-aws > > Cheers, > Till > > On Wed, Dec 2, 2020 at 11:31 AM sidhant gupta <sidhan...@gmail.com> wrote: > >> Hi All, >> >> I have 2 job managers in flink HA mode cluster setup. I have a load >> balancer forwarding request to both (leader and stand by) the job managers >> in default round-robin fashion. While uploading the job jar the Web UI is >> fluctuating between the leader and standby page. Its difficult to upload >> the job jar and also getting errors after submitting the job as below: >> >> Caused by: org.apache.flink.runtime.rest.handler.RestHandlerException: >> Jar file /tmp/flink-web-4be8d72f-6a7d-4ec9-b71e-516e58 >> c629d5/flink-web-upload/ce291201-31e4-4e55 >> -a68b-eae1a770518d_flinkfhirmapperjob-1.2.jar does not exist >> >> >> I tried using *web.upload.dir* property to use a folder location in S3 >> but getting below error: >> >> 2020-12-02 08:26:06,242 WARN >> org.apache.flink.runtime.rest.FileUploadHandler [] - File >> upload failed. >> >> java.lang.IllegalArgumentException: UploadDirectory is not absolute. >> >> at >> org.apache.flink.util.Preconditions.checkArgument(Preconditions.java:139) >> ~[flink-dist_2.11-1.11.2.jar:1.11.2] >> >> at >> org.apache.flink.runtime.rest.handler.FileUploads.<init>(FileUploads.java:58) >> ~[flink-dist_2.11-1.11.2.jar:1.11.2] >> >> at >> org.apache.flink.runtime.rest.FileUploadHandler.channelRead0(FileUploadHandler.java:159) >> [flink-dist_2.11-1.11.2.jar:1.11.2] >> >> at >> org.apache.flink.runtime.rest.FileUploadHandler.channelRead0(FileUploadHandler.java:68) >> [flink-dist_2.11-1.11.2.jar:1.11.2] >> >> at >> org.apache.flink.shaded.netty4.io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) >> [flink-dist_2.11-1.11.2.jar:1.11.2] >> >> >> Looks like web.upload.dir only supports local path. >> >> >> Any suggestions on how to upload and submit job jar in flink HA cluster >> setup via web UI and also it from the CLI ? >> >> >> Thanks and regards >> >> Sidhant Gupta >> >> >> >>
