using thin jar to replace fat jar on yarn cluster mode
Hi, Does flink on yarn support using thin jar to replace fat jar? I don't want the jar of each flink task to have hundreds of MB. I want to put all the dependent packages in a single directory,and then the size of each flink task jar will be tens of KB. zjfpla...@hotmail.com
Re: using thin jar to replace fat jar on yarn cluster mode
Hi, I think you can try specifying dependent jars with the -C option[1] when you submit the job, and see if that meets your needs. [1] https://ci.apache.org/projects/flink/flink-docs-stable/ops/cli.html#usage On Mon, Dec 23, 2019 at 10:09 AM zjfpla...@hotmail.com < zjfpla...@hotmail.com> wrote: > Hi, > Does flink on yarn support using thin jar to replace fat jar? > I don't want the jar of each flink task to have hundreds of MB. I > want to put all the dependent packages in a single directory,and then the > size of each flink task jar will be tens of KB. > > -- > zjfpla...@hotmail.com > -- Best regards! Rui Li
Re: using thin jar to replace fat jar on yarn cluster mode
Hi zjfplayer, First, your user fat jar should not contains any flink dependents, them should be "provided". If your goal is to reduce the deployment time of task, and you are currently using session mode, you can consider putting jars directly under lib. Best, Jingsong Lee On Mon, Dec 23, 2019 at 3:24 PM Rui Li wrote: > Hi, > > I think you can try specifying dependent jars with the -C option[1] when > you submit the job, and see if that meets your needs. > > [1] > https://ci.apache.org/projects/flink/flink-docs-stable/ops/cli.html#usage > > On Mon, Dec 23, 2019 at 10:09 AM zjfpla...@hotmail.com < > zjfpla...@hotmail.com> wrote: > >> Hi, >> Does flink on yarn support using thin jar to replace fat jar? >> I don't want the jar of each flink task to have hundreds of MB. I >> want to put all the dependent packages in a single directory,and then >> the size of each flink task jar will be tens of KB. >> >> -- >> zjfpla...@hotmail.com >> > > > -- > Best regards! > Rui Li > -- Best, Jingsong Lee
