Re: Where are uploaded Job jars stored?
Hi Zhu Zhu, as per my original post. If you have 3 Job nodes and you upload the job through the web UI on a specific node, then only the web UI for that node will show it. If you browse another job node you will not see it. As per Stevens suggestion, I already have a Gluster FS mounted folder on the Job nodes, I will point web.upload.dir to that folder. So that when I upload from one node all nodes will see it. On Fri, 11 Oct 2019 at 00:34, Steven Nelson wrote: > John, > > I think you are referring to the web upload directory. There is a setting > for that folder ‘web.upload.dir’. If you set that to a folder writeable > to both masters it will work as desired. I used an NFS mount (AWS EFS). > > -Steve > > > Sent from my iPhone > > On Oct 10, 2019, at 10:11 PM, Zhu Zhu wrote: > > Hi John, > > Not sure why you need to know the location of uploaded job jars? > > The job jar will be automatically localized to a taskmanager via > BlobService when a task belonging to the job is running on that taskmanager. > The localization dir is blob.storage.directory. If it is not specified, > it will be java.io.tmpdir in standalone mode. > > Thanks, > Zhu Zhu > > John Smith 于2019年10月11日周五 上午2:41写道： > >> And can that folder be shared so that all nodes see it? >> >> On Thu, 10 Oct 2019 at 14:36, Yun Tang wrote: >> >>> Hi John >>> >>> The jar is not stored in HA path, I think the answer [1] could help you. >>> >>> [1] >>> https://stackoverflow.com/questions/51936608/where-can-i-find-my-jar-on-apache-flink-server-which-i-submitted-using-apache-fl >>> >>> Best >>> Yun Tang >>> -- >>> *From:* John Smith >>> *Sent:* Friday, October 11, 2019 2:06 >>> *To:* user >>> *Subject:* Where are uploaded Job jars stored? >>> >>> Hi using 1.8.0 running on standalone cluster with Zookeeper HA. >>> >>> Are job JARs stored at: high-availability.storageDir ??? >>> >>> The thing is when you browse the individual nodes at port 8080 to go >>> submit the job only the node where you uploaded the JAR has it. >>> >>> - Go to any given node >>> - Upload a jar >>> - Browse another node >>> - Jar is not there. >>> >>> >>>
Re: Where are uploaded Job jars stored?
John, I think you are referring to the web upload directory. There is a setting for that folder ‘web.upload.dir’. If you set that to a folder writeable to both masters it will work as desired. I used an NFS mount (AWS EFS). -Steve Sent from my iPhone > On Oct 10, 2019, at 10:11 PM, Zhu Zhu wrote: > > Hi John, > > Not sure why you need to know the location of uploaded job jars? > > The job jar will be automatically localized to a taskmanager via BlobService > when a task belonging to the job is running on that taskmanager. > The localization dir is blob.storage.directory. If it is not specified, it > will be java.io.tmpdir in standalone mode. > > Thanks, > Zhu Zhu > > John Smith 于2019年10月11日周五 上午2:41写道： >> And can that folder be shared so that all nodes see it? >> >>> On Thu, 10 Oct 2019 at 14:36, Yun Tang wrote: >>> Hi John >>> >>> The jar is not stored in HA path, I think the answer [1] could help you. >>> >>> [1] >>> https://stackoverflow.com/questions/51936608/where-can-i-find-my-jar-on-apache-flink-server-which-i-submitted-using-apache-fl >>> >>> Best >>> Yun Tang >>> From: John Smith >>> Sent: Friday, October 11, 2019 2:06 >>> To: user >>> Subject: Where are uploaded Job jars stored? >>> >>> Hi using 1.8.0 running on standalone cluster with Zookeeper HA. >>> >>> Are job JARs stored at: high-availability.storageDir ??? >>> >>> The thing is when you browse the individual nodes at port 8080 to go submit >>> the job only the node where you uploaded the JAR has it. >>> >>> - Go to any given node >>> - Upload a jar >>> - Browse another node >>> - Jar is not there. >>> >>>
Re: Where are uploaded Job jars stored?
Hi John, Not sure why you need to know the location of uploaded job jars? The job jar will be automatically localized to a taskmanager via BlobService when a task belonging to the job is running on that taskmanager. The localization dir is blob.storage.directory. If it is not specified, it will be java.io.tmpdir in standalone mode. Thanks, Zhu Zhu John Smith 于2019年10月11日周五 上午2:41写道： > And can that folder be shared so that all nodes see it? > > On Thu, 10 Oct 2019 at 14:36, Yun Tang wrote: > >> Hi John >> >> The jar is not stored in HA path, I think the answer [1] could help you. >> >> [1] >> https://stackoverflow.com/questions/51936608/where-can-i-find-my-jar-on-apache-flink-server-which-i-submitted-using-apache-fl >> >> Best >> Yun Tang >> -- >> *From:* John Smith >> *Sent:* Friday, October 11, 2019 2:06 >> *To:* user >> *Subject:* Where are uploaded Job jars stored? >> >> Hi using 1.8.0 running on standalone cluster with Zookeeper HA. >> >> Are job JARs stored at: high-availability.storageDir ??? >> >> The thing is when you browse the individual nodes at port 8080 to go >> submit the job only the node where you uploaded the JAR has it. >> >> - Go to any given node >> - Upload a jar >> - Browse another node >> - Jar is not there. >> >> >>
Re: Where are uploaded Job jars stored?
And can that folder be shared so that all nodes see it? On Thu, 10 Oct 2019 at 14:36, Yun Tang wrote: > Hi John > > The jar is not stored in HA path, I think the answer [1] could help you. > > [1] > https://stackoverflow.com/questions/51936608/where-can-i-find-my-jar-on-apache-flink-server-which-i-submitted-using-apache-fl > > Best > Yun Tang > -- > *From:* John Smith > *Sent:* Friday, October 11, 2019 2:06 > *To:* user > *Subject:* Where are uploaded Job jars stored? > > Hi using 1.8.0 running on standalone cluster with Zookeeper HA. > > Are job JARs stored at: high-availability.storageDir ??? > > The thing is when you browse the individual nodes at port 8080 to go > submit the job only the node where you uploaded the JAR has it. > > - Go to any given node > - Upload a jar > - Browse another node > - Jar is not there. > > >
Re: Where are uploaded Job jars stored?
Hi John The jar is not stored in HA path, I think the answer [1] could help you. [1] https://stackoverflow.com/questions/51936608/where-can-i-find-my-jar-on-apache-flink-server-which-i-submitted-using-apache-fl Best Yun Tang From: John Smith Sent: Friday, October 11, 2019 2:06 To: user Subject: Where are uploaded Job jars stored? Hi using 1.8.0 running on standalone cluster with Zookeeper HA. Are job JARs stored at: high-availability.storageDir ??? The thing is when you browse the individual nodes at port 8080 to go submit the job only the node where you uploaded the JAR has it. - Go to any given node - Upload a jar - Browse another node - Jar is not there.
Where are uploaded Job jars stored?
Hi using 1.8.0 running on standalone cluster with Zookeeper HA. Are job JARs stored at: high-availability.storageDir ??? The thing is when you browse the individual nodes at port 8080 to go submit the job only the node where you uploaded the JAR has it. - Go to any given node - Upload a jar - Browse another node - Jar is not there.
