hi Yes I've specified the rest.address for the flink client to connect to the rest.address and the rest.address is valid and working fine but my question is why am I supposed to give jobmanager.rpc.address for flink client to connect to flink cluster if flink client depends only on rest.address?
On Thu, Jun 21, 2018 at 12:41 PM, Till Rohrmann <trohrm...@apache.org> wrote: > Hi, > > if the rest.address is different from the jobmanager.rpc.address, then you > should specify that in the flink-conf.yaml and Flink will connect to > rest.address. Only if rest.address is not specified, the system will fall > back to use the jobmanager.rpc.address. Currently, the rest server endpoint > runs in the same JVM as the cluster entrypoint and all JobMasters. > > Cheers, > Till > > On Thu, Jun 21, 2018 at 8:46 AM Sampath Bhat <sam414255p...@gmail.com> > wrote: > >> Hello Till >> >> Thanks for clarification. But I've few questions based on your reply. >> >> In non-HA setups we need the jobmanager.rpc.address to derive the hostname >> of the rest server. >> why is there dependency on jobmanager.rpc.address to get the hostname rest >> server? This holds good only for normal deployments such as on bare metal, >> virtual machine where flink cluster runs as another process in a machine. >> But if we try deploy flink on kubernetes then there could be possiblity >> where jobmanager.rpc.address and rest.address different from each other. >> >> So if rest.address is not provided in flink-conf.yaml then looking for >> jobmanager.rpc.address for deriving the hostname of rest server makes >> sense, but when the user has already provided the rest.address but flink >> still looks into jobmanager.rpc.address for getting hostname of rest >> server >> is an unwanted dependency IMO. >> >> In HA setup the rpc.address is obtained from zookeeper so user need not >> worry about unnecessary properties while submitting job. >> >> On Wed, Jun 20, 2018 at 1:25 PM, Till Rohrmann <trohrm...@apache.org> >> wrote: >> >> > It will, but it defaults to jobmanager.rpc.address if no rest.address >> has >> > been specified. >> > >> > On Wed, Jun 20, 2018 at 9:49 AM Chesnay Schepler <ches...@apache.org> >> > wrote: >> > >> >> Shouldn't the non-HA case be covered by rest.address? >> >> >> >> On 20.06.2018 09:40, Till Rohrmann wrote: >> >> >> >> Hi Sampath, >> >> >> >> it is no longer possible to not start the rest server endpoint by >> setting >> >> rest.port to -1. If you do this, then the cluster won't start. The >> comment >> >> in the flink-conf.yaml holds only true for the legacy mode. >> >> >> >> In non-HA setups we need the jobmanager.rpc.address to derive the >> >> hostname of the rest server. The jobmanager.rpc.port is no longer >> needed >> >> for the client but only for the other cluster components (TMs). When >> using >> >> the HA mode, then every address will be retrieved from ZooKeeper. >> >> >> >> I hope this clarifies things. >> >> >> >> Cheers, >> >> Till >> >> >> >> On Wed, Jun 20, 2018 at 9:24 AM Chesnay Schepler <ches...@apache.org> >> >> wrote: >> >> >> >>> I was worried this might be the case. >> >>> >> >>> The rest.port handling was simply copied from the legacy web-server, >> >>> which explicitly allowed shutting it down. >> >>> It may (I'm not entirely sure) also not be necessary for all >> deployment >> >>> modes; for example if the job is baked into the job/taskmanager >> images. >> >>> >> >>> I'm not quite sure whether the rpc address is actually required for >> the >> >>> REST job submission, or only since we still rely partly on some legacy >> >>> code (ClusterClient). Maybe Till (cc) knows the answer to that. >> >>> >> >>> > Adding on to this point you made - " the rpc address is still >> >>> *required *due >> >>> > to some technical implementations; it may be that you can set this >> to >> >>> some >> >>> > arbitrary value however." >> >>> > >> >>> > For job submission to happen successfully we should give specific >> rpc >> >>> > address and not any arbitrary value. If any arbitrary value is given >> >>> the >> >>> > job submission fails with the following error - >> >>> > org.apache.flink.client.deployment.ClusterRetrieveException: >> Couldn't >> >>> > retrieve standalone cluster >> >>> > at >> >>> > org.apache.flink.client.deployment.StandaloneClusterDescriptor. >> >>> retrieve(StandaloneClusterDescriptor.java:51) >> >>> > at >> >>> > org.apache.flink.client.deployment.StandaloneClusterDescriptor. >> >>> retrieve(StandaloneClusterDescriptor.java:31) >> >>> > at >> >>> > org.apache.flink.client.cli.CliFrontend.runProgram( >> >>> CliFrontend.java:249) >> >>> > at org.apache.flink.client.cli. >> CliFrontend.run(CliFrontend. >> >>> java:210) >> >>> > at >> >>> > org.apache.flink.client.cli.CliFrontend.parseParameters( >> >>> CliFrontend.java:1020) >> >>> > at >> >>> > org.apache.flink.client.cli.CliFrontend.lambda$main$9( >> >>> CliFrontend.java:1096) >> >>> > at >> >>> > org.apache.flink.runtime.security.NoOpSecurityContext.runSecured( >> >>> NoOpSecurityContext.java:30) >> >>> > at >> >>> > org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1096) >> >>> > Caused by: java.net.UnknownHostException: flinktest-flink- >> >>> jobmanager1233445: >> >>> > Name or service not known >> >>> > (Random name flinktest-flink-jobmanager1233445) >> >>> > at java.net.Inet6AddressImpl.lookupAllHostAddr(Native >> Method) >> >>> > at java.net.InetAddress$2.lookupAllHostAddr(InetAddress. >> >>> java:928) >> >>> > at >> >>> > java.net.InetAddress.getAddressesFromNameService( >> >>> InetAddress.java:1323) >> >>> > at java.net.InetAddress.getAllByName0(InetAddress. >> java:1276) >> >>> > at java.net.InetAddress.getAllByName(InetAddress.java: >> 1192) >> >>> > at java.net.InetAddress.getAllByName(InetAddress.java: >> 1126) >> >>> > at java.net.InetAddress.getByName(InetAddress.java:1076) >> >>> > at >> >>> > org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils. >> >>> getRpcUrl(AkkaRpcServiceUtils.java:171) >> >>> > at >> >>> > org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils. >> >>> getRpcUrl(AkkaRpcServiceUtils.java:136) >> >>> > at >> >>> > org.apache.flink.runtime.highavailability. >> >>> HighAvailabilityServicesUtils.createHighAvailabilityServices( >> >>> HighAvailabilityServicesUtils.java:83) >> >>> > at >> >>> > org.apache.flink.client.program.ClusterClient.<init>( >> >>> ClusterClient.java:158) >> >>> > at >> >>> > org.apache.flink.client.program.rest.RestClusterClient.<init>( >> >>> RestClusterClient.java:184) >> >>> > at >> >>> > org.apache.flink.client.program.rest.RestClusterClient.<init>( >> >>> RestClusterClient.java:157) >> >>> > at >> >>> > org.apache.flink.client.deployment.StandaloneClusterDescriptor. >> >>> retrieve(StandaloneClusterDescriptor.java:49) >> >>> > ... 7 more >> >>> > >> >>> > >> >>> > On Wed, Jun 20, 2018 at 11:18 AM, Sampath Bhat < >> >>> sam414255p...@gmail.com> >> >>> > wrote: >> >>> > >> >>> >> Hi Chesnay >> >>> >> >> >>> >> If REST API (i.e. the web server) is mandatory for submitting jobs >> >>> then >> >>> >> why is there an option to set rest.port to -1? I think it should be >> >>> >> mandatory to set some valid port for rest.port and make sure flink >> job >> >>> >> manager does not come up if valid port is not set for rest.port? Or >> >>> else >> >>> >> there must be some way to submit jobs even if REST API (i.e. the >> web >> >>> >> server) is not instantiated. >> >>> >> >> >>> >> If jobmanger.rpc.address is not required for flink client then why >> is >> >>> it >> >>> >> still looking for that property in flink-conf.yaml? Isn't it not a >> >>> bug? >> >>> >> Because if we comment out the jobmanger.rpc.address and >> >>> jobmanger.rpc.port >> >>> >> then flink client will not be able to submit the job. >> >>> >> >> >>> >> >> >>> >> On Tue, Jun 19, 2018 at 5:49 PM, Chesnay Schepler < >> ches...@apache.org >> >>> > >> >>> >> wrote: >> >>> >> >> >>> >>> In 1.5 we reworked the job-submission to go through the REST API >> >>> instead >> >>> >>> of akka. >> >>> >>> >> >>> >>> I believe the jobmanager rpc port shouldn't be necessary anymore, >> >>> the rpc >> >>> >>> address is still *required *due to some technical >> implementations; it >> >>> >>> may be that you can set this to some arbitrary value however. >> >>> >>> >> >>> >>> As a result the REST API (i.e. the web server) must be running in >> >>> order >> >>> >>> to submit jobs. >> >>> >>> >> >>> >>> >> >>> >>> On 19.06.2018 14:12, Sampath Bhat wrote: >> >>> >>> >> >>> >>> Hello >> >>> >>> >> >>> >>> I'm using Flink 1.5.0 version and Flink CLI to submit jar to flink >> >>> >>> cluster. >> >>> >>> >> >>> >>> In flink 1.4.2 only job manager rpc address and job manager rpc >> port >> >>> were >> >>> >>> sufficient for flink client to connect to job manager and submit >> the >> >>> job. >> >>> >>> >> >>> >>> But in flink 1.5.0 the flink client additionally requires the >> >>> >>> rest.address and rest.port for submitting the job to job manager. >> >>> What is >> >>> >>> the advantage of this new method over the 1.4.2 method of >> submitting >> >>> job? >> >>> >>> >> >>> >>> Moreover if we make rest.port = -1 the web server will not be >> >>> >>> instantiated then how should we submit the job? >> >>> >>> >> >>> >>> Regards >> >>> >>> Sampath >> >>> >>> >> >>> >>> >> >>> >>> >> >>> >> >>> >> >> >> >
