Re: Flink cli to upload flink jar and run flink jobs in the Jenkins pipeline

[sidhant gupta]

Thanks Yang for your help.

On Thu, Feb 4, 2021, 8:28 AM Yang Wang  wrote:

> Yes, if you are using the CLI(e.g. flink run/list/cancel -t yarn-session
> ...) for the job management,
> it will eventually call the RestClusterClient, which could retrieve the
> leader JobManager address from ZK.
>
> Please ensure that you have specified the HA related config options in CLI
> via -D or set them in the flink-conf.yaml.
>
> Best,
> Yang
>
> sidhant gupta  于2021年2月3日周三 下午10:02写道：
>
>> Is it possible to use flink CLI instead of flink client for connecting
>> zookeeper using network load balancer to retrieve the leader Jobmanager
>> address?
>>
>> On Wed, Feb 3, 2021, 12:42 PM Yang Wang  wrote:
>>
>>> I think the Flink client could make a connection with ZooKeeper via the
>>> network load balancer.
>>> Flink client is not aware of whether it is a network balancer or
>>> multiple ZooKeeper server address.
>>> After then Flink client will retrieve the active leader JobManager
>>> address via ZooKeeperHAService
>>> and submit the job successfully via rest client.
>>>
>>> Best,
>>> Yang
>>>
>>>
>>> sidhant gupta  于2021年2月2日周二 下午11:14写道：
>>>
 Hi

 I have a flink ECS cluster setup with HA mode using zookeeper where I
 have 2 jobmanagers out of which one of will be elected as leader using
 zookeeper leader election. I have one application load balancer in front of
 the jobmanagers and one network load balancer in front of zookeeper.

 As per [1]
 
  ,
 we can provide zookeeper address in the flink cli arguments and it would
 upload/ submit the jar to the leader jobmanager. But since I am using
 network load balancer in front of zookeeper, I guess it is not able to make
 connection with the zookeeper. Please provide suggestions or sample command
 for uploading the flink job jar or run the job.

 Is  there any way by which we can distinguish between leader and
 standby jobmanagers in terms of request or response ?

 Can we use flink cli in jenkins to upload the jar to the flink cluster
 and run the jobs?


 [1]
 http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Best-way-to-find-the-current-alive-jobmanager-with-HA-mode-zookeeper-td21787.html

 Thanks
 Sidhant Gupta

>>>

Re: Flink cli to upload flink jar and run flink jobs in the Jenkins pipeline

[Yang Wang]

Yes, if you are using the CLI(e.g. flink run/list/cancel -t yarn-session
...) for the job management,
it will eventually call the RestClusterClient, which could retrieve the
leader JobManager address from ZK.

Please ensure that you have specified the HA related config options in CLI
via -D or set them in the flink-conf.yaml.

Best,
Yang

sidhant gupta  于2021年2月3日周三 下午10:02写道：

> Is it possible to use flink CLI instead of flink client for connecting
> zookeeper using network load balancer to retrieve the leader Jobmanager
> address?
>
> On Wed, Feb 3, 2021, 12:42 PM Yang Wang  wrote:
>
>> I think the Flink client could make a connection with ZooKeeper via the
>> network load balancer.
>> Flink client is not aware of whether it is a network balancer or multiple
>> ZooKeeper server address.
>> After then Flink client will retrieve the active leader JobManager
>> address via ZooKeeperHAService
>> and submit the job successfully via rest client.
>>
>> Best,
>> Yang
>>
>>
>> sidhant gupta  于2021年2月2日周二 下午11:14写道：
>>
>>> Hi
>>>
>>> I have a flink ECS cluster setup with HA mode using zookeeper where I
>>> have 2 jobmanagers out of which one of will be elected as leader using
>>> zookeeper leader election. I have one application load balancer in front of
>>> the jobmanagers and one network load balancer in front of zookeeper.
>>>
>>> As per [1]
>>> 
>>>  ,
>>> we can provide zookeeper address in the flink cli arguments and it would
>>> upload/ submit the jar to the leader jobmanager. But since I am using
>>> network load balancer in front of zookeeper, I guess it is not able to make
>>> connection with the zookeeper. Please provide suggestions or sample command
>>> for uploading the flink job jar or run the job.
>>>
>>> Is  there any way by which we can distinguish between leader and standby
>>> jobmanagers in terms of request or response ?
>>>
>>> Can we use flink cli in jenkins to upload the jar to the flink cluster
>>> and run the jobs?
>>>
>>>
>>> [1]
>>> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Best-way-to-find-the-current-alive-jobmanager-with-HA-mode-zookeeper-td21787.html
>>>
>>> Thanks
>>> Sidhant Gupta
>>>
>>

Re: Flink cli to upload flink jar and run flink jobs in the Jenkins pipeline

[sidhant gupta]

Is it possible to use flink CLI instead of flink client for connecting
zookeeper using network load balancer to retrieve the leader Jobmanager
address?

On Wed, Feb 3, 2021, 12:42 PM Yang Wang  wrote:

> I think the Flink client could make a connection with ZooKeeper via the
> network load balancer.
> Flink client is not aware of whether it is a network balancer or multiple
> ZooKeeper server address.
> After then Flink client will retrieve the active leader JobManager address
> via ZooKeeperHAService
> and submit the job successfully via rest client.
>
> Best,
> Yang
>
>
> sidhant gupta  于2021年2月2日周二 下午11:14写道：
>
>> Hi
>>
>> I have a flink ECS cluster setup with HA mode using zookeeper where I
>> have 2 jobmanagers out of which one of will be elected as leader using
>> zookeeper leader election. I have one application load balancer in front of
>> the jobmanagers and one network load balancer in front of zookeeper.
>>
>> As per [1]
>> 
>>  ,
>> we can provide zookeeper address in the flink cli arguments and it would
>> upload/ submit the jar to the leader jobmanager. But since I am using
>> network load balancer in front of zookeeper, I guess it is not able to make
>> connection with the zookeeper. Please provide suggestions or sample command
>> for uploading the flink job jar or run the job.
>>
>> Is  there any way by which we can distinguish between leader and standby
>> jobmanagers in terms of request or response ?
>>
>> Can we use flink cli in jenkins to upload the jar to the flink cluster
>> and run the jobs?
>>
>>
>> [1]
>> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Best-way-to-find-the-current-alive-jobmanager-with-HA-mode-zookeeper-td21787.html
>>
>> Thanks
>> Sidhant Gupta
>>
>

Re: Flink cli to upload flink jar and run flink jobs in the Jenkins pipeline

[Yang Wang]

I think the Flink client could make a connection with ZooKeeper via the
network load balancer.
Flink client is not aware of whether it is a network balancer or multiple
ZooKeeper server address.
After then Flink client will retrieve the active leader JobManager address
via ZooKeeperHAService
and submit the job successfully via rest client.

Best,
Yang


sidhant gupta  于2021年2月2日周二 下午11:14写道：

> Hi
>
> I have a flink ECS cluster setup with HA mode using zookeeper where I have
> 2 jobmanagers out of which one of will be elected as leader using zookeeper
> leader election. I have one application load balancer in front of the
> jobmanagers and one network load balancer in front of zookeeper.
>
> As per [1]
> 
>  ,
> we can provide zookeeper address in the flink cli arguments and it would
> upload/ submit the jar to the leader jobmanager. But since I am using
> network load balancer in front of zookeeper, I guess it is not able to make
> connection with the zookeeper. Please provide suggestions or sample command
> for uploading the flink job jar or run the job.
>
> Is  there any way by which we can distinguish between leader and standby
> jobmanagers in terms of request or response ?
>
> Can we use flink cli in jenkins to upload the jar to the flink cluster and
> run the jobs?
>
>
> [1]
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Best-way-to-find-the-current-alive-jobmanager-with-HA-mode-zookeeper-td21787.html
>
> Thanks
> Sidhant Gupta
>

Flink cli to upload flink jar and run flink jobs in the Jenkins pipeline

[sidhant gupta]

Hi

I have a flink ECS cluster setup with HA mode using zookeeper where I have
2 jobmanagers out of which one of will be elected as leader using zookeeper
leader election. I have one application load balancer in front of the
jobmanagers and one network load balancer in front of zookeeper.

As per [1]

,
we can provide zookeeper address in the flink cli arguments and it would
upload/ submit the jar to the leader jobmanager. But since I am using
network load balancer in front of zookeeper, I guess it is not able to make
connection with the zookeeper. Please provide suggestions or sample command
for uploading the flink job jar or run the job.

Is  there any way by which we can distinguish between leader and standby
jobmanagers in terms of request or response ?

Can we use flink cli in jenkins to upload the jar to the flink cluster and
run the jobs?


[1]
http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Best-way-to-find-the-current-alive-jobmanager-with-HA-mode-zookeeper-td21787.html

Thanks
Sidhant Gupta