date:20190312

Awesome, thanks!

On Tue, Mar 12, 2019 at 11:53 AM Gary Yao  wrote:

> The RC artifacts are only deployed to the Maven Central Repository when
> the RC
> is promoted to a release. As written in the 1.8.0 RC1 voting email [1], you
> can find the maven artifacts, and the Flink binaries here:
>
> -
> https://repository.apache.org/content/repositories/orgapacheflink-1210/
> - https://dist.apache.org/repos/dist/dev/flink/flink-1.8.0-rc1/
>
> Alternatively, you can apply the patch yourself, and build Flink 1.7 from
> sources [2]. On my machine this takes around 10 minutes if tests are
> skipped.
>
> Best,
> Gary
>
> [1]
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/VOTE-Release-1-8-0-release-candidate-1-td27637.html
> [2]
> https://ci.apache.org/projects/flink/flink-docs-release-1.7/flinkDev/building.html#build-flink
>
> On Tue, Mar 12, 2019 at 4:01 PM Vishal Santoshi 
> wrote:
>
>> Do you have a mvn repository ( at mvn central )  set up for 1,8 release
>> candidate. We could test it for you.
>>
>> Without 1.8and this exit code we are essentially held up.
>>
>> On Tue, Mar 12, 2019 at 10:56 AM Gary Yao  wrote:
>>
>>> Nobody can tell with 100% certainty. We want to give the RC some exposure
>>> first, and there is also a release process that is prescribed by the ASF
>>> [1].
>>> You can look at past releases to get a feeling for how long the release
>>> process lasts [2].
>>>
>>> [1] http://www.apache.org/legal/release-policy.html#release-approval
>>> [2]
>>> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/template/NamlServlet.jtp?macro=search_page=1=%5BVOTE%5D+Release=0
>>>
>>>
>>> On Tue, Mar 12, 2019 at 3:38 PM Vishal Santoshi <
>>> vishal.santo...@gmail.com> wrote:
>>>
 And when is the 1.8.0 release expected ?

 On Tue, Mar 12, 2019 at 10:32 AM Vishal Santoshi <
 vishal.santo...@gmail.com> wrote:

> :) That makes so much more sense. Is  k8s native flink a part of this
> release ?
>
> On Tue, Mar 12, 2019 at 10:27 AM Gary Yao  wrote:
>
>> Hi Vishal,
>>
>> This issue was fixed recently [1], and the patch will be released
>> with 1.8. If
>> the Flink job gets cancelled, the JVM should exit with code 0. There
>> is a
>> release candidate [2], which you can test.
>>
>> Best,
>> Gary
>>
>> [1] https://issues.apache.org/jira/browse/FLINK-10743
>> [2]
>> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/VOTE-Release-1-8-0-release-candidate-1-td27637.html
>>
>> On Tue, Mar 12, 2019 at 3:21 PM Vishal Santoshi <
>> vishal.santo...@gmail.com> wrote:
>>
>>> Thanks Vijay,
>>>
>>> This is the larger issue.  The cancellation routine is itself broken.
>>>
>>> On cancellation flink does remove the checkpoint counter
>>>
>>> *2019-03-12 14:12:13,143
>>> INFO  org.apache.flink.runtime.checkpoint.ZooKeeperCheckpointIDCounter  
>>> -
>>> Removing /checkpoint-counter/ from
>>> ZooKeeper *
>>>
>>> but exist with a non zero code
>>>
>>> *2019-03-12 14:12:13,477
>>> INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint -
>>> Terminating cluster entrypoint process StandaloneJobClusterEntryPoint 
>>> with
>>> exit code 1444.*
>>>
>>>
>>> That I think is an issue. A cancelled job is a complete job and thus
>>> the exit code should be 0 for k8s to mark it complete.
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> On Tue, Mar 12, 2019 at 10:18 AM Vijay Bhaskar <
>>> bhaskar.eba...@gmail.com> wrote:
>>>
 Yes Vishal. Thats correct.

 Regards
 Bhaskar

 On Tue, Mar 12, 2019 at 7:14 PM Vishal Santoshi <
 vishal.santo...@gmail.com> wrote:

> This really not cool but here you go. This seems to work. Agreed
> that this cannot be this painful. The cancel does not exit with an 
> exit
> code pf 0 and thus the job has to manually delete. Vijay does this 
> align
> with what you have had to do ?
>
>
>- Take a save point . This returns a request id
>
>curl  --header "Content-Type: application/json" --request POST 
> --data 
> '{"target-directory":"hdfs://nn-crunchy:8020/tmp/xyz14","cancel-job":false}'
> 
> https://*/jobs//savepoints
>
>
>
>- Make sure the save point succeeded
>
>curl  --request GET   
> https:///jobs//savepoints/2c053ce3bea31276aa25e63784629687
>
>
>
>- cancel the job
>
>curl  --request PATCH 
> https://***/jobs/?mode=cancel
>
>

Re: K8s job cluster and cancel and resume from a save point ?

The RC artifacts are only deployed to the Maven Central Repository when the
RC
is promoted to a release. As written in the 1.8.0 RC1 voting email [1], you
can find the maven artifacts, and the Flink binaries here:

-
https://repository.apache.org/content/repositories/orgapacheflink-1210/
- https://dist.apache.org/repos/dist/dev/flink/flink-1.8.0-rc1/

Alternatively, you can apply the patch yourself, and build Flink 1.7 from
sources [2]. On my machine this takes around 10 minutes if tests are
skipped.

Best,
Gary

[1]
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/VOTE-Release-1-8-0-release-candidate-1-td27637.html
[2]
https://ci.apache.org/projects/flink/flink-docs-release-1.7/flinkDev/building.html#build-flink

On Tue, Mar 12, 2019 at 4:01 PM Vishal Santoshi 
wrote:

> Do you have a mvn repository ( at mvn central )  set up for 1,8 release
> candidate. We could test it for you.
>
> Without 1.8and this exit code we are essentially held up.
>
> On Tue, Mar 12, 2019 at 10:56 AM Gary Yao  wrote:
>
>> Nobody can tell with 100% certainty. We want to give the RC some exposure
>> first, and there is also a release process that is prescribed by the ASF
>> [1].
>> You can look at past releases to get a feeling for how long the release
>> process lasts [2].
>>
>> [1] http://www.apache.org/legal/release-policy.html#release-approval
>> [2]
>> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/template/NamlServlet.jtp?macro=search_page=1=%5BVOTE%5D+Release=0
>>
>>
>> On Tue, Mar 12, 2019 at 3:38 PM Vishal Santoshi <
>> vishal.santo...@gmail.com> wrote:
>>
>>> And when is the 1.8.0 release expected ?
>>>
>>> On Tue, Mar 12, 2019 at 10:32 AM Vishal Santoshi <
>>> vishal.santo...@gmail.com> wrote:
>>>
 :) That makes so much more sense. Is  k8s native flink a part of this
 release ?

 On Tue, Mar 12, 2019 at 10:27 AM Gary Yao  wrote:

> Hi Vishal,
>
> This issue was fixed recently [1], and the patch will be released with
> 1.8. If
> the Flink job gets cancelled, the JVM should exit with code 0. There
> is a
> release candidate [2], which you can test.
>
> Best,
> Gary
>
> [1] https://issues.apache.org/jira/browse/FLINK-10743
> [2]
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/VOTE-Release-1-8-0-release-candidate-1-td27637.html
>
> On Tue, Mar 12, 2019 at 3:21 PM Vishal Santoshi <
> vishal.santo...@gmail.com> wrote:
>
>> Thanks Vijay,
>>
>> This is the larger issue.  The cancellation routine is itself broken.
>>
>> On cancellation flink does remove the checkpoint counter
>>
>> *2019-03-12 14:12:13,143
>> INFO  org.apache.flink.runtime.checkpoint.ZooKeeperCheckpointIDCounter  -
>> Removing /checkpoint-counter/ from
>> ZooKeeper *
>>
>> but exist with a non zero code
>>
>> *2019-03-12 14:12:13,477
>> INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint -
>> Terminating cluster entrypoint process StandaloneJobClusterEntryPoint 
>> with
>> exit code 1444.*
>>
>>
>> That I think is an issue. A cancelled job is a complete job and thus
>> the exit code should be 0 for k8s to mark it complete.
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> On Tue, Mar 12, 2019 at 10:18 AM Vijay Bhaskar <
>> bhaskar.eba...@gmail.com> wrote:
>>
>>> Yes Vishal. Thats correct.
>>>
>>> Regards
>>> Bhaskar
>>>
>>> On Tue, Mar 12, 2019 at 7:14 PM Vishal Santoshi <
>>> vishal.santo...@gmail.com> wrote:
>>>
 This really not cool but here you go. This seems to work. Agreed
 that this cannot be this painful. The cancel does not exit with an exit
 code pf 0 and thus the job has to manually delete. Vijay does this 
 align
 with what you have had to do ?


- Take a save point . This returns a request id

curl  --header "Content-Type: application/json" --request POST 
 --data 
 '{"target-directory":"hdfs://nn-crunchy:8020/tmp/xyz14","cancel-job":false}'
 
 https://*/jobs//savepoints



- Make sure the save point succeeded

curl  --request GET   
 https:///jobs//savepoints/2c053ce3bea31276aa25e63784629687



- cancel the job

curl  --request PATCH 
 https://***/jobs/?mode=cancel



- Delete the job and deployment

kubectl delete -f 
 manifests/bf2-PRODUCTION/job-cluster-job-deployment.yaml

kubectl delete -f

Re: local disk cleanup after crash

Hi,

If no other TaskManager (TM) is running, you can delete everything. If
multiple TMs share the same host, as far as I know, you will have to parse
TM
logs to know what directories you can delete [1]. As for local recovery,
tasks
that were running on a crashed TM are lost. From the documentation [2]:

If a task manager is lost, the local state from all its task is lost.

Therefore, assuming that only one TM is running on each host, you can delete
everything.

Best,
Gary

[1]
http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/What-are-blobstore-files-and-why-do-they-keep-filling-up-tmp-directory-td26323.html
[2]
https://ci.apache.org/projects/flink/flink-docs-stable/ops/state/large_state_tuning.html#task-local-recovery

On Thu, Mar 7, 2019 at 10:45 PM Derek VerLee  wrote:

> I think that effort is put in to have task managers clean up their
> folders, however I have noticed that in some cases local folders are not
> cleaned up and can build up, eventually causing problems due to a full
> disk.  As far as I know this only happens with crashes and other
> out-of-happy-path scenarios.
>
> I am thinking of writing a script to clean up local folders that runs
> before task-manager starts between restarts in the case of a crash.
>
> Assuming local recovery is not configured, what should I delete and what
> should I leave around?
>
> What should I keep if local recovery is configured?
>
>
> Under the "taskmanager.tmp.dirs" I see:
>
> blobStore-*
> flink-dist-cache-*
> flink-io-*
> localState/*
> rocksdb-lib-*
>
>
> Thanks
>

Re: K8s job cluster and cancel and resume from a save point ?

Do you have a mvn repository ( at mvn central )  set up for 1,8 release
candidate. We could test it for you.

Without 1.8and this exit code we are essentially held up.

On Tue, Mar 12, 2019 at 10:56 AM Gary Yao  wrote:

> Nobody can tell with 100% certainty. We want to give the RC some exposure
> first, and there is also a release process that is prescribed by the ASF
> [1].
> You can look at past releases to get a feeling for how long the release
> process lasts [2].
>
> [1] http://www.apache.org/legal/release-policy.html#release-approval
> [2]
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/template/NamlServlet.jtp?macro=search_page=1=%5BVOTE%5D+Release=0
>
>
> On Tue, Mar 12, 2019 at 3:38 PM Vishal Santoshi 
> wrote:
>
>> And when is the 1.8.0 release expected ?
>>
>> On Tue, Mar 12, 2019 at 10:32 AM Vishal Santoshi <
>> vishal.santo...@gmail.com> wrote:
>>
>>> :) That makes so much more sense. Is  k8s native flink a part of this
>>> release ?
>>>
>>> On Tue, Mar 12, 2019 at 10:27 AM Gary Yao  wrote:
>>>
 Hi Vishal,

 This issue was fixed recently [1], and the patch will be released with
 1.8. If
 the Flink job gets cancelled, the JVM should exit with code 0. There is
 a
 release candidate [2], which you can test.

 Best,
 Gary

 [1] https://issues.apache.org/jira/browse/FLINK-10743
 [2]
 http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/VOTE-Release-1-8-0-release-candidate-1-td27637.html

 On Tue, Mar 12, 2019 at 3:21 PM Vishal Santoshi <
 vishal.santo...@gmail.com> wrote:

> Thanks Vijay,
>
> This is the larger issue.  The cancellation routine is itself broken.
>
> On cancellation flink does remove the checkpoint counter
>
> *2019-03-12 14:12:13,143
> INFO  org.apache.flink.runtime.checkpoint.ZooKeeperCheckpointIDCounter  -
> Removing /checkpoint-counter/ from
> ZooKeeper *
>
> but exist with a non zero code
>
> *2019-03-12 14:12:13,477
> INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint -
> Terminating cluster entrypoint process StandaloneJobClusterEntryPoint with
> exit code 1444.*
>
>
> That I think is an issue. A cancelled job is a complete job and thus
> the exit code should be 0 for k8s to mark it complete.
>
>
>
>
>
>
>
>
>
> On Tue, Mar 12, 2019 at 10:18 AM Vijay Bhaskar <
> bhaskar.eba...@gmail.com> wrote:
>
>> Yes Vishal. Thats correct.
>>
>> Regards
>> Bhaskar
>>
>> On Tue, Mar 12, 2019 at 7:14 PM Vishal Santoshi <
>> vishal.santo...@gmail.com> wrote:
>>
>>> This really not cool but here you go. This seems to work. Agreed
>>> that this cannot be this painful. The cancel does not exit with an exit
>>> code pf 0 and thus the job has to manually delete. Vijay does this align
>>> with what you have had to do ?
>>>
>>>
>>>- Take a save point . This returns a request id
>>>
>>>curl  --header "Content-Type: application/json" --request POST 
>>> --data 
>>> '{"target-directory":"hdfs://nn-crunchy:8020/tmp/xyz14","cancel-job":false}'
>>> 
>>> https://*/jobs//savepoints
>>>
>>>
>>>
>>>- Make sure the save point succeeded
>>>
>>>curl  --request GET   
>>> https:///jobs//savepoints/2c053ce3bea31276aa25e63784629687
>>>
>>>
>>>
>>>- cancel the job
>>>
>>>curl  --request PATCH 
>>> https://***/jobs/?mode=cancel
>>>
>>>
>>>
>>>- Delete the job and deployment
>>>
>>>kubectl delete -f 
>>> manifests/bf2-PRODUCTION/job-cluster-job-deployment.yaml
>>>
>>>kubectl delete -f 
>>> manifests/bf2-PRODUCTION/task-manager-deployment.yaml
>>>
>>>
>>>
>>>- Edit the job-cluster-job-deployment.yaml. Add/Edit
>>>
>>>args: ["job-cluster",
>>>
>>>   "--fromSavepoint",
>>>
>>>   
>>> "hdfs:///tmp/xyz14/savepoint-00-1d4f71345e22",
>>>   "--job-classname", .
>>>
>>>
>>>
>>>- Restart
>>>
>>>kubectl create -f 
>>> manifests/bf2-PRODUCTION/job-cluster-job-deployment.yaml
>>>
>>>kubectl create -f 
>>> manifests/bf2-PRODUCTION/task-manager-deployment.yaml
>>>
>>>
>>>
>>>- Make sure from the UI, that it restored from the specific save
>>>point.
>>>
>>>
>>> On Tue, Mar 12, 2019 at 7:26 AM Vijay Bhaskar <
>>> bhaskar.eba...@gmail.com> wrote:
>>>
 Yes Its supposed to work.  But unfortunately it was not working.
 Flink community needs to respond to this behavior.

Re: K8s job cluster and cancel and resume from a save point ?

Nobody can tell with 100% certainty. We want to give the RC some exposure
first, and there is also a release process that is prescribed by the ASF
[1].
You can look at past releases to get a feeling for how long the release
process lasts [2].

[1] http://www.apache.org/legal/release-policy.html#release-approval
[2]
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/template/NamlServlet.jtp?macro=search_page=1=%5BVOTE%5D+Release=0


On Tue, Mar 12, 2019 at 3:38 PM Vishal Santoshi 
wrote:

> And when is the 1.8.0 release expected ?
>
> On Tue, Mar 12, 2019 at 10:32 AM Vishal Santoshi <
> vishal.santo...@gmail.com> wrote:
>
>> :) That makes so much more sense. Is  k8s native flink a part of this
>> release ?
>>
>> On Tue, Mar 12, 2019 at 10:27 AM Gary Yao  wrote:
>>
>>> Hi Vishal,
>>>
>>> This issue was fixed recently [1], and the patch will be released with
>>> 1.8. If
>>> the Flink job gets cancelled, the JVM should exit with code 0. There is a
>>> release candidate [2], which you can test.
>>>
>>> Best,
>>> Gary
>>>
>>> [1] https://issues.apache.org/jira/browse/FLINK-10743
>>> [2]
>>> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/VOTE-Release-1-8-0-release-candidate-1-td27637.html
>>>
>>> On Tue, Mar 12, 2019 at 3:21 PM Vishal Santoshi <
>>> vishal.santo...@gmail.com> wrote:
>>>
 Thanks Vijay,

 This is the larger issue.  The cancellation routine is itself broken.

 On cancellation flink does remove the checkpoint counter

 *2019-03-12 14:12:13,143
 INFO  org.apache.flink.runtime.checkpoint.ZooKeeperCheckpointIDCounter  -
 Removing /checkpoint-counter/ from
 ZooKeeper *

 but exist with a non zero code

 *2019-03-12 14:12:13,477
 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint -
 Terminating cluster entrypoint process StandaloneJobClusterEntryPoint with
 exit code 1444.*


 That I think is an issue. A cancelled job is a complete job and thus
 the exit code should be 0 for k8s to mark it complete.









 On Tue, Mar 12, 2019 at 10:18 AM Vijay Bhaskar <
 bhaskar.eba...@gmail.com> wrote:

> Yes Vishal. Thats correct.
>
> Regards
> Bhaskar
>
> On Tue, Mar 12, 2019 at 7:14 PM Vishal Santoshi <
> vishal.santo...@gmail.com> wrote:
>
>> This really not cool but here you go. This seems to work. Agreed that
>> this cannot be this painful. The cancel does not exit with an exit code 
>> pf
>> 0 and thus the job has to manually delete. Vijay does this align with 
>> what
>> you have had to do ?
>>
>>
>>- Take a save point . This returns a request id
>>
>>curl  --header "Content-Type: application/json" --request POST --data 
>> '{"target-directory":"hdfs://nn-crunchy:8020/tmp/xyz14","cancel-job":false}'
>> 
>> https://*/jobs//savepoints
>>
>>
>>
>>- Make sure the save point succeeded
>>
>>curl  --request GET   
>> https:///jobs//savepoints/2c053ce3bea31276aa25e63784629687
>>
>>
>>
>>- cancel the job
>>
>>curl  --request PATCH 
>> https://***/jobs/?mode=cancel
>>
>>
>>
>>- Delete the job and deployment
>>
>>kubectl delete -f 
>> manifests/bf2-PRODUCTION/job-cluster-job-deployment.yaml
>>
>>kubectl delete -f 
>> manifests/bf2-PRODUCTION/task-manager-deployment.yaml
>>
>>
>>
>>- Edit the job-cluster-job-deployment.yaml. Add/Edit
>>
>>args: ["job-cluster",
>>
>>   "--fromSavepoint",
>>
>>   
>> "hdfs:///tmp/xyz14/savepoint-00-1d4f71345e22",
>>   "--job-classname", .
>>
>>
>>
>>- Restart
>>
>>kubectl create -f 
>> manifests/bf2-PRODUCTION/job-cluster-job-deployment.yaml
>>
>>kubectl create -f 
>> manifests/bf2-PRODUCTION/task-manager-deployment.yaml
>>
>>
>>
>>- Make sure from the UI, that it restored from the specific save
>>point.
>>
>>
>> On Tue, Mar 12, 2019 at 7:26 AM Vijay Bhaskar <
>> bhaskar.eba...@gmail.com> wrote:
>>
>>> Yes Its supposed to work.  But unfortunately it was not working.
>>> Flink community needs to respond to this behavior.
>>>
>>> Regards
>>> Bhaskar
>>>
>>> On Tue, Mar 12, 2019 at 3:45 PM Vishal Santoshi <
>>> vishal.santo...@gmail.com> wrote:
>>>
 Aah.
 Let me try this out and will get back to you.
 Though I would assume that save point with cancel is a single
 atomic step, rather then a save point *followed*  by a
 cancellation (

Re: K8s job cluster and cancel and resume from a save point ?

This really not cool but here you go. This seems to work. Agreed that this
cannot be this painful. The cancel does not exit with an exit code pf 0 and
thus the job has to manually delete. Vijay does this align with what you
have had to do ?

   - Take a save point . This returns a request id

   curl  --header "Content-Type: application/json" --request POST
--data 
'{"target-directory":"hdfs://nn-crunchy:8020/tmp/xyz14","cancel-job":false}'
   https://*/jobs//savepoints

   - Make sure the save point succeeded

   curl  --request GET
https:///jobs//savepoints/2c053ce3bea31276aa25e63784629687

   - cancel the job

   curl  --request PATCH
https://***/jobs/?mode=cancel

   - Delete the job and deployment

   kubectl delete -f manifests/bf2-PRODUCTION/job-cluster-job-deployment.yaml

   kubectl delete -f manifests/bf2-PRODUCTION/task-manager-deployment.yaml

   - Edit the job-cluster-job-deployment.yaml. Add/Edit

   args: ["job-cluster",

  "--fromSavepoint",

  "hdfs:///tmp/xyz14/savepoint-00-1d4f71345e22",
  "--job-classname", .

   - Restart

   kubectl create -f manifests/bf2-PRODUCTION/job-cluster-job-deployment.yaml

   kubectl create -f manifests/bf2-PRODUCTION/task-manager-deployment.yaml

   - Make sure from the UI, that it restored from the specific save point.

On Tue, Mar 12, 2019 at 7:26 AM Vijay Bhaskar 
wrote:

> Yes Its supposed to work.  But unfortunately it was not working. Flink
> community needs to respond to this behavior.
>
> Regards
> Bhaskar
>
> On Tue, Mar 12, 2019 at 3:45 PM Vishal Santoshi 
> wrote:
>
>> Aah.
>> Let me try this out and will get back to you.
>> Though I would assume that save point with cancel is a single atomic
>> step, rather then a save point *followed*  by a cancellation ( else why
>> would that be an option ).
>> Thanks again.
>>
>>
>> On Tue, Mar 12, 2019 at 4:50 AM Vijay Bhaskar 
>> wrote:
>>
>>> Hi Vishal,
>>>
>>> yarn-cancel doesn't mean to be for yarn cluster. It works for all
>>> clusters. Its recommended command.
>>>
>>> Use the following command to issue save point.
>>>  curl  --header "Content-Type: application/json" --request POST --data
>>> '{"target-directory":"hdfs://*:8020/tmp/xyz1","cancel-job":false}'
>>> \ https://
>>> .ingress.***/jobs//savepoints
>>>
>>> Then issue yarn-cancel.
>>> After that  follow the process to restore save point
>>>
>>> Regards
>>> Bhaskar
>>>
>>> On Tue, Mar 12, 2019 at 2:11 PM Vishal Santoshi <
>>> vishal.santo...@gmail.com> wrote:
>>>
 Hello Vijay,

Thank you for the reply. This though is k8s deployment (
 rather then yarn ) but may be they follow the same lifecycle.  I issue a*
 save point with cancel*  as documented here
 https://ci.apache.org/projects/flink/flink-docs-release-1.7/monitoring/rest_api.html#jobs-jobid-savepoints,
 a straight up
  curl  --header "Content-Type: application/json" --request POST --data
 '{"target-directory":"hdfs://*:8020/tmp/xyz1","cancel-job":true}'
 \ https://
 .ingress.***/jobs//savepoints

 I would assume that after taking the save point, the jvm should exit,
 after all the k8s deployment is of kind: job and if it is a job cluster
 then a cancellation should exit the jvm and hence the pod. It does seem to
 do some things right. It stops a bunch of stuff ( the JobMaster, the
 slotPol, zookeeper coordinator etc ) . It also remove the checkpoint
 counter but does not exit  the job. And after a little bit the job is
 restarted which does not make sense and absolutely not the right thing to
 do  ( to me at least ).

 Further if I delete the deployment and the job from k8s and restart the
 job and deployment fromSavePoint, it refuses to honor the fromSavePoint. I
 have to delete the zk chroot for it to consider the save point.

 Thus the process of cancelling and resuming from a SP on a k8s job
 cluster deployment  seems to be

- cancel with save point as defined hre

 https://ci.apache.org/projects/flink/flink-docs-release-1.7/monitoring/rest_api.html#jobs-jobid-savepoints
- delete the job manger job  and task manager deployments from k8s
almost immediately.
- clear the ZK chroot for the 000.. job  and may be the
checkpoints directory.
- resumeFromCheckPoint

 If some body can say that this indeed is the process ?

  Logs are attached.

 2019-03-12 08:10:43,871 INFO
 org.apache.flink.runtime.jobmaster.JobMaster  -
 Savepoint stored in

Re: K8s job cluster and cancel and resume from a save point ?

Hi Vishal,

I'm afraid not but there are open pull requests for that. You can track the
progress here:

https://issues.apache.org/jira/browse/FLINK-9953

Best,
Gary

On Tue, Mar 12, 2019 at 3:32 PM Vishal Santoshi 
wrote:

> :) That makes so much more sense. Is  k8s native flink a part of this
> release ?
>
> On Tue, Mar 12, 2019 at 10:27 AM Gary Yao  wrote:
>
>> Hi Vishal,
>>
>> This issue was fixed recently [1], and the patch will be released with
>> 1.8. If
>> the Flink job gets cancelled, the JVM should exit with code 0. There is a
>> release candidate [2], which you can test.
>>
>> Best,
>> Gary
>>
>> [1] https://issues.apache.org/jira/browse/FLINK-10743
>> [2]
>> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/VOTE-Release-1-8-0-release-candidate-1-td27637.html
>>
>> On Tue, Mar 12, 2019 at 3:21 PM Vishal Santoshi <
>> vishal.santo...@gmail.com> wrote:
>>
>>> Thanks Vijay,
>>>
>>> This is the larger issue.  The cancellation routine is itself broken.
>>>
>>> On cancellation flink does remove the checkpoint counter
>>>
>>> *2019-03-12 14:12:13,143
>>> INFO  org.apache.flink.runtime.checkpoint.ZooKeeperCheckpointIDCounter  -
>>> Removing /checkpoint-counter/ from
>>> ZooKeeper *
>>>
>>> but exist with a non zero code
>>>
>>> *2019-03-12 14:12:13,477
>>> INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint -
>>> Terminating cluster entrypoint process StandaloneJobClusterEntryPoint with
>>> exit code 1444.*
>>>
>>>
>>> That I think is an issue. A cancelled job is a complete job and thus the
>>> exit code should be 0 for k8s to mark it complete.
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> On Tue, Mar 12, 2019 at 10:18 AM Vijay Bhaskar 
>>> wrote:
>>>
 Yes Vishal. Thats correct.

 Regards
 Bhaskar

 On Tue, Mar 12, 2019 at 7:14 PM Vishal Santoshi <
 vishal.santo...@gmail.com> wrote:

> This really not cool but here you go. This seems to work. Agreed that
> this cannot be this painful. The cancel does not exit with an exit code pf
> 0 and thus the job has to manually delete. Vijay does this align with what
> you have had to do ?
>
>
>- Take a save point . This returns a request id
>
>curl  --header "Content-Type: application/json" --request POST --data 
> '{"target-directory":"hdfs://nn-crunchy:8020/tmp/xyz14","cancel-job":false}'
> https://*/jobs//savepoints
>
>
>
>- Make sure the save point succeeded
>
>curl  --request GET   
> https:///jobs//savepoints/2c053ce3bea31276aa25e63784629687
>
>
>
>- cancel the job
>
>curl  --request PATCH 
> https://***/jobs/?mode=cancel
>
>
>
>- Delete the job and deployment
>
>kubectl delete -f 
> manifests/bf2-PRODUCTION/job-cluster-job-deployment.yaml
>
>kubectl delete -f manifests/bf2-PRODUCTION/task-manager-deployment.yaml
>
>
>
>- Edit the job-cluster-job-deployment.yaml. Add/Edit
>
>args: ["job-cluster",
>
>   "--fromSavepoint",
>
>   
> "hdfs:///tmp/xyz14/savepoint-00-1d4f71345e22",
>   "--job-classname", .
>
>
>
>- Restart
>
>kubectl create -f 
> manifests/bf2-PRODUCTION/job-cluster-job-deployment.yaml
>
>kubectl create -f manifests/bf2-PRODUCTION/task-manager-deployment.yaml
>
>
>
>- Make sure from the UI, that it restored from the specific save
>point.
>
>
> On Tue, Mar 12, 2019 at 7:26 AM Vijay Bhaskar <
> bhaskar.eba...@gmail.com> wrote:
>
>> Yes Its supposed to work.  But unfortunately it was not working.
>> Flink community needs to respond to this behavior.
>>
>> Regards
>> Bhaskar
>>
>> On Tue, Mar 12, 2019 at 3:45 PM Vishal Santoshi <
>> vishal.santo...@gmail.com> wrote:
>>
>>> Aah.
>>> Let me try this out and will get back to you.
>>> Though I would assume that save point with cancel is a single atomic
>>> step, rather then a save point *followed*  by a cancellation ( else
>>> why would that be an option ).
>>> Thanks again.
>>>
>>>
>>> On Tue, Mar 12, 2019 at 4:50 AM Vijay Bhaskar <
>>> bhaskar.eba...@gmail.com> wrote:
>>>
 Hi Vishal,

 yarn-cancel doesn't mean to be for yarn cluster. It works for all
 clusters. Its recommended command.

 Use the following command to issue save point.
  curl  --header "Content-Type: application/json" --request POST
 --data '{"target-directory":"hdfs://*:8020/tmp/xyz1",
 "cancel-job":false}'  \ https://

Re: K8s job cluster and cancel and resume from a save point ?

And when is the 1.8.0 release expected ?

On Tue, Mar 12, 2019 at 10:32 AM Vishal Santoshi 
wrote:

> :) That makes so much more sense. Is  k8s native flink a part of this
> release ?
>
> On Tue, Mar 12, 2019 at 10:27 AM Gary Yao  wrote:
>
>> Hi Vishal,
>>
>> This issue was fixed recently [1], and the patch will be released with
>> 1.8. If
>> the Flink job gets cancelled, the JVM should exit with code 0. There is a
>> release candidate [2], which you can test.
>>
>> Best,
>> Gary
>>
>> [1] https://issues.apache.org/jira/browse/FLINK-10743
>> [2]
>> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/VOTE-Release-1-8-0-release-candidate-1-td27637.html
>>
>> On Tue, Mar 12, 2019 at 3:21 PM Vishal Santoshi <
>> vishal.santo...@gmail.com> wrote:
>>
>>> Thanks Vijay,
>>>
>>> This is the larger issue.  The cancellation routine is itself broken.
>>>
>>> On cancellation flink does remove the checkpoint counter
>>>
>>> *2019-03-12 14:12:13,143
>>> INFO  org.apache.flink.runtime.checkpoint.ZooKeeperCheckpointIDCounter  -
>>> Removing /checkpoint-counter/ from
>>> ZooKeeper *
>>>
>>> but exist with a non zero code
>>>
>>> *2019-03-12 14:12:13,477
>>> INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint -
>>> Terminating cluster entrypoint process StandaloneJobClusterEntryPoint with
>>> exit code 1444.*
>>>
>>>
>>> That I think is an issue. A cancelled job is a complete job and thus the
>>> exit code should be 0 for k8s to mark it complete.
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> On Tue, Mar 12, 2019 at 10:18 AM Vijay Bhaskar 
>>> wrote:
>>>
 Yes Vishal. Thats correct.

 Regards
 Bhaskar

 On Tue, Mar 12, 2019 at 7:14 PM Vishal Santoshi <
 vishal.santo...@gmail.com> wrote:

> This really not cool but here you go. This seems to work. Agreed that
> this cannot be this painful. The cancel does not exit with an exit code pf
> 0 and thus the job has to manually delete. Vijay does this align with what
> you have had to do ?
>
>
>- Take a save point . This returns a request id
>
>curl  --header "Content-Type: application/json" --request POST --data 
> '{"target-directory":"hdfs://nn-crunchy:8020/tmp/xyz14","cancel-job":false}'
> https://*/jobs//savepoints
>
>
>
>- Make sure the save point succeeded
>
>curl  --request GET   
> https:///jobs//savepoints/2c053ce3bea31276aa25e63784629687
>
>
>
>- cancel the job
>
>curl  --request PATCH 
> https://***/jobs/?mode=cancel
>
>
>
>- Delete the job and deployment
>
>kubectl delete -f 
> manifests/bf2-PRODUCTION/job-cluster-job-deployment.yaml
>
>kubectl delete -f manifests/bf2-PRODUCTION/task-manager-deployment.yaml
>
>
>
>- Edit the job-cluster-job-deployment.yaml. Add/Edit
>
>args: ["job-cluster",
>
>   "--fromSavepoint",
>
>   
> "hdfs:///tmp/xyz14/savepoint-00-1d4f71345e22",
>   "--job-classname", .
>
>
>
>- Restart
>
>kubectl create -f 
> manifests/bf2-PRODUCTION/job-cluster-job-deployment.yaml
>
>kubectl create -f manifests/bf2-PRODUCTION/task-manager-deployment.yaml
>
>
>
>- Make sure from the UI, that it restored from the specific save
>point.
>
>
> On Tue, Mar 12, 2019 at 7:26 AM Vijay Bhaskar <
> bhaskar.eba...@gmail.com> wrote:
>
>> Yes Its supposed to work.  But unfortunately it was not working.
>> Flink community needs to respond to this behavior.
>>
>> Regards
>> Bhaskar
>>
>> On Tue, Mar 12, 2019 at 3:45 PM Vishal Santoshi <
>> vishal.santo...@gmail.com> wrote:
>>
>>> Aah.
>>> Let me try this out and will get back to you.
>>> Though I would assume that save point with cancel is a single atomic
>>> step, rather then a save point *followed*  by a cancellation ( else
>>> why would that be an option ).
>>> Thanks again.
>>>
>>>
>>> On Tue, Mar 12, 2019 at 4:50 AM Vijay Bhaskar <
>>> bhaskar.eba...@gmail.com> wrote:
>>>
 Hi Vishal,

 yarn-cancel doesn't mean to be for yarn cluster. It works for all
 clusters. Its recommended command.

 Use the following command to issue save point.
  curl  --header "Content-Type: application/json" --request POST
 --data '{"target-directory":"hdfs://*:8020/tmp/xyz1",
 "cancel-job":false}'  \ https://
 .ingress.***/jobs//savepoints

 Then issue yarn-cancel.
 After that  follow the process to

Re: K8s job cluster and cancel and resume from a save point ?

:) That makes so much more sense. Is  k8s native flink a part of this
release ?

On Tue, Mar 12, 2019 at 10:27 AM Gary Yao  wrote:

> Hi Vishal,
>
> This issue was fixed recently [1], and the patch will be released with
> 1.8. If
> the Flink job gets cancelled, the JVM should exit with code 0. There is a
> release candidate [2], which you can test.
>
> Best,
> Gary
>
> [1] https://issues.apache.org/jira/browse/FLINK-10743
> [2]
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/VOTE-Release-1-8-0-release-candidate-1-td27637.html
>
> On Tue, Mar 12, 2019 at 3:21 PM Vishal Santoshi 
> wrote:
>
>> Thanks Vijay,
>>
>> This is the larger issue.  The cancellation routine is itself broken.
>>
>> On cancellation flink does remove the checkpoint counter
>>
>> *2019-03-12 14:12:13,143
>> INFO  org.apache.flink.runtime.checkpoint.ZooKeeperCheckpointIDCounter  -
>> Removing /checkpoint-counter/ from
>> ZooKeeper *
>>
>> but exist with a non zero code
>>
>> *2019-03-12 14:12:13,477
>> INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint -
>> Terminating cluster entrypoint process StandaloneJobClusterEntryPoint with
>> exit code 1444.*
>>
>>
>> That I think is an issue. A cancelled job is a complete job and thus the
>> exit code should be 0 for k8s to mark it complete.
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> On Tue, Mar 12, 2019 at 10:18 AM Vijay Bhaskar 
>> wrote:
>>
>>> Yes Vishal. Thats correct.
>>>
>>> Regards
>>> Bhaskar
>>>
>>> On Tue, Mar 12, 2019 at 7:14 PM Vishal Santoshi <
>>> vishal.santo...@gmail.com> wrote:
>>>
 This really not cool but here you go. This seems to work. Agreed that
 this cannot be this painful. The cancel does not exit with an exit code pf
 0 and thus the job has to manually delete. Vijay does this align with what
 you have had to do ?


- Take a save point . This returns a request id

curl  --header "Content-Type: application/json" --request POST --data 
 '{"target-directory":"hdfs://nn-crunchy:8020/tmp/xyz14","cancel-job":false}'
 https://*/jobs//savepoints



- Make sure the save point succeeded

curl  --request GET   
 https:///jobs//savepoints/2c053ce3bea31276aa25e63784629687



- cancel the job

curl  --request PATCH 
 https://***/jobs/?mode=cancel



- Delete the job and deployment

kubectl delete -f 
 manifests/bf2-PRODUCTION/job-cluster-job-deployment.yaml

kubectl delete -f manifests/bf2-PRODUCTION/task-manager-deployment.yaml



- Edit the job-cluster-job-deployment.yaml. Add/Edit

args: ["job-cluster",

   "--fromSavepoint",

   
 "hdfs:///tmp/xyz14/savepoint-00-1d4f71345e22",
   "--job-classname", .



- Restart

kubectl create -f 
 manifests/bf2-PRODUCTION/job-cluster-job-deployment.yaml

kubectl create -f manifests/bf2-PRODUCTION/task-manager-deployment.yaml



- Make sure from the UI, that it restored from the specific save
point.


 On Tue, Mar 12, 2019 at 7:26 AM Vijay Bhaskar 
 wrote:

> Yes Its supposed to work.  But unfortunately it was not working. Flink
> community needs to respond to this behavior.
>
> Regards
> Bhaskar
>
> On Tue, Mar 12, 2019 at 3:45 PM Vishal Santoshi <
> vishal.santo...@gmail.com> wrote:
>
>> Aah.
>> Let me try this out and will get back to you.
>> Though I would assume that save point with cancel is a single atomic
>> step, rather then a save point *followed*  by a cancellation ( else
>> why would that be an option ).
>> Thanks again.
>>
>>
>> On Tue, Mar 12, 2019 at 4:50 AM Vijay Bhaskar <
>> bhaskar.eba...@gmail.com> wrote:
>>
>>> Hi Vishal,
>>>
>>> yarn-cancel doesn't mean to be for yarn cluster. It works for all
>>> clusters. Its recommended command.
>>>
>>> Use the following command to issue save point.
>>>  curl  --header "Content-Type: application/json" --request POST
>>> --data '{"target-directory":"hdfs://*:8020/tmp/xyz1",
>>> "cancel-job":false}'  \ https://
>>> .ingress.***/jobs//savepoints
>>>
>>> Then issue yarn-cancel.
>>> After that  follow the process to restore save point
>>>
>>> Regards
>>> Bhaskar
>>>
>>> On Tue, Mar 12, 2019 at 2:11 PM Vishal Santoshi <
>>> vishal.santo...@gmail.com> wrote:
>>>
 Hello Vijay,

Thank you for the reply. This though is k8s
 deployment ( rather then yarn ) but

Re: K8s job cluster and cancel and resume from a save point ?

Hi Vishal,

This issue was fixed recently [1], and the patch will be released with 1.8.
If
the Flink job gets cancelled, the JVM should exit with code 0. There is a
release candidate [2], which you can test.

Best,
Gary

[1] https://issues.apache.org/jira/browse/FLINK-10743
[2]
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/VOTE-Release-1-8-0-release-candidate-1-td27637.html

On Tue, Mar 12, 2019 at 3:21 PM Vishal Santoshi 
wrote:

> Thanks Vijay,
>
> This is the larger issue.  The cancellation routine is itself broken.
>
> On cancellation flink does remove the checkpoint counter
>
> *2019-03-12 14:12:13,143
> INFO  org.apache.flink.runtime.checkpoint.ZooKeeperCheckpointIDCounter  -
> Removing /checkpoint-counter/ from
> ZooKeeper *
>
> but exist with a non zero code
>
> *2019-03-12 14:12:13,477
> INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint -
> Terminating cluster entrypoint process StandaloneJobClusterEntryPoint with
> exit code 1444.*
>
>
> That I think is an issue. A cancelled job is a complete job and thus the
> exit code should be 0 for k8s to mark it complete.
>
>
>
>
>
>
>
>
>
> On Tue, Mar 12, 2019 at 10:18 AM Vijay Bhaskar 
> wrote:
>
>> Yes Vishal. Thats correct.
>>
>> Regards
>> Bhaskar
>>
>> On Tue, Mar 12, 2019 at 7:14 PM Vishal Santoshi <
>> vishal.santo...@gmail.com> wrote:
>>
>>> This really not cool but here you go. This seems to work. Agreed that
>>> this cannot be this painful. The cancel does not exit with an exit code pf
>>> 0 and thus the job has to manually delete. Vijay does this align with what
>>> you have had to do ?
>>>
>>>
>>>- Take a save point . This returns a request id
>>>
>>>curl  --header "Content-Type: application/json" --request POST --data 
>>> '{"target-directory":"hdfs://nn-crunchy:8020/tmp/xyz14","cancel-job":false}'
>>> https://*/jobs//savepoints
>>>
>>>
>>>
>>>- Make sure the save point succeeded
>>>
>>>curl  --request GET   
>>> https:///jobs//savepoints/2c053ce3bea31276aa25e63784629687
>>>
>>>
>>>
>>>- cancel the job
>>>
>>>curl  --request PATCH 
>>> https://***/jobs/?mode=cancel
>>>
>>>
>>>
>>>- Delete the job and deployment
>>>
>>>kubectl delete -f 
>>> manifests/bf2-PRODUCTION/job-cluster-job-deployment.yaml
>>>
>>>kubectl delete -f manifests/bf2-PRODUCTION/task-manager-deployment.yaml
>>>
>>>
>>>
>>>- Edit the job-cluster-job-deployment.yaml. Add/Edit
>>>
>>>args: ["job-cluster",
>>>
>>>   "--fromSavepoint",
>>>
>>>   
>>> "hdfs:///tmp/xyz14/savepoint-00-1d4f71345e22",
>>>   "--job-classname", .
>>>
>>>
>>>
>>>- Restart
>>>
>>>kubectl create -f 
>>> manifests/bf2-PRODUCTION/job-cluster-job-deployment.yaml
>>>
>>>kubectl create -f manifests/bf2-PRODUCTION/task-manager-deployment.yaml
>>>
>>>
>>>
>>>- Make sure from the UI, that it restored from the specific save
>>>point.
>>>
>>>
>>> On Tue, Mar 12, 2019 at 7:26 AM Vijay Bhaskar 
>>> wrote:
>>>
 Yes Its supposed to work.  But unfortunately it was not working. Flink
 community needs to respond to this behavior.

 Regards
 Bhaskar

 On Tue, Mar 12, 2019 at 3:45 PM Vishal Santoshi <
 vishal.santo...@gmail.com> wrote:

> Aah.
> Let me try this out and will get back to you.
> Though I would assume that save point with cancel is a single atomic
> step, rather then a save point *followed*  by a cancellation ( else
> why would that be an option ).
> Thanks again.
>
>
> On Tue, Mar 12, 2019 at 4:50 AM Vijay Bhaskar <
> bhaskar.eba...@gmail.com> wrote:
>
>> Hi Vishal,
>>
>> yarn-cancel doesn't mean to be for yarn cluster. It works for all
>> clusters. Its recommended command.
>>
>> Use the following command to issue save point.
>>  curl  --header "Content-Type: application/json" --request POST
>> --data '{"target-directory":"hdfs://*:8020/tmp/xyz1",
>> "cancel-job":false}'  \ https://
>> .ingress.***/jobs//savepoints
>>
>> Then issue yarn-cancel.
>> After that  follow the process to restore save point
>>
>> Regards
>> Bhaskar
>>
>> On Tue, Mar 12, 2019 at 2:11 PM Vishal Santoshi <
>> vishal.santo...@gmail.com> wrote:
>>
>>> Hello Vijay,
>>>
>>>Thank you for the reply. This though is k8s
>>> deployment ( rather then yarn ) but may be they follow the same 
>>> lifecycle.
>>> I issue a* save point with cancel*  as documented here
>>> https://ci.apache.org/projects/flink/flink-docs-release-1.7/monitoring/rest_api.html#jobs-jobid-savepoints,
>>> a straight up
>>>  curl  --header "Content-Type: application/json"

Re: K8s job cluster and cancel and resume from a save point ?

Oh, Yeah this is larger issue indeed :)

Regards
Bhaskar

On Tue, Mar 12, 2019 at 7:51 PM Vishal Santoshi 
wrote:

> Thanks Vijay,
>
> This is the larger issue.  The cancellation routine is itself broken.
>
> On cancellation flink does remove the checkpoint counter
>
> *2019-03-12 14:12:13,143
> INFO  org.apache.flink.runtime.checkpoint.ZooKeeperCheckpointIDCounter  -
> Removing /checkpoint-counter/ from
> ZooKeeper *
>
> but exist with a non zero code
>
> *2019-03-12 14:12:13,477
> INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint -
> Terminating cluster entrypoint process StandaloneJobClusterEntryPoint with
> exit code 1444.*
>
>
> That I think is an issue. A cancelled job is a complete job and thus the
> exit code should be 0 for k8s to mark it complete.
>
>
>
>
>
>
>
>
>
> On Tue, Mar 12, 2019 at 10:18 AM Vijay Bhaskar 
> wrote:
>
>> Yes Vishal. Thats correct.
>>
>> Regards
>> Bhaskar
>>
>> On Tue, Mar 12, 2019 at 7:14 PM Vishal Santoshi <
>> vishal.santo...@gmail.com> wrote:
>>
>>> This really not cool but here you go. This seems to work. Agreed that
>>> this cannot be this painful. The cancel does not exit with an exit code pf
>>> 0 and thus the job has to manually delete. Vijay does this align with what
>>> you have had to do ?
>>>
>>>
>>>- Take a save point . This returns a request id
>>>
>>>curl  --header "Content-Type: application/json" --request POST --data 
>>> '{"target-directory":"hdfs://nn-crunchy:8020/tmp/xyz14","cancel-job":false}'
>>> https://*/jobs//savepoints
>>>
>>>
>>>
>>>- Make sure the save point succeeded
>>>
>>>curl  --request GET   
>>> https:///jobs//savepoints/2c053ce3bea31276aa25e63784629687
>>>
>>>
>>>
>>>- cancel the job
>>>
>>>curl  --request PATCH 
>>> https://***/jobs/?mode=cancel
>>>
>>>
>>>
>>>- Delete the job and deployment
>>>
>>>kubectl delete -f 
>>> manifests/bf2-PRODUCTION/job-cluster-job-deployment.yaml
>>>
>>>kubectl delete -f manifests/bf2-PRODUCTION/task-manager-deployment.yaml
>>>
>>>
>>>
>>>- Edit the job-cluster-job-deployment.yaml. Add/Edit
>>>
>>>args: ["job-cluster",
>>>
>>>   "--fromSavepoint",
>>>
>>>   
>>> "hdfs:///tmp/xyz14/savepoint-00-1d4f71345e22",
>>>   "--job-classname", .
>>>
>>>
>>>
>>>- Restart
>>>
>>>kubectl create -f 
>>> manifests/bf2-PRODUCTION/job-cluster-job-deployment.yaml
>>>
>>>kubectl create -f manifests/bf2-PRODUCTION/task-manager-deployment.yaml
>>>
>>>
>>>
>>>- Make sure from the UI, that it restored from the specific save
>>>point.
>>>
>>>
>>> On Tue, Mar 12, 2019 at 7:26 AM Vijay Bhaskar 
>>> wrote:
>>>
 Yes Its supposed to work.  But unfortunately it was not working. Flink
 community needs to respond to this behavior.

 Regards
 Bhaskar

 On Tue, Mar 12, 2019 at 3:45 PM Vishal Santoshi <
 vishal.santo...@gmail.com> wrote:

> Aah.
> Let me try this out and will get back to you.
> Though I would assume that save point with cancel is a single atomic
> step, rather then a save point *followed*  by a cancellation ( else
> why would that be an option ).
> Thanks again.
>
>
> On Tue, Mar 12, 2019 at 4:50 AM Vijay Bhaskar <
> bhaskar.eba...@gmail.com> wrote:
>
>> Hi Vishal,
>>
>> yarn-cancel doesn't mean to be for yarn cluster. It works for all
>> clusters. Its recommended command.
>>
>> Use the following command to issue save point.
>>  curl  --header "Content-Type: application/json" --request POST
>> --data '{"target-directory":"hdfs://*:8020/tmp/xyz1",
>> "cancel-job":false}'  \ https://
>> .ingress.***/jobs//savepoints
>>
>> Then issue yarn-cancel.
>> After that  follow the process to restore save point
>>
>> Regards
>> Bhaskar
>>
>> On Tue, Mar 12, 2019 at 2:11 PM Vishal Santoshi <
>> vishal.santo...@gmail.com> wrote:
>>
>>> Hello Vijay,
>>>
>>>Thank you for the reply. This though is k8s
>>> deployment ( rather then yarn ) but may be they follow the same 
>>> lifecycle.
>>> I issue a* save point with cancel*  as documented here
>>> https://ci.apache.org/projects/flink/flink-docs-release-1.7/monitoring/rest_api.html#jobs-jobid-savepoints,
>>> a straight up
>>>  curl  --header "Content-Type: application/json" --request POST
>>> --data
>>> '{"target-directory":"hdfs://*:8020/tmp/xyz1","cancel-job":true}'
>>> \ https://
>>> .ingress.***/jobs//savepoints
>>>
>>> I would assume that after taking the save point, the jvm should
>>> exit, after all the k8s deployment is

Re: K8s job cluster and cancel and resume from a save point ?

Thanks Vijay,

This is the larger issue.  The cancellation routine is itself broken.

On cancellation flink does remove the checkpoint counter

*2019-03-12 14:12:13,143
INFO  org.apache.flink.runtime.checkpoint.ZooKeeperCheckpointIDCounter  -
Removing /checkpoint-counter/ from
ZooKeeper *

but exist with a non zero code

*2019-03-12 14:12:13,477
INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint -
Terminating cluster entrypoint process StandaloneJobClusterEntryPoint with
exit code 1444.*


That I think is an issue. A cancelled job is a complete job and thus the
exit code should be 0 for k8s to mark it complete.









On Tue, Mar 12, 2019 at 10:18 AM Vijay Bhaskar 
wrote:

> Yes Vishal. Thats correct.
>
> Regards
> Bhaskar
>
> On Tue, Mar 12, 2019 at 7:14 PM Vishal Santoshi 
> wrote:
>
>> This really not cool but here you go. This seems to work. Agreed that
>> this cannot be this painful. The cancel does not exit with an exit code pf
>> 0 and thus the job has to manually delete. Vijay does this align with what
>> you have had to do ?
>>
>>
>>- Take a save point . This returns a request id
>>
>>curl  --header "Content-Type: application/json" --request POST --data 
>> '{"target-directory":"hdfs://nn-crunchy:8020/tmp/xyz14","cancel-job":false}' 
>>https://*/jobs//savepoints
>>
>>
>>
>>- Make sure the save point succeeded
>>
>>curl  --request GET   
>> https:///jobs//savepoints/2c053ce3bea31276aa25e63784629687
>>
>>
>>
>>- cancel the job
>>
>>curl  --request PATCH 
>> https://***/jobs/?mode=cancel
>>
>>
>>
>>- Delete the job and deployment
>>
>>kubectl delete -f manifests/bf2-PRODUCTION/job-cluster-job-deployment.yaml
>>
>>kubectl delete -f manifests/bf2-PRODUCTION/task-manager-deployment.yaml
>>
>>
>>
>>- Edit the job-cluster-job-deployment.yaml. Add/Edit
>>
>>args: ["job-cluster",
>>
>>   "--fromSavepoint",
>>
>>   
>> "hdfs:///tmp/xyz14/savepoint-00-1d4f71345e22",
>>   "--job-classname", .
>>
>>
>>
>>- Restart
>>
>>kubectl create -f manifests/bf2-PRODUCTION/job-cluster-job-deployment.yaml
>>
>>kubectl create -f manifests/bf2-PRODUCTION/task-manager-deployment.yaml
>>
>>
>>
>>- Make sure from the UI, that it restored from the specific save
>>point.
>>
>>
>> On Tue, Mar 12, 2019 at 7:26 AM Vijay Bhaskar 
>> wrote:
>>
>>> Yes Its supposed to work.  But unfortunately it was not working. Flink
>>> community needs to respond to this behavior.
>>>
>>> Regards
>>> Bhaskar
>>>
>>> On Tue, Mar 12, 2019 at 3:45 PM Vishal Santoshi <
>>> vishal.santo...@gmail.com> wrote:
>>>
 Aah.
 Let me try this out and will get back to you.
 Though I would assume that save point with cancel is a single atomic
 step, rather then a save point *followed*  by a cancellation ( else
 why would that be an option ).
 Thanks again.


 On Tue, Mar 12, 2019 at 4:50 AM Vijay Bhaskar 
 wrote:

> Hi Vishal,
>
> yarn-cancel doesn't mean to be for yarn cluster. It works for all
> clusters. Its recommended command.
>
> Use the following command to issue save point.
>  curl  --header "Content-Type: application/json" --request POST --data
> '{"target-directory":"hdfs://*:8020/tmp/xyz1",
> "cancel-job":false}'  \ https://
> .ingress.***/jobs//savepoints
>
> Then issue yarn-cancel.
> After that  follow the process to restore save point
>
> Regards
> Bhaskar
>
> On Tue, Mar 12, 2019 at 2:11 PM Vishal Santoshi <
> vishal.santo...@gmail.com> wrote:
>
>> Hello Vijay,
>>
>>Thank you for the reply. This though is k8s deployment
>> ( rather then yarn ) but may be they follow the same lifecycle.  I issue 
>> a*
>> save point with cancel*  as documented here
>> https://ci.apache.org/projects/flink/flink-docs-release-1.7/monitoring/rest_api.html#jobs-jobid-savepoints,
>> a straight up
>>  curl  --header "Content-Type: application/json" --request POST
>> --data
>> '{"target-directory":"hdfs://*:8020/tmp/xyz1","cancel-job":true}'
>> \ https://
>> .ingress.***/jobs//savepoints
>>
>> I would assume that after taking the save point, the jvm should exit,
>> after all the k8s deployment is of kind: job and if it is a job cluster
>> then a cancellation should exit the jvm and hence the pod. It does seem 
>> to
>> do some things right. It stops a bunch of stuff ( the JobMaster, the
>> slotPol, zookeeper coordinator etc ) . It also remove the checkpoint
>> counter but does not exit  the job. And after a little bit the job

Re: K8s job cluster and cancel and resume from a save point ?

Yes Vishal. Thats correct.

Regards
Bhaskar

On Tue, Mar 12, 2019 at 7:14 PM Vishal Santoshi 
wrote:

> This really not cool but here you go. This seems to work. Agreed that this
> cannot be this painful. The cancel does not exit with an exit code pf 0 and
> thus the job has to manually delete. Vijay does this align with what you
> have had to do ?
>
>
>- Take a save point . This returns a request id
>
>curl  --header "Content-Type: application/json" --request POST --data 
> '{"target-directory":"hdfs://nn-crunchy:8020/tmp/xyz14","cancel-job":false}'  
>   https://*/jobs//savepoints
>
>
>
>- Make sure the save point succeeded
>
>curl  --request GET   
> https:///jobs//savepoints/2c053ce3bea31276aa25e63784629687
>
>
>
>- cancel the job
>
>curl  --request PATCH 
> https://***/jobs/?mode=cancel
>
>
>
>- Delete the job and deployment
>
>kubectl delete -f manifests/bf2-PRODUCTION/job-cluster-job-deployment.yaml
>
>kubectl delete -f manifests/bf2-PRODUCTION/task-manager-deployment.yaml
>
>
>
>- Edit the job-cluster-job-deployment.yaml. Add/Edit
>
>args: ["job-cluster",
>
>   "--fromSavepoint",
>
>   
> "hdfs:///tmp/xyz14/savepoint-00-1d4f71345e22",
>   "--job-classname", .
>
>
>
>- Restart
>
>kubectl create -f manifests/bf2-PRODUCTION/job-cluster-job-deployment.yaml
>
>kubectl create -f manifests/bf2-PRODUCTION/task-manager-deployment.yaml
>
>
>
>- Make sure from the UI, that it restored from the specific save point.
>
>
> On Tue, Mar 12, 2019 at 7:26 AM Vijay Bhaskar 
> wrote:
>
>> Yes Its supposed to work.  But unfortunately it was not working. Flink
>> community needs to respond to this behavior.
>>
>> Regards
>> Bhaskar
>>
>> On Tue, Mar 12, 2019 at 3:45 PM Vishal Santoshi <
>> vishal.santo...@gmail.com> wrote:
>>
>>> Aah.
>>> Let me try this out and will get back to you.
>>> Though I would assume that save point with cancel is a single atomic
>>> step, rather then a save point *followed*  by a cancellation ( else why
>>> would that be an option ).
>>> Thanks again.
>>>
>>>
>>> On Tue, Mar 12, 2019 at 4:50 AM Vijay Bhaskar 
>>> wrote:
>>>
 Hi Vishal,

 yarn-cancel doesn't mean to be for yarn cluster. It works for all
 clusters. Its recommended command.

 Use the following command to issue save point.
  curl  --header "Content-Type: application/json" --request POST --data
 '{"target-directory":"hdfs://*:8020/tmp/xyz1",
 "cancel-job":false}'  \ https://
 .ingress.***/jobs//savepoints

 Then issue yarn-cancel.
 After that  follow the process to restore save point

 Regards
 Bhaskar

 On Tue, Mar 12, 2019 at 2:11 PM Vishal Santoshi <
 vishal.santo...@gmail.com> wrote:

> Hello Vijay,
>
>Thank you for the reply. This though is k8s deployment
> ( rather then yarn ) but may be they follow the same lifecycle.  I issue 
> a*
> save point with cancel*  as documented here
> https://ci.apache.org/projects/flink/flink-docs-release-1.7/monitoring/rest_api.html#jobs-jobid-savepoints,
> a straight up
>  curl  --header "Content-Type: application/json" --request POST --data
> '{"target-directory":"hdfs://*:8020/tmp/xyz1","cancel-job":true}'
> \ https://
> .ingress.***/jobs//savepoints
>
> I would assume that after taking the save point, the jvm should exit,
> after all the k8s deployment is of kind: job and if it is a job cluster
> then a cancellation should exit the jvm and hence the pod. It does seem to
> do some things right. It stops a bunch of stuff ( the JobMaster, the
> slotPol, zookeeper coordinator etc ) . It also remove the checkpoint
> counter but does not exit  the job. And after a little bit the job is
> restarted which does not make sense and absolutely not the right thing to
> do  ( to me at least ).
>
> Further if I delete the deployment and the job from k8s and restart
> the job and deployment fromSavePoint, it refuses to honor the
> fromSavePoint. I have to delete the zk chroot for it to consider the save
> point.
>
>
> Thus the process of cancelling and resuming from a SP on a k8s job
> cluster deployment  seems to be
>
>- cancel with save point as defined hre
>
> https://ci.apache.org/projects/flink/flink-docs-release-1.7/monitoring/rest_api.html#jobs-jobid-savepoints
>- delete the job manger job  and task manager deployments from k8s
>almost immediately.
>- clear the ZK chroot for the 000.. job  and may be the
>checkpoints directory.
>-

Re: K8s job cluster and cancel and resume from a save point ?

I must add that there has to be more love for k8s flink deployments. IMHO
that is the way to go.  Maintaining a captive/session cluster, if you have
k8s on premise is pretty much a no go  for various reasons.

On Tue, Mar 12, 2019 at 9:44 AM Vishal Santoshi 
wrote:

> This really not cool but here you go. This seems to work. Agreed that this
> cannot be this painful. The cancel does not exit with an exit code pf 0 and
> thus the job has to manually delete. Vijay does this align with what you
> have had to do ?
>
>
>- Take a save point . This returns a request id
>
>curl  --header "Content-Type: application/json" --request POST --data 
> '{"target-directory":"hdfs://nn-crunchy:8020/tmp/xyz14","cancel-job":false}'  
>   https://*/jobs//savepoints
>
>
>
>- Make sure the save point succeeded
>
>curl  --request GET   
> https:///jobs//savepoints/2c053ce3bea31276aa25e63784629687
>
>
>
>- cancel the job
>
>curl  --request PATCH 
> https://***/jobs/?mode=cancel
>
>
>
>- Delete the job and deployment
>
>kubectl delete -f manifests/bf2-PRODUCTION/job-cluster-job-deployment.yaml
>
>kubectl delete -f manifests/bf2-PRODUCTION/task-manager-deployment.yaml
>
>
>
>- Edit the job-cluster-job-deployment.yaml. Add/Edit
>
>args: ["job-cluster",
>
>   "--fromSavepoint",
>
>   
> "hdfs:///tmp/xyz14/savepoint-00-1d4f71345e22",
>   "--job-classname", .
>
>
>
>- Restart
>
>kubectl create -f manifests/bf2-PRODUCTION/job-cluster-job-deployment.yaml
>
>kubectl create -f manifests/bf2-PRODUCTION/task-manager-deployment.yaml
>
>
>
>- Make sure from the UI, that it restored from the specific save point.
>
>
> On Tue, Mar 12, 2019 at 7:26 AM Vijay Bhaskar 
> wrote:
>
>> Yes Its supposed to work.  But unfortunately it was not working. Flink
>> community needs to respond to this behavior.
>>
>> Regards
>> Bhaskar
>>
>> On Tue, Mar 12, 2019 at 3:45 PM Vishal Santoshi <
>> vishal.santo...@gmail.com> wrote:
>>
>>> Aah.
>>> Let me try this out and will get back to you.
>>> Though I would assume that save point with cancel is a single atomic
>>> step, rather then a save point *followed*  by a cancellation ( else why
>>> would that be an option ).
>>> Thanks again.
>>>
>>>
>>> On Tue, Mar 12, 2019 at 4:50 AM Vijay Bhaskar 
>>> wrote:
>>>
 Hi Vishal,

 yarn-cancel doesn't mean to be for yarn cluster. It works for all
 clusters. Its recommended command.

 Use the following command to issue save point.
  curl  --header "Content-Type: application/json" --request POST --data
 '{"target-directory":"hdfs://*:8020/tmp/xyz1",
 "cancel-job":false}'  \ https://
 .ingress.***/jobs//savepoints

 Then issue yarn-cancel.
 After that  follow the process to restore save point

 Regards
 Bhaskar

 On Tue, Mar 12, 2019 at 2:11 PM Vishal Santoshi <
 vishal.santo...@gmail.com> wrote:

> Hello Vijay,
>
>Thank you for the reply. This though is k8s deployment
> ( rather then yarn ) but may be they follow the same lifecycle.  I issue 
> a*
> save point with cancel*  as documented here
> https://ci.apache.org/projects/flink/flink-docs-release-1.7/monitoring/rest_api.html#jobs-jobid-savepoints,
> a straight up
>  curl  --header "Content-Type: application/json" --request POST --data
> '{"target-directory":"hdfs://*:8020/tmp/xyz1","cancel-job":true}'
> \ https://
> .ingress.***/jobs//savepoints
>
> I would assume that after taking the save point, the jvm should exit,
> after all the k8s deployment is of kind: job and if it is a job cluster
> then a cancellation should exit the jvm and hence the pod. It does seem to
> do some things right. It stops a bunch of stuff ( the JobMaster, the
> slotPol, zookeeper coordinator etc ) . It also remove the checkpoint
> counter but does not exit  the job. And after a little bit the job is
> restarted which does not make sense and absolutely not the right thing to
> do  ( to me at least ).
>
> Further if I delete the deployment and the job from k8s and restart
> the job and deployment fromSavePoint, it refuses to honor the
> fromSavePoint. I have to delete the zk chroot for it to consider the save
> point.
>
>
> Thus the process of cancelling and resuming from a SP on a k8s job
> cluster deployment  seems to be
>
>- cancel with save point as defined hre
>
> https://ci.apache.org/projects/flink/flink-docs-release-1.7/monitoring/rest_api.html#jobs-jobid-savepoints
>- delete the job manger job  and task manager deployments

Re: Expressing Flink array aggregation using Table / SQL API

2019-03-12 Thread Piyush Narang

Thanks for getting back Kurt. Yeah this might be an option to try out. I was 
hoping there would be a way to express this directly in the SQL though ☹.

-- Piyush

From: Kurt Young 
Date: Tuesday, March 12, 2019 at 2:25 AM
To: Piyush Narang 
Cc: "user@flink.apache.org" 
Subject: Re: Expressing Flink array aggregation using Table / SQL API

Hi Piyush,

Could you try to add clientId into your aggregate function, and to track the 
map of  inside your new aggregate 
function, and assemble what ever result when emit.
The SQL will looks like:

SELECT userId, some_aggregation(clientId, eventType, `timestamp`, dataField)
FROM my_kafka_stream_table
GROUP BY userId,  HOP(`timestamp`, INTERVAL '1' MINUTE, INTERVAL '1' HOUR)

Kurt

On Tue, Mar 12, 2019 at 11:03 AM Piyush Narang 
mailto:p.nar...@criteo.com>> wrote:
Hi folks,

I’m getting started with Flink and trying to figure out how to express 
aggregating some rows into an array to finally sink data into an 
AppendStreamTableSink.
My data looks something like this:
userId, clientId, eventType, timestamp, dataField

I need to compute some custom aggregations using a UDAF while grouping by 
userId, clientId over a sliding window (10 mins, triggered every 1 min). My 
first attempt is:
SELECT userId, clientId, my_aggregation(eventType, `timestamp`, dataField) as 
custom_aggregated
FROM my_kafka_stream_table
GROUP BY userId, clientId, HOP(`timestamp`, INTERVAL '1' MINUTE, INTERVAL '1' 
HOUR)

This query works as I expect it to. In every time window I end up with inserts 
for unique userId + clientId combinations. What I want to do though, is 
generate a single row per userId in each time window and this is what I’m 
struggling with expressing along with the restriction that I want to sink this 
to an AppendStreamTableSink. I was hoping to do something like this:

SELECT userId, COLLECT(client_custom_aggregated)
FROM
(
  SELECT userId, MAP[clientId, my_aggregation(eventType, `timestamp`, 
dataField) as custom_aggregated] as client_custom_aggregated
  FROM my_kafka_stream_table
  GROUP BY userId, clientId, HOP(`timestamp`, INTERVAL '1' MINUTE, INTERVAL '1' 
HOUR)
) GROUP BY userId

Unfortunately when I try this (and a few other variants), I run into the error, 
“AppendStreamTableSink requires that Table has only insert changes”. Does 
anyone know if there’s a way for me to compute my collect aggregation to 
produce one row per userId for a given time window?

Thanks,

-- Piyush

Re: Backoff strategies for async IO functions?

2019-03-12 Thread Konstantin Knauf

Hi Shuyi,

I am not sure. You could handle retries in the user code within
org.apache.flink.streaming.api.functions.async.AsyncFunction#asyncInvoke
without using a DLQ as described in my original answer to William.  On the
other hand, I agree that it could easier for the user and it is indeed a
common scenario.

Two follow up questions come to mind:

   - When a Flink job fails and restarts, would you expect the "retry
   counter" to be reset or to continue where it left off?
   - With AsyncStream.orderedWait() when would you expect a record to be
   skipped? After the final timeout, after the first timeout?

Would you like to create a JIRA ticket [1] for this improvement with
answers to the questions above and we can continue to discuss it there.

Best,

Konstantin

[1]
https://issues.apache.org/jira/projects/FLINK/issues/FLINK-11835?filter=allopenissues


On Sun, Mar 10, 2019 at 9:20 AM Shuyi Chen  wrote:

> Hi Konstantin,
>
> (cc Till since he owns the code)
>
> For async-IO, IO failure and retry is a common & expected pattern. In most
> of the use cases, users will need to deal with IO failure and retry.
> Therefore, I think it's better to address the problem in Flink rather than
> user implementing its custom logic in user code for a better dev
> experience. We do have similar problem in many of our use cases. To enable
> backoff and retry, we need to put the failed message to a DLQ (another
> Kafka topic) and re-ingest the message from the DLQ topic to retry, which
> is manual/cumbersome and require setting up extra Kafka topic.
>
> Can we add multiple strategies to handle async IO failure in the
> AsyncWaitOperator? I propose the following strategies:
>
>
>- FAIL_OPERATOR (default & current behavior)
>- FIX_INTERVAL_RETRY (retry with configurable fixed interval up to N
>times)
>- EXP_BACKOFF_RETRY (retry with exponential backoff up to N times)
>
> What do you guys think? Thanks a lot.
>
> Shuyi
>
> On Fri, Mar 8, 2019 at 3:17 PM Konstantin Knauf 
> wrote:
>
>> Hi William,
>>
>> the AsyncOperator does not have such a setting. It is "merely" a wrapper
>> around an asynchronous call, which provides integration with Flink's state
>> & time management.
>>
>> I think, the way to go would be to do the exponential back-off in the
>> user code and set the timeout of the AsyncOperator to the sum of the
>> timeouts in the user code (e.g. 2s + 4s + 8s + 16s).
>>
>> Cheers,
>>
>> Konstantin
>>
>>
>> On Thu, Mar 7, 2019 at 5:20 PM William Saar  wrote:
>>
>>> Hi,
>>> Is there a way to specify an exponential backoff strategy for when async
>>> function calls fail?
>>>
>>> I have an async function that does web requests to a rate-limited API.
>>> Can you handle that with settings on the async function call?
>>>
>>> Thanks,
>>> William
>>>
>>>
>>>
>>
>> --
>>
>> Konstantin Knauf | Solutions Architect
>>
>> +49 160 91394525
>>
>> 
>>
>> Follow us @VervericaData
>>
>> --
>>
>> Join Flink Forward  - The Apache Flink
>> Conference
>>
>> Stream Processing | Event Driven | Real Time
>>
>> --
>>
>> Data Artisans GmbH | Invalidenstrasse 115, 10115 Berlin, Germany
>>
>> --
>> Data Artisans GmbH
>> Registered at Amtsgericht Charlottenburg: HRB 158244 B
>> Managing Directors: Dr. Kostas Tzoumas, Dr. Stephan Ewen
>>
>

-- 

Konstantin Knauf | Solutions Architect

+49 160 91394525



Follow us @VervericaData

--

Join Flink Forward  - The Apache Flink
Conference

Stream Processing | Event Driven | Real Time

--

Data Artisans GmbH | Invalidenstrasse 115, 10115 Berlin, Germany

--
Data Artisans GmbH
Registered at Amtsgericht Charlottenburg: HRB 158244 B
Managing Directors: Dr. Kostas Tzoumas, Dr. Stephan Ewen

Re: What does "Continuous incremental cleanup" mean in Flink 1.8 release notes

2019-03-12 Thread Konstantin Knauf

Hi Tony,

yes, when taking a savepoint  the same strategy as the during a
non-incremental checkpoint is used.

Best,

Konstantin

On Mon, Mar 11, 2019 at 2:29 AM Tony Wei  wrote:

> Hi Konstantin,
>
> That is really helpful. Thanks.
>
> Another follow-up question: The document said "Cleanup in full
> snapshot" is not applicable for
> the incremental checkpointing in the RocksDB state backend. However, when
> user manually
> trigger a savepoint and restart job from it, the expired states should be
> clean up as well based
> on Flink 1.6's implementation. Am I right?
>
> Best,
> Tony Wei
>
> Konstantin Knauf  於 2019年3月9日 週六 上午7:00寫道：
>
>> Hi Tony,
>>
>> before Flink 1.8 expired state is only cleaned up, when you try to access
>> it after expiration, i.e. when user code tries to access the expired state,
>> the state value is cleaned and "null" is returned. There was also already
>> the option to clean up expired state during full snapshots (
>> https://github.com/apache/flink/pull/6460). With Flink 1.8 expired state
>> is cleaned up continuously in the background regardless of checkpointing or
>> any attempt to access it after expiration.
>>
>> As a reference the linked JIRA tickets should be a good starting point.
>>
>> Hope this helps.
>>
>> Konstantin
>>
>>
>>
>>
>> On Fri, Mar 8, 2019 at 10:45 AM Tony Wei  wrote:
>>
>>> Hi everyone,
>>>
>>> I read the Flink 1.8 release notes about state [1], and it said
>>>
>>> *Continuous incremental cleanup of old Keyed State with TTL*
 We introduced TTL (time-to-live) for Keyed state in Flink 1.6 (
 FLINK-9510 ). This
 feature allowed to clean up and make inaccessible keyed state entries when
 accessing them. In addition state would now also being cleaned up when
 writing a savepoint/checkpoint.
 Flink 1.8 introduces continous cleanup of old entries for both the
 RocksDB state backend (FLINK-10471
 ) and the heap
 state backend (FLINK-10473
 ). This means that
 old entries (according to the ttl setting) are continously being cleanup 
 up.
>>>
>>>
>>> I'm not familiar with TTL's implementation in Flink 1.6 and what new
>>> features introduced in Flink
>>> 1.8. I don't understand what difference between these two release
>>> version after reading the
>>> release notes. Did they change the outcome of TTL feature, or provide
>>> new TTL features, or just
>>> change the behavior of executing TTL mechanism.
>>>
>>> Could you give me more references to learn about it? A simple example
>>> to illustrate it is more
>>> appreciated. Thank you.
>>>
>>> Best,
>>> Tony Wei
>>>
>>> [1]
>>> https://ci.apache.org/projects/flink/flink-docs-master/release-notes/flink-1.8.html#state
>>>
>>
>>
>> --
>>
>> Konstantin Knauf | Solutions Architect
>>
>> +49 160 91394525
>>
>> 
>>
>> Follow us @VervericaData
>>
>> --
>>
>> Join Flink Forward  - The Apache Flink
>> Conference
>>
>> Stream Processing | Event Driven | Real Time
>>
>> --
>>
>> Data Artisans GmbH | Invalidenstrasse 115, 10115 Berlin, Germany
>>
>> --
>> Data Artisans GmbH
>> Registered at Amtsgericht Charlottenburg: HRB 158244 B
>> Managing Directors: Dr. Kostas Tzoumas, Dr. Stephan Ewen
>>
>

-- 

Konstantin Knauf | Solutions Architect

+49 160 91394525



Follow us @VervericaData

--

Join Flink Forward  - The Apache Flink
Conference

Stream Processing | Event Driven | Real Time

--

Data Artisans GmbH | Invalidenstrasse 115, 10115 Berlin, Germany

--
Data Artisans GmbH
Registered at Amtsgericht Charlottenburg: HRB 158244 B
Managing Directors: Dr. Kostas Tzoumas, Dr. Stephan Ewen

Re: K8s job cluster and cancel and resume from a save point ?

Yes Its supposed to work.  But unfortunately it was not working. Flink
community needs to respond to this behavior.

Regards
Bhaskar

On Tue, Mar 12, 2019 at 3:45 PM Vishal Santoshi 
wrote:

> Aah.
> Let me try this out and will get back to you.
> Though I would assume that save point with cancel is a single atomic step,
> rather then a save point *followed*  by a cancellation ( else why would
> that be an option ).
> Thanks again.
>
>
> On Tue, Mar 12, 2019 at 4:50 AM Vijay Bhaskar 
> wrote:
>
>> Hi Vishal,
>>
>> yarn-cancel doesn't mean to be for yarn cluster. It works for all
>> clusters. Its recommended command.
>>
>> Use the following command to issue save point.
>>  curl  --header "Content-Type: application/json" --request POST --data
>> '{"target-directory":"hdfs://*:8020/tmp/xyz1","cancel-job":false}'
>> \ https://
>> .ingress.***/jobs//savepoints
>>
>> Then issue yarn-cancel.
>> After that  follow the process to restore save point
>>
>> Regards
>> Bhaskar
>>
>> On Tue, Mar 12, 2019 at 2:11 PM Vishal Santoshi <
>> vishal.santo...@gmail.com> wrote:
>>
>>> Hello Vijay,
>>>
>>>Thank you for the reply. This though is k8s deployment (
>>> rather then yarn ) but may be they follow the same lifecycle.  I issue a*
>>> save point with cancel*  as documented here
>>> https://ci.apache.org/projects/flink/flink-docs-release-1.7/monitoring/rest_api.html#jobs-jobid-savepoints,
>>> a straight up
>>>  curl  --header "Content-Type: application/json" --request POST --data
>>> '{"target-directory":"hdfs://*:8020/tmp/xyz1","cancel-job":true}'
>>> \ https://
>>> .ingress.***/jobs//savepoints
>>>
>>> I would assume that after taking the save point, the jvm should exit,
>>> after all the k8s deployment is of kind: job and if it is a job cluster
>>> then a cancellation should exit the jvm and hence the pod. It does seem to
>>> do some things right. It stops a bunch of stuff ( the JobMaster, the
>>> slotPol, zookeeper coordinator etc ) . It also remove the checkpoint
>>> counter but does not exit  the job. And after a little bit the job is
>>> restarted which does not make sense and absolutely not the right thing to
>>> do  ( to me at least ).
>>>
>>> Further if I delete the deployment and the job from k8s and restart the
>>> job and deployment fromSavePoint, it refuses to honor the fromSavePoint. I
>>> have to delete the zk chroot for it to consider the save point.
>>>
>>>
>>> Thus the process of cancelling and resuming from a SP on a k8s job
>>> cluster deployment  seems to be
>>>
>>>- cancel with save point as defined hre
>>>
>>> https://ci.apache.org/projects/flink/flink-docs-release-1.7/monitoring/rest_api.html#jobs-jobid-savepoints
>>>- delete the job manger job  and task manager deployments from k8s
>>>almost immediately.
>>>- clear the ZK chroot for the 000.. job  and may be the
>>>checkpoints directory.
>>>- resumeFromCheckPoint
>>>
>>> If some body can say that this indeed is the process ?
>>>
>>>
>>>
>>>  Logs are attached.
>>>
>>>
>>>
>>> 2019-03-12 08:10:43,871 INFO
>>> org.apache.flink.runtime.jobmaster.JobMaster  -
>>> Savepoint stored in
>>> hdfs://*:8020/tmp/xyz3/savepoint-00-6d5bdc9b53ae. Now
>>> cancelling .
>>>
>>> 2019-03-12 08:10:43,871 INFO
>>> org.apache.flink.runtime.executiongraph.ExecutionGraph- Job
>>> anomaly_echo () switched from state RUNNING
>>> to CANCELLING.
>>>
>>> 2019-03-12 08:10:44,227 INFO  
>>> org.apache.flink.runtime.checkpoint.CheckpointCoordinator
>>> - Completed checkpoint 10 for job 
>>> (7238 bytes in 311 ms).
>>>
>>> 2019-03-12 08:10:44,232 INFO
>>> org.apache.flink.runtime.executiongraph.ExecutionGraph- Source:
>>> Barnacle Anomalies Kafka topic -> Map -> Sink: Logging Sink (1/1)
>>> (e2d02ca40a9a6c96a0c1882f5a2e4dd6) switched from RUNNING to CANCELING.
>>>
>>> 2019-03-12 08:10:44,274 INFO
>>> org.apache.flink.runtime.executiongraph.ExecutionGraph- Source:
>>> Barnacle Anomalies Kafka topic -> Map -> Sink: Logging Sink (1/1)
>>> (e2d02ca40a9a6c96a0c1882f5a2e4dd6) switched from CANCELING to CANCELED.
>>>
>>> 2019-03-12 08:10:44,276 INFO
>>> org.apache.flink.runtime.executiongraph.ExecutionGraph- Job
>>> anomaly_echo () switched from state
>>> CANCELLING to CANCELED.
>>>
>>> 2019-03-12 08:10:44,276 INFO  
>>> org.apache.flink.runtime.checkpoint.CheckpointCoordinator
>>> - Stopping checkpoint coordinator for job
>>> .
>>>
>>> 2019-03-12 08:10:44,277 INFO
>>> org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore  -
>>> Shutting down
>>>
>>> 2019-03-12 08:10:44,323 INFO  
>>> org.apache.flink.runtime.checkpoint.CompletedCheckpoint
>>>   - Checkpoint with ID 8 at
>>>

Re: Random forest - Flink ML

2019-03-12 Thread Benoît Paris

There has been some developments at  Apache SAMOA
   for a forest of decision trees.

This is not regular Random Forest, but a form of trees that can be
incrementally learned fast. If I recall correctly they also have adaptive
algorithms as well. Here are some resources:

*  VHT: Vertical Hoeffding Tree   

*  Apache SAMOA   

Now I don't know the status of the project nor have I tried them, nor have I
ever tried SAMOA; but this is something that could fit your needs.



--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: K8s job cluster and cancel and resume from a save point ?

Aah.
Let me try this out and will get back to you.
Though I would assume that save point with cancel is a single atomic step,
rather then a save point *followed*  by a cancellation ( else why would
that be an option ).
Thanks again.


On Tue, Mar 12, 2019 at 4:50 AM Vijay Bhaskar 
wrote:

> Hi Vishal,
>
> yarn-cancel doesn't mean to be for yarn cluster. It works for all
> clusters. Its recommended command.
>
> Use the following command to issue save point.
>  curl  --header "Content-Type: application/json" --request POST --data
> '{"target-directory":"hdfs://*:8020/tmp/xyz1","cancel-job":false}'
> \ https://
> .ingress.***/jobs//savepoints
>
> Then issue yarn-cancel.
> After that  follow the process to restore save point
>
> Regards
> Bhaskar
>
> On Tue, Mar 12, 2019 at 2:11 PM Vishal Santoshi 
> wrote:
>
>> Hello Vijay,
>>
>>Thank you for the reply. This though is k8s deployment (
>> rather then yarn ) but may be they follow the same lifecycle.  I issue a*
>> save point with cancel*  as documented here
>> https://ci.apache.org/projects/flink/flink-docs-release-1.7/monitoring/rest_api.html#jobs-jobid-savepoints,
>> a straight up
>>  curl  --header "Content-Type: application/json" --request POST --data
>> '{"target-directory":"hdfs://*:8020/tmp/xyz1","cancel-job":true}'
>> \ https://
>> .ingress.***/jobs//savepoints
>>
>> I would assume that after taking the save point, the jvm should exit,
>> after all the k8s deployment is of kind: job and if it is a job cluster
>> then a cancellation should exit the jvm and hence the pod. It does seem to
>> do some things right. It stops a bunch of stuff ( the JobMaster, the
>> slotPol, zookeeper coordinator etc ) . It also remove the checkpoint
>> counter but does not exit  the job. And after a little bit the job is
>> restarted which does not make sense and absolutely not the right thing to
>> do  ( to me at least ).
>>
>> Further if I delete the deployment and the job from k8s and restart the
>> job and deployment fromSavePoint, it refuses to honor the fromSavePoint. I
>> have to delete the zk chroot for it to consider the save point.
>>
>>
>> Thus the process of cancelling and resuming from a SP on a k8s job
>> cluster deployment  seems to be
>>
>>- cancel with save point as defined hre
>>
>> https://ci.apache.org/projects/flink/flink-docs-release-1.7/monitoring/rest_api.html#jobs-jobid-savepoints
>>- delete the job manger job  and task manager deployments from k8s
>>almost immediately.
>>- clear the ZK chroot for the 000.. job  and may be the
>>checkpoints directory.
>>- resumeFromCheckPoint
>>
>> If some body can say that this indeed is the process ?
>>
>>
>>
>>  Logs are attached.
>>
>>
>>
>> 2019-03-12 08:10:43,871 INFO
>> org.apache.flink.runtime.jobmaster.JobMaster  -
>> Savepoint stored in
>> hdfs://*:8020/tmp/xyz3/savepoint-00-6d5bdc9b53ae. Now
>> cancelling .
>>
>> 2019-03-12 08:10:43,871 INFO
>> org.apache.flink.runtime.executiongraph.ExecutionGraph- Job
>> anomaly_echo () switched from state RUNNING
>> to CANCELLING.
>>
>> 2019-03-12 08:10:44,227 INFO  
>> org.apache.flink.runtime.checkpoint.CheckpointCoordinator
>> - Completed checkpoint 10 for job 
>> (7238 bytes in 311 ms).
>>
>> 2019-03-12 08:10:44,232 INFO
>> org.apache.flink.runtime.executiongraph.ExecutionGraph- Source:
>> Barnacle Anomalies Kafka topic -> Map -> Sink: Logging Sink (1/1)
>> (e2d02ca40a9a6c96a0c1882f5a2e4dd6) switched from RUNNING to CANCELING.
>>
>> 2019-03-12 08:10:44,274 INFO
>> org.apache.flink.runtime.executiongraph.ExecutionGraph- Source:
>> Barnacle Anomalies Kafka topic -> Map -> Sink: Logging Sink (1/1)
>> (e2d02ca40a9a6c96a0c1882f5a2e4dd6) switched from CANCELING to CANCELED.
>>
>> 2019-03-12 08:10:44,276 INFO
>> org.apache.flink.runtime.executiongraph.ExecutionGraph- Job
>> anomaly_echo () switched from state
>> CANCELLING to CANCELED.
>>
>> 2019-03-12 08:10:44,276 INFO  
>> org.apache.flink.runtime.checkpoint.CheckpointCoordinator
>> - Stopping checkpoint coordinator for job
>> .
>>
>> 2019-03-12 08:10:44,277 INFO
>> org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore  -
>> Shutting down
>>
>> 2019-03-12 08:10:44,323 INFO  
>> org.apache.flink.runtime.checkpoint.CompletedCheckpoint
>>   - Checkpoint with ID 8 at
>> 'hdfs://nn-crunchy:8020/tmp/xyz2/savepoint-00-859e626cbb00' not
>> discarded.
>>
>> 2019-03-12 08:10:44,437 INFO
>> org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore  -
>> Removing
>> /k8s_anomalyecho/k8s_anomalyecho/checkpoints/
>> from ZooKeeper
>>
>> 2019-03-12 08:10:44,437 INFO  
>>

Re: Random forest - Flink ML

2019-03-12 Thread Avi Levi

Thanks Flavio,
I will definitely check it out. But from a quick glance , it seems that it
is missing implementation of "random forest" which is something that we are
looking for .
If anyone can recommend/suggest/share that will be greatly appreciated.

Best Regards
Avi

On Mon, Mar 11, 2019 at 10:01 PM Flavio Pompermaier 
wrote:

> I know there's an outgoing promising effort on improving Flink ML in the
> Streamline project [1] but I don't know why it's not very
> considered/advertised.
>
> Best,
> Flavio
>
> [1] https://h2020-streamline-project.eu/apache-flink/
>
> Il Lun 11 Mar 2019, 15:40 Avi Levi  ha scritto:
>
>> HI ,
>>  According to Tills comment
>> 
>> I understand that flink-ml is going to be ditched. What will be the
>> alternative ?
>> Looking for a "random forest" method that we can add to our pipeline
>> (scala). any suggestions?
>>
>> Thanks
>> Avi
>>
>>
>>
>>

Re: Set partition number of Flink DataSet

2019-03-12 Thread qi luo

Hi Ken,

Thanks for your reply. I may not make myself clear: our problem is not about 
reading but rather writing. 

We need to write to N files based on key partitioning. We have to use 
setParallelism() to set the output partition/file number, but when the 
partition number is too large (~100K), the parallelism would be too high. Is 
there any other way to achieve this?

Thanks,
Qi

> On Mar 11, 2019, at 11:22 PM, Ken Krugler  wrote:
> 
> Hi Qi,
> 
> I’m guessing you’re calling createInput() for each input file.
> 
> If so, then instead you want to do something like:
> 
>   Job job = Job.getInstance();
> 
>   for each file…
>   FileInputFormat.addInputPath(job, new 
> org.apache.hadoop.fs.Path(file path));
> 
>   env.createInput(HadoopInputs.createHadoopInput(…, job)
> 
> Flink/Hadoop will take care of parallelizing the reads from the files, given 
> the parallelism that you’re specifying.
> 
> — Ken
> 
> 
>> On Mar 11, 2019, at 5:42 AM, qi luo > > wrote:
>> 
>> Hi,
>> 
>> We’re trying to distribute batch input data to (N) HDFS files partitioning 
>> by hash using DataSet API. What I’m doing is like:
>> 
>> env.createInput(…)
>>   .partitionByHash(0)
>>   .setParallelism(N)
>>   .output(…)
>> 
>> This works well for small number of files. But when we need to distribute to 
>> large number of files (say 100K), the parallelism becomes too large and we 
>> could not afford that many TMs.
>> 
>> In spark we can write something like ‘rdd.partitionBy(N)’ and control the 
>> parallelism separately (using dynamic allocation). Is there anything similar 
>> in Flink or other way we can achieve similar result? Thank you!
>> 
>> Qi
> 
> --
> Ken Krugler
> +1 530-210-6378
> http://www.scaleunlimited.com 
> Custom big data solutions & training
> Flink, Solr, Hadoop, Cascading & Cassandra
>

Re: K8s job cluster and cancel and resume from a save point ?

Hi Vishal,

yarn-cancel doesn't mean to be for yarn cluster. It works for all clusters.
Its recommended command.

Use the following command to issue save point.
 curl  --header "Content-Type: application/json" --request POST --data
'{"target-directory":"hdfs://*:8020/tmp/xyz1","cancel-job":false}'
\ https://
.ingress.***/jobs//savepoints

Then issue yarn-cancel.
After that  follow the process to restore save point

Regards
Bhaskar

On Tue, Mar 12, 2019 at 2:11 PM Vishal Santoshi 
wrote:

> Hello Vijay,
>
>Thank you for the reply. This though is k8s deployment (
> rather then yarn ) but may be they follow the same lifecycle.  I issue a*
> save point with cancel*  as documented here
> https://ci.apache.org/projects/flink/flink-docs-release-1.7/monitoring/rest_api.html#jobs-jobid-savepoints,
> a straight up
>  curl  --header "Content-Type: application/json" --request POST --data
> '{"target-directory":"hdfs://*:8020/tmp/xyz1","cancel-job":true}'
> \ https://
> .ingress.***/jobs//savepoints
>
> I would assume that after taking the save point, the jvm should exit,
> after all the k8s deployment is of kind: job and if it is a job cluster
> then a cancellation should exit the jvm and hence the pod. It does seem to
> do some things right. It stops a bunch of stuff ( the JobMaster, the
> slotPol, zookeeper coordinator etc ) . It also remove the checkpoint
> counter but does not exit  the job. And after a little bit the job is
> restarted which does not make sense and absolutely not the right thing to
> do  ( to me at least ).
>
> Further if I delete the deployment and the job from k8s and restart the
> job and deployment fromSavePoint, it refuses to honor the fromSavePoint. I
> have to delete the zk chroot for it to consider the save point.
>
>
> Thus the process of cancelling and resuming from a SP on a k8s job cluster
> deployment  seems to be
>
>- cancel with save point as defined hre
>
> https://ci.apache.org/projects/flink/flink-docs-release-1.7/monitoring/rest_api.html#jobs-jobid-savepoints
>- delete the job manger job  and task manager deployments from k8s
>almost immediately.
>- clear the ZK chroot for the 000.. job  and may be the
>checkpoints directory.
>- resumeFromCheckPoint
>
> If some body can say that this indeed is the process ?
>
>
>
>  Logs are attached.
>
>
>
> 2019-03-12 08:10:43,871 INFO  org.apache.flink.runtime.jobmaster.JobMaster
> - Savepoint stored in
> hdfs://*:8020/tmp/xyz3/savepoint-00-6d5bdc9b53ae. Now
> cancelling .
>
> 2019-03-12 08:10:43,871 INFO
> org.apache.flink.runtime.executiongraph.ExecutionGraph- Job
> anomaly_echo () switched from state RUNNING
> to CANCELLING.
>
> 2019-03-12 08:10:44,227 INFO  
> org.apache.flink.runtime.checkpoint.CheckpointCoordinator
> - Completed checkpoint 10 for job 
> (7238 bytes in 311 ms).
>
> 2019-03-12 08:10:44,232 INFO
> org.apache.flink.runtime.executiongraph.ExecutionGraph- Source:
> Barnacle Anomalies Kafka topic -> Map -> Sink: Logging Sink (1/1)
> (e2d02ca40a9a6c96a0c1882f5a2e4dd6) switched from RUNNING to CANCELING.
>
> 2019-03-12 08:10:44,274 INFO
> org.apache.flink.runtime.executiongraph.ExecutionGraph- Source:
> Barnacle Anomalies Kafka topic -> Map -> Sink: Logging Sink (1/1)
> (e2d02ca40a9a6c96a0c1882f5a2e4dd6) switched from CANCELING to CANCELED.
>
> 2019-03-12 08:10:44,276 INFO
> org.apache.flink.runtime.executiongraph.ExecutionGraph- Job
> anomaly_echo () switched from state
> CANCELLING to CANCELED.
>
> 2019-03-12 08:10:44,276 INFO  
> org.apache.flink.runtime.checkpoint.CheckpointCoordinator
> - Stopping checkpoint coordinator for job
> .
>
> 2019-03-12 08:10:44,277 INFO
> org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore  -
> Shutting down
>
> 2019-03-12 08:10:44,323 INFO  
> org.apache.flink.runtime.checkpoint.CompletedCheckpoint
>   - Checkpoint with ID 8 at
> 'hdfs://nn-crunchy:8020/tmp/xyz2/savepoint-00-859e626cbb00' not
> discarded.
>
> 2019-03-12 08:10:44,437 INFO
> org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore  -
> Removing
> /k8s_anomalyecho/k8s_anomalyecho/checkpoints/
> from ZooKeeper
>
> 2019-03-12 08:10:44,437 INFO  
> org.apache.flink.runtime.checkpoint.CompletedCheckpoint
>   - Checkpoint with ID 10 at
> 'hdfs://*:8020/tmp/xyz3/savepoint-00-6d5bdc9b53ae' not
> discarded.
>
> 2019-03-12 08:10:44,447 INFO
> org.apache.flink.runtime.checkpoint.ZooKeeperCheckpointIDCounter  -
> Shutting down.
>
> 2019-03-12 08:10:44,447 INFO
> org.apache.flink.runtime.checkpoint.ZooKeeperCheckpointIDCounter  -
> Removing

Re: K8s job cluster and cancel and resume from a save point ?

Hello Vijay,

Thank you for the reply. This though is k8s deployment (
rather then yarn ) but may be they follow the same lifecycle. I issue a*
save point with cancel* as documented here
https://ci.apache.org/projects/flink/flink-docs-release-1.7/monitoring/rest_api.html#jobs-jobid-savepoints,
a straight up
curl --header "Content-Type: application/json" --request POST --data
'{"target-directory":"hdfs://*:8020/tmp/xyz1","cancel-job":true}'
\ https://
.ingress.***/jobs//savepoints

I would assume that after taking the save point, the jvm should exit, after
all the k8s deployment is of kind: job and if it is a job cluster then a
cancellation should exit the jvm and hence the pod. It does seem to do some
things right. It stops a bunch of stuff ( the JobMaster, the slotPol,
zookeeper coordinator etc ) . It also remove the checkpoint counter but
does not exit the job. And after a little bit the job is restarted which
does not make sense and absolutely not the right thing to do ( to me at
least ).

Further if I delete the deployment and the job from k8s and restart the job
and deployment fromSavePoint, it refuses to honor the fromSavePoint. I have
to delete the zk chroot for it to consider the save point.

Thus the process of cancelling and resuming from a SP on a k8s job cluster
deployment seems to be

- cancel with save point as defined hre

https://ci.apache.org/projects/flink/flink-docs-release-1.7/monitoring/rest_api.html#jobs-jobid-savepoints
- delete the job manger job and task manager deployments from k8s
almost immediately.
- clear the ZK chroot for the 000.. job and may be the
checkpoints directory.
- resumeFromCheckPoint

If some body can say that this indeed is the process ?

Logs are attached.

2019-03-12 08:10:43,871 INFO org.apache.flink.runtime.jobmaster.JobMaster
- Savepoint stored in
hdfs://*:8020/tmp/xyz3/savepoint-00-6d5bdc9b53ae. Now
cancelling .

2019-03-12 08:10:43,871 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph- Job
anomaly_echo () switched from state RUNNING
to CANCELLING.

2019-03-12 08:10:44,227 INFO
org.apache.flink.runtime.checkpoint.CheckpointCoordinator
- Completed checkpoint 10 for job
(7238 bytes in 311 ms).

2019-03-12 08:10:44,232 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph- Source:
Barnacle Anomalies Kafka topic -> Map -> Sink: Logging Sink (1/1)
(e2d02ca40a9a6c96a0c1882f5a2e4dd6) switched from RUNNING to CANCELING.

2019-03-12 08:10:44,274 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph- Source:
Barnacle Anomalies Kafka topic -> Map -> Sink: Logging Sink (1/1)
(e2d02ca40a9a6c96a0c1882f5a2e4dd6) switched from CANCELING to CANCELED.

2019-03-12 08:10:44,276 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph- Job
anomaly_echo () switched from state
CANCELLING to CANCELED.

2019-03-12 08:10:44,276 INFO
org.apache.flink.runtime.checkpoint.CheckpointCoordinator
- Stopping checkpoint coordinator for job
.

2019-03-12 08:10:44,277 INFO
org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore -
Shutting down

2019-03-12 08:10:44,323 INFO
org.apache.flink.runtime.checkpoint.CompletedCheckpoint
- Checkpoint with ID 8 at
'hdfs://nn-crunchy:8020/tmp/xyz2/savepoint-00-859e626cbb00' not
discarded.

2019-03-12 08:10:44,437 INFO
org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore -
Removing
/k8s_anomalyecho/k8s_anomalyecho/checkpoints/
from ZooKeeper

2019-03-12 08:10:44,437 INFO
org.apache.flink.runtime.checkpoint.CompletedCheckpoint
- Checkpoint with ID 10 at
'hdfs://*:8020/tmp/xyz3/savepoint-00-6d5bdc9b53ae' not
discarded.

2019-03-12 08:10:44,447 INFO
org.apache.flink.runtime.checkpoint.ZooKeeperCheckpointIDCounter -
Shutting down.

2019-03-12 08:10:44,447 INFO
org.apache.flink.runtime.checkpoint.ZooKeeperCheckpointIDCounter -
Removing /checkpoint-counter/ from ZooKeeper

2019-03-12 08:10:44,463 INFO
org.apache.flink.runtime.dispatcher.MiniDispatcher- Job
reached globally terminal state CANCELED.

2019-03-12 08:10:44,467 INFO org.apache.flink.runtime.jobmaster.JobMaster
- Stopping the JobMaster for job
anomaly_echo().

2019-03-12 08:10:44,468 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint
- Shutting StandaloneJobClusterEntryPoint down with application
status CANCELED. Diagnostics null.

2019-03-12 08:10:44,468 INFO
org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint - Shutting
down rest endpoint.

2019-03-12 08:10:44,473 INFO

Could not resolve ResourceManager address on Flink 1.7.1

2019-03-12 Thread Le Xu

Hello:

I am trying to set up a standalone flink cluster (1.7.1) and I'm getting a
very similar error as the user reported in
this

thread. However, I believe the root cause should be different -- as I tried
start job manager using both start-cluster.sh and jobmanager.sh but both of
them failed with the same error.
The error I got is on task manager (flink-worker1) is similar to the
following:

6:6123/user/resourcemanager, retrying in 1 ms: Could not connect to rpc
endpoint under address akka.tcp://flink@10.0.0.6:6123/user/resourcemanager..
2019-03-12 07:39:42,884 INFO
org.apache.flink.runtime.taskexecutor.TaskExecutor- Could not
resolve ResourceManager address akka.tcp://
flink@10.0.0.6:6123/user/resourcemanager, retrying in 1 ms: Could not
connect to rpc endpoint under address akka.tcp://
flink@10.0.0.6:6123/user/resourcemanager..
2019-03-12 07:39:52,901 INFO
org.apache.flink.runtime.taskexecutor.TaskExecutor- Could not
resolve ResourceManager address akka.tcp://
flink@10.0.0.6:6123/user/resourcemanager, retrying in 1 ms: Could not
connect to rpc endpoint under address akka.tcp://
flink@10.0.0.6:6123/user/resourcemanager..
2019-03-12 07:40:02,925 INFO
org.apache.flink.runtime.taskexecutor.TaskExecutor- Could not
resolve ResourceManager address akka.tcp://
flink@10.0.0.6:6123/user/resourcemanager, retrying in 1 ms: Could not
connect to rpc endpoint under address akka.tcp://
flink@10.0.0.6:6123/user/resourcemanager..
2019-03-12 07:40:12,939 INFO
org.apache.flink.runtime.taskexecutor.TaskExecutor- Could not
resolve ResourceManager address akka.tcp://
flink@10.0.0.6:6123/user/resourcemanager, retrying in 1 ms: Could not
connect to rpc endpoint under address akka.tcp://
flink@10.0.0.6:6123/user/resourcemanager..
2019-03-12 07:40:22,963 INFO
org.apache.flink.runtime.taskexecutor.TaskExecutor- Could not
resolve ResourceManager address akka.tcp://
flink@10.0.0.6:6123/user/resourcemanager, retrying in 1 ms: Could not
connect to rpc endpoint under address akka.tcp://
flink@10.0.0.6:6123/user/resourcemanager..
2019-03-12 07:40:32,978 INFO
org.apache.flink.runtime.taskexecutor.TaskExecutor- Could not
resolve ResourceManager address akka.tcp://
flink@10.0.0.6:6123/user/resourcemanager, retrying in 1 ms: Could not
connect to rpc endpoint under address akka.tcp://
flink@10.0.0.6:6123/user/resourcemanager..


But the job manager seems to start up ok:

2019-03-12 07:38:36,643 INFO
akka.remote.Remoting  - Remoting
started; listening on addresses :[akka.tcp://flink@10.0.0.6:6123]
2019-03-12 07:38:36,659 INFO
org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils - Actor
system started at akka.tcp://flink@10.0.0.6:6123
2019-03-12 07:38:36,690 INFO
org.apache.flink.runtime.blob.BlobServer  - Created
BLOB server storage directory
C:\cygwin64\tmp\blobStore-85b28100-fa08-4488-9f79-d0d712f34733
2019-03-12 07:38:36,690 INFO
org.apache.flink.runtime.blob.BlobServer  - Started
BLOB server at 0.0.0.0:54072 - max concurrent requests: 50 - max backlog:
1000
2019-03-12 07:38:36,705 INFO
org.apache.flink.runtime.metrics.MetricRegistryImpl   - No metrics
reporter configured, no metrics will be exposed/reported.
2019-03-12 07:38:36,721 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Trying to
start actor system at 10.0.0.6:0
2019-03-12 07:38:36,737 INFO
akka.event.slf4j.Slf4jLogger  - Slf4jLogger
started
2019-03-12 07:38:36,752 INFO
akka.remote.Remoting  - Starting
remoting
2019-03-12 07:38:36,768 INFO
akka.remote.Remoting  - Remoting
started; listening on addresses :[akka.tcp://flink-metrics@10.0.0.6:54085]
2019-03-12 07:38:36,768 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Actor
system started at akka.tcp://flink-metrics@10.0.0.6:54085
2019-03-12 07:38:36,784 INFO
org.apache.flink.runtime.dispatcher.FileArchivedExecutionGraphStore  -
Initializing FileArchivedExecutionGraphStore: Storage directory
C:\cygwin64\tmp\executionGraphStore-550bff8d-314e-4a04-b10e-93bdc7af80c6,
expiration time 360, maximum cache size 52428800 bytes.
2019-03-12 07:38:36,815 INFO
org.apache.flink.runtime.blob.TransientBlobCache  - Created
BLOB cache storage directory
C:\cygwin64\tmp\blobStore-608a5134-9f0d-44dd-8e3d-d9fbe4185d21
2019-03-12 07:38:36,830 WARN
org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint- Upload
directory
C:\cygwin64\tmp\flink-web-2d9712e2-54cb-428a-a27a-826fa2214dad\flink-web-upload
does not exist, or has been deleted externally. Previously uploaded files
are no longer available.
2019-03-12 07:38:36,830 INFO

Re: K8s job cluster and cancel and resume from a save point ?