Flink on kubernetes flink-conf.yaml

2018-03-19 Thread Sampath Bhat
Hello In the official documentation of flink https://ci.apache.org/projects/flink/flink-docs-release-1.4/ops/deployment/kubernetes.html The task manager yaml exposes ports such as task manager rpc port /data port and all. So for these ports to take effect on flink the same has to be mentioned in

Re: [Proposal] CEP library changes - review request

2018-03-19 Thread Dawid Wysakowicz
Hi Shailesh, Thanks for your interest in the CEP library and sorry for late response. I must say I am not fun of this approach. After this change, the Processing time is no longer a processing time, plus it will work differently in any other place of Flink. It will also not sort the events etc.

Re: [DISCUSS] Flink security improvements

2018-03-19 Thread Till Rohrmann
Hi Shuyi, sorry for the unresponsiveness on your proposal. Currently, the community is strongly focused on fixing and testing Flink 1.5 so that we can release it soon. My gut feeling is that the community will pick up the security improvements thread once most of the blocking issues are resolved.

[jira] [Created] (FLINK-9020) Move test projects of end-to-end tests in separate modules

2018-03-19 Thread Florian Schmidt (JIRA)
Florian Schmidt created FLINK-9020: -- Summary: Move test projects of end-to-end tests in separate modules Key: FLINK-9020 URL: https://issues.apache.org/jira/browse/FLINK-9020 Project: Flink

Re: FLIP-6 / Kubernetes

2018-03-19 Thread Till Rohrmann
Hi Thomas, I think the one way to get started would be to adapt the Flink docker images [1,2] to run with Flink 1.5. Per default, they will use the Flip-6 components. Flink 1.5 won't come with a dedicated integration with Kubernetes which is able to start new pods. However, it should work that yo

Re: FLIP-6 / Kubernetes

2018-03-19 Thread Till Rohrmann
I forgot to add Flink's K8 documentation [1] which might also be helpful with getting started. [1] https://ci.apache.org/projects/flink/flink-docs-master/ops/deployment/kubernetes.html Cheers, Till On Mon, Mar 19, 2018 at 10:54 AM, Till Rohrmann wrote: > Hi Thomas, > > I think the one way to g

Re: Flink on Kubernetes issue

2018-03-19 Thread Till Rohrmann
Hi Sampath, you can find the file under flink/docs/ops/deployment/kubernetes.md. Feel free to apply the change and open a PR against Flink's repository. Cheers, Till On Mon, Mar 19, 2018 at 7:57 AM, Sampath Bhat wrote: > Adding on, it would be great if this is updated. If given the permission

Re: Flink on kubernetes flink-conf.yaml

2018-03-19 Thread Till Rohrmann
Hi Sampath, agreed, Flink's K8 documentation can still be improved. As part of the release testing for Flink 1.5 we want to setup a K8 cluster running Flink and update the documentation with more details. The community would really appreciate if you want to help with that. Cheers, Till On Mon, M

Re: [Proposal] CEP library changes - review request

2018-03-19 Thread Shailesh Jain
Thanks for your reply, Dawid. I understand that the approach I've tried out is not generic enough, and would need a lot more thought to be put into w.r.t parallelism considerations, out of order events, effects on downstream operators etc. The intention was to do a quick implementation to check th

[jira] [Created] (FLINK-9021) org.apache.flink.table.codegen.CodeGenException: Unsupported call: TUMBLE

2018-03-19 Thread yizhou (JIRA)
yizhou created FLINK-9021: - Summary: org.apache.flink.table.codegen.CodeGenException: Unsupported call: TUMBLE Key: FLINK-9021 URL: https://issues.apache.org/jira/browse/FLINK-9021 Project: Flink I

[jira] [Created] (FLINK-9022) fix resource close in `StreamTaskStateInitializerImpl.streamOperatorStateContext()`

2018-03-19 Thread Sihua Zhou (JIRA)
Sihua Zhou created FLINK-9022: - Summary: fix resource close in `StreamTaskStateInitializerImpl.streamOperatorStateContext()` Key: FLINK-9022 URL: https://issues.apache.org/jira/browse/FLINK-9022 Project:

[jira] [Created] (FLINK-9023) Make full thread dump in case of test timeout

2018-03-19 Thread Piotr Nowojski (JIRA)
Piotr Nowojski created FLINK-9023: - Summary: Make full thread dump in case of test timeout Key: FLINK-9023 URL: https://issues.apache.org/jira/browse/FLINK-9023 Project: Flink Issue Type: Imp

Re: Flink on Kubernetes issue

2018-03-19 Thread Sampath Bhat
Hi I have modified the documentation. Please refer the link: https://github.com/apache/flink/pull/5721 Let me know if I have to do any further action On Mon, Mar 19, 2018 at 3:28 PM, Till Rohrmann wrote: > Hi Sampath, > > you can find the file under flink/docs/ops/deployment/kubernetes.md. Fee

Job manager - Zookeeper HA

2018-03-19 Thread Sampath Bhat
Hello I'm configuring flink cluster with Zookeeper for high availability for Job managers. These are the HA related configurations- high-availability: zookeeper high-availability.cluster-id: flink high-availability.zookeeper.quorum: localhost:2181 high-availability.zookeeper.storageDir: file:///h

[jira] [Created] (FLINK-9024) IntelliJ IDEA support run and debug Flink code

2018-03-19 Thread chillon.m (JIRA)
chillon.m created FLINK-9024: Summary: IntelliJ IDEA support run and debug Flink code Key: FLINK-9024 URL: https://issues.apache.org/jira/browse/FLINK-9024 Project: Flink Issue Type: Bug

Re: Job manager - Zookeeper HA

2018-03-19 Thread Liangfei Su
You're right. It's job-manager read and distribute jar files. If you are using HDFS, you should be able to check no connection from TM to hadoop. On Mon, Mar 19, 2018 at 9:36 PM, Sampath Bhat wrote: > Hello > > I'm configuring flink cluster with Zookeeper for high availability for Job > managers

Re: Job manager - Zookeeper HA

2018-03-19 Thread Till Rohrmann
Hi Sampath, it is correct that `high-availability.zookeeper.storageDir` is used to persist job data which is needed to recover a job. This is the JobGraph itself, its jars as well as checkpoint meta data. The TaskManager does not need access to this directory, only the JobManagers do. However, th

[jira] [Created] (FLINK-9025) Problem with convert String to Row

2018-03-19 Thread ChenDonghe (JIRA)
ChenDonghe created FLINK-9025: - Summary: Problem with convert String to Row Key: FLINK-9025 URL: https://issues.apache.org/jira/browse/FLINK-9025 Project: Flink Issue Type: Bug Componen

[jira] [Created] (FLINK-9026) Unregister finished tasks from TaskManagerMetricGroup and close it

2018-03-19 Thread Till Rohrmann (JIRA)
Till Rohrmann created FLINK-9026: Summary: Unregister finished tasks from TaskManagerMetricGroup and close it Key: FLINK-9026 URL: https://issues.apache.org/jira/browse/FLINK-9026 Project: Flink

[jira] [Created] (FLINK-9027) Web UI does not cleanup temporary files

2018-03-19 Thread Nico Kruber (JIRA)
Nico Kruber created FLINK-9027: -- Summary: Web UI does not cleanup temporary files Key: FLINK-9027 URL: https://issues.apache.org/jira/browse/FLINK-9027 Project: Flink Issue Type: Bug C

[ANNOUNCE] Weekly community update #12

2018-03-19 Thread Till Rohrmann
Dear community, I've noticed that Flink has grown quite a bit in the past. As a consequence it can be quite challenging to stay up to date. Especially for community members who don't follow Flink's MLs on a daily basis. In order to keep a bigger part of the community in the loop, I wanted to try