Re: [DISCUSS] Flink Cluster Overview Dashboard Improvement Proposal

2018-10-11 Thread Till Rohrmann
Hi Fabian, yes the community is very much open and thankful for contributions to the web UI including the technology used. What it could use is a person who would really like to drive this since so far it was if it all someone's side project. Cheers, Till On Wed, Oct 10, 2018 at 11:28 AM Fabian

Re: [DISCUSS] Integrate Flink SQL well with Hive ecosystem

2018-10-11 Thread Timo Walther
Hi Xuefu, thanks for your proposal, it is a nice summary. Here are my thoughts to your list: 1. I think this is also on our current mid-term roadmap. Flink lacks a poper catalog support for a very long time. Before we can connect catalogs we need to define how to map all the information from

Re: [VOTE] Release flink-shaded 5.0, release candidate #1

2018-10-11 Thread Timo Walther
+1 - I built locally and checked the JAR files for suspicious things. - I went throught the change diff between 4 and 5 as well. Could not find anything blocking this release. Thanks, Timo Am 10.10.18 um 17:22 schrieb Aljoscha Krettek: +1 I did - verify all changes between 4.0 and 5.0 -

[jira] [Created] (FLINK-10528) Remove deprecated APIs from Table API for Flink 1.7.0

2018-10-11 Thread Fabian Hueske (JIRA)
Fabian Hueske created FLINK-10528: - Summary: Remove deprecated APIs from Table API for Flink 1.7.0 Key: FLINK-10528 URL: https://issues.apache.org/jira/browse/FLINK-10528 Project: Flink Issue

Re: [VOTE] Release flink-shaded 5.0, release candidate #1

2018-10-11 Thread Tzu-Li (Gordon) Tai
+1 - Checked signature and hashes - Built locally with current master branch Couldn't find anything blocking either. Cheers, Gordon On Thu, Oct 11, 2018 at 4:33 PM Timo Walther wrote: > +1 > > - I built locally and checked the JAR files for suspicious things. > - I went throught the change di

Re: [VOTE] Release flink-shaded 5.0, release candidate #1

2018-10-11 Thread vino yang
+1, - checked signature and hash - checked the version number in the source code tag - checked version number for all the artifacts Thanks, vino. Tzu-Li (Gordon) Tai 于2018年10月11日周四 下午6:21写道: > +1 > > - Checked signature and hashes > - Built locally with current master branch > > Couldn't find

[jira] [Created] (FLINK-10529) Add flink-s3-fs-base to the connectors in the travis stage file.

2018-10-11 Thread Kostas Kloudas (JIRA)
Kostas Kloudas created FLINK-10529: -- Summary: Add flink-s3-fs-base to the connectors in the travis stage file. Key: FLINK-10529 URL: https://issues.apache.org/jira/browse/FLINK-10529 Project: Flink

[jira] [Created] (FLINK-10530) ProcessFailureCancelingITCase.testCancelingOnProcessFailure failed on Travis.

2018-10-11 Thread Kostas Kloudas (JIRA)
Kostas Kloudas created FLINK-10530: -- Summary: ProcessFailureCancelingITCase.testCancelingOnProcessFailure failed on Travis. Key: FLINK-10530 URL: https://issues.apache.org/jira/browse/FLINK-10530 Pro

[jira] [Created] (FLINK-10531) State TTL RocksDb backend end-to-end test end-to-end test failed on Travis

2018-10-11 Thread Till Rohrmann (JIRA)
Till Rohrmann created FLINK-10531: - Summary: State TTL RocksDb backend end-to-end test end-to-end test failed on Travis Key: FLINK-10531 URL: https://issues.apache.org/jira/browse/FLINK-10531 Project:

Re: [DISCUSS] Integrate Flink SQL well with Hive ecosystem

2018-10-11 Thread Zhang, Xuefu
Hi Jörn, Thanks for your feedback. Yes, I think Hive on Flink makes sense and in fact it is one of the two approaches that I named in the beginning of the thread. As also pointed out there, this isn't mutually exclusive from work we proposed inside Flink and they target at different user groups

Re: [DISCUSS] Integrate Flink SQL well with Hive ecosystem

2018-10-11 Thread Rong Rong
Hi Xuefu, Thanks for putting together the overview. I would like to add some more on top of Timo's comments. 1,2. I agree with Timo that a proper catalog support should also address the metadata compatibility issues. I was actually wondering if you are referring to something like utilizing table s

Spreading Tasks across TaskManagers

2018-10-11 Thread Maximilian Michels
Hi everyone, I've recently come across a cluster scheduling problem users are facing. Clusters where TaskManagers have more slots than the parallelism (#tm_slots > job_parallelism), tend to schedule all job tasks on a single TaskManager. This is not good for spreading load and has been discu

[jira] [Created] (FLINK-10532) Broken links in documentation

2018-10-11 Thread Chesnay Schepler (JIRA)
Chesnay Schepler created FLINK-10532: Summary: Broken links in documentation Key: FLINK-10532 URL: https://issues.apache.org/jira/browse/FLINK-10532 Project: Flink Issue Type: Bug

Re: [DISCUSS] Integrate Flink SQL well with Hive ecosystem

2018-10-11 Thread Zhang, Xuefu
Hi Timo, Thank you for your input. It's exciting to see that the community has already initiated some of the topics. We'd certainly like to leverage the current and previous work and make progress in phases. Here I'd like to comment on a few things on top of your feedback. 1. I think there are

Re: [DISCUSS] Integrate Flink SQL well with Hive ecosystem

2018-10-11 Thread Zhang, Xuefu
Hi Rong, Thanks for your feedback. Some of my earlier comments might have addressed some of your points, so here I'd like to cover some specifics. 1. Yes, I expect that table stats stored in Hive will be used in Flink plan optimization, but it's not part of compatibility concern (yet). 2. Both

Become a contributor

2018-10-11 Thread Zhang, Xuefu
Hi there, Could anyone kindly add me as a contributor to Flink project? Thanks, Xuefu

Re: [DISCUSS] Integrate Flink SQL well with Hive ecosystem

2018-10-11 Thread Taher Koitawala
I think integrating Flink with Hive would be an amazing option and also to get Flink's SQL up to pace would be amazing. Current Flink Sql syntax to prepare and process a table is too verbose, users manually need to retype table definitions and that's a pain. Hive metastore integration should be do

Re: [DISCUSS] Integrate Flink SQL well with Hive ecosystem

2018-10-11 Thread Taher Koitawala
One other thought on the same lines was to use hive tables to store kafka information to process streaming tables. Something like "create table streaming_table ( bootstrapServers string, topic string, keySerialiser string, ValueSerialiser string)" Insert into streaming_table values(,"10.17.1.1:90

[jira] [Created] (FLINK-10533) job parallelism equals task slot number but not use all tasl slot

2018-10-11 Thread sean.miao (JIRA)
sean.miao created FLINK-10533: - Summary: job parallelism equals task slot number but not use all tasl slot Key: FLINK-10533 URL: https://issues.apache.org/jira/browse/FLINK-10533 Project: Flink

[jira] [Created] (FLINK-10534) Add idle timeout for a flink session cluster

2018-10-11 Thread ouyangzhe (JIRA)
ouyangzhe created FLINK-10534: - Summary: Add idle timeout for a flink session cluster Key: FLINK-10534 URL: https://issues.apache.org/jira/browse/FLINK-10534 Project: Flink Issue Type: New Featur

[jira] [Created] (FLINK-10535) User jar is present in the flink job manager's class path

2018-10-11 Thread yinhua.dai (JIRA)
yinhua.dai created FLINK-10535: -- Summary: User jar is present in the flink job manager's class path Key: FLINK-10535 URL: https://issues.apache.org/jira/browse/FLINK-10535 Project: Flink Issue T

Re: [DISCUSS] Integrate Flink SQL well with Hive ecosystem

2018-10-11 Thread Zhang, Xuefu
Hi Taher, Thank you for your input. I think you emphasized two important points: 1. Hive metastore could be used for storing Flink metadata 2. There are some usability issues around Flink SQL configuration I think we all agree on #1. #2 may be well true and the usability should be improved. How