Re: Ask for SQL using kafka in Flink

Yes, I am also looking for examples for Kafka avro table examples in java and command line. Also, Kafka avro table sink is still missing. In addition, once we have Kafka topic, the API should read the schema directly from schema file or schema registry. The way of current API supporting lacks of

[jira] [Created] (FLINK-9525) Missing META-INF/services/*FileSystemFactory in flink-hadoop-fs module

Hai Zhou created FLINK-9525: --- Summary: Missing META-INF/services/*FileSystemFactory in flink-hadoop-fs module Key: FLINK-9525 URL: https://issues.apache.org/jira/browse/FLINK-9525 Project: Flink I

Re: [DISCUSS] Flink 1.6 features

Hi Stephan, Will [ https://issues.apache.org/jira/browse/FLINK-5479 ] (Per-partition watermarks in FlinkKafkaConsumer should consider idle partitions) be included in 1.6? As we are seeing more users with this issue on the mailing lists. Thanks. Ben 2018-06-05 5:29 GMT+08:00 Che Lui Shum : > H

Re: [Discuss] Proposing FLIP-25 - Support User State TTL Natively in Flink

Hi, I like that this design combines the few previously discussed approaches together to address a much broader scope of TTL use cases. Each previously discussed approach alone focuses on a narrower scenario than this one, and has its own limitations. I'm glad to see the community is willing to s

Re: Flink Cassandra Sink - Setting TTL for the record

Thanks. I was able get this working with insert statement with "using TTL" https://docs.datastax.com/en/cql/3.3/cql/cql_using/useExpireExample.html Rohan On Mon, Jun 4, 2018 at 11:45 AM, miki haiat wrote: > if you dont have this option in the conector API you need to implement a > richs

Fwd: [DISCUSS] FLIP-6 Problems

-- Forwarded message - From: Renjie Liu Date: Tue, Jun 5, 2018 at 10:43 AM Subject: [DISCUSS] FLIP-6 Problems To: user Hi: We've deployed flink 1.5.0 and tested the new cluster manager, it's really great for flink to be elastic. However we've also found some problems that block

[jira] [Created] (FLINK-9524) NPE from ProcTimeBoundedRangeOver.scala

yan zhou created FLINK-9524: --- Summary: NPE from ProcTimeBoundedRangeOver.scala Key: FLINK-9524 URL: https://issues.apache.org/jira/browse/FLINK-9524 Project: Flink Issue Type: Bug Compone

[jira] [Created] (FLINK-9523) Add Kafka examples for Flink Table/SQL API

Shuyi Chen created FLINK-9523: - Summary: Add Kafka examples for Flink Table/SQL API Key: FLINK-9523 URL: https://issues.apache.org/jira/browse/FLINK-9523 Project: Flink Issue Type: Task

Re: [DISCUSS] Flink 1.6 features

Hi Stephan, Will FLINK-7129 (Support dynamically changing CEP patterns) be included in 1.6? There were discussions about possibly including it in 1.6: http://mail-archives.apache.org/mod_mbox/flink-user/201803.mbox/%3cCAMq=ou7gru2o9jtowxn1lc1f7nkcxayn6a3e58kxctb4b50...@mail.gmail.com%3e Thanks,

[jira] [Created] (FLINK-9522) Rework Flink website

Fabian Hueske created FLINK-9522: Summary: Rework Flink website Key: FLINK-9522 URL: https://issues.apache.org/jira/browse/FLINK-9522 Project: Flink Issue Type: Improvement Componen

Re: Flink Cassandra Sink - Setting TTL for the record

if you dont have this option in the conector API you need to implement a richsinkfunction with your desire logic. https://ci.apache.org/projects/flink/flink-docs-master/api/java/org/apache/flink/streaming/api/functions/sink/RichSinkFunction.html On Mon, 4 Jun 2018, 21:32 Rohan Thimmappa, wrot

Flink Cassandra Sink - Setting TTL for the record

Hi there, i am using timeseries data in cassandra by reading data from kafka. I didn't see any option set the TTL. Any help would be of great help! -- Thanks Rohan

Re: Ask for SQL using kafka in Flink

Given the popularity of Flink SQL and Kafka as streaming source, I think we can add some examples of using Kafka[XXX]TableSource in flink-examples/flink-examples-table module. What do you guys think? Cheers Shuyi On Mon, Jun 4, 2018 at 12:57 AM, Timo Walther wrote: > Hi, > > as you can see in c

[jira] [Created] (FLINK-9521) Add build plugin to package table example jar

vinoyang created FLINK-9521: --- Summary: Add build plugin to package table example jar Key: FLINK-9521 URL: https://issues.apache.org/jira/browse/FLINK-9521 Project: Flink Issue Type: Bug Affects

[jira] [Created] (FLINK-9520) JarFileCreator does not support Java MethodReferences

Timo Walther created FLINK-9520: --- Summary: JarFileCreator does not support Java MethodReferences Key: FLINK-9520 URL: https://issues.apache.org/jira/browse/FLINK-9520 Project: Flink Issue Type:

[jira] [Created] (FLINK-9519) table example jar did not copy into example dir

vinoyang created FLINK-9519: --- Summary: table example jar did not copy into example dir Key: FLINK-9519 URL: https://issues.apache.org/jira/browse/FLINK-9519 Project: Flink Issue Type: Bug Affec

Re: [DISCUSS] Flink 1.6 features

Before removing the legacy code, I would still wait a bit and see what the user feedback is. The legacy mode is a good safety net against severe deployment regressions. Thus, it should be a very conscious decision to remove the code. As far as I know, there is currently nobody actively working on

[jira] [Created] (FLINK-9518) SSL setup Docs config example has wrong keys password

Yazdan Shirvany created FLINK-9518: -- Summary: SSL setup Docs config example has wrong keys password Key: FLINK-9518 URL: https://issues.apache.org/jira/browse/FLINK-9518 Project: Flink Issu

[jira] [Created] (FLINK-9517) Fixing broken links on CLI Docs

Yazdan Shirvany created FLINK-9517: -- Summary: Fixing broken links on CLI Docs Key: FLINK-9517 URL: https://issues.apache.org/jira/browse/FLINK-9517 Project: Flink Issue Type: Improvement

Re:[DISCUSS] Flink 1.6 features

Hi Stephan, could you please also consider the "Elastic Filter " feature discussioned in thread http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/PROPOSAL-Introduce-Elastic-Bloom-Filter-For-Flink-td22430.html ? Best, Sihua On 06/4/2018 17:21，Stephan Ewen wrote： Hi Flink Comm

[DISCUSS] Flink 1.6 features

Hi Flink Community! The release of Apache Flink 1.5 has happened (yay!) - so it is a good time to start talking about what to do for release 1.6. *== Suggested release timeline ==* I would propose to release around *end of July* (that is 8-9 weeks from now). The rational behind that: There was

Re: [Discuss] Proposing FLIP-25 - Support User State TTL Natively in Flink

Hi, I like the separation of visibility and clean-up. So far the design only addressed the clean-up aspect and aimed to have state accessible as long as possible, i.e., until it was cleared. We did not consider the use case of compliance. Support for strict visibility is a good idea, IMO. Moreove

[ANNOUNCE] Weekly community update #23

Dear community, this is the weekly community update thread #23. Please post any news and updates you want to share with the community to this thread. # Releasing flink-shaded 4.0 The community is currently testing the RC 2 for flink-shaded 4.0 [1] which bumps the shaded Netty version to 4.1.24 a

Re: [TABLE][SQL] Unify UniqueKeyExtractor and DataStreamRetractionRules

Hi Piotr, thanks for bringing up this discussion. I was not involved in the design discussions at that time but I also find the logic about upserts and retractions in multiple stages quite confusing. So in general +1 for simplification, however, by using a RelShuttle instead of rules we might

[jira] [Created] (FLINK-9516) Create wrapper with TTL logic for map state

Andrey Zagrebin created FLINK-9516: -- Summary: Create wrapper with TTL logic for map state Key: FLINK-9516 URL: https://issues.apache.org/jira/browse/FLINK-9516 Project: Flink Issue Type: Sub

[jira] [Created] (FLINK-9515) Create wrapper with TTL logic for list state

Andrey Zagrebin created FLINK-9515: -- Summary: Create wrapper with TTL logic for list state Key: FLINK-9515 URL: https://issues.apache.org/jira/browse/FLINK-9515 Project: Flink Issue Type: Su

[jira] [Created] (FLINK-9514) Create wrapper with TTL logic for value state

Andrey Zagrebin created FLINK-9514: -- Summary: Create wrapper with TTL logic for value state Key: FLINK-9514 URL: https://issues.apache.org/jira/browse/FLINK-9514 Project: Flink Issue Type: S

[jira] [Created] (FLINK-9513) Wrap state binder with TTL logic

Andrey Zagrebin created FLINK-9513: -- Summary: Wrap state binder with TTL logic Key: FLINK-9513 URL: https://issues.apache.org/jira/browse/FLINK-9513 Project: Flink Issue Type: Sub-task

Re: [Discuss] Proposing FLIP-25 - Support User State TTL Natively in Flink

Hi andrey, Thanks for this doc! TBH, personally I prefer the approach you outlined in the doc over the previous one that purly based on timers. I think this approach looks very similar to the approach I outlined in this thread before, so it still face the challenges that @Bowen outlined, but I

[jira] [Created] (FLINK-9512) StateDescriptor TTL config should be part of versioned meta info snapshot

Andrey Zagrebin created FLINK-9512: -- Summary: StateDescriptor TTL config should be part of versioned meta info snapshot Key: FLINK-9512 URL: https://issues.apache.org/jira/browse/FLINK-9512 Project:

[jira] [Created] (FLINK-9511) Make StateDescriptor configurable with optional TTL

Andrey Zagrebin created FLINK-9511: -- Summary: Make StateDescriptor configurable with optional TTL Key: FLINK-9511 URL: https://issues.apache.org/jira/browse/FLINK-9511 Project: Flink Issue T

[jira] [Created] (FLINK-9510) State TTL API with cleanup on explicit access

Andrey Zagrebin created FLINK-9510: -- Summary: State TTL API with cleanup on explicit access Key: FLINK-9510 URL: https://issues.apache.org/jira/browse/FLINK-9510 Project: Flink Issue Type: N

Re: [Discuss] Proposing FLIP-25 - Support User State TTL Natively in Flink

Hi everybody, We have been recently brainstorming ideas around state TTL in Flink and compiled our thoughts in the following design doc: https://docs.google.com/document/d/1SI_WoXAfOd4_NKpGyk4yh3mf59g12pSGNXRtNFi-tgM