Re: Request for assignment: FLINK-18119 Fix unlimitedly growing state for ...

2020-06-15 Thread Benchao Li
Hi Hyeonseop, I'm sorry to hear that you got no reactions in time. We can move to the issue you mentioned for further discussions. Hyeonseop Lee 于2020年6月16日周二 下午1:00写道: > Hello, > > > I have experienced an unlimitedly growing state in my streaming query in > table API and identified that the cu

Does FlinkKinesisConsumer not retry on NoHttpResponseException?

2020-06-15 Thread Singh Aulakh, Karanpreet KP
Hello! (Apache Flink1.8 on AWS EMR release label 5.28.x) Our data source is an AWS Kinesis stream (with 450 shards if that matters). We use the FlinkKinesisConsumer to read the kinesis stream. Our application occasionally (once every couple of days) crashes with a "Target server failed to resp

Request for assignment: FLINK-18119 Fix unlimitedly growing state for ...

2020-06-15 Thread Hyeonseop Lee
Hello, I have experienced an unlimitedly growing state in my streaming query in table API and identified that the current implementation of time range bounded over aggregate function was the cause. I was able to fix it by modifying a couple of functions in flink-table-runtime-blink. I am runnin

[jira] [Created] (FLINK-18316) Add a dynamic state registration primitive for Stateful Functions

2020-06-15 Thread Tzu-Li (Gordon) Tai (Jira)
Tzu-Li (Gordon) Tai created FLINK-18316: --- Summary: Add a dynamic state registration primitive for Stateful Functions Key: FLINK-18316 URL: https://issues.apache.org/jira/browse/FLINK-18316 Proje

[jira] [Created] (FLINK-18315) Insert into partitioned table can fail with values

2020-06-15 Thread Danny Chen (Jira)
Danny Chen created FLINK-18315: -- Summary: Insert into partitioned table can fail with values Key: FLINK-18315 URL: https://issues.apache.org/jira/browse/FLINK-18315 Project: Flink Issue Type: Ta

[jira] [Created] (FLINK-18314) There are some problems in docs

2020-06-15 Thread jinxin (Jira)
jinxin created FLINK-18314: -- Summary: There are some problems in docs Key: FLINK-18314 URL: https://issues.apache.org/jira/browse/FLINK-18314 Project: Flink Issue Type: Bug Components: Doc

[jira] [Created] (FLINK-18312) SavepointStatusHandler and StaticFileServerHandler not redirect

2020-06-15 Thread Yu Wang (Jira)
Yu Wang created FLINK-18312: --- Summary: SavepointStatusHandler and StaticFileServerHandler not redirect Key: FLINK-18312 URL: https://issues.apache.org/jira/browse/FLINK-18312 Project: Flink Issue

Re: [DISCUSS] SQL Syntax for Table API StatementSet

2020-06-15 Thread Jark Wu
Hi Fabian, Thanks for starting this discussion. I think this is a very important syntax to support file mode and multi-statement for SQL Client. I'm +1 to introduce a syntax to group SQL statements to execute together. As a reference, traditional database systems also have similar syntax, such as

[jira] [Created] (FLINK-18313) Hive dialect doc should mention that views created in Flink cannot be used in Hive

2020-06-15 Thread Rui Li (Jira)
Rui Li created FLINK-18313: -- Summary: Hive dialect doc should mention that views created in Flink cannot be used in Hive Key: FLINK-18313 URL: https://issues.apache.org/jira/browse/FLINK-18313 Project: Flink

[jira] [Created] (FLINK-18311) StreamingKafkaITCase stalls indefinitely

2020-06-15 Thread Dian Fu (Jira)
Dian Fu created FLINK-18311: --- Summary: StreamingKafkaITCase stalls indefinitely Key: FLINK-18311 URL: https://issues.apache.org/jira/browse/FLINK-18311 Project: Flink Issue Type: Bug Comp

Re: Moving in and Modify Dependency Source

2020-06-15 Thread Austin Cawley-Edwards
Ah, missed Till's response -- thanks as well! I'll add those headers to the files, so just now wondering about including the licenses/ notice in the RMQ connector resources. On Mon, Jun 15, 2020 at 7:40 PM Austin Cawley-Edwards < austin.caw...@gmail.com> wrote: > Hey Robert, > > Thanks for getti

Re: Moving in and Modify Dependency Source

2020-06-15 Thread Austin Cawley-Edwards
Hey Robert, Thanks for getting back to me! Just wasn't sure on the license header requirements for the CI checks in Flink. Not too experienced with working with licenses, especially in large open-source projects. Since we would be using APL 2 (and from this link[1] we need to state changes, includ

Re: Rest handler redirect problem

2020-06-15 Thread Chesnay Schepler
I think this is not unintentional and simply a case we did not consider. Please file a JIRA. On 15/06/2020 19:01, Wong Lucent wrote: Hi, Recently, our team upgraged our filnk cluster from version 1.7 to 1.10. And we met some problem when calling the flink rest api. 1) We deploy our flink cl

[jira] [Created] (FLINK-18310) Failure while parsing reporter interval does not reliably revert back to default

2020-06-15 Thread Chesnay Schepler (Jira)
Chesnay Schepler created FLINK-18310: Summary: Failure while parsing reporter interval does not reliably revert back to default Key: FLINK-18310 URL: https://issues.apache.org/jira/browse/FLINK-18310

Rest handler redirect problem

2020-06-15 Thread Wong Lucent
Hi, Recently, our team upgraged our filnk cluster from version 1.7 to 1.10. And we met some problem when calling the flink rest api. 1) We deploy our flink cluster in standlone mode on kubernetes and use two Jobmanagers for HA. 2) We deployed a kubernetes service for the two jobmanagers to pr

Re: Any python example with json data from Kafka using flink-statefun

2020-06-15 Thread Sunil Sattiraju
Thanks Igal, I dont have control over the data source inside kafka ( current kafka topic contains either json or avro formats only, i am trying to reproduce this scenario using my test data generator ). is it possible to convert the json to proto at the receiving end of statefun applicaiton?

[jira] [Created] (FLINK-18309) Recommend avoiding uppercase to emphasise statements in doc style

2020-06-15 Thread Andrey Zagrebin (Jira)
Andrey Zagrebin created FLINK-18309: --- Summary: Recommend avoiding uppercase to emphasise statements in doc style Key: FLINK-18309 URL: https://issues.apache.org/jira/browse/FLINK-18309 Project: Flin

[jira] [Created] (FLINK-18308) KafkaProducerTestBase->Kafka011ProducerExactlyOnceITCase. testExactlyOnceCustomOperator hangs in Azure

2020-06-15 Thread Andrey Zagrebin (Jira)
Andrey Zagrebin created FLINK-18308: --- Summary: KafkaProducerTestBase->Kafka011ProducerExactlyOnceITCase. testExactlyOnceCustomOperator hangs in Azure Key: FLINK-18308 URL: https://issues.apache.org/jira/browse/F

[jira] [Created] (FLINK-18307) Replace "slave" file name with "workers"

2020-06-15 Thread Stephan Ewen (Jira)
Stephan Ewen created FLINK-18307: Summary: Replace "slave" file name with "workers" Key: FLINK-18307 URL: https://issues.apache.org/jira/browse/FLINK-18307 Project: Flink Issue Type: Sub-task

Re: Any python example with json data from Kafka using flink-statefun

2020-06-15 Thread Igal Shilman
Hi, The values must be valid encoded Protobuf messages [1], while in your attached code snippet you are sending utf-8 encoded JSON strings. You can take a look at this example with a generator that produces Protobuf messages [2][3] [1] https://developers.google.com/protocol-buffers/docs/pythontut

Any python example with json data from Kafka using flink-statefun

2020-06-15 Thread Sunil Sattiraju
Hi, Based on the example from https://github.com/apache/flink-statefun/tree/master/statefun-examples/statefun-python-greeter-example I am trying to ingest json data in kafka, but unable to achieve based on the examples. event-generator.py def produce(): request = {} request['id'] = "ab

NullPointer Exception while trying to access or read ReadOnly ctx in processElement method in KeyedBroadCastProcessFunction in Apache Flink

2020-06-15 Thread bujjirahul45 .
I have some interesting scenario i am working on pattern matching in flink evaluating the incoming data against a set of patterns using keyedbroadcastprocessfunction, when i am running the program in IDE i am getting null pointer exception in processElements method when trying to access ReadOnlyCon

[ANNOUNCE] Weekly Community Update 2020/23-24

2020-06-15 Thread Konstantin Knauf
Dear community, happy to share this community update on the last two weeks including the release of Stateful Functions 2.1, a table source for ElasticSearch, and a bit more. The community is still working on release testing for Apache Flink 1.11, so still comparably quite. Expecting the first feat

[jira] [Created] (FLINK-18306) how to satisfy the node-sass dependency when compiling runtime-web?

2020-06-15 Thread appleyuchi (Jira)
appleyuchi created FLINK-18306: -- Summary: how to satisfy the node-sass dependency when compiling runtime-web? Key: FLINK-18306 URL: https://issues.apache.org/jira/browse/FLINK-18306 Project: Flink

[jira] [Created] (FLINK-18305) Add interval configuration too all reporter examples that support it

2020-06-15 Thread Chesnay Schepler (Jira)
Chesnay Schepler created FLINK-18305: Summary: Add interval configuration too all reporter examples that support it Key: FLINK-18305 URL: https://issues.apache.org/jira/browse/FLINK-18305 Project:

[jira] [Created] (FLINK-18304) Document default reporter interval

2020-06-15 Thread Chesnay Schepler (Jira)
Chesnay Schepler created FLINK-18304: Summary: Document default reporter interval Key: FLINK-18304 URL: https://issues.apache.org/jira/browse/FLINK-18304 Project: Flink Issue Type: Improv

[DISCUSS] SQL Syntax for Table API StatementSet

2020-06-15 Thread Fabian Hueske
Hi everyone, FLIP-84 [1] added the concept of a "statement set" to group multiple INSERT INTO statements (SQL or Table API) together. The statements in a statement set are jointly optimized and executed as a single Flink job. I would like to start a discussion about a SQL syntax to group multiple

[jira] [Created] (FLINK-18303) Filesystem connector doesn't flush part files after rolling interval

2020-06-15 Thread Jark Wu (Jira)
Jark Wu created FLINK-18303: --- Summary: Filesystem connector doesn't flush part files after rolling interval Key: FLINK-18303 URL: https://issues.apache.org/jira/browse/FLINK-18303 Project: Flink I

[jira] [Created] (FLINK-18302) Sql client uses wrong class loader when execute INSERT statements

2020-06-15 Thread Jark Wu (Jira)
Jark Wu created FLINK-18302: --- Summary: Sql client uses wrong class loader when execute INSERT statements Key: FLINK-18302 URL: https://issues.apache.org/jira/browse/FLINK-18302 Project: Flink Issu

Re: [DISCUSS] Re-renaming "Flink Master" back to JobManager

2020-06-15 Thread Konstantin Knauf
Hi Aljoscha, sounds good to me. Let’s also make sure we don’t refer to the JobMaster as Jobmanager anywhere then (code, config). I am not sure we can avoid mentioning the Flink ResourceManagers in user facing docs completely. For JobMaster and Dispatcher this seems doable. Best, Konstantin On

[jira] [Created] (FLINK-18301) Backup Kafka logs on e2e failure

2020-06-15 Thread Chesnay Schepler (Jira)
Chesnay Schepler created FLINK-18301: Summary: Backup Kafka logs on e2e failure Key: FLINK-18301 URL: https://issues.apache.org/jira/browse/FLINK-18301 Project: Flink Issue Type: Improvem

[DISCUSS] Re-renaming "Flink Master" back to JobManager

2020-06-15 Thread Aljoscha Krettek
Hi All, This came to my mind because of the master/slave discussion in [1] and the larger discussions about inequality/civil rights happening right now in the world. I think for this reason alone we should use a name that does not include "master". We could rename it back to JobManager, whic

[jira] [Created] (FLINK-18300) SQL Client doesn't support ALTER VIEW

2020-06-15 Thread Rui Li (Jira)
Rui Li created FLINK-18300: -- Summary: SQL Client doesn't support ALTER VIEW Key: FLINK-18300 URL: https://issues.apache.org/jira/browse/FLINK-18300 Project: Flink Issue Type: Bug Component

[jira] [Created] (FLINK-18299) Add keyword in json format to parse timestamp in different standard

2020-06-15 Thread Shengkai Fang (Jira)
Shengkai Fang created FLINK-18299: - Summary: Add keyword in json format to parse timestamp in different standard Key: FLINK-18299 URL: https://issues.apache.org/jira/browse/FLINK-18299 Project: Flink

[jira] [Created] (FLINK-18298) Rename TableResult headers of SHOW statements

2020-06-15 Thread Fabian Hueske (Jira)
Fabian Hueske created FLINK-18298: - Summary: Rename TableResult headers of SHOW statements Key: FLINK-18298 URL: https://issues.apache.org/jira/browse/FLINK-18298 Project: Flink Issue Type: I

[jira] [Created] (FLINK-18297) SQL client: setting execution.type to invalid value shuts down the session

2020-06-15 Thread David Anderson (Jira)
David Anderson created FLINK-18297: -- Summary: SQL client: setting execution.type to invalid value shuts down the session Key: FLINK-18297 URL: https://issues.apache.org/jira/browse/FLINK-18297 Projec

[jira] [Created] (FLINK-18295) Remove the hack logics of result consumers

2020-06-15 Thread Zhu Zhu (Jira)
Zhu Zhu created FLINK-18295: --- Summary: Remove the hack logics of result consumers Key: FLINK-18295 URL: https://issues.apache.org/jira/browse/FLINK-18295 Project: Flink Issue Type: Improvement

[jira] [Created] (FLINK-18296) Add support for datetype TIMESTAMP_WITH_LOCAL_ZONE for Json format

2020-06-15 Thread Shengkai Fang (Jira)
Shengkai Fang created FLINK-18296: - Summary: Add support for datetype TIMESTAMP_WITH_LOCAL_ZONE for Json format Key: FLINK-18296 URL: https://issues.apache.org/jira/browse/FLINK-18296 Project: Flink

Re: Flink 1.10 with GCS for checkpoints

2020-06-15 Thread Till Rohrmann
Hi Ramya, it looks as if Flink cannot find the Hadoop dependencies. Could you make sure that you start Flink with HADOOP_CLASSPATH defined or pointing it to the Hadoop conf directory via HADOOP_CONF_DIR. See this link [1] for more information on how to add Hadoop support. [1] https://ci.apache.or

[jira] [Created] (FLINK-18294) Log java processes and disk space after each e2e test

2020-06-15 Thread Chesnay Schepler (Jira)
Chesnay Schepler created FLINK-18294: Summary: Log java processes and disk space after each e2e test Key: FLINK-18294 URL: https://issues.apache.org/jira/browse/FLINK-18294 Project: Flink

[jira] [Created] (FLINK-18293) TaskExecutor offering non empty slots can lead to resource violation

2020-06-15 Thread Till Rohrmann (Jira)
Till Rohrmann created FLINK-18293: - Summary: TaskExecutor offering non empty slots can lead to resource violation Key: FLINK-18293 URL: https://issues.apache.org/jira/browse/FLINK-18293 Project: Flink

Flink 1.10 with GCS for checkpoints

2020-06-15 Thread Ramya Ramamurthy
Hi, We are trying to upgrade our Flink from 1.7 to 1.10. We have our checkpoints on Google Cloud Storage today. But this is not working well with 1.10. And below is the error we get. any help here would be appreciated. We followed the below blog for GCS related configurations. https://www.ververic

Re: Moving in and Modify Dependency Source

2020-06-15 Thread Robert Metzger
Hi Austin, Thanks for working on the RMQ connector! There seem to be a few users affected by that issue. The GitHub page confirms that users can choose from the three licenses: https://github.com/rabbitmq/rabbitmq-java-client#license: > This means that the user can consider the library to be lice

Re: request create flip permission for flink es bounded source/lookup source connector

2020-06-15 Thread Robert Metzger
Ah, sorry. The permission system of Confluence is a bit annoying. I gave you "Add Attachment" permissions. On Mon, Jun 15, 2020 at 8:57 AM Jacky Lau wrote: > Hi Robert: > When i edit the FLIP, and upload the images. It will show the > prompting > message like this "You'll need to ask permis

Interested in contributing to Apache Flink

2020-06-15 Thread Vivin Peris
Respected Sir I am currently a final year undergrad student at National Institute of Technology Surathkal, India. I have extensively worked on Apache Flink in my previous internship where I had to create a database of billions of IPs in the world along with data associated with it. The power of F

Re: [DISCUSS] Semantics of our JIRA fields

2020-06-15 Thread Piotr Nowojski
> On 12 Jun 2020, at 15:44, Robert Metzger wrote: > > Piotrek, do you agree with my "affects version" explanation? I would like > to bring this discussion to a conclusion. > +0 for this semantic from my side. > On Tue, May 26, 2020 at 4:51 PM Till Rohrmann wrote: > >> If we change the mean

[jira] [Created] (FLINK-18292) Savepoint for job error

2020-06-15 Thread Kayo (Jira)
Kayo created FLINK-18292: Summary: Savepoint for job error Key: FLINK-18292 URL: https://issues.apache.org/jira/browse/FLINK-18292 Project: Flink Issue Type: Bug Components: Command Line Cl