Re: Confused window operation

Thanks hequn & acqua.csq On Wed, Nov 14, 2018 at 2:17 PM Hequn Cheng wrote: > Hi Jeff, > > The window is not a global window. It is related to a specified key. You > would have 6 windows after flatMap() and keyBy(). > key: hello with 3 windows > key: world with 1 window > key: flink with 1 wind

Re: Field could not be resolved by the field mapping when using kafka connector

Thanks hequn, it is very helpful On Wed, Nov 14, 2018 at 2:32 PM Hequn Cheng wrote: > Hi jeff, > > We need a different field name for the rowtime indicator, something looks > like: > >> new Schema() >> .field("status", Types.STRING) >> .field("direction", Types.STRING) >>

Re: Field could not be resolved by the field mapping when using kafka connector

Hi jeff, We need a different field name for the rowtime indicator, something looks like: > new Schema() > .field("status", Types.STRING) > .field("direction", Types.STRING) > .field("rowtime", Types.SQL_TIMESTAMP).rowtime( > new > Rowtime().timestampsFromFiel

Re: Confused window operation

Hi Jeff, The window is not a global window. It is related to a specified key. You would have 6 windows after flatMap() and keyBy(). key: hello with 3 windows key: world with 1 window key: flink with 1 window key: hadoop with 1 window Best, Hequn On Wed, Nov 14, 2018 at 10:31 AM Jeff Zhang wrot

Re: Flink 1.7 RC missing flink-scala-shell jar

I do not see flink-scala-shell jar under flink opt directory. To run scala shell, do I have to include the flink-scala-shell jar in my program jar? Why the error is saying Could not find or load main class org.apache.flink.api.scala.FlinkShell On Tue, Nov 13, 2018 at 4:48 PM Tzu-Li Chen wrote: >

How flink table api to join with mysql dimtable

hi, I want to use flink sql to left join static dimension table from mysql currently, so I converted the mysql table into data stream to join with datastream which has converted to flink table. While I found that the real-time stream data is not joined correctly with mysql data at the beginnin

Confused window operation

Hi all, I am a little confused with the following windows operation. Here's the code, val senv = StreamExecutionEnvironment.getExecutionEnvironment senv.setParallelism(1) val data = senv.fromElements("hello world", "hello flink", "hello hadoop") data.flatMap(line => line.split("\\s")) .map(w =

?????? How to get the ActorSystem in custom operator

yes ??I??m trying to have a "signaling" input channel to control the operator behaviour in the following case. The StreamOperators will subscribe to the MQ ??not support broadcast?? in the same group?? One of the operator will receive the massage, then dispatch or broadcast the message the sp

Field could not be resolved by the field mapping when using kafka connector

Hi, I hit the following error when I try to use kafka connector in flink table api. There's very little document about how to use kafka connector in flink table api, could anyone help me on that ? Thanks Exception in thread "main" org.apache.flink.table.api.ValidationException: Field 'event_ts' c

Re: Flink 1.7 RC missing flink-scala-shell jar

Hi, Till is the release manager for 1.7, so ping him here. Best, tison. Hao Sun 于2018年11月14日周三 上午3:07写道： > Sorry I mean the scala-2.12 version is missing > > On Tue, Nov 13, 2018 at 10:58 AM Hao Sun wrote: > >> I can not find the jar here: >> >> https://repository.apache.org/content/reposito

Re: Flink 1.7 RC missing flink-scala-shell jar

Sorry I mean the scala-2.12 version is missing On Tue, Nov 13, 2018 at 10:58 AM Hao Sun wrote: > I can not find the jar here: > > https://repository.apache.org/content/repositories/orgapacheflink-1191/org/apache/flink/ > > Here is the error: > bash-4.4# ./bin/start-scala-shell.sh local > Error:

Flink 1.7 RC missing flink-scala-shell jar

I can not find the jar here: https://repository.apache.org/content/repositories/orgapacheflink-1191/org/apache/flink/ Here is the error: bash-4.4# ./bin/start-scala-shell.sh local Error: Could not find or load main class org.apache.flink.api.scala.FlinkShell I think somehow I have to include the

Custom state serializer in the Scala DataStream API

I need to have states that support schema migration. For that I intend to serialize them using protobuf. For that I’ve create a custom TypeSerializer. Unfortunately, I couldn’t find how to set the serializer used when I use mapWithState in a KeyedStream. So any one could point me in the right dire

Re: FLINK Kinesis Connector Error - ProvisionedThroughputExceededException

Hi all, I think Steve's occurrence of the warning was from the consumer side. For the Flink Kinesis Consumer, this could most likely occur due to excessive ListShard API calls on the target Kinesis stream. The consumer uses this API to discover shards, at a fixed interval. The problem with the cu

Get savepoint status fails - Flink 1.6.2

Hello, I am trying to get the status for a savepoint using the rest api but the GET request is failing with an error as depicted bellow. /curl -k https://localhost:8081/jobs/c78511cf0dc10c1e9f7db17566522d5b/savepoints/51c174eab1efd2c1354282f52f37fadb {"errors":["Operation not found under key: org

Re: How to get the ActorSystem in custom operator

Hi, I don't think that is possible. Are you trying to "bootstrap" your operator state, or trying to have a "signaling" input channel to control the operator behaviour? If so, there are other better design patterns for these kind of Flink jobs.If possible, you can perhaps briefly describe your use

Re: ***UNCHECKED*** Table To String

I would recommend the training exercises by data Artisans [1]. They have challenging exercises and also nice solutions in Java and Scala. Flink's end-to-end test also contain a lot of good example code [2]. I hope this helps. Regards, Timo [1] http://training.data-artisans.com/ [2] https://gi

Re: Rich variant for Async IO in Scala

It should compile against a RichAsyncFunction as well. Can you open an issue on JIRA for this? Including the compiler issues that you observe. Thank you. Am 13.11.18 um 15:40 schrieb Bruno Aranda: Hi, Tried again last night. The problem is that I was trying to use org.apache.flink.streaming.

Re: Rich variant for Async IO in Scala

Hi, Tried again last night. The problem is that I was trying to use org.apache.flink.streaming.api.*scala*.AsyncDataStream, and that won't compile against the RichAsyncFunction. I could change it to use org.apache.flink.streaming.api.*datastream*.AsyncDataStream instead, but it is not as elegant a

Re: Task Manager allocation issue when upgrading 1.6.0 to 1.6.2

Good to hear Cliff. You're right that it's not a nice user experience. The problem with queryable state is that one would need to take a look at the actual user job to decide whether the user uses queryable state or not. But then it's already too late for starting the respective infrastructure nee

Field could not be resolved by the field mapping when using kafka connector

Hi, I hit the following error when I try to use kafka connector in flink table api. There's very little document about how to use kafka connector in flink table api, could anyone help me on that ? Thanks Exception in thread "main" org.apache.flink.table.api.ValidationException: Field 'event_ts' c

Rescaling Flink job from an External Checkpoint

Hi, I'm using Flink 1.5 with RocksDB State Backend for Checkpointing. According to the docs, Checkpoints do not support Job rescaling. However, when i tried to restart a job from an externalised checkpoint with a different parallelism, the job resumes correctly without any issues. I would like to

Re: Ingesting data from an API

Hi Aarti, I would imagine that the described approach (sharing the same object instance with the API and the Flink runtime) would only work in toy executions, such as executing the job within the IDE. Moreover, you would not be able to have exactly-once semantics with this source, which for most

Re: Auto/Dynamic scaling in Flink

Hi, Yet Flink does not support auto-scaling. However, there is an umbrella JIRA issue[1] to cover the discussion about it. And I think the design doc(draft) attached[2] could help. Best, tison. [1] https://issues.apache.org/jira/browse/FLINK-10407 [2] https://docs.google.com/document/d/1XKDXnrp8

Re: Auto/Dynamic scaling in Flink

Hi, Flink does not support auto-scaling, yet. Rescaling operations currently are always manual, i.e take a savepoint of the Flink job, and when restoring from the savepoint, define a new parallelism for the job. As for the metrics to be used for auto-scaling, I can imagine that it would be possibl

Re: Kinesis Shards and Parallelism

Hi, Another detail not that apparent in the description is that the assignment would only be evenly distributed assuming that the open Kinesis shards have consecutive shard ids, and are of the same Kinesis stream. Once you reshard a Kinesis stream, it could be that the shard ids are no longer cons

Re: Flink Streaming sink to InfluxDB

Hi, This is most likely an exception that indicates either that 1) you are using mismatching versions of Flink in your application code and the installed Flink cluster, or 2) your application code isn't properly packaged. >From your exception, I'm guessing it is the latter case. If so, I would sug

Re: Queryable state when key is UUID - getting Kyro Exception

Hi Jayant, could you maybe setup a small Github project with the client and server code? Otherwise it is really hard to reproduce the problem. Thanks a lot! Cheers, Till On Tue, Nov 13, 2018 at 11:29 AM Jayant Ameta wrote: > Getting the same error even when I added flink-avro dependency to the

Re: Implementation error: Unhandled exception - "Implementation error: Unhandled exception."

Hi Richard, could you share with us the complete logs to better debug the problem. What do you mean exactly with upgrading your job? Cancel with savepoint and then resuming the new job from the savepoint? Thanks a lot. Cheers, Till On Mon, Nov 12, 2018 at 5:08 PM Timo Walther wrote: > Hi Richa

Re: Queryable state when key is UUID - getting Kyro Exception

Getting the same error even when I added flink-avro dependency to the client. Jayant Ameta On Tue, Nov 13, 2018 at 2:28 PM bupt_ljy wrote: > Hi Jayant, > >I don’t know why flink uses the Avro serializer, which is usually used > in POJO class, but from the error messages, I think you can ad

Quickstart - TaskExecutor ERROR

Hi, I'm following the Quickstart instructions, but the local Flink cluster doesn't start up correctly. The web frontend is available at http://localhost:8081, but shows 0 Task Managers, 0 Task Slots and

回复: What if not to keep containers across attempts in HA setup?(Internet mail)

Hi Paul， Could you check out your YARN property "yarn.resourcemanager.work-preserving-recovery.enabled"? if value is false, set true and try it again. Best, Devin 发件人： Paul Lam 发送时间： 2018-11-13 12:55 收件人： Flink ML 主题： What if no

Re: running flink job cluster on kubernetes with HA

Its looks like in the next version 1.7 you can achieve HA on Kubernetes without zookeeper . Anyway for now you can configure one zookeeper path to save the data , the path should be some distribute FS like HDFS ,S3 fs. Thanks , Miki On Tue, No

Last batch of stream data could not be sinked when data comes very slow

Hi Experts, When we implement a sink, usually we implement a batch, according to the record number or when reaching a time interval, however this may lead to data of last batch do not write to sink. Because it is triggered by the incoming record. I also test the JDBCOutputFormat

Re: Queryable state when key is UUID - getting Kyro Exception

Hi Jayant, I don’t know why flink uses the Avro serializer, which is usually used in POJO class, but from the error messages, I think you can add flink-avro as a dependency and try again. Best, Jiayi Liao Original Message Sender:Jayant ametawittyam...@gmail.com Recipient:bupt_ljybupt_...@163

Re: ***UNCHECKED*** Table To String

Hi Steve, if you are ok with using the DataStream API you can simply use a map() function [1] and call row.toString(). However, usually people want custom logic to construct a string. This logic could either be in SQL using the concat operator `||` or in the DataStream API. Regards, Timo [1

running flink job cluster on kubernetes with HA

Hi, I want to run several jobs under kubernetes using "flink job cluster" (see - https://ci.apache.org/projects/flink/flink-docs-stable/ops/deployment/kubernetes.html#flink-job-cluster-on-kubernetes), meaning each job is running on a different flink cluster. I want to configure the cluster with H

Re: Queryable state when key is UUID - getting Kyro Exception

Thanks Jiayi, I updated the client code to use keyed stream key. The key is a Tuple2 CompletableFuture> resultFuture = client.getKvState(JobID.fromHexString("337f4476388fabc6f29bb4fb9107082c"), "rules", Tuple2.of(uuid, "test"), TypeInformation.of(new TypeHint>() { }), descripto

Flink Streaming sink to InfluxDB

Hello Team, I'm new to Flink and writing a Flink job that will take data from Kafka and sink it to InfluxDB. I tried using the concept this guy is using https://github.com/apache/bahir-flink/blob/master/flink-connector-influxdb/examples/src/main/java/org/apache/flink/streaming/examples/influxdb