jobmanager rpc inside kubernetes

when trying to submit a job to my k8s cluster using `flink run -m localhost:6123 app.jar` i get the following error in jobmanager log: - dropping message [class akka.actor.ActorSelectionMessage] for non-local recipient [Actor[akka.tcp://flink@localhost:6123/]] arriving at

Re: Measure End-to-End latency/delay for each record

Ok thanks Michael for all your help! -- Dhruv Kumar PhD Candidate Department of Computer Science and Engineering University of Minnesota www.dhruvkumar.me > On Apr 26, 2018, at 19:24, TechnoMage wrote: > > Yes, Kafka for

Re: Measure End-to-End latency/delay for each record

Yes, Kafka for source and sink which makes monitoring the Flink in/out easy. Michael > On Apr 26, 2018, at 5:27 PM, Dhruv Kumar wrote: > > Ok that answers my questions. > > What are you keeping the source and sink as? Is it Kafka for both? > >

Re: Measure End-to-End latency/delay for each record

Ok that answers my questions. What are you keeping the source and sink as? Is it Kafka for both? -- Dhruv Kumar PhD Candidate Department of Computer Science and Engineering University of Minnesota www.dhruvkumar.me > On Apr 26, 2018, at 16:37,

coordinate watermarks between jobs?

Hi All, I am trying to reply events from 3 different sources and hopefully in time sequence, say Stream1, Stream2, Stream3. Since their size vary a lot, the watermarks on one stream is much faster than other streams. Is there any way to coordinate the watermarks between different input streams.

Re: Measure End-to-End latency/delay for each record

Yes NTP can still have skew. It may be measured in fractions of a second, but with Flink that can be significant if you care about sub-second latency accuracy. Since I have a 20 stage stream with 0.002 second latency it can matter. Back pressure is the limiting of input due to the inability

Re: KafkaProducer with generic (Avro) serialization schema

Hi Bill, Thanks for your answer. However this proposal isn't going to solve my issue, since the problem here is that the context bounds I need to give in order to serialize it to Avro (SchemaFor, ToRecord and FromRecord) aren't serializable classes. This results in Flink not being able to

Re: Multiple Streams Connect Watermark

You can also merge all three types into an nary-Either type and union all three inputs together. However, Flink only supports a binary Either, so you'd have to implement a custom TypeInformation and TypeSerializer to make that work. Best, Fabian 2018-04-26 20:44 GMT+02:00 Chengzhi Zhao

Apache Flink Examples

Hi I have been exploring Apache Flink for sometime now and I notice that although the documentation has good amount of information, there may not be sufficient examples (code snippets) which quickly explain what a particular feature will do. A good example which comes to my mind is Plotly

Re: Multiple Streams Connect Watermark

Thanks Fabian for the explanation. If I have data with different schemas, it seems the only option I have is to use connect to perform joins (inner, outer), is there any operators that can put more than two streams together (all different schema)? Best, Chengzhi On Thu, Apr 26, 2018 at 6:05 AM,

Re: Measure End-to-End latency/delay for each record

What do you mean by the time skew from one machine(source) to another(sink)? Do you mean the system time clocks of the source and sink may not be in sync. If I regularly use NTP to keep the system clocks in sync, will time skew still happen? Could you also elaborate on what do you mean by back

Re: Measure End-to-End latency/delay for each record

In a single machine system this may work ok. In a multi-machine system this is not as reliable as the time skew from one machine (source) to another (sink) can impact the measurements. This also does not account for back presure on the source. We are using an external process to in parallel

Measure End-to-End latency/delay for each record

Hi I was trying to compute the end-to-end-latency for each record processed by Flink. By end-to-end latency, I mean the difference between the time at which the record entered the Flink system (came at source) and the time at which the record is finally emitted into the sink. What is the best

Re: Setting the parallelism in a cluster of machines properly

On Thu, Apr 26, 2018 at 10:47 AM, Makis Pap wrote: > OK Michael! > > I will look into it and will come back at you! Thanks for the help. I > agree that it is quite suspicious the par = 8 > > Jps? Meaning? > jps is a tool that comes with JDK (see $JAVA_HOME/bin). This is

Re: Setting the parallelism in a cluster of machines properly

OK Michael! I will look into it and will come back at you! Thanks for the help. I agree that it is quite suspicious the par = 8 Jps? Meaning? Oh I should mention that the JobManager node is also a TaskManager. Best, Max > On 27 Apr 2018, at 01:39, TechnoMage wrote: >

Re: Setting the parallelism in a cluster of machines properly

Check that you have slaves and masters set correctly on all machines, and in particular the one submitting jobs. Make sure that from the machine submitting the job that it is talking to the correct job manager (jobmanager.rpc.address). It really sounds like you are some how submitting jobs to

Re: Setting the parallelism in a cluster of machines properly

You need to verify your configs are correct. Check that the local machine sees all the task managers, that is the most likely reason it will reject a higher parallelism. I use a java program to submit to a 3 node 18 slot cluster without issue on a job with 18 parallelism. I have not used the

Re: Setting the parallelism in a cluster of machines properly

No man. I have 17 TaskManagers and each has a number of 8 slots. Do you think it is better to have 8 TaskManager (1 slot each) ? Best, Max -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: Setting the parallelism in a cluster of machines properly

Go to the web UI and verify all 136 TaskManagers are visible in the machine you are submitting the job from. I have encountered issues where not all TaskManagers start, or you may not have all 17 configured properly to be one cluster vs 17 clusters of 8. Michael > On Apr 26, 2018, at 10:48

Setting the parallelism in a cluster of machines properly

Hello Flinkers, I have deployed Flink in a cluster of 17 nodes, each having 8 CPUs. Thus, in total there are 136 CPUs available. I have set the parameter askmanager.numberOfTaskSlots = 8 in all machines, since they have 8 CPUs. And when I am going to run ./flink run -c classpath jarFile -p 136

Re: Class loading issues when using Remote Execution Environment

Thanks Chesnay for your incredible help! I will try out the suggestions again. A few questions: - What version of Flink are you trying with? I have had issues when I placed the gradoop-demo-shaded.jar in the lib folder on Flink installation (1.4 even refused to start!). - Are there other config

Re: Different result on running Flink in local mode and Yarn cluster

Not knowing the library or the config needs I do not have a suggestion. If the config is accumulated from inputs and needs to see all inputs I would suggest setting parallelism to 1 as an experiment, but it would need a redesign to run in parallel. Michael Sent from my iPad > On Apr 26,

Question regarding refreshing KafkaConsumer in FlinkKafkaConnector

Hi Flink support, I'm trying to use FlinkKafkaConnector010 to do the streaming from KafkaBroker Sasl. Currently, I got problem with Token refreshing every 24hours that's why I want restart the KafkaConsumer every 24hours. Since KafkaConsumer is wrapped by FlinkKafkaConnector010, I'm trying to

Re: Using RocksDB as State Backend over a Distributed File System

Thanks a lot Stefan. This clarifies everything. Regards, Chirag  On Thursday, 26 April, 2018, 7:16:52 PM IST, Stefan Richter wrote: Adding one thing, the format of the non-incremental is similar but unfortunately not (yet) identical with the FS backend.

Re: Help with OneInputStreamOperatorTestHarness

Hi Fabian, I created FLINK-9262 . FYI, - Chris > On Apr 26, 2018, at 3:07 AM, Fabian Hueske wrote: > > Thanks for reporting the issue Chris! > Would you mind opening a JIRA issue [1] to track the bug for Flink 1.4? > >

Re: Using RocksDB as State Backend over a Distributed File System

Adding one thing, the format of the non-incremental is similar but unfortunately not (yet) identical with the FS backend. This is because of some internal implementation details that allow the FS checkpoints to be slightly more consise in the file format but we might „de-optimize“ this minor

Re: Beam quickstart

For some reason it only seems to work if I put my jars in the Flink lib folder. I am not sure why though... Gyula Jörn Franke ezt írta (időpont: 2018. ápr. 25., Sze, 16:50): > Tried with a fat jar to see if it works in general ? > > > On 25. Apr 2018, at 15:32, Gyula Fóra

Re: Using RocksDB as State Backend over a Distributed File System

On the local disk you have the normal RocksDB working directory consisting mainly of the SSTable files. In the checkpoint directory on distributed storage it depends on whether or not you are using incremental checkpoints. For incremental checkpoints, the files are essentially the SSTables

Re: Insert data into Cassandra without Flink Cassandra connection

Here is my code stream.flatMap(new FlatMapFunction() { @Override public void flatMap(byte[] value, Collector out) throws Exception { Parser.setInsert(true); CassandraConnection.connect(); Parser.setInsert(true);

Re: Using RocksDB as State Backend over a Distributed File System

Wow never considered it that way.  Thanks a lot for clarifying Stefan. This gives rise to another question. Whats the format of this data? Is it the same format which is used to store checkpoints when FS state backend is used? Regards, Chirag Sent from Yahoo Mail on Android On Thu, 26 Apr

Re: Can't send kafka message with timestamp

Hi, I think the problem is the SerializationSchema parameter. Best, QIngxiang Ma. 2018-04-26 20:59 GMT+08:00 Alexander Smirnov : > Hi, > > > I'm creating kafka producer with timestamps enabled following instructions at >

Re: Using RocksDB as State Backend over a Distributed File System

Hi, I'm agree with Stefan. I think you can look at this document, given here: Apache Flink 1.4 Documentation：Checkpointing Best, Qingxiang Ma. 2018-04-26 20:00 GMT+08:00 Stefan Richter

Can't send kafka message with timestamp

Hi, I'm creating kafka producer with timestamps enabled following instructions at https://ci.apache.org/projects/flink/flink-docs-release-1.4/dev/connectors/kafka.html#kafka-producer Optional customPartitioner = Optional.empty(); FlinkKafkaProducer011

Consumed input splits

Hi to all, is there a way to see from the Flink UI the number of procesed splits o a source? For example, I'm reading data from a (batch) JDBC input and Flink creates 50 splits. However I don't know how many of them have been consumed and which one is waiting to be processed. Best, Flavio

Insert data into Cassandra without Flink Cassandra connection

I want to use Cassandra native connection (Not Flink Cassandra connection) to insert some data into Cassandra. According to the design of the code, the connection to Cassandra will open once at the start and all taskmanager use it to write data. It's ok running in local mode. The problem is when

RE: use of values of previously accepted event

Hi Or is it possible to use global or local variables inside in pattern sequence ? And how (by Scala) ? Best, Esa From: Esa Heikkinen Sent: Wednesday, April 25, 2018 4:16 PM To: user@flink.apache.org Subject: CEP: use of values of previously accepted event Hi I

Re: Using RocksDB as State Backend over a Distributed File System

Hi, I think there is a misunderstanding. RocksDB state backend always operates on local disk of the node that runs your task to give you optimal performance. You can think of this as a transient working area that does not require any durability. Durability always happens through checkpoints

Using RocksDB as State Backend over a Distributed File System

Hi, I am working on a use case where I need to store a large amount of data in state. I am using RocksDB as my state backend. Now to ensure data replication, I want to store the RocksDB files in some distributed file system. >From the documentation I can see that Flink recommends a list of

Re: data enrichment with SQL use case

Hi all, @Ken, the approach of telling the operator which input to read from would cause problems with the current checkpointing mechanism because checkpoint barriers are not allowed to overtake regular records. Chaining wouldn't be an issue, because operators with two inputs are not chained to

Re: Class loading issues when using Remote Execution Environment

Small update: I could reproduce your problems locally when submitting the fat-jar. I could get the job to run after placing the gradoop-demo-shaded.jar into the lib folder. I have not tried yet placing only the gradoop jars into lib (but my guess is you missed a gradoop jar) Note that the

Re: Help with OneInputStreamOperatorTestHarness

Thanks for reporting the issue Chris! Would you mind opening a JIRA issue [1] to track the bug for Flink 1.4? Thank you, Fabian [1] https://issues.apache.org/jira/browse/FLINK 2018-04-25 21:11 GMT+02:00 Chris Schneider : > Hi Gang, > > FWIW, the code below works

Re: Multiple Streams Connect Watermark

Hi Chengzhi, Functions in Flink are implemented in a way to preserve the timestamps of elements or assign timestamps which are aligned with the existing watermarks. For example, the result of a time window aggregation has the end timestamp of the window as a timestamp and records emitted by the

Re: Externalized checkpoints and metadata

Hi Juan, I think you are right and there maybe more then 3 companies implementing different solutions for this...I created a ticket to address it here https://issues.apache.org/jira/browse/FLINK-9260. Hope this could help to reduce other's redundant efforts on this...(If it could be

Re: Externalized checkpoints and metadata

Hello all, Thank you all for your responses, I’ll take a look at your code Hao, and probably implement something similar. I’d like to ask though, so as to know what we could expect from Flink in the future, if this issue will be addressed somehow, considering that we have already 3 different

Re: Different result on running Flink in local mode and Yarn cluster

Only the anonymous FlatMapFunction instance is sent to the TaskManager. Move the static field to that class. Michael Sent from my iPad > On Apr 25, 2018, at 10:42 PM, Soheil Pourbafrani > wrote: > > I run a code using Flink Java API that gets some bytes from Kafka