Re: Watermark through Rest Api

Hi, Watermarks are tracked per Task/Operator level: https://ci.apache.org/projects/flink/flink-docs-stable/monitoring/metrics.html#io Tracking watermarks on the job level would be problematic, since it would

Re: Between Checkpoints in Kafka 11

Hi, I have nothing more to add. You (Dawid) and Vino explained it correctly :) Piotrek > On 24 Sep 2018, at 15:16, Dawid Wysakowicz wrote: > > Hi Harshvardhan, > > Flink won't buffer all the events between checkpoints. Flink uses Kafka's > transaction, which are committed only on

Re: How does flink read data from kafka number of TM's are more than topic partitions

+91 8407979163 > > > On Fri, Sep 21, 2018 at 6:26 PM Piotr Nowojski <mailto:pi...@data-artisans.com>> wrote: > Hi, > > Yes, in your case half of the Kafka source tasks wouldn’t read/process any > records (you can check that in web UI). This shouldn’t harm you, u

Re: How does flink read data from kafka number of TM's are more than topic partitions

Hi, Yes, in your case half of the Kafka source tasks wouldn’t read/process any records (you can check that in web UI). This shouldn’t harm you, unless your records will be redistributed after the source. For example: source.keyBy(..).process(new MyVeryHeavyOperator()).print() Should be fine,

Re: ***UNCHECKED*** Re: Standalone cluster instability

t; task manager process when it discovers that it has been lost? It reports that > there are no active task managers and available slots are 0. We're running on > flink version 1.4.2. > > I've attached the syslog and jobmanager log, the crash happened at Sep 18 > 23:31:14. > &

Re: OneInputStreamOperatorTestHarness.snapshot doesn't include timers?

g I should have noticed. > > — Ken > > >> On Aug 16, 2018, at 4:37 AM, Piotr Nowojski > <mailto:pi...@data-artisans.com>> wrote: >> >> Hi, >> >> You made a small mistake when restoring from state using test harness, that >> I myself

Re: How to submit flink job on yarn by java code

Hi, Is this path accessible on the container? If not, use some distributed file system, nfs or -yt —yarnship option of the cli. Please also take a look at https://lists.apache.org/thread.html/%3CCAF=1nJ8GONoqux7czxpUxAf7L3p=-E_ePSTHk0uWa=GRyG=2...@mail.gmail.com%3E

Re: How to compare two window ?

Hi, Could you rephrase your question? Maybe by posting some code examples? Piotrek > On 16 Aug 2018, at 08:26, 苗元君 wrote: > > Hi, Flink guys, > U really to a quick release, it's fantastic ! > > I'v got a situation , > window 1 is time driven, slice is 1min, trigger is 1 count > window 2

Re: Standalone cluster instability

y. > > Are there any heuristics or best practices which you (or anyone in the > community) recommend to benchmark memory requirements of a particular flink > job? > > Thanks, > Shailesh > > > On Tue, Aug 14, 2018 at 6:08 PM, Piotr Nowojski <mailto:pi...@data-

Re: OneInputStreamOperatorTestHarness.snapshot doesn't include timers?

Hi, You made a small mistake when restoring from state using test harness, that I myself have also done in the past. Problem is with an ordering of those calls: result.open(); if (savedState != null) { result.initializeState(savedState); } Open is supposed

Re: Scala 2.12 Support

Hi, Scala 2.12 support is high on our priority list and we hope to have it included for the 1.7 release (as you can see in the ticket itself), which should happen later this year. Piotrek > On 15 Aug 2018, at 17:59, Aaron Levin wrote: > > Hello! > > I'm wondering if there is anywhere I can

Re: Standalone cluster instability

fka source/sink, ~8 Window operators, and the rest CEP > operators). > > Thanks, > Shailesh > > On Fri, Aug 10, 2018 at 4:48 PM, Piotr Nowojski <mailto:pi...@data-artisans.com>> wrote: > Hi, > > Please post full TaskManager logs, including stderr and stdout. (Have

Re: Standalone cluster instability

nd anything special in the logs before the failure. > Here are the logs, please take a look: > > https://drive.google.com/drive/folders/1zlUDMpbO9xZjjJzf28lUX-bkn_x7QV59?usp=sharing > > <https://drive.google.com/drive/folders/1zlUDMpbO9xZjjJzf28lUX-bkn_x7QV59?usp=sharing> > > T

Re: Heap Problem with Checkpoints

Hi, Thanks for getting back with more information. Apparently this is a known bug of JDK since 2003 and is still not resolved: https://bugs.java.com/view_bug.do?bug_id=4872014 https://bugs.java.com/view_bug.do?bug_id=6664633

Re: How to submit a job with dependency jars by flink cli in Flink 1.4.2?

t; > It seems I have to find a way to shorten the time my users cost. > > Yours Sincerely > > Joshua > > On Fri, Aug 3, 2018 at 9:08 PM, Piotr Nowojski <mailto:pi...@data-artisans.com>> wrote: > Hi, > > -yt,--yarnship Ship files in the spe

Re: How to submit a job with dependency jars by flink cli in Flink 1.4.2?

b can not run in cluster on yarn, yt just > can not submit. > > I am trying to change the "-C > file:/home/work/xxx/lib/commons-math3-3.5.jar" to "-C > hdfs://namenode1/home/work/xxx/lib/commons-math3-3.5.jar", but Clifrontend > error was caught. > I

Re: How to submit a job with dependency jars by flink cli in Flink 1.4.2?

Hi, Are those paths: file: '/home/work/xxx/lib/commons-math3-3.5.jar' (missing) file: '/home/work/xxx/lib/flink-connector-kafka-0.8_2.11-1.4.2.jar' (missing) accessible from the inside of your container? bin/flink run --help (…) -C,--classpath Adds a URL to each

Re: Flink job hangs/deadlocks (possibly related to out of memory)

ks, > > Gerard > > On Mon, Jul 16, 2018 at 4:12 PM Piotr Nowojski <mailto:pi...@data-artisans.com>> wrote: > Hi Gerard, > > I second to what Zhijiang wrote. Please check GC pauses, either via GC > logging, 3rd party tool like jconsole (or some

Re: Flink job hangs/deadlocks (possibly related to out of memory)

Hi Gerard, I second to what Zhijiang wrote. Please check GC pauses, either via GC logging, 3rd party tool like jconsole (or some memory profiler) or via enabling resource logging in Flink. After confirming that this is not the issue next time this happens, instead of cancelling the job,

Re: Heap Problem with Checkpoints

ian Wollert > Zalando SE > > E-Mail: fabian.woll...@zalando.de > <mailto:fabian.woll...@zalando.de> > > Am Di., 19. Juni 2018 um 11:55 Uhr schrieb Piotr Nowojski > mailto:pi...@data-artisans.com>>: > Hi, > > Can you search the logs/std err/std output for

Re: Heap Problem with Checkpoints

? I > tried also using a bigger instance, but this never stabilizes, it just keeps > increasing (gave it already 10GB+ Heap) ... > > Cheers > > -- > > Fabian Wollert > Zalando SE > > E-Mail: fabian.woll...@zalando.de <mailto:fabian.woll...@zalando.de> > >

Re: [BucketingSink] notify on moving into pending/ final state

.} > public BucketingSink > registerOnPendingStateChangeCallback(FileStateChangeCallback... callbacks) > {...} > > I’m ready to discuss the best ways, how such hooks could be implemented in > the core impl or any other improvements, that will help us to add such > functionality into ou

Re: how to emit record to downstream operator in snapshotState and/or onProcessingTime

mer proposal can work. > > I was originally register timer like this. ProcessingTimeCallback interface > doesn't supply the Collector parameter > > ((StreamingRuntimeContext) getRuntimeContext()) > .getProcessingTimeService() > .registerTimer(..., this); > >

Re: Flink kafka consumer stopped committing offsets

atic way to obfuscate the logs so that I'm sure > that there isn't anything sensitive left. Any way, there isn't anything else > to share really. I wrote: "As you can see, it didn't log anything until > ~2018-06-07 22:08. Also that's where the log ends". > > Thanks once

Re: Akka version conflict running on Flink cluster

Hi, Please take a look on this thread first: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Akka-Http-used-in-custom-RichSourceFunction-td20314.html

Re: Kafka to Flink to Hive - Writes failing

Yes, BucketingSink is a better option. You can start from looking at the BucketingSink java docs. Please also take a look on this: https://stackoverflow.com/questions/47669729/how-to-write-to-orc-files-using-bucketingsink-in-apache-flink

Re: Take elements from window

Hi, Do I understand you correctly, that you just want to have a three different sliding windows (for 3 rules) with duration of 10, 20 and 30 minutes? If so, I haven’t tested it but I would guess that there are at least two solutions for the problem: 1. just create three different sliding

Re: Heap Problem with Checkpoints

Hi, What kind of messages are those “logs about S3 operations”? Did you try to google search them? Maybe it’s a known S3 issue? Another approach is please use some heap space analyser from which you can backtrack classes that are referencing those “memory leaks” and again try to google any

Re: [BucketingSink] notify on moving into pending/ final state

Hi, I see that could be a useful feature. What exactly now is preventing you from inheriting from BucketingSink? Maybe it would be just enough to make the BucketingSink easier extendable. One thing now that could collide with such feature is that Kostas is now working on larger BucketingSink

Re: Flink kafka consumer stopped committing offsets

Hi, What’s your KafkaConsumer configuration? Especially values for: - is checkpointing enabled? - enable.auto.commit (or auto.commit.enable for Kafka 0.8) / auto.commit.interval.ms - did you set setCommitOffsetsOnCheckpoints() ? Please also refer to

Re: [flink-connector-filesystem] OutOfMemory in checkpointless environment

Hi, BucketingSink is designed to provide exactly-once writes to file system, which is inherently tied to checkpointing. As you just saw, without checkpointing, BucketingSink is never notified that it can commit pending files. If you do not want to use checkpointing for some reasons, you could

Re: Conceptual question

gt; > Best Regards, > Tony Wei > > 2018-06-07 21:43 GMT+08:00 Piotr Nowojski <mailto:pi...@data-artisans.com>>: > Hi, > > Oh, I see now. Yes indeed getKeyedStateBackened() is not exposed to the > function and you can not migrate your state that way. > >

Re: Conceptual question

gration state like the manner showed in `WindowOperator`? > > Best Regards, > Tony Wei > > 2018-06-07 20:28 GMT+08:00 Piotr Nowojski <mailto:pi...@data-artisans.com>>: > What function are you implementing and how are you using it? > > Usually it’s enou

Re: Conceptual question

e migration that you > mentioned? > And are there something concerned and difficulties that will leads to > restored state failed or other problems? Thank you! > > Best Regards, > Tony Wei > > 2018-06-07 16:10 GMT+08:00 Piotr Nowojski <mailto:pi...@data-artisans.com>>

Re: Conceptual question

Hi, General solution for state/schema migration is under development and it might be released with Flink 1.6.0. Before that, you need to manually handle the state migration in your operator’s open method. Lets assume that your OperatorV1 has a state field “stateV1”. Your OperatorV2 defines

Re: Extending stream events with a an aggregate value

Hi, Ńo worries :) You probably need to write your own process function to do exactly that, maybe something like this: DataStream> test; DataStream> max = test.keyBy(0) .process(new KeyedProcessFunction, Tuple3>() { public ValueState max; @Override public void

Re: Question about JVM exit caused by timeout exception with the asynchronous IO of flink 1.4.2

Hi, You can increase a timeout, that’s one way to tackle it. In Flink 1.6.0 there will be possibility to override default Flink’s behaviour regarding handling timeouts: https://issues.apache.org/jira/browse/FLINK-7789 to handle them, instead

Re: Odd job failure

t; On Mon, May 28, 2018 at 1:48 AM, Piotr Nowojski <mailto:pi...@data-artisans.com>> wrote: > Most likely suspect is the standard java problem of some dependency > convergence issue. Please check if you are not pulling in multiple Kafka > versions into your class path. Especially

Re: Odd job failure

/flink-docs-master/ops/config.html#task-checkpoint-alignment-max-size > > <https://ci.apache.org/projects/flink/flink-docs-master/ops/config.html#task-checkpoint-alignment-max-size> > > On Wed, May 2, 2018 at 1:37 PM, Piotr Nowojski <pi...@data-artisans.com > <ma

Re: Akka Http used in custom RichSourceFunction

bsocketclient> > > Niels > > > > On Fri, May 25, 2018 at 11:00 AM Piotr Nowojski <pi...@data-artisans.com > <mailto:pi...@data-artisans.com>> wrote: > Hi, > > Yes, this might be the cause of the issue, because indeed it looks like your >

Re: Akka Http used in custom RichSourceFunction

ironment.scala:629) > at net.vankaam.flink.WebSocketSample$.main(WebSocketSample.scala:42) > at net.vankaam.flink.WebSocketSample.main(WebSocketSample.scala) > > > Cheers, > Niels > > > On Thu, May 24, 2018 at 4:08 PM Piotr Nowojski <pi...@data-artisans.com &

Re: Multiple stream operator watermark handling

arsome.lucid...@gmail.com> wrote: > > On Thu, May 24, 2018 at 9:20 AM, Elias Levy <fearsome.lucid...@gmail.com > <mailto:fearsome.lucid...@gmail.com>> wrote: > On Thu, May 24, 2018 at 7:26 AM, Piotr Nowojski <pi...@data-artisans.com > <mailto:pi...@data-artisans.com&

Re: When is 1.5 coming out

; Thanks. Do we have a timeline for when we would be out with 1.5. It seems got > delayed again ? > > On Thu, May 24, 2018 at 8:07 AM, Piotr Nowojski <pi...@data-artisans.com > <mailto:pi...@data-artisans.com>> wrote: > It seems like Flink-9349 will make it to the 1.

Re: Multiple stream operator watermark handling

Hi, From top of my head I can imagine two solutions: 1. Override the default behaviour of the operator via for example org.apache.flink.streaming.api.datastream.ConnectedStreams#transform 2. Can you set control stream’s watermark to Watermark#MAX_WATERMARK or maybe Watermark#MAX_WATERMARK - 1

Re: Akka Http used in custom RichSourceFunction

Hi, Please take a look on https://ci.apache.org/projects/flink/flink-docs-release-1.4/monitoring/debugging_classloading.html . Especially check if you are using child-first class loading

Re: FlinkKinesisProducer weird behaviour

is-kpl-consumer-deaggregation.html > > <https://docs.aws.amazon.com/streams/latest/dev/kinesis-kpl-consumer-deaggregation.html> > [2] > https://docs.aws.amazon.com/streams/latest/dev/kinesis-kpl-integration.html > <https://docs.aws.amazon.com/streams/latest/dev/kinesis-kpl-integ

Re: When is 1.5 coming out

It seems like Flink-9349 will make it to the 1.5, because RC5 was cancelled. Piotrek > On 24 May 2018, at 14:03, Vishal Santoshi wrote: > > Flink-9349 can we out this to 1.5 release. > > On Thu, May 24, 2018, 7:59 AM Vishal Santoshi

Re: Starting beam pipeline from savepoint

Hi, I am not sure, but it probably brings down to the question, whether you can restore from a save point when using RemoteEnvironment - and answer for this question is unfortunately no. Maybe Aljoscha will know something more. Piotrek > On 23 May 2018, at 19:25, borisbolvig

Re: FlinkKinesisProducer weird behaviour

Hi, Have you tried to write the same records, with exactly the same configuration to the Kinesis, but outside of Flink (with some standalone Java application)? Piotrek > On 24 May 2018, at 09:40, Rafi Aroch wrote: > > Hi, > > We're using Kinesis as our input & output

Re: Decrease initial source read speed

n out the imbalance of the assumption of record size being proportional to number of bytes. Piotrek > On 23 May 2018, at 15:40, Piotr Nowojski <pi...@data-artisans.com> wrote: > > Hi, > > Yes if you have mixed workload in your pipeline, it is matter of finding a

Re: Decrease initial source read speed

sed data later. > Changing buffer sizes globally will probably affect throughput of later > steps, as you wrote. > > >> On 23 May 2018, at 14:48, Piotr Nowojski <pi...@data-artisans.com >> <mailto:pi...@data-artisans.com>> wrote: >> >> Hi, >> >&g

Re: Decrease initial source read speed

Hi, Yes, Flink 1.5.0 will come with better tools to handle this problem. Namely you will be able to limit the “in flight” data, by controlling the number of assigned credits per channel/input gate. Even without any configuring Flink 1.5.0 will out of the box buffer less data, thus mitigating

Re: Question about the behavior of TM when it lost the zookeeper client session in HA mode

Hi, It looks like there was an error in asynchronous job of sending the records to Kafka. Probably this is a collateral damage of loosing connection to zookeeper. Piotrek > On 15 May 2018, at 13:33, Ufuk Celebi wrote: > > Hey Tony, > > thanks for the detailed report. > >

Re: Checkpoint is not triggering as per configuration

n skip the checkpoint of "Source: Custom File Source" but still > having checkpoint on "Split Reader: Custom File Source"? > Thanks, > Tao > > On Fri, May 11, 2018 at 4:34 AM, Piotr Nowojski <pi...@data-artisans.com > <mailto:pi...@data-artisans.com>>

Re: How to broadcast messages to all task manager instances in cluster?

> message is broadcast to each task manager, the data streams will see the > change. The logic in data streams is quite simple, just get some parameters > from the static variable. So I think to add connect and flatmap to each of > them is too verbose. I am wondering is there any better

Re: Late data before window end is even close

ebug futher what's causing late data on a production > stream job, please let me know. > > On Fri, May 11, 2018 at 2:18 PM, Piotr Nowojski <pi...@data-artisans.com > <mailto:pi...@data-artisans.com>> wrote: > Hey, > > Actually I think Fabian initial messag

Re: Flink Forward SF 2018 Videos

Hi, Previous videos were always uploaded there, so I guess the new one should appear there shortly. Laura might now something more about it. Thanks, Piotrek > On 10 May 2018, at 23:44, Rafi Aroch wrote: > > Hi, > > Are there any plans to upload the videos to the Flink

Re: Checkpoint is not triggering as per configuration

Hi, It’s not considered as a bug, only a missing not yet implemented feature (check my previous responses for the Jira ticket). Generally speaking using file input stream for DataStream programs is not very popular, thus this was so far low on our priority list. Piotrek > On 10 May 2018, at

Re: How to broadcast messages to all task manager instances in cluster?

Hi, I don’t quite understand your problem. If you broadcast message as an input to your operator that depends on this configuration, each instance of your operator will receive this configuration. It shouldn't matter whether Flink scheduled your operator on one, some or all of the

Re: Late data before window end is even close

Hey, Actually I think Fabian initial message was incorrect. As far as I can see in the code of WindowOperator (last lines of org.apache.flink.streaming.runtime.operators.windowing.WindowOperator#processElement ), the element is sent to late side output if it is late AND it wasn’t assigned to

Re: This server is not the leader for that topic-partition

.x. Piotrek > On 7 May 2018, at 12:12, Alexander Smirnov <alexander.smirn...@gmail.com> > wrote: > > Hi Piotr, using 0.11.0 Kafka version > > On Sat, May 5, 2018 at 10:19 AM Piotr Nowojski <pi...@data-artisans.com > <mailto:pi...@data-artisans.com>> wr

Re: This server is not the leader for that topic-partition

t; final Properties props = new Properties(); > ... > props.put(ProducerConfig.RETRIES_CONFIG, 10); > props.put(ConsumerConfig.MAX_POLL_INTERVAL_MS_CONFIG, > Integer.toString(Integer.MAX_VALUE)); > props.put(ConsumerConfig.SESSION_TIMEOUT_MS_CONFIG, 2); > > > > On

Re: PartitionNotFoundException after deployment

Ufuk: I don’t know why. +1 for your other suggestions. Piotrek > On 4 May 2018, at 14:52, Ufuk Celebi wrote: > > Hey Gyula! > > I'm including Piotr and Nico (cc'd) who have worked on the network > stack in the last releases. > > Registering the network structures

Re: intentional back-pressure (or a poor man's side-input)

s this initial state load in open()... not sure how > to work out the keying. > > I'll post back if I get anywhere with these ideas. > > On 5/3/18 10:49 AM, Piotr Nowojski wrote: >> Maybe it could work with Flink’s 1.5 credit base flow control. But you

Re: intentional back-pressure (or a poor man's side-input)

Maybe it could work with Flink’s 1.5 credit base flow control. But you would need a way to express state “block one input side of the CoProcessFunction”, pass this information up to the input gate and handle it probably similar to how `org.apache.flink.streaming.runtime.io.CachedBufferBlocker`

Re: KafkaProducer with generic (Avro) serialization schema

Hi, My Scala knowledge is very limited (and my Scala's serialization knowledge is non existent), but one way or another you have to make your SerializationSchema serialisable. If indeed this is the problem, maybe a better place to ask this question is on Stack Overflow or some scala specific

Re: ConnectedIterativeStreams and processing state 1.4.2

Hi, Why can not you use simple CoProcessFunction and handle cache updates within it’s processElement1 or processElement2 method? Piotrek > On 1 May 2018, at 10:20, Lasse Nedergaard wrote: > > Hi. > > I have a case where I have a input stream that I want to enrich

Re: Fat jar fails deployment (streaming job too large)

Short answer: could be that your job is simply too big to be serialised, distributed and deserialised in the given time and you would have to increase timeouts even more. Long answer: Do you have the same problem when you try to submit smaller job? Does your cluster work for simpler jobs?

Re: Apache Flink - Flink Forward SF 2018 - Scaling stream data pipelines (source code)

Hi, Till, do have this code somewhere? M Singh: Till is out of the office and will be back on next week, so he will probably not be able to respond for couple of days. Piotrek > On 30 Apr 2018, at 13:51, M Singh wrote: > > Hi: > > I was looking at the flink-forward sf

Re: Odd job failure

Hi, It might be some Kafka issue. From what you described your reasoning seems sound. For some reason TM3 fails and is unable to restart and process any data, thus forcing spilling on checkpoint barriers on TM1 and TM2. I don’t know the reason behind java.lang.NoClassDefFoundError:

Re: Insert data into Cassandra without Flink Cassandra connection

Hi, The only way that I can think of is if you keep your flatMap operator with parallelism 1, but that might defeat the purpose. Otherwise there is no way to open one single connection and share it across multiple TaskManagers (which can be running on different physical machines). Please

Re: Restart strategy defined in flink-conf.yaml is ignored

> > > Alex > > > On Thu, Apr 5, 2018 at 11:28 PM Piotr Nowojski <pi...@data-artisans.com > <mailto:pi...@data-artisans.com>> wrote: > Hi, > > Thanks for the details! I can confirm this behaviour. flink-conf.yaml > restart-strategy value is being co

Re: Restart strategy defined in flink-conf.yaml is ignored

ckpointingMode.EXACTLY_ONCE); > > DataStream stream = env.fromCollection(Arrays.asList("test")); > > stream.map(new MapFunction<String, String>(){ > @Override > public String map(String obj) { > throw new Null

Re: Checkpoints very slow with high backpressure

Thanks for the explanation. I hope that either 1.5 will solve your issue (please let us know if it doesn’t!) or if you can’t wait, that decreasing memory buffers can mitigate the problem. Piotrek > On 5 Apr 2018, at 08:13, Edward wrote: > > Thanks for the update Piotr. >

Re: Restart strategy defined in flink-conf.yaml is ignored

Hi, Can you provide more details, like post your configuration/log files/screen shots from web UI and Flink version being used? Piotrek > On 5 Apr 2018, at 06:07, Alexander Smirnov > wrote: > > Hello, > > I've defined restart strategy in flink-conf.yaml as

Re: Checkpoints very slow with high backpressure

-the-network-buffers <https://ci.apache.org/projects/flink/flink-docs-release-1.4/ops/config.html#configuring-the-network-buffers> On the other hand, why does it prevents you from using a checkpointing at all? Piotr Nowojski > On 5 Apr 2018, at 06:10, Edward <egb...@hotmail.com>

Re: Confluent Schema Registry DeserializationSchema

Hi, It looks like to me that kafka.utils.VerifiableProperties comes from org.apache.kafka:kafka package - please check and solve (if possible) dependency conflicts in your pom.xml regarding this package. Probably there is some version collision. Piotrek > On 21 Mar 2018, at 16:40, dim5b

Re: Error running on Hadoop 2.7

Hi, Have you replaced all of your old Flink binaries with freshly downloaded Hadoop 2.7 versions? Are you sure that something hasn't mix in the process? Does some simple word count example works on the cluster after the upgrade? Piotrek > On 21 Mar

Re: Kafka ProducerFencedException after checkpointing

t; Anyway thank you very much for the detailed explanation! > > Best, > > Dongwon > > > > On Tue, Mar 20, 2018 at 8:10 PM, Piotr Nowojski <pi...@data-artisans.com > <mailto:pi...@data-artisans.com>> wrote: > Hi, > > Please increase transaction.timeo

Re: Kafka ProducerFencedException after checkpointing

Hi, What’s your Kafka’s transaction timeout setting? Please both check Kafka producer configuration (transaction.timeout.ms property) and Kafka broker configuration. The most likely cause of such error message is when Kafka's timeout is smaller then Flink’s checkpoint interval and transactions

Re: Checkpoint is not triggering as per configuration

uhamamd Abrar Akber > MS110400027 > > On Mon, Feb 5, 2018 at 5:33 PM, Piotr Nowojski <pi...@data-artisans.com > <mailto:pi...@data-artisans.com>> wrote: > Hi, > > Did you check task manager and job manager logs for any problems? > > Piotrek > > &

Re: Too many open files on Bucketing sink

Hi, There is an open similar issue: https://issues.apache.org/jira/browse/FLINK-8707 It’s still under investigation and it would be helpful if you could follow up the discussion there, run same diagnostics commands as Alexander Gardner did

Re: Extremely large job serialization produced by union operator

Hi, Could you provide more details about your queries and setup? Logs could be helpful as well. Piotrek > On 9 Mar 2018, at 11:00, 杨力 wrote: > > I wrote a flink-sql app with following topography. > > KafkaJsonTableSource -> SQL -> toAppendStream -> Map ->

Re: "Close()" aborts last transaction in TwoPhaseCommitSinkFunction

Hi, Short answer is: no, at the moment clean shutdown is not implemented for the streaming, but it’s on our to do list for the future. Hacky answer: you could implement some custom code, that would wait for at least one completed checkpoint after the last input data. But that would require

Re: Emulate Tumbling window in Event Time Space

Hi, As Xingcan responded, you could use already built in operator for that. If you really want to implement something on your own (need custom feature? For fun?), you would have to implement some variation of a InternalTimerService from Flink (you can browse the code for an inspiration). On

Re: Does Flink support Hadoop (HDFS) 2.9 ?

Hi, You can build Flink against Hadoop 2.9: https://issues.apache.org/jira/browse/FLINK-8177 It seems like convenience binaries will be built by us only since 1.5: https://issues.apache.org/jira/browse/FLINK-8363

Re: Hi Flink Team

Hi, timeWindowAll is a non parallel operation, since it gathers all of the elements and process them together:

Re: Slow Flink program

Hi, First of all learn about what’s going with your job: check the status of the machines, cpu/network usage on the cluster. If CPU is not ~100%, analyse what is preventing the machines to work faster (network bottleneck, locking, blocking operations etc). If CPU is ~100%, profile the

Re: Python and Scala

don’t get it > to work yet. > > Esa > > From: Piotr Nowojski [mailto:pi...@data-artisans.com] > Sent: Wednesday, February 14, 2018 10:55 AM > To: Esa Heikkinen <esa.heikki...@student.tut.fi> > Cc: Esa Heikkinen <esa.heikki...@student.tut.fi>; user@flink.apache.o

Re: Python and Scala

Hi Scala REPL uses the same code as compiled library so they should work the same. Piotrek > On 13 Feb 2018, at 18:32, Esa Heikkinen <heikk...@student.tut.fi> wrote: > > Hi > > And what about the differences between Scala REPL and Scala (compiled) ? > Esa >

Re: Python and Scala

Hi, 1. Flink’s Python Batch API is not complete and it’s not on pair with Scala. 2. As for know there is no Python API for Flink Streaming, however there is some ongoing work with that: https://issues.apache.org/jira/browse/FLINK-5886 3. CEP

Re: Ceph configuration for checkpoints?

Hi, Have you tried to refer to ceph documentation? http://docs.ceph.com/docs/jewel/cephfs/hadoop/ It claims to be: > a drop-in replacement for the Hadoop File System (HDFS) So I would first try to configure ceph according to their documentation

Re: Rebalance to subtasks in same TaskManager instance

Hi, Unfortunately I don’t think it’s currently possible in the Flink. Please feel free to submit a feature request for it on our JIRA https://issues.apache.org/jira/projects/FLINK/summary Have you tried out the setup using rebalance? In

Re: Reduce parallelism without network transfer.

ducing parallelism is rebalance, and > not rescale ? > > Regards, > Kien > > Sent from TypeApp <http://www.typeapp.com/r?b=11979> > On Feb 5, 2018, at 15:28, Piotr Nowojski <pi...@data-artisans.com > <mailto:pi...@data-artisans.com>> wrote: > Hi, > &

Re: Why does jobmanager running needs slot ?

implemented this before. > > Rice. > > > > > > At 2018-02-05 17:56:49, "Piotr Nowojski" <pi...@data-artisans.com> wrote: > It seems so - but I’m saying this only basing on a annotations when this > method was added (in the last couple of months). I’m no

Re: Why does jobmanager running needs slot ?

ow. Is it a new design at FLIP6 ? > > Rice. > > > > > > At 2018-02-05 17:49:05, "Piotr Nowojski" <pi...@data-artisans.com> wrote: > I might be wrong but I think it is other way around and the naming of this > method is correct - it does exactly

Re: Why does jobmanager running needs slot ?

e better to rename it to > requestSlotsFromJobManager. I dont know whether it is sounds OKay for that. I > just feel like offerSlotToJobManager sounds strange.. What do you think of > this ? > > Rice. > > > > > > At 2018-02-05 17:30:3

Re: Why does jobmanager running needs slot ?

org.apache.flink.runtime.jobmaster.JobMaster#offerSlots is a receiver side of an RPC call that is being initiated on the sender side: org.apache.flink.runtime.taskexecutor.TaskExecutor#offerSlotsToJobManager. In other words, JobMasterGateway.offerSlots is called by a TaskManager and it is a

Re: Getting Key from keyBy() in ProcessFunction

I think now it’s not easily possible, however it might be a valid suggestion to add `OnTimerContext#getCurrentKey()` method. Besides using ValueState as you discussed before, as a some kind of a walk around you could copy and modify KeyedProcessOperator to suits your needs, but this would be

Re: Global window keyBy

Hi, FIRE_AND_PURGE triggers `org.apache.flink.api.common.state.State#clear()` call and it "Removes the value mapped under the current key.”. So other keys should remain unmodified. I hope this solves your problem/question? Piotrek > On 4 Feb 2018, at 15:39, miki haiat

<    1   2   3   4   5   6   7   >