RE: [E] writeBufferLowWaterMark cannot be greater than writeBufferHighWaterMark error

Hi Fabian, In my case, Since it is stream processing, after that error , that task manager stuck and not taking any new messages. Can you let me know how many task managers can I run in 8 core x 32GB machine.? I am using Heap size as 4096 for each task manager. Thanks

Re: [E] writeBufferLowWaterMark cannot be greater than writeBufferHighWaterMark error

Hi, I reproduced the issue with Flink 1.1.4 and the 1.2.0 release branch. The WARN log statement and the IllegalArgumentException are thrown by Netty. Not sure what the implications are. My batch jobs finished successfully, so maybe the bad configuration options are just ignored. Would be good

Re: Monitoring REST API

Although Flink exposes some metrics in the API/UI, it probably only does that because it was easy to do and convenient for users. However, I don't think Flink is intended to be a complete monitoring solution for your cluster. Instead, you should take a look at collectd https://collectd.org/

Re: Reading worker-local input files

Hi Robert, this is indeed a bit tricky to do. The problem is mostly with the generation of the input splits, setup of Flink, and the scheduling of tasks. 1) you have to ensure that on each worker at least one DataSource task is scheduled. The easiest way to do this is to have a bare metal setup

RE: [E] writeBufferLowWaterMark cannot be greater than writeBufferHighWaterMark error

Thanks Fabian, For default 65536 also got the same error . That is the reason increased to see it is helping or not. Thanks [Verizon] Kanagaraj Vengidasamy RTCI 7701 E Telecom PKWY Temple Terrace, FL 33637 O 813.978.4372 | M 813.455.9757

Re: Is there some way to use broadcastSet in streaming ?

Hi, no, broadcast sets are not available in the DataStream API. There might be other ways to achieve similar functionality, but the optimal solution depends on the use case. If you give a few details about what you would like to do, we might be able to suggest alternatives. Best, Fabian

Re: [E] writeBufferLowWaterMark cannot be greater than writeBufferHighWaterMark error

Hi Kanagaraj, I would assume that the issue is caused by this configuration parameter: taskmanager.memory.segment-size: 131072 I think the maximum possible value given Netty's "writeBufferHighWaterMark" parameter is 65536. There might be a way to tune Netty's parameters but I don't know how to

Re: Compiling Flink for Scala 2.11

Hi Markus, thanks for reporting this issue. This bug was introduced when the opt.xml file was added to the repository a few days ago. There are two open JIRAs, FLINK-5392 and FLINK-5396, each one with a pull request to fix the problem. Best, Fabian 2016-12-26 2:16 GMT+01:00 M. Dale

Reading worker-local input files

Hi everyone, I'm using Flink and/or Hadoop on my cluster, and I'm having them generate log data in each worker node's /local folder (regular mount point). Now I would like to process these files using Flink, but I'm not quite sure how I could tell Flink to use each worker node's /local folder as