RE: [E] Re: unable to add more servers in zookeeper quorum peers in flink 1.2
That worked.. Thanks Chesnay, [Verizon]<http://www.verizon.com/> Kanagaraj Vengidasamy RTCI 7701 E Telecom PKWY Temple Terrace, FL 33637 O 813.978.4372 | M 813.455.9757 <http://www.facebook.com/verizon> [Twitter] <http://twitter.com/verizon> [LinkedIn] <http://www.linkedin.com/company/verizon> From: Chesnay Schepler [mailto:ches...@apache.org] Sent: Wednesday, March 22, 2017 11:23 AM To: user@flink.apache.org Subject: Re: [E] Re: unable to add more servers in zookeeper quorum peers in flink 1.2 I guess that's because the grouping is wrong. ^server\.([0-9])+[[:space:]]*\=([^: \#]+) should probably be ^server\.([0-9]+)[[:space:]]*\=([^: \#]+) Could you modify the .sh script as such and try again? Regards, Chesnay On 22.03.2017 16:10, kanagaraj.vengidas...@verizon.com<mailto:kanagaraj.vengidas...@verizon.com> wrote: Greg, Sorry about that.. when I copy the config and replaced the real server names . I don't have space in my configurations. The issue is when I have server.10=, server.11= And tried to start the zookeeper, Myid replaced with 0 and 1 for these servers which is conflicting with server.0 and server.1 Thanks [Verizon]<http://www.verizon.com/> Kanagaraj Vengidasamy RTCI 7701 E Telecom PKWY Temple Terrace, FL 33637 O 813.978.4372 | M 813.455.9757 <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.facebook.com_verizon=DwMD-g=udBTRvFvXC5Dhqg7UHpJlPps3mZ3LRxpb6__0PomBTQ=vykaCRzOoltktnhRTDFL-6nA55ddeOEzGwp9gbKhlfRrUojHU0nRm-mt29N20zWv=8K9Z7ArBnvYZL6fAHHu5u1r_-ezNyqdayG6uUQ8Wpqc=gcE_OOjlAoAmneDnyY7lTkUL_wVW8XyzeRjhTyhbuHw=> [Twitter] <https://urldefense.proofpoint.com/v2/url?u=http-3A__twitter.com_verizon=DwMD-g=udBTRvFvXC5Dhqg7UHpJlPps3mZ3LRxpb6__0PomBTQ=vykaCRzOoltktnhRTDFL-6nA55ddeOEzGwp9gbKhlfRrUojHU0nRm-mt29N20zWv=8K9Z7ArBnvYZL6fAHHu5u1r_-ezNyqdayG6uUQ8Wpqc=tqZxl_Osm7Nl_TMH5fn6uLqMmYwzWMRmIY0a6Cv94XE=> [LinkedIn] <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.linkedin.com_company_verizon=DwMD-g=udBTRvFvXC5Dhqg7UHpJlPps3mZ3LRxpb6__0PomBTQ=vykaCRzOoltktnhRTDFL-6nA55ddeOEzGwp9gbKhlfRrUojHU0nRm-mt29N20zWv=8K9Z7ArBnvYZL6fAHHu5u1r_-ezNyqdayG6uUQ8Wpqc=YbbS1mTqYCQNadmBvLZ9SoaBM-myrHL3BE_dj_yp8DM=> From: Greg Hogan [mailto:c...@greghogan.com] Sent: Wednesday, March 22, 2017 10:08 AM To: user@flink.apache.org<mailto:user@flink.apache.org> Subject: [E] Re: unable to add more servers in zookeeper quorum peers in flink 1.2 Kanagaraj, None of the server lines are matching since the regex in start-zookeeper-quorum.sh does not allow for spaces after the equals sign. ^server\.([0-9])+[[:space:]]*\=([^: \#]+) Greg On Mar 22, 2017, at 8:49 AM, kanagaraj.vengidas...@verizon.com<mailto:kanagaraj.vengidas...@verizon.com> wrote: Hi All, We are using flink 1.2 . Unable to add more than one digit in server.x. when I have more than one digit it is not allowing to start the zookeeper. What I need to do if I want to keep more servers? # ZooKeeper quorum peers server.0=server1:2888:3888 server.1=server2:2888:3888 server.2=server3:2888:3888 server.3=server4:2888:3888 server.4=server5:2888:3888 server.5=server6:2888:3888 server.6=server7:2888:3888 server.7=server8:2888:3888 server.8=server9:2888:3888 server.9=server10:2888:3888 #server.10=server11:2888:3888 #server.11=server12:2888:3888 Thanks Kanagaraj
RE: [E] Re: unable to add more servers in zookeeper quorum peers in flink 1.2
Greg, Sorry about that.. when I copy the config and replaced the real server names . I don't have space in my configurations. The issue is when I have server.10=, server.11= And tried to start the zookeeper, Myid replaced with 0 and 1 for these servers which is conflicting with server.0 and server.1 Thanks [Verizon]<http://www.verizon.com/> Kanagaraj Vengidasamy RTCI 7701 E Telecom PKWY Temple Terrace, FL 33637 O 813.978.4372 | M 813.455.9757 <http://www.facebook.com/verizon> [Twitter] <http://twitter.com/verizon> [LinkedIn] <http://www.linkedin.com/company/verizon> From: Greg Hogan [mailto:c...@greghogan.com] Sent: Wednesday, March 22, 2017 10:08 AM To: user@flink.apache.org Subject: [E] Re: unable to add more servers in zookeeper quorum peers in flink 1.2 Kanagaraj, None of the server lines are matching since the regex in start-zookeeper-quorum.sh does not allow for spaces after the equals sign. ^server\.([0-9])+[[:space:]]*\=([^: \#]+) Greg On Mar 22, 2017, at 8:49 AM, kanagaraj.vengidas...@verizon.com<mailto:kanagaraj.vengidas...@verizon.com> wrote: Hi All, We are using flink 1.2 . Unable to add more than one digit in server.x. when I have more than one digit it is not allowing to start the zookeeper. What I need to do if I want to keep more servers? # ZooKeeper quorum peers server.0=server1:2888:3888 server.1=server2:2888:3888 server.2=server3:2888:3888 server.3=server4:2888:3888 server.4=server5:2888:3888 server.5=server6:2888:3888 server.6=server7:2888:3888 server.7=server8:2888:3888 server.8=server9:2888:3888 server.9=server10:2888:3888 #server.10=server11:2888:3888 #server.11=server12:2888:3888 Thanks Kanagaraj
RE: [E] writeBufferLowWaterMark cannot be greater than writeBufferHighWaterMark error
Hi Fabian, In my case, Since it is stream processing, after that error , that task manager stuck and not taking any new messages. Can you let me know how many task managers can I run in 8 core x 32GB machine.? I am using Heap size as 4096 for each task manager. Thanks [Verizon]<http://www.verizon.com/> Kanagaraj Vengidasamy RTCI 7701 E Telecom PKWY Temple Terrace, FL 33637 O 813.978.4372 | M 813.455.9757 <http://www.facebook.com/verizon> [Twitter] <http://twitter.com/verizon> [LinkedIn] <http://www.linkedin.com/company/verizon> From: Fabian Hueske [mailto:fhue...@gmail.com] Sent: Tuesday, December 27, 2016 2:53 PM To: user@flink.apache.org Cc: Ufuk Celebi Subject: Re: [E] writeBufferLowWaterMark cannot be greater than writeBufferHighWaterMark error Hi, I reproduced the issue with Flink 1.1.4 and the 1.2.0 release branch. The WARN log statement and the IllegalArgumentException are thrown by Netty. Not sure what the implications are. My batch jobs finished successfully, so maybe the bad configuration options are just ignored. Would be good to check this though. Thanks, Fabian 2016-12-27 16:10 GMT+01:00 <kanagaraj.vengidas...@verizon.com<mailto:kanagaraj.vengidas...@verizon.com>>: Thanks Fabian, For default 65536 also got the same error . That is the reason increased to see it is helping or not. Thanks [Verizon]<http://www.verizon.com/> Kanagaraj Vengidasamy RTCI 7701 E Telecom PKWY Temple Terrace, FL 33637 O 813.978.4372<tel:(813)%20978-4372> | M 813.455.9757<tel:(813)%20455-9757> <http://www.facebook.com/verizon> [Twitter] <http://twitter.com/verizon> [LinkedIn] <http://www.linkedin.com/company/verizon> From: Fabian Hueske [mailto:fhue...@gmail.com<mailto:fhue...@gmail.com>] Sent: Tuesday, December 27, 2016 10:01 AM To: user@flink.apache.org<mailto:user@flink.apache.org> Cc: Ufuk Celebi Subject: Re: [E] writeBufferLowWaterMark cannot be greater than writeBufferHighWaterMark error Hi Kanagaraj, I would assume that the issue is caused by this configuration parameter: taskmanager.memory.segment-size: 131072 I think the maximum possible value given Netty's "writeBufferHighWaterMark" parameter is 65536. There might be a way to tune Netty's parameters but I don't know how to do that. Maybe Ufuk (in CC) knows better. Is there any particular reason why want to increase the network buffer size instead of keeping the default? Did you find that the default parameter gives insufficient performance? Best, Fabian 2016-12-26 20:52 GMT+01:00 <kanagaraj.vengidas...@verizon.com<mailto:kanagaraj.vengidas...@verizon.com>>: Hi, I am getting writeBufferLowWaterMark cannot be greater than writeBufferHighWaterMark error frequently ... and those task managers not processing messages after that error. What could be wrong in my configuration? What I need to do to avoid this error.? Have 8x32 VM's - 8 machines ( running 35 task managers - each one has 8 slots) taskmanager.heap.mb: 4096 taskmanager.numberOfTaskSlots: 8 taskmanager.network.numberOfBuffers: 22000 taskmanager.memory.segment-size: 131072 2016-12-26 14:26:06,548| |WARN io.netty.bootstrap.ServerBootstrap - Failed to set a channel option: [id: 0xf1ef59e6, /138.83.31.4:60812<http://138.83.31.4:60812> => /138.83.31.9:41304<http://138.83.31.9:41304>] java.lang.IllegalArgumentException: writeBufferLowWaterMark cannot be greater than writeBufferHighWaterMark (65536): 131073 at io.netty.channel.DefaultChannelConfig.setWriteBufferLowWaterMark(DefaultChannelConfig.java:334) at io.netty.channel.socket.DefaultSocketChannelConfig.setWriteBufferLowWaterMark(DefaultSocketChannelConfig.java:332) at io.netty.channel.socket.DefaultSocketChannelConfig.setWriteBufferLowWaterMark(DefaultSocketChannelConfig.java:35) at io.netty.channel.DefaultChannelConfig.setOption(DefaultChannelConfig.java:183) at io.netty.channel.socket.DefaultSocketChannelConfig.setOption(DefaultSocketChannelConfig.java:121) at io.netty.bootstrap.ServerBootstrap$ServerBootstrapAcceptor.channelRead(ServerBootstrap.java:238) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:847) at io.netty.channel.nio.AbstractNioMessageChannel$NioMessageUnsafe.read(AbstractNioMessageChannel.java:93) at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511) at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468) at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382) at io.netty
RE: [E] writeBufferLowWaterMark cannot be greater than writeBufferHighWaterMark error
Thanks Fabian,
For default 65536 also got the same error . That is the reason
increased to see it is helping or not.
Thanks
[Verizon]<http://www.verizon.com/>
Kanagaraj Vengidasamy
RTCI
7701 E Telecom PKWY
Temple Terrace, FL 33637
O 813.978.4372 | M 813.455.9757
<http://www.facebook.com/verizon> [Twitter] <http://twitter.com/verizon>
[LinkedIn] <http://www.linkedin.com/company/verizon>
From: Fabian Hueske [mailto:fhue...@gmail.com]
Sent: Tuesday, December 27, 2016 10:01 AM
To: user@flink.apache.org
Cc: Ufuk Celebi
Subject: Re: [E] writeBufferLowWaterMark cannot be greater than
writeBufferHighWaterMark error
Hi Kanagaraj,
I would assume that the issue is caused by this configuration parameter:
taskmanager.memory.segment-size: 131072
I think the maximum possible value given Netty's "writeBufferHighWaterMark"
parameter is 65536.
There might be a way to tune Netty's parameters but I don't know how to do
that. Maybe Ufuk (in CC) knows better.
Is there any particular reason why want to increase the network buffer size
instead of keeping the default?
Did you find that the default parameter gives insufficient performance?
Best, Fabian
2016-12-26 20:52 GMT+01:00
<kanagaraj.vengidas...@verizon.com<mailto:kanagaraj.vengidas...@verizon.com>>:
Hi,
I am getting writeBufferLowWaterMark cannot be greater than
writeBufferHighWaterMark error frequently ... and those task managers not
processing messages after that error.
What could be wrong in my configuration? What I need to do to avoid this
error.?
Have 8x32 VM's - 8 machines ( running 35 task managers - each one has 8 slots)
taskmanager.heap.mb: 4096
taskmanager.numberOfTaskSlots: 8
taskmanager.network.numberOfBuffers: 22000
taskmanager.memory.segment-size: 131072
2016-12-26 14:26:06,548| |WARN io.netty.bootstrap.ServerBootstrap - Failed
to set a channel option: [id: 0xf1ef59e6,
/138.83.31.4:60812<http://138.83.31.4:60812> =>
/138.83.31.9:41304<http://138.83.31.9:41304>]
java.lang.IllegalArgumentException: writeBufferLowWaterMark cannot be greater
than writeBufferHighWaterMark (65536): 131073
at
io.netty.channel.DefaultChannelConfig.setWriteBufferLowWaterMark(DefaultChannelConfig.java:334)
at
io.netty.channel.socket.DefaultSocketChannelConfig.setWriteBufferLowWaterMark(DefaultSocketChannelConfig.java:332)
at
io.netty.channel.socket.DefaultSocketChannelConfig.setWriteBufferLowWaterMark(DefaultSocketChannelConfig.java:35)
at
io.netty.channel.DefaultChannelConfig.setOption(DefaultChannelConfig.java:183)
at
io.netty.channel.socket.DefaultSocketChannelConfig.setOption(DefaultSocketChannelConfig.java:121)
at
io.netty.bootstrap.ServerBootstrap$ServerBootstrapAcceptor.channelRead(ServerBootstrap.java:238)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339)
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324)
at
io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:847)
at
io.netty.channel.nio.AbstractNioMessageChannel$NioMessageUnsafe.read(AbstractNioMessageChannel.java:93)
at
io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
at
io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
at
io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
at
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
at java.lang.Thread.run(Thread.java:745)
Thanks
Kanagaraj Vengidasamy
RTCI
7701 E Telecom PKWY
Temple Terrace, FL 33637
O 813.978.4372 | M 813.455.9757
RE: [E] writeBufferLowWaterMark cannot be greater than writeBufferHighWaterMark error
Hi, I am getting writeBufferLowWaterMark cannot be greater than writeBufferHighWaterMark error frequently ... and those task managers not processing messages after that error. What could be wrong in my configuration? What I need to do to avoid this error.? Have 8x32 VM's - 8 machines ( running 35 task managers - each one has 8 slots) taskmanager.heap.mb: 4096 taskmanager.numberOfTaskSlots: 8 taskmanager.network.numberOfBuffers: 22000 taskmanager.memory.segment-size: 131072 2016-12-26 14:26:06,548| |WARN io.netty.bootstrap.ServerBootstrap - Failed to set a channel option: [id: 0xf1ef59e6, /138.83.31.4:60812 => /138.83.31.9:41304] java.lang.IllegalArgumentException: writeBufferLowWaterMark cannot be greater than writeBufferHighWaterMark (65536): 131073 at io.netty.channel.DefaultChannelConfig.setWriteBufferLowWaterMark(DefaultChannelConfig.java:334) at io.netty.channel.socket.DefaultSocketChannelConfig.setWriteBufferLowWaterMark(DefaultSocketChannelConfig.java:332) at io.netty.channel.socket.DefaultSocketChannelConfig.setWriteBufferLowWaterMark(DefaultSocketChannelConfig.java:35) at io.netty.channel.DefaultChannelConfig.setOption(DefaultChannelConfig.java:183) at io.netty.channel.socket.DefaultSocketChannelConfig.setOption(DefaultSocketChannelConfig.java:121) at io.netty.bootstrap.ServerBootstrap$ServerBootstrapAcceptor.channelRead(ServerBootstrap.java:238) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:847) at io.netty.channel.nio.AbstractNioMessageChannel$NioMessageUnsafe.read(AbstractNioMessageChannel.java:93) at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511) at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468) at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354) at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111) at java.lang.Thread.run(Thread.java:745) Thanks Kanagaraj Vengidasamy RTCI 7701 E Telecom PKWY Temple Terrace, FL 33637 O 813.978.4372 | M 813.455.9757
