[
https://issues.apache.org/jira/browse/NIFI-1626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15195241#comment-15195241
]
Matthew Clarke edited comment on NIFI-1626 at 3/15/16 1:01 PM:
---------------------------------------------------------------
So i saw some discussions on line where people suggested breaking messages
larger than 1MB in to multiple requests to zookeeper. I saw nothing that said
how to do this. Feel like zookeeper set caps and said we will just fail fast
if they are exceeded. Good news is that i think we figured out the
snapshot/transaction log rolling thing. We need to set the snapCount and
should probably add this to the zookeeper.properties file. If not set it
defaults to 100,000 events. We set it to 1,000 in our tests with listHDFS
using "snapCount=1000". First couple logs rolled at ~129MB and then each after
that got a larger and larger (my guess is because of the increasing size of the
events). So when a single event gets large, the transaction log size
skyrockets. Even when zookeepers fails, the transaction log still continues to
record events (even the large ones of 1MB). It too eventually fails. Not sure
what is the ultimate trigger there.
was (Author: msclarke):
So i saw some discussions on line where people suggested breaking messages
larger than 1MB in to multiple requests to zookeeper. I saw nothing that said
how to do this. Feel like zookeeper set caps and said we will just fail fast
if they are exceeded. Good news is that i think we figured out the
snapshot/transaction log rolling thing. We need to set the snapCount and
should probably add this to the zookeeper.properties file. If not set it
defaults to 10,000 events. We set it to 1,000 in our tests with listHDFS using
"snapCount=1000". First couple logs rolled at ~129MB and then each after that
got a larger and larger (my guess is because of the increasing size of the
events). So when a single event gets large, the transaction log size
skyrockets. Even when zookeepers fails, the transaction log still continues to
record events (even the large ones of 1MB). It too eventually fails. Not sure
what is the ultimate trigger there.
> ZooKeeper State Provider should block requests > 1 MB
> -----------------------------------------------------
>
> Key: NIFI-1626
> URL: https://issues.apache.org/jira/browse/NIFI-1626
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
> Reporter: Mark Payne
> Assignee: Mark Payne
> Fix For: 0.6.0
>
>
> ZooKeeper does not support setting data on a ZNode to a value greater than 1
> MB. However, the client doesn't does not check this - it sends the data to
> the server and lets the server reject the request. This can cause problems
> with ZooKeeper if the requests grow extremely large. As a result, we should
> prevent this from ever happening by throwing an Exception ourselves if
> attempting to set state to a value > 1 MB
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
