[jira] [Commented] (NIFI-6501) Add config to limit buffer queue size in CaptureChangeMySQL
[ https://issues.apache.org/jira/browse/NIFI-6501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17677997#comment-17677997 ] ASF subversion and git services commented on NIFI-6501: --- Commit 8986f4b33a86e9bd646b7c20da852610e9f7d5ea in nifi's branch refs/heads/main from Matt Burgess [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=8986f4b33a ] NIFI-6501 Refactor CaptureChangeMySQL with bounded queue of 1000 This closes #6791 Signed-off-by: David Handermann > Add config to limit buffer queue size in CaptureChangeMySQL > --- > > Key: NIFI-6501 > URL: https://issues.apache.org/jira/browse/NIFI-6501 > Project: Apache NiFi > Issue Type: Bug > Components: Configuration >Affects Versions: 1.9.2, 1.14.0, 1.13.1, 1.13.2 >Reporter: Purushotham Pushpavanthar >Assignee: Purushotham Pushpavanthar >Priority: Critical > Labels: easyfix > Attachments: image-2019-08-02-11-29-10-829.png > > Time Spent: 3h 50m > Remaining Estimate: 0h > > CaptureChangeMySQL processor registers a listener with Blocking Queue as > buffer with the BinLogClient. When the thread polling from the Queue is > slower compared to writer, the queue grows uncontrollably and brings down the > node. > Since the flow writing to listeners in > [mysql-binlog-connector-java|[https://github.com/shyiko/mysql-binlog-connector-java]] > is blocking, we should initialize the queue with *initial size* and *queue > offer timeout* specified by the user based in cluster configuration. > [http://apache-nifi-developer-list.39713.n7.nabble.com/NiFi-Cluster-crashes-while-running-CaptureChangeMySQL-for-CDC-td20895.html] -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (NIFI-6501) Add config to limit buffer queue size in CaptureChangeMySQL
[ https://issues.apache.org/jira/browse/NIFI-6501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17429331#comment-17429331 ] Lei Wang commented on NIFI-6501: I have used NIFI-6501 by setting offertimeout as a high value in our prod environment for about two years. It runs good. I think we can accept the pull request Thanks, Lei > Add config to limit buffer queue size in CaptureChangeMySQL > --- > > Key: NIFI-6501 > URL: https://issues.apache.org/jira/browse/NIFI-6501 > Project: Apache NiFi > Issue Type: Bug > Components: Configuration >Affects Versions: 1.9.2, 1.14.0, 1.13.1, 1.13.2 >Reporter: Purushotham Pushpavanthar >Assignee: Purushotham Pushpavanthar >Priority: Critical > Labels: easyfix > Attachments: image-2019-08-02-11-29-10-829.png > > Time Spent: 2h 10m > Remaining Estimate: 0h > > CaptureChangeMySQL processor registers a listener with Blocking Queue as > buffer with the BinLogClient. When the thread polling from the Queue is > slower compared to writer, the queue grows uncontrollably and brings down the > node. > Since the flow writing to listeners in > [mysql-binlog-connector-java|[https://github.com/shyiko/mysql-binlog-connector-java]] > is blocking, we should initialize the queue with *initial size* and *queue > offer timeout* specified by the user based in cluster configuration. > [http://apache-nifi-developer-list.39713.n7.nabble.com/NiFi-Cluster-crashes-while-running-CaptureChangeMySQL-for-CDC-td20895.html] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (NIFI-6501) Add config to limit buffer queue size in CaptureChangeMySQL
[ https://issues.apache.org/jira/browse/NIFI-6501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17388601#comment-17388601 ] Przemyslaw Dubaniewicz commented on NIFI-6501: -- [~leiwangouc], [~bbende], [~pushpavanthar] do you think any further changes are needed here? This is a kind of serious bug so it would be great to have it fixed. Can we restart the discussion here? Or should I contact somebody else? > Add config to limit buffer queue size in CaptureChangeMySQL > --- > > Key: NIFI-6501 > URL: https://issues.apache.org/jira/browse/NIFI-6501 > Project: Apache NiFi > Issue Type: Bug > Components: Configuration >Affects Versions: 1.9.2 >Reporter: Purushotham Pushpavanthar >Assignee: Purushotham Pushpavanthar >Priority: Critical > Labels: easyfix > Attachments: image-2019-08-02-11-29-10-829.png > > Time Spent: 1h 50m > Remaining Estimate: 0h > > CaptureChangeMySQL processor registers a listener with Blocking Queue as > buffer with the BinLogClient. When the thread polling from the Queue is > slower compared to writer, the queue grows uncontrollably and brings down the > node. > Since the flow writing to listeners in > [mysql-binlog-connector-java|[https://github.com/shyiko/mysql-binlog-connector-java]] > is blocking, we should initialize the queue with *initial size* and *queue > offer timeout* specified by the user based in cluster configuration. > [http://apache-nifi-developer-list.39713.n7.nabble.com/NiFi-Cluster-crashes-while-running-CaptureChangeMySQL-for-CDC-td20895.html] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (NIFI-6501) Add config to limit buffer queue size in CaptureChangeMySQL
[ https://issues.apache.org/jira/browse/NIFI-6501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17344364#comment-17344364 ] Pawel Chudzik commented on NIFI-6501: - Hey, Why hasn't this PR been merged ? The proposed solution seems to be good enough and the probability of data loss can be decreased by setting a high timeout. > Add config to limit buffer queue size in CaptureChangeMySQL > --- > > Key: NIFI-6501 > URL: https://issues.apache.org/jira/browse/NIFI-6501 > Project: Apache NiFi > Issue Type: Bug > Components: Configuration >Affects Versions: 1.9.2 >Reporter: Purushotham Pushpavanthar >Assignee: Purushotham Pushpavanthar >Priority: Critical > Labels: easyfix > Attachments: image-2019-08-02-11-29-10-829.png > > Time Spent: 1.5h > Remaining Estimate: 0h > > CaptureChangeMySQL processor registers a listener with Blocking Queue as > buffer with the BinLogClient. When the thread polling from the Queue is > slower compared to writer, the queue grows uncontrollably and brings down the > node. > Since the flow writing to listeners in > [mysql-binlog-connector-java|[https://github.com/shyiko/mysql-binlog-connector-java]] > is blocking, we should initialize the queue with *initial size* and *queue > offer timeout* specified by the user based in cluster configuration. > [http://apache-nifi-developer-list.39713.n7.nabble.com/NiFi-Cluster-crashes-while-running-CaptureChangeMySQL-for-CDC-td20895.html] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (NIFI-6501) Add config to limit buffer queue size in CaptureChangeMySQL
[ https://issues.apache.org/jira/browse/NIFI-6501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16901911#comment-16901911 ] Purushotham Pushpavanthar commented on NIFI-6501: - [~leiwangouc], [~bende] To guarantee that there is no data loss, the BinLogClient has to be disconnected when the buffer is full even after Max Queue Offer Time has expired. When the processor is triggered next, it'll have to check first whether is the Queue.size() >= Max Queue Size, if the queue is full, don't do anything. If not check if BinLogClient is disconnected, seek to binlog_filename and binlog_position and establish connection, else continue processing events from queue. What do you guys think? > Add config to limit buffer queue size in CaptureChangeMySQL > --- > > Key: NIFI-6501 > URL: https://issues.apache.org/jira/browse/NIFI-6501 > Project: Apache NiFi > Issue Type: Bug > Components: Configuration >Affects Versions: 1.9.2 >Reporter: Purushotham Pushpavanthar >Assignee: Purushotham Pushpavanthar >Priority: Critical > Labels: easyfix > Attachments: image-2019-08-02-11-29-10-829.png > > Time Spent: 1h > Remaining Estimate: 0h > > CaptureChangeMySQL processor registers a listener with Blocking Queue as > buffer with the BinLogClient. When the thread polling from the Queue is > slower compared to writer, the queue grows uncontrollably and brings down the > node. > Since the flow writing to listeners in > [mysql-binlog-connector-java|[https://github.com/shyiko/mysql-binlog-connector-java]] > is blocking, we should initialize the queue with *initial size* and *queue > offer timeout* specified by the user based in cluster configuration. > [http://apache-nifi-developer-list.39713.n7.nabble.com/NiFi-Cluster-crashes-while-running-CaptureChangeMySQL-for-CDC-td20895.html] -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (NIFI-6501) Add config to limit buffer queue size in CaptureChangeMySQL
[ https://issues.apache.org/jira/browse/NIFI-6501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16898885#comment-16898885 ] Bryan Bende commented on NIFI-6501: --- If the BinaryLogClient is also blocked waiting for the call to onEvent to complete, then you can just set the offer timeout really high, and unless the the downstream processors are truly stopped, then you should eventually get space to complete the offer and not really lose date. The main two reasons the blocking queue would be full are 1) the next processor can't keep up and back-pressure happened on the queue between the processors, which caused CaptureChangeMySql to temporarily not execute, which caused the internal queue to fill up, or 2) the next processor is stopped which caused the same thing. The difference is that in #1 there will be space at some point, it will be going back and forth between back-pressure and no back-pressure, where as #2 there will never be space until the downstream processor is started. > Add config to limit buffer queue size in CaptureChangeMySQL > --- > > Key: NIFI-6501 > URL: https://issues.apache.org/jira/browse/NIFI-6501 > Project: Apache NiFi > Issue Type: Bug > Components: Configuration >Affects Versions: 1.9.2 >Reporter: Purushotham Pushpavanthar >Assignee: Purushotham Pushpavanthar >Priority: Critical > Labels: easyfix > Attachments: image-2019-08-02-11-29-10-829.png > > Time Spent: 1h > Remaining Estimate: 0h > > CaptureChangeMySQL processor registers a listener with Blocking Queue as > buffer with the BinLogClient. When the thread polling from the Queue is > slower compared to writer, the queue grows uncontrollably and brings down the > node. > Since the flow writing to listeners in > [mysql-binlog-connector-java|[https://github.com/shyiko/mysql-binlog-connector-java]] > is blocking, we should initialize the queue with *initial size* and *queue > offer timeout* specified by the user based in cluster configuration. > [http://apache-nifi-developer-list.39713.n7.nabble.com/NiFi-Cluster-crashes-while-running-CaptureChangeMySQL-for-CDC-td20895.html] -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (NIFI-6501) Add config to limit buffer queue size in CaptureChangeMySQL
[ https://issues.apache.org/jira/browse/NIFI-6501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16898675#comment-16898675 ] Purushotham Pushpavanthar commented on NIFI-6501: - Yes. But the state ( binlog filename and position) will be retained. I guess if the downstream processor is stopped, there is no point in keeping binlogClient running. We need to stop the client and restart to particular binlog filename and position when triggered next. Correct me if I'm wrong. > Add config to limit buffer queue size in CaptureChangeMySQL > --- > > Key: NIFI-6501 > URL: https://issues.apache.org/jira/browse/NIFI-6501 > Project: Apache NiFi > Issue Type: Bug > Components: Configuration >Affects Versions: 1.9.2 >Reporter: Purushotham Pushpavanthar >Assignee: Purushotham Pushpavanthar >Priority: Critical > Labels: easyfix > Attachments: image-2019-08-02-11-29-10-829.png > > Time Spent: 1h > Remaining Estimate: 0h > > CaptureChangeMySQL processor registers a listener with Blocking Queue as > buffer with the BinLogClient. When the thread polling from the Queue is > slower compared to writer, the queue grows uncontrollably and brings down the > node. > Since the flow writing to listeners in > [mysql-binlog-connector-java|[https://github.com/shyiko/mysql-binlog-connector-java]] > is blocking, we should initialize the queue with *initial size* and *queue > offer timeout* specified by the user based in cluster configuration. > [http://apache-nifi-developer-list.39713.n7.nabble.com/NiFi-Cluster-crashes-while-running-CaptureChangeMySQL-for-CDC-td20895.html] -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (NIFI-6501) Add config to limit buffer queue size in CaptureChangeMySQL
[ https://issues.apache.org/jira/browse/NIFI-6501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16898552#comment-16898552 ] Lei Wang commented on NIFI-6501: So in this senario, every one minutes，one binlog message will be lost and unable to be sent to downstream, Is this right? > Add config to limit buffer queue size in CaptureChangeMySQL > --- > > Key: NIFI-6501 > URL: https://issues.apache.org/jira/browse/NIFI-6501 > Project: Apache NiFi > Issue Type: Bug > Components: Configuration >Affects Versions: 1.9.2 >Reporter: Purushotham Pushpavanthar >Assignee: Purushotham Pushpavanthar >Priority: Critical > Labels: easyfix > Attachments: image-2019-08-02-11-29-10-829.png > > Time Spent: 1h > Remaining Estimate: 0h > > CaptureChangeMySQL processor registers a listener with Blocking Queue as > buffer with the BinLogClient. When the thread polling from the Queue is > slower compared to writer, the queue grows uncontrollably and brings down the > node. > Since the flow writing to listeners in > [mysql-binlog-connector-java|[https://github.com/shyiko/mysql-binlog-connector-java]] > is blocking, we should initialize the queue with *initial size* and *queue > offer timeout* specified by the user based in cluster configuration. > [http://apache-nifi-developer-list.39713.n7.nabble.com/NiFi-Cluster-crashes-while-running-CaptureChangeMySQL-for-CDC-td20895.html] -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (NIFI-6501) Add config to limit buffer queue size in CaptureChangeMySQL
[ https://issues.apache.org/jira/browse/NIFI-6501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16898541#comment-16898541 ] Purushotham Pushpavanthar commented on NIFI-6501: - Hi Lei, This is expected behavior. # Yes, the frequency is same as max queue offer time. # This exception leads to data loss. We're compromising on data guarantee for node availability. > Add config to limit buffer queue size in CaptureChangeMySQL > --- > > Key: NIFI-6501 > URL: https://issues.apache.org/jira/browse/NIFI-6501 > Project: Apache NiFi > Issue Type: Bug > Components: Configuration >Affects Versions: 1.9.2 >Reporter: Purushotham Pushpavanthar >Assignee: Purushotham Pushpavanthar >Priority: Critical > Labels: easyfix > Attachments: image-2019-08-02-11-29-10-829.png > > Time Spent: 1h > Remaining Estimate: 0h > > CaptureChangeMySQL processor registers a listener with Blocking Queue as > buffer with the BinLogClient. When the thread polling from the Queue is > slower compared to writer, the queue grows uncontrollably and brings down the > node. > Since the flow writing to listeners in > [mysql-binlog-connector-java|[https://github.com/shyiko/mysql-binlog-connector-java]] > is blocking, we should initialize the queue with *initial size* and *queue > offer timeout* specified by the user based in cluster configuration. > [http://apache-nifi-developer-list.39713.n7.nabble.com/NiFi-Cluster-crashes-while-running-CaptureChangeMySQL-for-CDC-td20895.html] -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (NIFI-6501) Add config to limit buffer queue size in CaptureChangeMySQL
[ https://issues.apache.org/jira/browse/NIFI-6501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16898521#comment-16898521 ] Lei Wang commented on NIFI-6501: Hi Purushotham, I have rebuild the nifi-cdc-mysql-processors using your new code. Seems it works, no OOM error again. After the down stream is blocked， the processor will throw an error every one minites because the queue is full: !image-2019-08-02-11-29-10-829.png! java.lang.RuntimeException: Unable to add event to the queue at org.apache.nifi.cdc.mysql.event.BinlogEventListener.onEvent(BinlogEventListener.java:59) at com.github.shyiko.mysql.binlog.BinaryLogClient.notifyEventListeners(BinaryLogClient.java:1055) at com.github.shyiko.mysql.binlog.BinaryLogClient.listenForEventPackets(BinaryLogClient.java:913) at com.github.shyiko.mysql.binlog.BinaryLogClient.connect(BinaryLogClient.java:559) at com.github.shyiko.mysql.binlog.BinaryLogClient$7.run(BinaryLogClient.java:793) at java.lang.Thread.run(Thread.java:748) I have two quesion: * The frequency of the exception is the same with the property Max Queue Offer Time. Is this right? * Will the exception cause data loss? > Add config to limit buffer queue size in CaptureChangeMySQL > --- > > Key: NIFI-6501 > URL: https://issues.apache.org/jira/browse/NIFI-6501 > Project: Apache NiFi > Issue Type: Bug > Components: Configuration >Affects Versions: 1.9.2 >Reporter: Purushotham Pushpavanthar >Assignee: Purushotham Pushpavanthar >Priority: Critical > Labels: easyfix > Attachments: image-2019-08-02-11-29-10-829.png > > Time Spent: 1h > Remaining Estimate: 0h > > CaptureChangeMySQL processor registers a listener with Blocking Queue as > buffer with the BinLogClient. When the thread polling from the Queue is > slower compared to writer, the queue grows uncontrollably and brings down the > node. > Since the flow writing to listeners in > [mysql-binlog-connector-java|[https://github.com/shyiko/mysql-binlog-connector-java]] > is blocking, we should initialize the queue with *initial size* and *queue > offer timeout* specified by the user based in cluster configuration. > [http://apache-nifi-developer-list.39713.n7.nabble.com/NiFi-Cluster-crashes-while-running-CaptureChangeMySQL-for-CDC-td20895.html] -- This message was sent by Atlassian JIRA (v7.6.14#76016)
