[jira] [Updated] (NIFI-5583) Make Change Data Capture (CDC) processor for MySQL refer to GTID
[ https://issues.apache.org/jira/browse/NIFI-5583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yoshiaki Takahashi updated NIFI-5583: - Status: Patch Available (was: Open) https://github.com/apache/nifi/pull/2997 > Make Change Data Capture (CDC) processor for MySQL refer to GTID > > > Key: NIFI-5583 > URL: https://issues.apache.org/jira/browse/NIFI-5583 > Project: Apache NiFi > Issue Type: New Feature > Components: Extensions >Affects Versions: 1.7.1 >Reporter: Yoshiaki Takahashi >Priority: Major > > When connecting to a MySQL cluster consisting of multiple hosts, existing CDC > processor can not automatically switch to other host when the connected host > goes down. > The reason is that file names and positions of binary log may be different > for each host in the same cluster. > In such a case it is impossible to continue reading files from the same > position. > In order to continue reading in such cases, a processor referring to the GTID > (Global Transaction ID) is necessary. > GTID is ID that uniquely identifies a transaction, and it is recorded in > binary log of MySQL. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi-minifi-cpp pull request #428: MINIFICPP-652: Add traceability to reposi...
Github user arpadboda commented on a diff in the pull request:
https://github.com/apache/nifi-minifi-cpp/pull/428#discussion_r228848352
--- Diff: libminifi/src/FlowController.cpp ---
@@ -911,11 +911,18 @@ uint64_t FlowController::getUptime() {
}
std::vector FlowController::getTraces() {
-std::vector traces;
+ std::vector traces;
auto timer_driven = timer_scheduler_->getTraces();
traces.insert(traces.end(),
std::make_move_iterator(timer_driven.begin()),
std::make_move_iterator(timer_driven.end()));
auto event_driven = event_scheduler_->getTraces();
traces.insert(traces.end(),
std::make_move_iterator(event_driven.begin()),
std::make_move_iterator(event_driven.end()));
--- End diff --
Okay, I can live with that.
---
[jira] [Commented] (MINIFICPP-652) Add monitored threads to stacks protocol
[
https://issues.apache.org/jira/browse/MINIFICPP-652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1904#comment-1904
]
ASF GitHub Bot commented on MINIFICPP-652:
--
Github user arpadboda commented on a diff in the pull request:
https://github.com/apache/nifi-minifi-cpp/pull/428#discussion_r228848352
--- Diff: libminifi/src/FlowController.cpp ---
@@ -911,11 +911,18 @@ uint64_t FlowController::getUptime() {
}
std::vector FlowController::getTraces() {
-std::vector traces;
+ std::vector traces;
auto timer_driven = timer_scheduler_->getTraces();
traces.insert(traces.end(),
std::make_move_iterator(timer_driven.begin()),
std::make_move_iterator(timer_driven.end()));
auto event_driven = event_scheduler_->getTraces();
traces.insert(traces.end(),
std::make_move_iterator(event_driven.begin()),
std::make_move_iterator(event_driven.end()));
--- End diff --
Okay, I can live with that.
> Add monitored threads to stacks protocol
>
>
> Key: MINIFICPP-652
> URL: https://issues.apache.org/jira/browse/MINIFICPP-652
> Project: NiFi MiNiFi C++
> Issue Type: Improvement
>Reporter: Mr TheSegfault
>Assignee: Mr TheSegfault
>Priority: Major
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] nifi-minifi-cpp pull request #428: MINIFICPP-652: Add traceability to reposi...
Github user arpadboda commented on a diff in the pull request: https://github.com/apache/nifi-minifi-cpp/pull/428#discussion_r228848961 --- Diff: libminifi/include/core/TraceableResource.h --- @@ -0,0 +1,53 @@ +/** + * + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +#ifndef LIBMINIFI_INCLUDE_CORE_TRACEABLERESOURCE_H_ --- End diff -- I don't think it's important to immediately change the project as it works, however for newly introduced files I would prefer pragma given it cannot be mistyped. ---
[jira] [Commented] (MINIFICPP-652) Add monitored threads to stacks protocol
[ https://issues.apache.org/jira/browse/MINIFICPP-652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1907#comment-1907 ] ASF GitHub Bot commented on MINIFICPP-652: -- Github user arpadboda commented on a diff in the pull request: https://github.com/apache/nifi-minifi-cpp/pull/428#discussion_r228848961 --- Diff: libminifi/include/core/TraceableResource.h --- @@ -0,0 +1,53 @@ +/** + * + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +#ifndef LIBMINIFI_INCLUDE_CORE_TRACEABLERESOURCE_H_ --- End diff -- I don't think it's important to immediately change the project as it works, however for newly introduced files I would prefer pragma given it cannot be mistyped. > Add monitored threads to stacks protocol > > > Key: MINIFICPP-652 > URL: https://issues.apache.org/jira/browse/MINIFICPP-652 > Project: NiFi MiNiFi C++ > Issue Type: Improvement >Reporter: Mr TheSegfault >Assignee: Mr TheSegfault >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi-minifi-cpp pull request #428: MINIFICPP-652: Add traceability to reposi...
Github user arpadboda commented on a diff in the pull request:
https://github.com/apache/nifi-minifi-cpp/pull/428#discussion_r228855333
--- Diff: libminifi/src/FlowController.cpp ---
@@ -911,11 +911,18 @@ uint64_t FlowController::getUptime() {
}
std::vector FlowController::getTraces() {
-std::vector traces;
+ std::vector traces;
auto timer_driven = timer_scheduler_->getTraces();
traces.insert(traces.end(),
std::make_move_iterator(timer_driven.begin()),
std::make_move_iterator(timer_driven.end()));
auto event_driven = event_scheduler_->getTraces();
traces.insert(traces.end(),
std::make_move_iterator(event_driven.begin()),
std::make_move_iterator(event_driven.end()));
+ // repositories
+ auto prov_repo_trace = provenance_repo_->getTraces();
+ traces.emplace_back(std::move(prov_repo_trace));
--- End diff --
Just for the record as I have already approved the PR, this is fine as is.
push_back also supports move, so in such case there is no difference.
The reason I prefer to avoid emplace in such cases is calling explicit
ctors:
```
class A
{
public:
bool mb;
explicit A(bool b): mb(b) {}
};
int main()
{
A* ap = new A(false);
std::vector vec;
vec.emplace_back(ap);
std::cout << vec[0].mb; //prints true
}
```
---
[jira] [Commented] (MINIFICPP-652) Add monitored threads to stacks protocol
[
https://issues.apache.org/jira/browse/MINIFICPP-652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1923#comment-1923
]
ASF GitHub Bot commented on MINIFICPP-652:
--
Github user arpadboda commented on a diff in the pull request:
https://github.com/apache/nifi-minifi-cpp/pull/428#discussion_r228855333
--- Diff: libminifi/src/FlowController.cpp ---
@@ -911,11 +911,18 @@ uint64_t FlowController::getUptime() {
}
std::vector FlowController::getTraces() {
-std::vector traces;
+ std::vector traces;
auto timer_driven = timer_scheduler_->getTraces();
traces.insert(traces.end(),
std::make_move_iterator(timer_driven.begin()),
std::make_move_iterator(timer_driven.end()));
auto event_driven = event_scheduler_->getTraces();
traces.insert(traces.end(),
std::make_move_iterator(event_driven.begin()),
std::make_move_iterator(event_driven.end()));
+ // repositories
+ auto prov_repo_trace = provenance_repo_->getTraces();
+ traces.emplace_back(std::move(prov_repo_trace));
--- End diff --
Just for the record as I have already approved the PR, this is fine as is.
push_back also supports move, so in such case there is no difference.
The reason I prefer to avoid emplace in such cases is calling explicit
ctors:
```
class A
{
public:
bool mb;
explicit A(bool b): mb(b) {}
};
int main()
{
A* ap = new A(false);
std::vector vec;
vec.emplace_back(ap);
std::cout << vec[0].mb; //prints true
}
```
> Add monitored threads to stacks protocol
>
>
> Key: MINIFICPP-652
> URL: https://issues.apache.org/jira/browse/MINIFICPP-652
> Project: NiFi MiNiFi C++
> Issue Type: Improvement
>Reporter: Mr TheSegfault
>Assignee: Mr TheSegfault
>Priority: Major
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] nifi-minifi-cpp pull request #427: MINIFICPP-653: Check if empty content, if...
Github user arpadboda commented on a diff in the pull request: https://github.com/apache/nifi-minifi-cpp/pull/427#discussion_r228861915 --- Diff: docker/test/integration/test_zero_file.py --- @@ -0,0 +1,38 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the \"License\"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an \"AS IS\" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from minifi import * +from minifi.test import * --- End diff -- Not a big deal here, not sure about having pyling would be more gain than pain. :) ---
[jira] [Commented] (MINIFICPP-653) Log message will segfault client if no content produced
[ https://issues.apache.org/jira/browse/MINIFICPP-653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1949#comment-1949 ] ASF GitHub Bot commented on MINIFICPP-653: -- Github user arpadboda commented on a diff in the pull request: https://github.com/apache/nifi-minifi-cpp/pull/427#discussion_r228861915 --- Diff: docker/test/integration/test_zero_file.py --- @@ -0,0 +1,38 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the \"License\"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an \"AS IS\" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from minifi import * +from minifi.test import * --- End diff -- Not a big deal here, not sure about having pyling would be more gain than pain. :) > Log message will segfault client if no content produced > --- > > Key: MINIFICPP-653 > URL: https://issues.apache.org/jira/browse/MINIFICPP-653 > Project: NiFi MiNiFi C++ > Issue Type: Improvement >Reporter: Mr TheSegfault >Assignee: Mr TheSegfault >Priority: Blocker > > Log message will segfault client if no content produced -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (NIFI-5763) ListSftp
dav created NIFI-5763: - Summary: ListSftp Key: NIFI-5763 URL: https://issues.apache.org/jira/browse/NIFI-5763 Project: Apache NiFi Issue Type: Wish Components: Extensions Affects Versions: 1.6.0 Reporter: dav ListSftp and other Sftp processors should be able to add parameters in order to edit the type of connection and the number of simultaneously connections on Sftp Server. For instance: I have this error on nifi: -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (NIFI-5764) ListSftp
dav created NIFI-5764:
-
Summary: ListSftp
Key: NIFI-5764
URL: https://issues.apache.org/jira/browse/NIFI-5764
Project: Apache NiFi
Issue Type: Wish
Components: Extensions
Affects Versions: 1.6.0
Reporter: dav
ListSftp and other Sftp processors should be able to add parameters
(like [-B buffer_size] [-b batchfile] [-c cipher]
[-D sftp_server_path] [-F ssh_config] [-i identity_file] [-l limit]
[-o ssh_option] [-P port] [-R num_requests] [-S program]
[-s subsystem | sftp_server] host
sftp [user@]host[:file ...]
sftp [user@]host[:dir[/]]
sftp -b batchfile [user@]host)
in order to edit the type of connection on Sftp Server.
For instance, I have this error on nifi:
2018-10-29 11:06:09,462 ERROR [Timer-Driven Process Thread-5]
SimpleProcessLogger.java:254 ListSFTP[id=766ac418-27ce-335a-5b13-52abe3495592]
Failed to perform listing on remote host due to java.io.IOException: Failed to
obtain connection to remote host due to com.jcraft.jsch.JSchException: Auth
fail: {}
java.io.IOException: Failed to obtain connection to remote host due to
com.jcraft.jsch.JSchException: Auth fail
at
org.apache.nifi.processors.standard.util.SFTPTransfer.getChannel(SFTPTransfer.java:468)
at
org.apache.nifi.processors.standard.util.SFTPTransfer.getListing(SFTPTransfer.java:192)
at
org.apache.nifi.processors.standard.util.SFTPTransfer.getListing(SFTPTransfer.java:156)
at
org.apache.nifi.processors.standard.ListFileTransfer.performListing(ListFileTransfer.java:105)
at
org.apache.nifi.processor.util.list.AbstractListProcessor.onTrigger(AbstractListProcessor.java:401)
at
org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
at
org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1147)
at
org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:175)
at
org.apache.nifi.controller.scheduling.QuartzSchedulingAgent$2.run(QuartzSchedulingAgent.java:140)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: com.jcraft.jsch.JSchException: Auth fail
at com.jcraft.jsch.Session.connect(Session.java:519)
at com.jcraft.jsch.Session.connect(Session.java:183)
at
org.apache.nifi.processors.standard.util.SFTPTransfer.getChannel(SFTPTransfer.java:448)
... 15 common frames omitted
This can be avoided by connect to Sftp server with this string:
*sftp -o “controlmaster auto” username@sftp_server*
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (NIFI-5764) Allow ListSftp
[
https://issues.apache.org/jira/browse/NIFI-5764?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
dav updated NIFI-5764:
--
Summary: Allow ListSftp (was: ListSftp)
> Allow ListSftp
> --
>
> Key: NIFI-5764
> URL: https://issues.apache.org/jira/browse/NIFI-5764
> Project: Apache NiFi
> Issue Type: Wish
> Components: Extensions
>Affects Versions: 1.6.0
>Reporter: dav
>Priority: Critical
> Labels: SFTP, customization, sftp
>
> ListSftp and other Sftp processors should be able to add parameters
> (like [-B buffer_size] [-b batchfile] [-c cipher]
> [-D sftp_server_path] [-F ssh_config] [-i identity_file] [-l limit]
> [-o ssh_option] [-P port] [-R num_requests] [-S program]
> [-s subsystem | sftp_server] host
> sftp [user@]host[:file ...]
> sftp [user@]host[:dir[/]]
> sftp -b batchfile [user@]host)
> in order to edit the type of connection on Sftp Server.
> For instance, I have this error on nifi:
> 2018-10-29 11:06:09,462 ERROR [Timer-Driven Process Thread-5]
> SimpleProcessLogger.java:254
> ListSFTP[id=766ac418-27ce-335a-5b13-52abe3495592] Failed to perform listing
> on remote host due to java.io.IOException: Failed to obtain connection to
> remote host due to com.jcraft.jsch.JSchException: Auth fail: {}
> java.io.IOException: Failed to obtain connection to remote host due to
> com.jcraft.jsch.JSchException: Auth fail
> at
> org.apache.nifi.processors.standard.util.SFTPTransfer.getChannel(SFTPTransfer.java:468)
> at
> org.apache.nifi.processors.standard.util.SFTPTransfer.getListing(SFTPTransfer.java:192)
> at
> org.apache.nifi.processors.standard.util.SFTPTransfer.getListing(SFTPTransfer.java:156)
> at
> org.apache.nifi.processors.standard.ListFileTransfer.performListing(ListFileTransfer.java:105)
> at
> org.apache.nifi.processor.util.list.AbstractListProcessor.onTrigger(AbstractListProcessor.java:401)
> at
> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
> at
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1147)
> at
> org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:175)
> at
> org.apache.nifi.controller.scheduling.QuartzSchedulingAgent$2.run(QuartzSchedulingAgent.java:140)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: com.jcraft.jsch.JSchException: Auth fail
> at com.jcraft.jsch.Session.connect(Session.java:519)
> at com.jcraft.jsch.Session.connect(Session.java:183)
> at
> org.apache.nifi.processors.standard.util.SFTPTransfer.getChannel(SFTPTransfer.java:448)
> ... 15 common frames omitted
> This can be avoided by connect to Sftp server with this string:
> *sftp -o “controlmaster auto” username@sftp_server*
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (NIFI-5764) Allow ListSftp connection parameter
[
https://issues.apache.org/jira/browse/NIFI-5764?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
dav updated NIFI-5764:
--
Summary: Allow ListSftp connection parameter (was: Allow ListSftp)
> Allow ListSftp connection parameter
> ---
>
> Key: NIFI-5764
> URL: https://issues.apache.org/jira/browse/NIFI-5764
> Project: Apache NiFi
> Issue Type: Wish
> Components: Extensions
>Affects Versions: 1.6.0
>Reporter: dav
>Priority: Critical
> Labels: SFTP, customization, sftp
>
> ListSftp and other Sftp processors should be able to add parameters
> (like [-B buffer_size] [-b batchfile] [-c cipher]
> [-D sftp_server_path] [-F ssh_config] [-i identity_file] [-l limit]
> [-o ssh_option] [-P port] [-R num_requests] [-S program]
> [-s subsystem | sftp_server] host
> sftp [user@]host[:file ...]
> sftp [user@]host[:dir[/]]
> sftp -b batchfile [user@]host)
> in order to edit the type of connection on Sftp Server.
> For instance, I have this error on nifi:
> 2018-10-29 11:06:09,462 ERROR [Timer-Driven Process Thread-5]
> SimpleProcessLogger.java:254
> ListSFTP[id=766ac418-27ce-335a-5b13-52abe3495592] Failed to perform listing
> on remote host due to java.io.IOException: Failed to obtain connection to
> remote host due to com.jcraft.jsch.JSchException: Auth fail: {}
> java.io.IOException: Failed to obtain connection to remote host due to
> com.jcraft.jsch.JSchException: Auth fail
> at
> org.apache.nifi.processors.standard.util.SFTPTransfer.getChannel(SFTPTransfer.java:468)
> at
> org.apache.nifi.processors.standard.util.SFTPTransfer.getListing(SFTPTransfer.java:192)
> at
> org.apache.nifi.processors.standard.util.SFTPTransfer.getListing(SFTPTransfer.java:156)
> at
> org.apache.nifi.processors.standard.ListFileTransfer.performListing(ListFileTransfer.java:105)
> at
> org.apache.nifi.processor.util.list.AbstractListProcessor.onTrigger(AbstractListProcessor.java:401)
> at
> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
> at
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1147)
> at
> org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:175)
> at
> org.apache.nifi.controller.scheduling.QuartzSchedulingAgent$2.run(QuartzSchedulingAgent.java:140)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: com.jcraft.jsch.JSchException: Auth fail
> at com.jcraft.jsch.Session.connect(Session.java:519)
> at com.jcraft.jsch.Session.connect(Session.java:183)
> at
> org.apache.nifi.processors.standard.util.SFTPTransfer.getChannel(SFTPTransfer.java:448)
> ... 15 common frames omitted
> This can be avoided by connect to Sftp server with this string:
> *sftp -o “controlmaster auto” username@sftp_server*
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] nifi pull request #3112: NIFI-5761 ReplaceText processor can stop processing...
Github user gardellajuanpablo commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3112#discussion_r228913385
--- Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ReplaceText.java
---
@@ -297,16 +297,27 @@ public void onTrigger(final ProcessContext context,
final ProcessSession session
} catch (StackOverflowError e) {
// Some regular expressions can produce many matches on large
input data size using recursive code
// do not log the StackOverflowError stack trace
-logger.info("Transferred {} to 'failure' due to {}", new
Object[] {flowFile, e.toString()});
-session.transfer(flowFile, REL_FAILURE);
+sendToFailure(session, flowFile, logger, e);
+return;
+} catch (RuntimeException e) {
--- End diff --
Done, thanks for the suggestion.
---
[jira] [Commented] (NIFI-5761) ReplaceText processor can stop processing data if it evaluates invalid expressions
[
https://issues.apache.org/jira/browse/NIFI-5761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16667166#comment-16667166
]
ASF GitHub Bot commented on NIFI-5761:
--
Github user gardellajuanpablo commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3112#discussion_r228913385
--- Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ReplaceText.java
---
@@ -297,16 +297,27 @@ public void onTrigger(final ProcessContext context,
final ProcessSession session
} catch (StackOverflowError e) {
// Some regular expressions can produce many matches on large
input data size using recursive code
// do not log the StackOverflowError stack trace
-logger.info("Transferred {} to 'failure' due to {}", new
Object[] {flowFile, e.toString()});
-session.transfer(flowFile, REL_FAILURE);
+sendToFailure(session, flowFile, logger, e);
+return;
+} catch (RuntimeException e) {
--- End diff --
Done, thanks for the suggestion.
> ReplaceText processor can stop processing data if it evaluates invalid
> expressions
> --
>
> Key: NIFI-5761
> URL: https://issues.apache.org/jira/browse/NIFI-5761
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
>Affects Versions: 1.5.0, 1.7.1
> Environment: ALL
>Reporter: Gardella Juan Pablo
>Assignee: Gardella Juan Pablo
>Priority: Major
> Original Estimate: 3h
> Remaining Estimate: 3h
>
> Given a flowfile body with nifi expression, when _ReplaceText_ processor
> evaluates it and the expression throws an exception, the processor will
> rollback the flowfile and keep trying to evaluate instead of send the
> flowfile to _failure_ relationshipt.
> Discussion Thread:
> http://apache-nifi-users-list.2361937.n4.nabble.com/ReplaceText-cannot-consume-messages-if-Regex-does-not-match-td5986.html
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] nifi-minifi-cpp pull request #429: MINIFICPP-654 - C API: failure callback i...
GitHub user arpadboda opened a pull request: https://github.com/apache/nifi-minifi-cpp/pull/429 MINIFICPP-654 - C API: failure callback improvements Thank you for submitting a contribution to Apache NiFi - MiNiFi C++. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [ ] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [ ] Does your PR title start with MINIFICPP- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically master)? - [ ] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file? - [ ] If applicable, have you updated the NOTICE file? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/arpadboda/nifi-minifi-cpp MINIFICPP-654 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi-minifi-cpp/pull/429.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #429 commit e229d2abe141f8e84b010f3d6500a3be894f20ac Author: Arpad Boda Date: 2018-10-24T13:14:05Z MINIFICPP-654 - C API: failure callback improvements ---
[GitHub] nifi-minifi-cpp issue #429: MINIFICPP-654 - C API: failure callback improvem...
Github user arpadboda commented on the issue: https://github.com/apache/nifi-minifi-cpp/pull/429 It's not final, further changes expected as I'm not yet satisfied with the code quality, but the concept can be reviewed at least. ---
[jira] [Commented] (MINIFICPP-654) C API: failure callback improvements
[ https://issues.apache.org/jira/browse/MINIFICPP-654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16667195#comment-16667195 ] ASF GitHub Bot commented on MINIFICPP-654: -- GitHub user arpadboda opened a pull request: https://github.com/apache/nifi-minifi-cpp/pull/429 MINIFICPP-654 - C API: failure callback improvements Thank you for submitting a contribution to Apache NiFi - MiNiFi C++. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [ ] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [ ] Does your PR title start with MINIFICPP- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically master)? - [ ] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file? - [ ] If applicable, have you updated the NOTICE file? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/arpadboda/nifi-minifi-cpp MINIFICPP-654 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi-minifi-cpp/pull/429.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #429 commit e229d2abe141f8e84b010f3d6500a3be894f20ac Author: Arpad Boda Date: 2018-10-24T13:14:05Z MINIFICPP-654 - C API: failure callback improvements > C API: failure callback improvements > > > Key: MINIFICPP-654 > URL: https://issues.apache.org/jira/browse/MINIFICPP-654 > Project: NiFi MiNiFi C++ > Issue Type: Improvement >Reporter: Arpad Boda >Assignee: Arpad Boda >Priority: Minor > Fix For: 0.6.0 > > > Improvements and further discussion of failure callbacks. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (MINIFICPP-654) C API: failure callback improvements
[ https://issues.apache.org/jira/browse/MINIFICPP-654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16667199#comment-16667199 ] ASF GitHub Bot commented on MINIFICPP-654: -- Github user arpadboda commented on the issue: https://github.com/apache/nifi-minifi-cpp/pull/429 It's not final, further changes expected as I'm not yet satisfied with the code quality, but the concept can be reviewed at least. > C API: failure callback improvements > > > Key: MINIFICPP-654 > URL: https://issues.apache.org/jira/browse/MINIFICPP-654 > Project: NiFi MiNiFi C++ > Issue Type: Improvement >Reporter: Arpad Boda >Assignee: Arpad Boda >Priority: Minor > Fix For: 0.6.0 > > > Improvements and further discussion of failure callbacks. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (MINIFICPP-657) Begin removing Instance class in favor of a C based design.
Mr TheSegfault created MINIFICPP-657: Summary: Begin removing Instance class in favor of a C based design. Key: MINIFICPP-657 URL: https://issues.apache.org/jira/browse/MINIFICPP-657 Project: NiFi MiNiFi C++ Issue Type: Epic Reporter: Mr TheSegfault We currently use an execution plan as instance variables in the C code. We should begin porting these constructs to C or eliminate the need for them altogether. This will mean that sub components should be slowly ported. Could also use this as a way to begin integrating C functionality into integration tests -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (MINIFICPP-658) Port Site to Site to C
Mr TheSegfault created MINIFICPP-658: Summary: Port Site to Site to C Key: MINIFICPP-658 URL: https://issues.apache.org/jira/browse/MINIFICPP-658 Project: NiFi MiNiFi C++ Issue Type: Improvement Reporter: Mr TheSegfault Assignee: Mr TheSegfault Most code is already written in C, so there's no reason this should be a tough task. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (MINIFICPP-659) Move CAPI code out of libminifi
Mr TheSegfault created MINIFICPP-659: Summary: Move CAPI code out of libminifi Key: MINIFICPP-659 URL: https://issues.apache.org/jira/browse/MINIFICPP-659 Project: NiFi MiNiFi C++ Issue Type: Improvement Reporter: Mr TheSegfault Assignee: Mr TheSegfault -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi-minifi pull request #140: MINIFI-473 Formatted Admin Guide and Java Qui...
GitHub user andrewmlim opened a pull request: https://github.com/apache/nifi-minifi/pull/140 MINIFI-473 Formatted Admin Guide and Java Quick Start Guide to be con⦠â¦sistent with NiFi docs Also made some edits for readability and minor errors You can merge this pull request into a Git repository by running: $ git pull https://github.com/andrewmlim/nifi-minifi MINIFI-473 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi-minifi/pull/140.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #140 commit ee99654ddaf6cea4aba78992f86107606c0dfc74 Author: Andrew Lim Date: 2018-10-29T15:51:54Z MINIFI-473 Formatted Admin Guide and Java Quick Start Guide to be consistent with NiFi docs ---
[GitHub] nifi-minifi-cpp issue #429: MINIFICPP-654 - C API: failure callback improvem...
Github user phrocker commented on the issue: https://github.com/apache/nifi-minifi-cpp/pull/429 @arpadboda is this ready for review? You stated, "it's not final" ---
[jira] [Commented] (MINIFICPP-654) C API: failure callback improvements
[ https://issues.apache.org/jira/browse/MINIFICPP-654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16667407#comment-16667407 ] ASF GitHub Bot commented on MINIFICPP-654: -- Github user phrocker commented on the issue: https://github.com/apache/nifi-minifi-cpp/pull/429 @arpadboda is this ready for review? You stated, "it's not final" > C API: failure callback improvements > > > Key: MINIFICPP-654 > URL: https://issues.apache.org/jira/browse/MINIFICPP-654 > Project: NiFi MiNiFi C++ > Issue Type: Improvement >Reporter: Arpad Boda >Assignee: Arpad Boda >Priority: Minor > Fix For: 0.6.0 > > > Improvements and further discussion of failure callbacks. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi issue #3111: NIFI-5757 AvroRecordSetWriter - Fix for slow synchronized ...
Github user markap14 commented on the issue: https://github.com/apache/nifi/pull/3111 @arkadius thanks for the contribution. I don't think this is the right approach, however, As-is, the code ensures that we don't hold any more than a set number of entries in the cache. The proposed change removes this limitation, which introduces a memory leak that can quickly cause a system to run out of memory if they are using an unbounded number of schemas (as can occur if a system is dynamically generating schemas). The synchronization here is limited to an extremely small scope. Can you explain how you arrived at the determination that the existing code is slow? What kind of performance are you seeing? What performance do you expect? What kind of difference do you see after applying this change? ---
[jira] [Commented] (NIFI-5757) AvroRecordSetWriter synchronize every access to compiledAvroSchemaCache
[
https://issues.apache.org/jira/browse/NIFI-5757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16667412#comment-16667412
]
ASF GitHub Bot commented on NIFI-5757:
--
Github user markap14 commented on the issue:
https://github.com/apache/nifi/pull/3111
@arkadius thanks for the contribution. I don't think this is the right
approach, however, As-is, the code ensures that we don't hold any more than a
set number of entries in the cache. The proposed change removes this
limitation, which introduces a memory leak that can quickly cause a system to
run out of memory if they are using an unbounded number of schemas (as can
occur if a system is dynamically generating schemas).
The synchronization here is limited to an extremely small scope. Can you
explain how you arrived at the determination that the existing code is slow?
What kind of performance are you seeing? What performance do you expect? What
kind of difference do you see after applying this change?
> AvroRecordSetWriter synchronize every access to compiledAvroSchemaCache
> ---
>
> Key: NIFI-5757
> URL: https://issues.apache.org/jira/browse/NIFI-5757
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Core Framework
>Affects Versions: 1.7.1
>Reporter: Arek Burdach
>Priority: Major
>
> Avro record serialization is a quite expensive operation.
> This stack trace I very often see in thread dumps:
> {noformat}
> Thread 48583: (state = BLOCKED)
> -
> org.apache.nifi.avro.AvroRecordSetWriter.compileAvroSchema(java.lang.String)
> @bci=9, line=124 (Compiled frame)
> -
> org.apache.nifi.avro.AvroRecordSetWriter.createWriter(org.apache.nifi.logging.ComponentLog,
> org.apache.nifi.serialization.record.RecordSchema, java.io.OutputStream)
> @bci=96, line=92 (Compiled frame)
> - sun.reflect.GeneratedMethodAccessor183.invoke(java.lang.Object,
> java.lang.Object[]) @bci=56 (Compiled frame)
> - sun.reflect.DelegatingMethodAccessorImpl.invoke(java.lang.Object,
> java.lang.Object[]) @bci=6, line=43 (Compiled frame)
> - java.lang.reflect.Method.invoke(java.lang.Object, java.lang.Object[])
> @bci=56, line=498 (Compiled frame)
> -
> org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler.invoke(java.lang.Object,
> java.lang.reflect.Method, java.lang.Object[]) @bci=309, line=89 (Compiled
> frame)
> - com.sun.proxy.$Proxy100.createWriter(org.apache.nifi.logging.ComponentLog,
> org.apache.nifi.serialization.record.RecordSchema, java.io.OutputStream)
> @bci=24 (Compiled frame)
> -
> org.apache.nifi.processors.kafka.pubsub.PublisherLease.publish(org.apache.nifi.flowfile.FlowFile,
> org.apache.nifi.serialization.record.RecordSet,
> org.apache.nifi.serialization.RecordSetWriterFactory,
> org.apache.nifi.serialization.record.RecordSchema, java.lang.String,
> java.lang.String) @bci=71, line=169 (Compiled frame)
> -
> org.apache.nifi.processors.kafka.pubsub.PublishKafkaRecord_1_0$1.process(java.io.InputStream)
> @bci=94, line=412 (Compiled frame)
> {noformat}
> The reason why it happens is because {{AvroRecordSetWriter}} synchronizing
> every access to cache of compiled schemas.
> I've prepared PR that is fixing this issue by using {{ConcurrentHashMap}}
> instead: https://github.com/apache/nifi/pull/3111
> It is not a perfect fix because it removes cache size limitation which BTW
> was hardcoded to {{20}}. Services can be reusable by many flows so such a
> hard limit is not a good choice.
> What do you think about such an improvement?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] nifi-minifi pull request #140: MINIFI-473 Formatted Admin Guide and Java Qui...
Github user asfgit closed the pull request at: https://github.com/apache/nifi-minifi/pull/140 ---
[GitHub] nifi issue #3105: NIFI-5621: Added Cassandra connection provider service
Github user mattyb149 commented on the issue: https://github.com/apache/nifi/pull/3105 Mind rebasing this and setting all 1.8.0-SNAPSHOT versions to 1.9.0-SNAPSHOT? Please and thanks! ---
[jira] [Commented] (NIFI-5621) Create a Connection Pooling service implementation to be used in Cassandra processors
[ https://issues.apache.org/jira/browse/NIFI-5621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16667496#comment-16667496 ] ASF GitHub Bot commented on NIFI-5621: -- Github user mattyb149 commented on the issue: https://github.com/apache/nifi/pull/3105 Mind rebasing this and setting all 1.8.0-SNAPSHOT versions to 1.9.0-SNAPSHOT? Please and thanks! > Create a Connection Pooling service implementation to be used in Cassandra > processors > - > > Key: NIFI-5621 > URL: https://issues.apache.org/jira/browse/NIFI-5621 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: Sivaprasanna Sethuraman >Assignee: Sivaprasanna Sethuraman >Priority: Major > > Like how the Relational Database processors leverage 'DBCPConnectionPool' > controller service, there should be one that could be used by the processors > from Cassandra bundle. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi issue #3041: NIFI-5224 Added SolrClientService.
Github user mattyb149 commented on the issue: https://github.com/apache/nifi/pull/3041 Mind updating the versions to 1.9.0-SNAPSHOT? Please and thanks! ---
[jira] [Commented] (NIFI-5224) Add SolrClientService
[ https://issues.apache.org/jira/browse/NIFI-5224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16667498#comment-16667498 ] ASF GitHub Bot commented on NIFI-5224: -- Github user mattyb149 commented on the issue: https://github.com/apache/nifi/pull/3041 Mind updating the versions to 1.9.0-SNAPSHOT? Please and thanks! > Add SolrClientService > - > > Key: NIFI-5224 > URL: https://issues.apache.org/jira/browse/NIFI-5224 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Johannes Peter >Assignee: Mike Thomsen >Priority: Major > > The Solr CRUD functions that are currently included in SolrUtils should be > moved to a controller service. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi-minifi-cpp pull request #429: MINIFICPP-654 - C API: failure callback i...
Github user phrocker commented on a diff in the pull request:
https://github.com/apache/nifi-minifi-cpp/pull/429#discussion_r229018124
--- Diff: libminifi/include/capi/Plan.h ---
@@ -46,11 +48,50 @@
#include "capi/cstructs.h"
#include "capi/api.h"
+enum FailureStrategy { AS_IS, ROLLBACK };
+
+bool intToFailureStragey(int in, FailureStrategy *out);
+
+using callback_type = std::function;
+using content_repo_sptr = std::shared_ptr;
+
+namespace {
+
+ void failureStrategyAsIs(core::ProcessSession *session, callback_type
user_callback, content_repo_sptr cr_ptr) {
+auto ff = session->get();
+if (ff == nullptr) {
+ return;
+}
+
+auto claim = ff->getResourceClaim();
+
+if (claim != nullptr && user_callback != nullptr) {
+ claim->increaseFlowFileRecordOwnedCount();
+ // create a flow file.
+ auto path = claim->getContentFullPath();
+ auto ffr = create_ff_object_na(path.c_str(), path.length(),
ff->getSize());
+ ffr->attributes = ff->getAttributesPtr();
+ ffr->ffp = ff.get();
+ auto content_repo_ptr =
static_cast*>(ffr->crp);
+ *content_repo_ptr = cr_ptr;
+ user_callback(ffr);
+}
+session->remove(ff);
+ }
+
+ void failureStrategyRollback(core::ProcessSession *session,
callback_type user_callback, content_repo_sptr cr_ptr) {
+session->rollback();
+failureStrategyAsIs(session, user_callback, cr_ptr);
+ }
+}
+
+static const std::map>
FailureStrategies =
--- End diff --
If there are only two is an ordered map or any map at all necessary? Would
a simple switch suffice?
---
[GitHub] nifi-minifi-cpp pull request #429: MINIFICPP-654 - C API: failure callback i...
Github user phrocker commented on a diff in the pull request: https://github.com/apache/nifi-minifi-cpp/pull/429#discussion_r229018550 --- Diff: libminifi/src/capi/api.cpp --- @@ -170,31 +165,31 @@ flow_file_record* create_ff_object_na(const char *file, const size_t len, const new_ff->attributes = nullptr; new_ff->contentLocation = new char[len + 1]; snprintf(new_ff->contentLocation, len + 1, "%s", file); - std::ifstream in(file, std::ifstream::ate | std::ifstream::binary); // set the size of the flow file. new_ff->size = size; + new_ff->crp = static_cast(new std::shared_ptr); --- End diff -- make_shared ? ---
[GitHub] nifi-minifi-cpp pull request #429: MINIFICPP-654 - C API: failure callback i...
Github user phrocker commented on a diff in the pull request:
https://github.com/apache/nifi-minifi-cpp/pull/429#discussion_r229017394
--- Diff: libminifi/include/capi/Plan.h ---
@@ -46,11 +48,50 @@
#include "capi/cstructs.h"
#include "capi/api.h"
+enum FailureStrategy { AS_IS, ROLLBACK };
+
+bool intToFailureStragey(int in, FailureStrategy *out);
+
+using callback_type = std::function;
--- End diff --
^I'm not inherently against aliasing but I think 'callback_type' does
little to improve readability, and since readability is in the eye of the
beholder ( and I don't have a preference if others use it ) is there a way to
at least better indicate that this is a function used for a certain type of
callback?
---
[GitHub] nifi-minifi-cpp pull request #429: MINIFICPP-654 - C API: failure callback i...
Github user phrocker commented on a diff in the pull request:
https://github.com/apache/nifi-minifi-cpp/pull/429#discussion_r229020814
--- Diff: libminifi/src/capi/api.cpp ---
@@ -400,11 +395,19 @@ processor *add_processor_with_linkage(flow *flow,
const char *processor_name) {
return nullptr;
}
-int add_failure_callback(flow *flow, void (*onerror_callback)(const
flow_file_record*)) {
+int add_failure_callback(flow *flow, void
(*onerror_callback)(flow_file_record*)) {
ExecutionPlan *plan = static_cast(flow->plan);
return plan->setFailureCallback(onerror_callback) ? 0 : 1;
}
+int set_failure_strategy(flow *flow, int strategy) {
--- End diff --
Why is strategy not typed to an enumeration? Might change the dynamics of
the return value but it might make conversions more safe.
---
[GitHub] nifi-minifi-cpp pull request #429: MINIFICPP-654 - C API: failure callback i...
Github user phrocker commented on a diff in the pull request:
https://github.com/apache/nifi-minifi-cpp/pull/429#discussion_r229016579
--- Diff: libminifi/include/capi/Plan.h ---
@@ -46,11 +48,50 @@
#include "capi/cstructs.h"
#include "capi/api.h"
+enum FailureStrategy { AS_IS, ROLLBACK };
--- End diff --
Do you think AS_IS could be clarified in the scope of the enumeration to
better indicate the intent?
---
[GitHub] nifi-minifi-cpp pull request #429: MINIFICPP-654 - C API: failure callback i...
Github user phrocker commented on a diff in the pull request: https://github.com/apache/nifi-minifi-cpp/pull/429#discussion_r229015052 --- Diff: libminifi/include/capi/Plan.h --- @@ -16,6 +16,8 @@ * limitations under the License. */ +#pragma once --- End diff -- The more I think about this for a public, open source project, we should probably avoid pragma once given that it's implementation defined and not language spec. We don't control all compiler variants, especially for what will be this portion of the API. I realize it's nearly "universally supported" but it seems like an unnecessary risk, albeit small. ---
[GitHub] nifi-minifi-cpp pull request #429: MINIFICPP-654 - C API: failure callback i...
Github user phrocker commented on a diff in the pull request:
https://github.com/apache/nifi-minifi-cpp/pull/429#discussion_r229021535
--- Diff: libminifi/include/capi/Plan.h ---
@@ -46,11 +48,50 @@
#include "capi/cstructs.h"
#include "capi/api.h"
+enum FailureStrategy { AS_IS, ROLLBACK };
+
+bool intToFailureStragey(int in, FailureStrategy *out);
+
+using callback_type = std::function;
+using content_repo_sptr = std::shared_ptr;
+
+namespace {
+
+ void failureStrategyAsIs(core::ProcessSession *session, callback_type
user_callback, content_repo_sptr cr_ptr) {
+auto ff = session->get();
+if (ff == nullptr) {
+ return;
+}
+
+auto claim = ff->getResourceClaim();
+
+if (claim != nullptr && user_callback != nullptr) {
+ claim->increaseFlowFileRecordOwnedCount();
+ // create a flow file.
+ auto path = claim->getContentFullPath();
+ auto ffr = create_ff_object_na(path.c_str(), path.length(),
ff->getSize());
+ ffr->attributes = ff->getAttributesPtr();
+ ffr->ffp = ff.get();
+ auto content_repo_ptr =
static_cast*>(ffr->crp);
+ *content_repo_ptr = cr_ptr;
+ user_callback(ffr);
+}
+session->remove(ff);
+ }
+
+ void failureStrategyRollback(core::ProcessSession *session,
callback_type user_callback, content_repo_sptr cr_ptr) {
+session->rollback();
+failureStrategyAsIs(session, user_callback, cr_ptr);
+ }
+}
+
+static const std::map>
FailureStrategies =
+{ { FailureStrategy::AS_IS, failureStrategyAsIs },
{FailureStrategy::ROLLBACK, failureStrategyRollback } };
class ExecutionPlan {
public:
- explicit ExecutionPlan(std::shared_ptr
content_repo, std::shared_ptr flow_repo,
std::shared_ptr prov_repo);
+ explicit ExecutionPlan(content_repo_sptr content_repo,
std::shared_ptr flow_repo, std::shared_ptr
prov_repo);
--- End diff --
^ Seems harder to read. Not sure if it helps but we have our linter set to
200 line characters ( to match Apache NiFi ), so we're not that tight on line
space.
---
[GitHub] nifi-minifi-cpp pull request #429: MINIFICPP-654 - C API: failure callback i...
Github user phrocker commented on a diff in the pull request:
https://github.com/apache/nifi-minifi-cpp/pull/429#discussion_r229015497
--- Diff: libminifi/include/capi/Plan.h ---
@@ -46,11 +48,50 @@
#include "capi/cstructs.h"
#include "capi/api.h"
+enum FailureStrategy { AS_IS, ROLLBACK };
+
+bool intToFailureStragey(int in, FailureStrategy *out);
--- End diff --
*spelling.
Is this something to be used in C? Perhaps we should avoid booleans?
stdbool wasn't introduced until C99..
---
[jira] [Commented] (MINIFICPP-654) C API: failure callback improvements
[
https://issues.apache.org/jira/browse/MINIFICPP-654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16667512#comment-16667512
]
ASF GitHub Bot commented on MINIFICPP-654:
--
Github user phrocker commented on a diff in the pull request:
https://github.com/apache/nifi-minifi-cpp/pull/429#discussion_r229018124
--- Diff: libminifi/include/capi/Plan.h ---
@@ -46,11 +48,50 @@
#include "capi/cstructs.h"
#include "capi/api.h"
+enum FailureStrategy { AS_IS, ROLLBACK };
+
+bool intToFailureStragey(int in, FailureStrategy *out);
+
+using callback_type = std::function;
+using content_repo_sptr = std::shared_ptr;
+
+namespace {
+
+ void failureStrategyAsIs(core::ProcessSession *session, callback_type
user_callback, content_repo_sptr cr_ptr) {
+auto ff = session->get();
+if (ff == nullptr) {
+ return;
+}
+
+auto claim = ff->getResourceClaim();
+
+if (claim != nullptr && user_callback != nullptr) {
+ claim->increaseFlowFileRecordOwnedCount();
+ // create a flow file.
+ auto path = claim->getContentFullPath();
+ auto ffr = create_ff_object_na(path.c_str(), path.length(),
ff->getSize());
+ ffr->attributes = ff->getAttributesPtr();
+ ffr->ffp = ff.get();
+ auto content_repo_ptr =
static_cast*>(ffr->crp);
+ *content_repo_ptr = cr_ptr;
+ user_callback(ffr);
+}
+session->remove(ff);
+ }
+
+ void failureStrategyRollback(core::ProcessSession *session,
callback_type user_callback, content_repo_sptr cr_ptr) {
+session->rollback();
+failureStrategyAsIs(session, user_callback, cr_ptr);
+ }
+}
+
+static const std::map>
FailureStrategies =
--- End diff --
If there are only two is an ordered map or any map at all necessary? Would
a simple switch suffice?
> C API: failure callback improvements
>
>
> Key: MINIFICPP-654
> URL: https://issues.apache.org/jira/browse/MINIFICPP-654
> Project: NiFi MiNiFi C++
> Issue Type: Improvement
>Reporter: Arpad Boda
>Assignee: Arpad Boda
>Priority: Minor
> Fix For: 0.6.0
>
>
> Improvements and further discussion of failure callbacks.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (MINIFICPP-654) C API: failure callback improvements
[
https://issues.apache.org/jira/browse/MINIFICPP-654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16667507#comment-16667507
]
ASF GitHub Bot commented on MINIFICPP-654:
--
Github user phrocker commented on a diff in the pull request:
https://github.com/apache/nifi-minifi-cpp/pull/429#discussion_r229021535
--- Diff: libminifi/include/capi/Plan.h ---
@@ -46,11 +48,50 @@
#include "capi/cstructs.h"
#include "capi/api.h"
+enum FailureStrategy { AS_IS, ROLLBACK };
+
+bool intToFailureStragey(int in, FailureStrategy *out);
+
+using callback_type = std::function;
+using content_repo_sptr = std::shared_ptr;
+
+namespace {
+
+ void failureStrategyAsIs(core::ProcessSession *session, callback_type
user_callback, content_repo_sptr cr_ptr) {
+auto ff = session->get();
+if (ff == nullptr) {
+ return;
+}
+
+auto claim = ff->getResourceClaim();
+
+if (claim != nullptr && user_callback != nullptr) {
+ claim->increaseFlowFileRecordOwnedCount();
+ // create a flow file.
+ auto path = claim->getContentFullPath();
+ auto ffr = create_ff_object_na(path.c_str(), path.length(),
ff->getSize());
+ ffr->attributes = ff->getAttributesPtr();
+ ffr->ffp = ff.get();
+ auto content_repo_ptr =
static_cast*>(ffr->crp);
+ *content_repo_ptr = cr_ptr;
+ user_callback(ffr);
+}
+session->remove(ff);
+ }
+
+ void failureStrategyRollback(core::ProcessSession *session,
callback_type user_callback, content_repo_sptr cr_ptr) {
+session->rollback();
+failureStrategyAsIs(session, user_callback, cr_ptr);
+ }
+}
+
+static const std::map>
FailureStrategies =
+{ { FailureStrategy::AS_IS, failureStrategyAsIs },
{FailureStrategy::ROLLBACK, failureStrategyRollback } };
class ExecutionPlan {
public:
- explicit ExecutionPlan(std::shared_ptr
content_repo, std::shared_ptr flow_repo,
std::shared_ptr prov_repo);
+ explicit ExecutionPlan(content_repo_sptr content_repo,
std::shared_ptr flow_repo, std::shared_ptr
prov_repo);
--- End diff --
^ Seems harder to read. Not sure if it helps but we have our linter set to
200 line characters ( to match Apache NiFi ), so we're not that tight on line
space.
> C API: failure callback improvements
>
>
> Key: MINIFICPP-654
> URL: https://issues.apache.org/jira/browse/MINIFICPP-654
> Project: NiFi MiNiFi C++
> Issue Type: Improvement
>Reporter: Arpad Boda
>Assignee: Arpad Boda
>Priority: Minor
> Fix For: 0.6.0
>
>
> Improvements and further discussion of failure callbacks.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (MINIFICPP-654) C API: failure callback improvements
[
https://issues.apache.org/jira/browse/MINIFICPP-654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16667509#comment-16667509
]
ASF GitHub Bot commented on MINIFICPP-654:
--
Github user phrocker commented on a diff in the pull request:
https://github.com/apache/nifi-minifi-cpp/pull/429#discussion_r229016579
--- Diff: libminifi/include/capi/Plan.h ---
@@ -46,11 +48,50 @@
#include "capi/cstructs.h"
#include "capi/api.h"
+enum FailureStrategy { AS_IS, ROLLBACK };
--- End diff --
Do you think AS_IS could be clarified in the scope of the enumeration to
better indicate the intent?
> C API: failure callback improvements
>
>
> Key: MINIFICPP-654
> URL: https://issues.apache.org/jira/browse/MINIFICPP-654
> Project: NiFi MiNiFi C++
> Issue Type: Improvement
>Reporter: Arpad Boda
>Assignee: Arpad Boda
>Priority: Minor
> Fix For: 0.6.0
>
>
> Improvements and further discussion of failure callbacks.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (MINIFICPP-654) C API: failure callback improvements
[
https://issues.apache.org/jira/browse/MINIFICPP-654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16667510#comment-16667510
]
ASF GitHub Bot commented on MINIFICPP-654:
--
Github user phrocker commented on a diff in the pull request:
https://github.com/apache/nifi-minifi-cpp/pull/429#discussion_r229015497
--- Diff: libminifi/include/capi/Plan.h ---
@@ -46,11 +48,50 @@
#include "capi/cstructs.h"
#include "capi/api.h"
+enum FailureStrategy { AS_IS, ROLLBACK };
+
+bool intToFailureStragey(int in, FailureStrategy *out);
--- End diff --
*spelling.
Is this something to be used in C? Perhaps we should avoid booleans?
stdbool wasn't introduced until C99..
> C API: failure callback improvements
>
>
> Key: MINIFICPP-654
> URL: https://issues.apache.org/jira/browse/MINIFICPP-654
> Project: NiFi MiNiFi C++
> Issue Type: Improvement
>Reporter: Arpad Boda
>Assignee: Arpad Boda
>Priority: Minor
> Fix For: 0.6.0
>
>
> Improvements and further discussion of failure callbacks.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (MINIFICPP-654) C API: failure callback improvements
[ https://issues.apache.org/jira/browse/MINIFICPP-654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16667511#comment-16667511 ] ASF GitHub Bot commented on MINIFICPP-654: -- Github user phrocker commented on a diff in the pull request: https://github.com/apache/nifi-minifi-cpp/pull/429#discussion_r229015052 --- Diff: libminifi/include/capi/Plan.h --- @@ -16,6 +16,8 @@ * limitations under the License. */ +#pragma once --- End diff -- The more I think about this for a public, open source project, we should probably avoid pragma once given that it's implementation defined and not language spec. We don't control all compiler variants, especially for what will be this portion of the API. I realize it's nearly "universally supported" but it seems like an unnecessary risk, albeit small. > C API: failure callback improvements > > > Key: MINIFICPP-654 > URL: https://issues.apache.org/jira/browse/MINIFICPP-654 > Project: NiFi MiNiFi C++ > Issue Type: Improvement >Reporter: Arpad Boda >Assignee: Arpad Boda >Priority: Minor > Fix For: 0.6.0 > > > Improvements and further discussion of failure callbacks. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (MINIFICPP-654) C API: failure callback improvements
[
https://issues.apache.org/jira/browse/MINIFICPP-654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16667508#comment-16667508
]
ASF GitHub Bot commented on MINIFICPP-654:
--
Github user phrocker commented on a diff in the pull request:
https://github.com/apache/nifi-minifi-cpp/pull/429#discussion_r229017394
--- Diff: libminifi/include/capi/Plan.h ---
@@ -46,11 +48,50 @@
#include "capi/cstructs.h"
#include "capi/api.h"
+enum FailureStrategy { AS_IS, ROLLBACK };
+
+bool intToFailureStragey(int in, FailureStrategy *out);
+
+using callback_type = std::function;
--- End diff --
^I'm not inherently against aliasing but I think 'callback_type' does
little to improve readability, and since readability is in the eye of the
beholder ( and I don't have a preference if others use it ) is there a way to
at least better indicate that this is a function used for a certain type of
callback?
> C API: failure callback improvements
>
>
> Key: MINIFICPP-654
> URL: https://issues.apache.org/jira/browse/MINIFICPP-654
> Project: NiFi MiNiFi C++
> Issue Type: Improvement
>Reporter: Arpad Boda
>Assignee: Arpad Boda
>Priority: Minor
> Fix For: 0.6.0
>
>
> Improvements and further discussion of failure callbacks.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (MINIFICPP-654) C API: failure callback improvements
[
https://issues.apache.org/jira/browse/MINIFICPP-654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16667513#comment-16667513
]
ASF GitHub Bot commented on MINIFICPP-654:
--
Github user phrocker commented on a diff in the pull request:
https://github.com/apache/nifi-minifi-cpp/pull/429#discussion_r229020814
--- Diff: libminifi/src/capi/api.cpp ---
@@ -400,11 +395,19 @@ processor *add_processor_with_linkage(flow *flow,
const char *processor_name) {
return nullptr;
}
-int add_failure_callback(flow *flow, void (*onerror_callback)(const
flow_file_record*)) {
+int add_failure_callback(flow *flow, void
(*onerror_callback)(flow_file_record*)) {
ExecutionPlan *plan = static_cast(flow->plan);
return plan->setFailureCallback(onerror_callback) ? 0 : 1;
}
+int set_failure_strategy(flow *flow, int strategy) {
--- End diff --
Why is strategy not typed to an enumeration? Might change the dynamics of
the return value but it might make conversions more safe.
> C API: failure callback improvements
>
>
> Key: MINIFICPP-654
> URL: https://issues.apache.org/jira/browse/MINIFICPP-654
> Project: NiFi MiNiFi C++
> Issue Type: Improvement
>Reporter: Arpad Boda
>Assignee: Arpad Boda
>Priority: Minor
> Fix For: 0.6.0
>
>
> Improvements and further discussion of failure callbacks.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (MINIFICPP-654) C API: failure callback improvements
[ https://issues.apache.org/jira/browse/MINIFICPP-654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16667514#comment-16667514 ] ASF GitHub Bot commented on MINIFICPP-654: -- Github user phrocker commented on a diff in the pull request: https://github.com/apache/nifi-minifi-cpp/pull/429#discussion_r229018550 --- Diff: libminifi/src/capi/api.cpp --- @@ -170,31 +165,31 @@ flow_file_record* create_ff_object_na(const char *file, const size_t len, const new_ff->attributes = nullptr; new_ff->contentLocation = new char[len + 1]; snprintf(new_ff->contentLocation, len + 1, "%s", file); - std::ifstream in(file, std::ifstream::ate | std::ifstream::binary); // set the size of the flow file. new_ff->size = size; + new_ff->crp = static_cast(new std::shared_ptr); --- End diff -- make_shared ? > C API: failure callback improvements > > > Key: MINIFICPP-654 > URL: https://issues.apache.org/jira/browse/MINIFICPP-654 > Project: NiFi MiNiFi C++ > Issue Type: Improvement >Reporter: Arpad Boda >Assignee: Arpad Boda >Priority: Minor > Fix For: 0.6.0 > > > Improvements and further discussion of failure callbacks. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (MINIFICPP-654) C API: failure callback improvements
[ https://issues.apache.org/jira/browse/MINIFICPP-654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16667520#comment-16667520 ] ASF GitHub Bot commented on MINIFICPP-654: -- Github user phrocker commented on the issue: https://github.com/apache/nifi-minifi-cpp/pull/429 Looks good. minor comments, nothing that's a big deal > C API: failure callback improvements > > > Key: MINIFICPP-654 > URL: https://issues.apache.org/jira/browse/MINIFICPP-654 > Project: NiFi MiNiFi C++ > Issue Type: Improvement >Reporter: Arpad Boda >Assignee: Arpad Boda >Priority: Minor > Fix For: 0.6.0 > > > Improvements and further discussion of failure callbacks. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi-minifi-cpp issue #429: MINIFICPP-654 - C API: failure callback improvem...
Github user phrocker commented on the issue: https://github.com/apache/nifi-minifi-cpp/pull/429 Looks good. minor comments, nothing that's a big deal ---
[GitHub] nifi issue #3111: NIFI-5757 AvroRecordSetWriter - Fix for slow synchronized ...
Github user arkadius commented on the issue: https://github.com/apache/nifi/pull/3111 Hi @markap14 . Thank you for your reply. We have a quite heavy-loaded environment. We are processing about 5 GB / min of messages on 32 core host. Avro serialization was our main bottleneck. We had avg 30 threads concurrently using this service. CPU usage was on 85%, iowaits near 0, load on 50. Increasing of number of threads didn't help. As I wrote in issue, I found out that the synchronization is an issue by sampling (doing occasional thread dumps). Most of thread was blocked on it. After this modification, we stopped to occur such a performance issue. Regards to removing of limitation of entries in the cache. I agree with you that it isn't the best approach. Limitation of 20 schemas wasn't either. In systems with multiple flows, commonly using the same service, this limit would be constantly hit. What do you think about introducing two parameters to this service: Cache Size Limit and Time To Leave? After hitting one of them, entry would be evicted. Condition would be checked during each entry get. Similar to `cachedSchema.isOlderThan` check in `CachingSchemaRegistryClient`. If we assume that these limits would be soft one, it won't need synchronization. ---
[jira] [Commented] (NIFI-5757) AvroRecordSetWriter synchronize every access to compiledAvroSchemaCache
[
https://issues.apache.org/jira/browse/NIFI-5757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16667538#comment-16667538
]
ASF GitHub Bot commented on NIFI-5757:
--
Github user arkadius commented on the issue:
https://github.com/apache/nifi/pull/3111
Hi @markap14 . Thank you for your reply. We have a quite heavy-loaded
environment. We are processing about 5 GB / min of messages on 32 core host.
Avro serialization was our main bottleneck. We had avg 30 threads concurrently
using this service. CPU usage was on 85%, iowaits near 0, load on 50.
Increasing of number of threads didn't help. As I wrote in issue, I found out
that the synchronization is an issue by sampling (doing occasional thread
dumps). Most of thread was blocked on it. After this modification, we stopped
to occur such a performance issue.
Regards to removing of limitation of entries in the cache. I agree with you
that it isn't the best approach. Limitation of 20 schemas wasn't either. In
systems with multiple flows, commonly using the same service, this limit would
be constantly hit. What do you think about introducing two parameters to this
service: Cache Size Limit and Time To Leave? After hitting one of them, entry
would be evicted. Condition would be checked during each entry get. Similar to
`cachedSchema.isOlderThan` check in `CachingSchemaRegistryClient`. If we assume
that these limits would be soft one, it won't need synchronization.
> AvroRecordSetWriter synchronize every access to compiledAvroSchemaCache
> ---
>
> Key: NIFI-5757
> URL: https://issues.apache.org/jira/browse/NIFI-5757
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Core Framework
>Affects Versions: 1.7.1
>Reporter: Arek Burdach
>Priority: Major
>
> Avro record serialization is a quite expensive operation.
> This stack trace I very often see in thread dumps:
> {noformat}
> Thread 48583: (state = BLOCKED)
> -
> org.apache.nifi.avro.AvroRecordSetWriter.compileAvroSchema(java.lang.String)
> @bci=9, line=124 (Compiled frame)
> -
> org.apache.nifi.avro.AvroRecordSetWriter.createWriter(org.apache.nifi.logging.ComponentLog,
> org.apache.nifi.serialization.record.RecordSchema, java.io.OutputStream)
> @bci=96, line=92 (Compiled frame)
> - sun.reflect.GeneratedMethodAccessor183.invoke(java.lang.Object,
> java.lang.Object[]) @bci=56 (Compiled frame)
> - sun.reflect.DelegatingMethodAccessorImpl.invoke(java.lang.Object,
> java.lang.Object[]) @bci=6, line=43 (Compiled frame)
> - java.lang.reflect.Method.invoke(java.lang.Object, java.lang.Object[])
> @bci=56, line=498 (Compiled frame)
> -
> org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler.invoke(java.lang.Object,
> java.lang.reflect.Method, java.lang.Object[]) @bci=309, line=89 (Compiled
> frame)
> - com.sun.proxy.$Proxy100.createWriter(org.apache.nifi.logging.ComponentLog,
> org.apache.nifi.serialization.record.RecordSchema, java.io.OutputStream)
> @bci=24 (Compiled frame)
> -
> org.apache.nifi.processors.kafka.pubsub.PublisherLease.publish(org.apache.nifi.flowfile.FlowFile,
> org.apache.nifi.serialization.record.RecordSet,
> org.apache.nifi.serialization.RecordSetWriterFactory,
> org.apache.nifi.serialization.record.RecordSchema, java.lang.String,
> java.lang.String) @bci=71, line=169 (Compiled frame)
> -
> org.apache.nifi.processors.kafka.pubsub.PublishKafkaRecord_1_0$1.process(java.io.InputStream)
> @bci=94, line=412 (Compiled frame)
> {noformat}
> The reason why it happens is because {{AvroRecordSetWriter}} synchronizing
> every access to cache of compiled schemas.
> I've prepared PR that is fixing this issue by using {{ConcurrentHashMap}}
> instead: https://github.com/apache/nifi/pull/3111
> It is not a perfect fix because it removes cache size limitation which BTW
> was hardcoded to {{20}}. Services can be reusable by many flows so such a
> hard limit is not a good choice.
> What do you think about such an improvement?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (NIFI-5765) WriteJsonResult fails with Class Cast Exception when Choice data type resolves to Array
[ https://issues.apache.org/jira/browse/NIFI-5765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph Percivall updated NIFI-5765: --- Affects Version/s: 1.8.0 > WriteJsonResult fails with Class Cast Exception when Choice data type > resolves to Array > --- > > Key: NIFI-5765 > URL: https://issues.apache.org/jira/browse/NIFI-5765 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.8.0 >Reporter: Joseph Percivall >Priority: Major > Attachments: Screen Shot 2018-10-29 at 2.35.50 PM.png, > WriteJsonResult_Choice_Array_example.xml > > > The problem is this line[1]. For the casting, it uses the passed in value > instead of the chosen data type. > A template demonstrating the problem is attached. The corner case is hit when > there is a choice of data types, and the data type chosen is Array. It > properly does everything else but fails when casting it due to: > org.apache.nifi.serialization.record.type.ChoiceDataType cannot be cast to > org.apache.nifi.serialization.record.type.ArrayDataType. > [1] > https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-standard-services/nifi-record-serialization-services-bundle/nifi-record-serialization-services/src/main/java/org/apache/nifi/json/WriteJsonResult.java#L379 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (NIFI-5765) WriteJsonResult fails with Class Cast Exception when Choice data type resolves to Array
Joseph Percivall created NIFI-5765: -- Summary: WriteJsonResult fails with Class Cast Exception when Choice data type resolves to Array Key: NIFI-5765 URL: https://issues.apache.org/jira/browse/NIFI-5765 Project: Apache NiFi Issue Type: Bug Reporter: Joseph Percivall Attachments: Screen Shot 2018-10-29 at 2.35.50 PM.png, WriteJsonResult_Choice_Array_example.xml The problem is this line[1]. For the casting, it uses the passed in value instead of the chosen data type. A template demonstrating the problem is attached. The corner case is hit when there is a choice of data types, and the data type chosen is Array. It properly does everything else but fails when casting it due to: org.apache.nifi.serialization.record.type.ChoiceDataType cannot be cast to org.apache.nifi.serialization.record.type.ArrayDataType. [1] https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-standard-services/nifi-record-serialization-services-bundle/nifi-record-serialization-services/src/main/java/org/apache/nifi/json/WriteJsonResult.java#L379 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (MINIFICPP-660) Create docker test for python library
Mr TheSegfault created MINIFICPP-660: Summary: Create docker test for python library Key: MINIFICPP-660 URL: https://issues.apache.org/jira/browse/MINIFICPP-660 Project: NiFi MiNiFi C++ Issue Type: Improvement Reporter: Mr TheSegfault Assignee: Mr TheSegfault -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (MINIFICPP-659) Move CAPI code out of libminifi
[ https://issues.apache.org/jira/browse/MINIFICPP-659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16667601#comment-16667601 ] ASF GitHub Bot commented on MINIFICPP-659: -- GitHub user phrocker opened a pull request: https://github.com/apache/nifi-minifi-cpp/pull/430 MINIFICPP-659: Break out CAPI into nanofi Break out structure -- facilitating Python and new nanofi structure Thank you for submitting a contribution to Apache NiFi - MiNiFi C++. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [ ] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [ ] Does your PR title start with MINIFICPP- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically master)? - [ ] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file? - [ ] If applicable, have you updated the NOTICE file? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/phrocker/nifi-minifi-cpp MINIFICPP-659 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi-minifi-cpp/pull/430.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #430 commit 362fc9142a5d09ab6d8c4d84472dd43b9151a00e Author: Marc Parisi Date: 2018-10-29T18:53:37Z MINIFICPP-659: Break out CAPI into nanofi Break out structure -- facilitating Python and new nanofi structure > Move CAPI code out of libminifi > --- > > Key: MINIFICPP-659 > URL: https://issues.apache.org/jira/browse/MINIFICPP-659 > Project: NiFi MiNiFi C++ > Issue Type: Improvement >Reporter: Mr TheSegfault >Assignee: Mr TheSegfault >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi-minifi-cpp pull request #430: MINIFICPP-659: Break out CAPI into nanofi
GitHub user phrocker opened a pull request: https://github.com/apache/nifi-minifi-cpp/pull/430 MINIFICPP-659: Break out CAPI into nanofi Break out structure -- facilitating Python and new nanofi structure Thank you for submitting a contribution to Apache NiFi - MiNiFi C++. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [ ] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [ ] Does your PR title start with MINIFICPP- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically master)? - [ ] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file? - [ ] If applicable, have you updated the NOTICE file? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/phrocker/nifi-minifi-cpp MINIFICPP-659 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi-minifi-cpp/pull/430.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #430 commit 362fc9142a5d09ab6d8c4d84472dd43b9151a00e Author: Marc Parisi Date: 2018-10-29T18:53:37Z MINIFICPP-659: Break out CAPI into nanofi Break out structure -- facilitating Python and new nanofi structure ---
[GitHub] nifi issue #3111: NIFI-5757 AvroRecordSetWriter - Fix for slow synchronized ...
Github user markap14 commented on the issue: https://github.com/apache/nifi/pull/3111 @arkadius fair enough. I'd be okay with a soft limit. ---
[jira] [Commented] (NIFI-5757) AvroRecordSetWriter synchronize every access to compiledAvroSchemaCache
[
https://issues.apache.org/jira/browse/NIFI-5757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16667632#comment-16667632
]
ASF GitHub Bot commented on NIFI-5757:
--
Github user markap14 commented on the issue:
https://github.com/apache/nifi/pull/3111
@arkadius fair enough. I'd be okay with a soft limit.
> AvroRecordSetWriter synchronize every access to compiledAvroSchemaCache
> ---
>
> Key: NIFI-5757
> URL: https://issues.apache.org/jira/browse/NIFI-5757
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Core Framework
>Affects Versions: 1.7.1
>Reporter: Arek Burdach
>Priority: Major
>
> Avro record serialization is a quite expensive operation.
> This stack trace I very often see in thread dumps:
> {noformat}
> Thread 48583: (state = BLOCKED)
> -
> org.apache.nifi.avro.AvroRecordSetWriter.compileAvroSchema(java.lang.String)
> @bci=9, line=124 (Compiled frame)
> -
> org.apache.nifi.avro.AvroRecordSetWriter.createWriter(org.apache.nifi.logging.ComponentLog,
> org.apache.nifi.serialization.record.RecordSchema, java.io.OutputStream)
> @bci=96, line=92 (Compiled frame)
> - sun.reflect.GeneratedMethodAccessor183.invoke(java.lang.Object,
> java.lang.Object[]) @bci=56 (Compiled frame)
> - sun.reflect.DelegatingMethodAccessorImpl.invoke(java.lang.Object,
> java.lang.Object[]) @bci=6, line=43 (Compiled frame)
> - java.lang.reflect.Method.invoke(java.lang.Object, java.lang.Object[])
> @bci=56, line=498 (Compiled frame)
> -
> org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler.invoke(java.lang.Object,
> java.lang.reflect.Method, java.lang.Object[]) @bci=309, line=89 (Compiled
> frame)
> - com.sun.proxy.$Proxy100.createWriter(org.apache.nifi.logging.ComponentLog,
> org.apache.nifi.serialization.record.RecordSchema, java.io.OutputStream)
> @bci=24 (Compiled frame)
> -
> org.apache.nifi.processors.kafka.pubsub.PublisherLease.publish(org.apache.nifi.flowfile.FlowFile,
> org.apache.nifi.serialization.record.RecordSet,
> org.apache.nifi.serialization.RecordSetWriterFactory,
> org.apache.nifi.serialization.record.RecordSchema, java.lang.String,
> java.lang.String) @bci=71, line=169 (Compiled frame)
> -
> org.apache.nifi.processors.kafka.pubsub.PublishKafkaRecord_1_0$1.process(java.io.InputStream)
> @bci=94, line=412 (Compiled frame)
> {noformat}
> The reason why it happens is because {{AvroRecordSetWriter}} synchronizing
> every access to cache of compiled schemas.
> I've prepared PR that is fixing this issue by using {{ConcurrentHashMap}}
> instead: https://github.com/apache/nifi/pull/3111
> It is not a perfect fix because it removes cache size limitation which BTW
> was hardcoded to {{20}}. Services can be reusable by many flows so such a
> hard limit is not a good choice.
> What do you think about such an improvement?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (NIFI-5765) WriteJsonResult fails with Class Cast Exception when Choice data type resolves to Array
[ https://issues.apache.org/jira/browse/NIFI-5765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph Percivall reassigned NIFI-5765: -- Assignee: Joseph Percivall > WriteJsonResult fails with Class Cast Exception when Choice data type > resolves to Array > --- > > Key: NIFI-5765 > URL: https://issues.apache.org/jira/browse/NIFI-5765 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.8.0 >Reporter: Joseph Percivall >Assignee: Joseph Percivall >Priority: Major > Attachments: Screen Shot 2018-10-29 at 2.35.50 PM.png, > WriteJsonResult_Choice_Array_example.xml > > > The problem is this line[1]. For the casting, it uses the passed in value > instead of the chosen data type. > A template demonstrating the problem is attached. The corner case is hit when > there is a choice of data types, and the data type chosen is Array. It > properly does everything else but fails when casting it due to: > org.apache.nifi.serialization.record.type.ChoiceDataType cannot be cast to > org.apache.nifi.serialization.record.type.ArrayDataType. > [1] > https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-standard-services/nifi-record-serialization-services-bundle/nifi-record-serialization-services/src/main/java/org/apache/nifi/json/WriteJsonResult.java#L379 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi pull request #3114: NIFI-5765 Fixing WriteJsonResult to use chosenDataT...
GitHub user JPercivall opened a pull request: https://github.com/apache/nifi/pull/3114 NIFI-5765 Fixing WriteJsonResult to use chosenDataType when writing a⦠â¦n Array value Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [X] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [X] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [X] Has your PR been rebased against the latest commit within the target branch (typically master)? - [X] Is your initial contribution a single, squashed commit? ### For code changes: - [X] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [X] Have you written or updated unit tests to verify your changes? - [X] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [X] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [X] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [X] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [X] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/JPercivall/nifi NIFI-5765_handling_WriteJsonResult_choice_array Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/3114.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3114 commit 6babefda6a533c0699a20d02a0dc84b25acc7ae8 Author: Joe Percivall Date: 2018-10-29T20:04:05Z NIFI-5765 Fixing WriteJsonResult to use chosenDataType when writing an Array value ---
[jira] [Commented] (NIFI-5765) WriteJsonResult fails with Class Cast Exception when Choice data type resolves to Array
[ https://issues.apache.org/jira/browse/NIFI-5765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16667681#comment-16667681 ] ASF GitHub Bot commented on NIFI-5765: -- GitHub user JPercivall opened a pull request: https://github.com/apache/nifi/pull/3114 NIFI-5765 Fixing WriteJsonResult to use chosenDataType when writing a… …n Array value Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [X] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [X] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [X] Has your PR been rebased against the latest commit within the target branch (typically master)? - [X] Is your initial contribution a single, squashed commit? ### For code changes: - [X] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [X] Have you written or updated unit tests to verify your changes? - [X] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [X] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [X] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [X] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [X] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/JPercivall/nifi NIFI-5765_handling_WriteJsonResult_choice_array Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/3114.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3114 commit 6babefda6a533c0699a20d02a0dc84b25acc7ae8 Author: Joe Percivall Date: 2018-10-29T20:04:05Z NIFI-5765 Fixing WriteJsonResult to use chosenDataType when writing an Array value > WriteJsonResult fails with Class Cast Exception when Choice data type > resolves to Array > --- > > Key: NIFI-5765 > URL: https://issues.apache.org/jira/browse/NIFI-5765 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.8.0 >Reporter: Joseph Percivall >Assignee: Joseph Percivall >Priority: Major > Attachments: Screen Shot 2018-10-29 at 2.35.50 PM.png, > WriteJsonResult_Choice_Array_example.xml > > > The problem is this line[1]. For the casting, it uses the passed in value > instead of the chosen data type. > A template demonstrating the problem is attached. The corner case is hit when > there is a choice of data types, and the data type chosen is Array. It > properly does everything else but fails when casting it due to: > org.apache.nifi.serialization.record.type.ChoiceDataType cannot be cast to > org.apache.nifi.serialization.record.type.ArrayDataType. > [1] > https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-standard-services/nifi-record-serialization-services-bundle/nifi-record-serialization-services/src/main/java/org/apache/nifi/json/WriteJsonResult.java#L379 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5765) WriteJsonResult fails with Class Cast Exception when Choice data type resolves to Array
[ https://issues.apache.org/jira/browse/NIFI-5765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16667684#comment-16667684 ] ASF GitHub Bot commented on NIFI-5765: -- Github user JPercivall commented on the issue: https://github.com/apache/nifi/pull/3114 Template for testing on the ticket: https://issues.apache.org/jira/browse/NIFI-5765 > WriteJsonResult fails with Class Cast Exception when Choice data type > resolves to Array > --- > > Key: NIFI-5765 > URL: https://issues.apache.org/jira/browse/NIFI-5765 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.8.0 >Reporter: Joseph Percivall >Assignee: Joseph Percivall >Priority: Major > Attachments: Screen Shot 2018-10-29 at 2.35.50 PM.png, > WriteJsonResult_Choice_Array_example.xml > > > The problem is this line[1]. For the casting, it uses the passed in value > instead of the chosen data type. > A template demonstrating the problem is attached. The corner case is hit when > there is a choice of data types, and the data type chosen is Array. It > properly does everything else but fails when casting it due to: > org.apache.nifi.serialization.record.type.ChoiceDataType cannot be cast to > org.apache.nifi.serialization.record.type.ArrayDataType. > [1] > https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-standard-services/nifi-record-serialization-services-bundle/nifi-record-serialization-services/src/main/java/org/apache/nifi/json/WriteJsonResult.java#L379 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi issue #3114: NIFI-5765 Fixing WriteJsonResult to use chosenDataType whe...
Github user JPercivall commented on the issue: https://github.com/apache/nifi/pull/3114 Template for testing on the ticket: https://issues.apache.org/jira/browse/NIFI-5765 ---
[jira] [Created] (NIFI-5766) Make formatting in User Guide consistent with Sys Admin Guide
Andrew Lim created NIFI-5766: Summary: Make formatting in User Guide consistent with Sys Admin Guide Key: NIFI-5766 URL: https://issues.apache.org/jira/browse/NIFI-5766 Project: Apache NiFi Issue Type: Improvement Components: Documentation & Website Reporter: Andrew Lim Similar to changes made for Sys Admin guide, format file references in _italics_ and directories in preformatted. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (NIFI-5766) Make formatting in User Guide consistent with Sys Admin Guide
[ https://issues.apache.org/jira/browse/NIFI-5766?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Lim reassigned NIFI-5766: Assignee: Andrew Lim > Make formatting in User Guide consistent with Sys Admin Guide > - > > Key: NIFI-5766 > URL: https://issues.apache.org/jira/browse/NIFI-5766 > Project: Apache NiFi > Issue Type: Improvement > Components: Documentation & Website >Reporter: Andrew Lim >Assignee: Andrew Lim >Priority: Trivial > > Similar to changes made for Sys Admin guide, format file references in > _italics_ and directories in preformatted. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi-minifi-cpp pull request #429: MINIFICPP-654 - C API: failure callback i...
Github user arpadboda commented on a diff in the pull request:
https://github.com/apache/nifi-minifi-cpp/pull/429#discussion_r229107427
--- Diff: libminifi/include/capi/Plan.h ---
@@ -46,11 +48,50 @@
#include "capi/cstructs.h"
#include "capi/api.h"
+enum FailureStrategy { AS_IS, ROLLBACK };
+
+bool intToFailureStragey(int in, FailureStrategy *out);
--- End diff --
Nope, it's hidden behind the C API, the API wraps it to int return value.
---
[jira] [Commented] (MINIFICPP-654) C API: failure callback improvements
[
https://issues.apache.org/jira/browse/MINIFICPP-654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16667783#comment-16667783
]
ASF GitHub Bot commented on MINIFICPP-654:
--
Github user arpadboda commented on a diff in the pull request:
https://github.com/apache/nifi-minifi-cpp/pull/429#discussion_r229107427
--- Diff: libminifi/include/capi/Plan.h ---
@@ -46,11 +48,50 @@
#include "capi/cstructs.h"
#include "capi/api.h"
+enum FailureStrategy { AS_IS, ROLLBACK };
+
+bool intToFailureStragey(int in, FailureStrategy *out);
--- End diff --
Nope, it's hidden behind the C API, the API wraps it to int return value.
> C API: failure callback improvements
>
>
> Key: MINIFICPP-654
> URL: https://issues.apache.org/jira/browse/MINIFICPP-654
> Project: NiFi MiNiFi C++
> Issue Type: Improvement
>Reporter: Arpad Boda
>Assignee: Arpad Boda
>Priority: Minor
> Fix For: 0.6.0
>
>
> Improvements and further discussion of failure callbacks.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] nifi-minifi-cpp pull request #429: MINIFICPP-654 - C API: failure callback i...
Github user arpadboda commented on a diff in the pull request:
https://github.com/apache/nifi-minifi-cpp/pull/429#discussion_r229107701
--- Diff: libminifi/include/capi/Plan.h ---
@@ -46,11 +48,50 @@
#include "capi/cstructs.h"
#include "capi/api.h"
+enum FailureStrategy { AS_IS, ROLLBACK };
--- End diff --
I agree with you that it's not the best naming as doesn't properly describe
what happens, but no better idea. @apiri - you are our last hope. :)
---
[jira] [Commented] (MINIFICPP-654) C API: failure callback improvements
[
https://issues.apache.org/jira/browse/MINIFICPP-654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16667784#comment-16667784
]
ASF GitHub Bot commented on MINIFICPP-654:
--
Github user arpadboda commented on a diff in the pull request:
https://github.com/apache/nifi-minifi-cpp/pull/429#discussion_r229107701
--- Diff: libminifi/include/capi/Plan.h ---
@@ -46,11 +48,50 @@
#include "capi/cstructs.h"
#include "capi/api.h"
+enum FailureStrategy { AS_IS, ROLLBACK };
--- End diff --
I agree with you that it's not the best naming as doesn't properly describe
what happens, but no better idea. @apiri - you are our last hope. :)
> C API: failure callback improvements
>
>
> Key: MINIFICPP-654
> URL: https://issues.apache.org/jira/browse/MINIFICPP-654
> Project: NiFi MiNiFi C++
> Issue Type: Improvement
>Reporter: Arpad Boda
>Assignee: Arpad Boda
>Priority: Minor
> Fix For: 0.6.0
>
>
> Improvements and further discussion of failure callbacks.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (MINIFICPP-654) C API: failure callback improvements
[
https://issues.apache.org/jira/browse/MINIFICPP-654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16667785#comment-16667785
]
ASF GitHub Bot commented on MINIFICPP-654:
--
Github user arpadboda commented on a diff in the pull request:
https://github.com/apache/nifi-minifi-cpp/pull/429#discussion_r229107909
--- Diff: libminifi/include/capi/Plan.h ---
@@ -46,11 +48,50 @@
#include "capi/cstructs.h"
#include "capi/api.h"
+enum FailureStrategy { AS_IS, ROLLBACK };
+
+bool intToFailureStragey(int in, FailureStrategy *out);
+
+using callback_type = std::function;
--- End diff --
Sure, I will rename this, should be failure callback type or something
alike.
> C API: failure callback improvements
>
>
> Key: MINIFICPP-654
> URL: https://issues.apache.org/jira/browse/MINIFICPP-654
> Project: NiFi MiNiFi C++
> Issue Type: Improvement
>Reporter: Arpad Boda
>Assignee: Arpad Boda
>Priority: Minor
> Fix For: 0.6.0
>
>
> Improvements and further discussion of failure callbacks.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] nifi-minifi-cpp pull request #429: MINIFICPP-654 - C API: failure callback i...
Github user arpadboda commented on a diff in the pull request:
https://github.com/apache/nifi-minifi-cpp/pull/429#discussion_r229107909
--- Diff: libminifi/include/capi/Plan.h ---
@@ -46,11 +48,50 @@
#include "capi/cstructs.h"
#include "capi/api.h"
+enum FailureStrategy { AS_IS, ROLLBACK };
+
+bool intToFailureStragey(int in, FailureStrategy *out);
+
+using callback_type = std::function;
--- End diff --
Sure, I will rename this, should be failure callback type or something
alike.
---
[GitHub] nifi-minifi-cpp pull request #429: MINIFICPP-654 - C API: failure callback i...
Github user arpadboda commented on a diff in the pull request: https://github.com/apache/nifi-minifi-cpp/pull/429#discussion_r229108406 --- Diff: libminifi/src/capi/api.cpp --- @@ -170,31 +165,31 @@ flow_file_record* create_ff_object_na(const char *file, const size_t len, const new_ff->attributes = nullptr; new_ff->contentLocation = new char[len + 1]; snprintf(new_ff->contentLocation, len + 1, "%s", file); - std::ifstream in(file, std::ifstream::ate | std::ifstream::binary); // set the size of the flow file. new_ff->size = size; + new_ff->crp = static_cast(new std::shared_ptr); --- End diff -- As there are no arguments provided, I couldn't save specifying the template parameter. Moreover it's an empty shared ptr, so it doesn't even allocate twice. ---
[jira] [Commented] (MINIFICPP-654) C API: failure callback improvements
[ https://issues.apache.org/jira/browse/MINIFICPP-654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16667786#comment-16667786 ] ASF GitHub Bot commented on MINIFICPP-654: -- Github user arpadboda commented on a diff in the pull request: https://github.com/apache/nifi-minifi-cpp/pull/429#discussion_r229108406 --- Diff: libminifi/src/capi/api.cpp --- @@ -170,31 +165,31 @@ flow_file_record* create_ff_object_na(const char *file, const size_t len, const new_ff->attributes = nullptr; new_ff->contentLocation = new char[len + 1]; snprintf(new_ff->contentLocation, len + 1, "%s", file); - std::ifstream in(file, std::ifstream::ate | std::ifstream::binary); // set the size of the flow file. new_ff->size = size; + new_ff->crp = static_cast(new std::shared_ptr); --- End diff -- As there are no arguments provided, I couldn't save specifying the template parameter. Moreover it's an empty shared ptr, so it doesn't even allocate twice. > C API: failure callback improvements > > > Key: MINIFICPP-654 > URL: https://issues.apache.org/jira/browse/MINIFICPP-654 > Project: NiFi MiNiFi C++ > Issue Type: Improvement >Reporter: Arpad Boda >Assignee: Arpad Boda >Priority: Minor > Fix For: 0.6.0 > > > Improvements and further discussion of failure callbacks. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi-minifi-cpp pull request #429: MINIFICPP-654 - C API: failure callback i...
Github user arpadboda commented on a diff in the pull request:
https://github.com/apache/nifi-minifi-cpp/pull/429#discussion_r229108728
--- Diff: libminifi/src/capi/api.cpp ---
@@ -400,11 +395,19 @@ processor *add_processor_with_linkage(flow *flow,
const char *processor_name) {
return nullptr;
}
-int add_failure_callback(flow *flow, void (*onerror_callback)(const
flow_file_record*)) {
+int add_failure_callback(flow *flow, void
(*onerror_callback)(flow_file_record*)) {
ExecutionPlan *plan = static_cast(flow->plan);
return plan->setFailureCallback(onerror_callback) ? 0 : 1;
}
+int set_failure_strategy(flow *flow, int strategy) {
--- End diff --
The reason is pretty simple: I didn't have enough coffee when I wrote this,
so forgot that C also supports enums. :) Will replace.
---
[jira] [Commented] (MINIFICPP-654) C API: failure callback improvements
[
https://issues.apache.org/jira/browse/MINIFICPP-654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16667787#comment-16667787
]
ASF GitHub Bot commented on MINIFICPP-654:
--
Github user arpadboda commented on a diff in the pull request:
https://github.com/apache/nifi-minifi-cpp/pull/429#discussion_r229108728
--- Diff: libminifi/src/capi/api.cpp ---
@@ -400,11 +395,19 @@ processor *add_processor_with_linkage(flow *flow,
const char *processor_name) {
return nullptr;
}
-int add_failure_callback(flow *flow, void (*onerror_callback)(const
flow_file_record*)) {
+int add_failure_callback(flow *flow, void
(*onerror_callback)(flow_file_record*)) {
ExecutionPlan *plan = static_cast(flow->plan);
return plan->setFailureCallback(onerror_callback) ? 0 : 1;
}
+int set_failure_strategy(flow *flow, int strategy) {
--- End diff --
The reason is pretty simple: I didn't have enough coffee when I wrote this,
so forgot that C also supports enums. :) Will replace.
> C API: failure callback improvements
>
>
> Key: MINIFICPP-654
> URL: https://issues.apache.org/jira/browse/MINIFICPP-654
> Project: NiFi MiNiFi C++
> Issue Type: Improvement
>Reporter: Arpad Boda
>Assignee: Arpad Boda
>Priority: Minor
> Fix For: 0.6.0
>
>
> Improvements and further discussion of failure callbacks.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] nifi issue #3114: NIFI-5765 Fixing WriteJsonResult to use chosenDataType whe...
Github user JPercivall commented on the issue: https://github.com/apache/nifi/pull/3114 Follow-up commit is b/c I found the same bug in PutElasticsearchHttpRecord. I believe that's the only other occurrence. ---
[jira] [Commented] (NIFI-5765) WriteJsonResult fails with Class Cast Exception when Choice data type resolves to Array
[ https://issues.apache.org/jira/browse/NIFI-5765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16667800#comment-16667800 ] ASF GitHub Bot commented on NIFI-5765: -- Github user JPercivall commented on the issue: https://github.com/apache/nifi/pull/3114 Follow-up commit is b/c I found the same bug in PutElasticsearchHttpRecord. I believe that's the only other occurrence. > WriteJsonResult fails with Class Cast Exception when Choice data type > resolves to Array > --- > > Key: NIFI-5765 > URL: https://issues.apache.org/jira/browse/NIFI-5765 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.8.0 >Reporter: Joseph Percivall >Assignee: Joseph Percivall >Priority: Major > Attachments: Screen Shot 2018-10-29 at 2.35.50 PM.png, > WriteJsonResult_Choice_Array_example.xml > > > The problem is this line[1]. For the casting, it uses the passed in value > instead of the chosen data type. > A template demonstrating the problem is attached. The corner case is hit when > there is a choice of data types, and the data type chosen is Array. It > properly does everything else but fails when casting it due to: > org.apache.nifi.serialization.record.type.ChoiceDataType cannot be cast to > org.apache.nifi.serialization.record.type.ArrayDataType. > [1] > https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-standard-services/nifi-record-serialization-services-bundle/nifi-record-serialization-services/src/main/java/org/apache/nifi/json/WriteJsonResult.java#L379 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi-minifi-cpp pull request #430: MINIFICPP-659: Break out CAPI into nanofi
Github user arpadboda commented on a diff in the pull request:
https://github.com/apache/nifi-minifi-cpp/pull/430#discussion_r229111455
--- Diff: nanofi/examples/CMakeLists.txt ---
@@ -61,18 +61,18 @@ add_executable(generate_flow generate_flow.c)
add_executable(terminate_handler terminate_handler.c)
-target_link_libraries(generate_flow capi ${CMAKE_THREAD_LIBS_INIT}
${LINK_FLAGS} minifi ${LINK_END_FLAGS})
+target_link_libraries(generate_flow nanofi ${CMAKE_THREAD_LIBS_INIT})
-target_link_libraries(terminate_handler capi ${CMAKE_THREAD_LIBS_INIT}
${LINK_FLAGS} minifi ${LINK_END_FLAGS})
+target_link_libraries(terminate_handler nanofi ${CMAKE_THREAD_LIBS_INIT})
if (NOT WIN32)
add_executable(transmit_flow transmit_flow.c)
-target_link_libraries(transmit_flow capi ${CMAKE_THREAD_LIBS_INIT}
${LINK_FLAGS} minifi ${LINK_END_FLAGS})
+target_link_libraries(transmit_flow nanofi ${CMAKE_THREAD_LIBS_INIT})
add_executable(monitor_directory monitor_directory.c)
-target_link_libraries(monitor_directory capi ${CMAKE_THREAD_LIBS_INIT}
${LINK_FLAGS} minifi ${LINK_END_FLAGS})
+target_link_libraries(monitor_directory nanofi ${CMAKE_THREAD_LIBS_INIT})
--- End diff --
Are you sure this is fine?
I see that nanofi is linked with whole-archive flag to make sure it
contains all symbols from minifi, however when linking happens here and
monitor_directory binary is created, the linker is still free to drop unused
symbols.
Fix me, I would also be happy if we could skip doing that ugly flag-magic
here. :)
---
[GitHub] nifi-minifi-cpp pull request #430: MINIFICPP-659: Break out CAPI into nanofi
Github user arpadboda commented on a diff in the pull request:
https://github.com/apache/nifi-minifi-cpp/pull/430#discussion_r229112853
--- Diff: thirdparty/rocksdb/CMakeLists.txt ---
@@ -592,16 +592,16 @@ if(WIN32)
set(LIBS ${ROCKSDB_STATIC_LIB} ${THIRDPARTY_LIBS} ${SYSTEM_LIBS})
else()
set(SYSTEM_LIBS ${CMAKE_THREAD_LIBS_INIT})
- set(LIBS ${ROCKSDB_SHARED_LIB} ${THIRDPARTY_LIBS} ${SYSTEM_LIBS})
- add_library(${ROCKSDB_SHARED_LIB} SHARED ${SOURCES})
+ set(LIBS ${ROCKSDB_STATIC_LIB} ${THIRDPARTY_LIBS} ${SYSTEM_LIBS})
+ #add_library(${ROCKSDB_SHARED_LIB} SHARED ${SOURCES})
# target_link_libraries(${ROCKSDB_SHARED_LIB}
#${THIRDPARTY_LIBS} ${SYSTEM_LIBS})
- set_target_properties(${ROCKSDB_SHARED_LIB} PROPERTIES
-LINKER_LANGUAGE CXX
-VERSION ${ROCKSDB_VERSION}
-SOVERSION ${ROCKSDB_VERSION_MAJOR}
-CXX_STANDARD 11
-OUTPUT_NAME "rocksdb")
+# set_target_properties(${ROCKSDB_SHARED_LIB} PROPERTIES
--- End diff --
Why commented?
If we comment it, even temporary, let's add some comment that justifies.
---
[jira] [Commented] (MINIFICPP-659) Move CAPI code out of libminifi
[
https://issues.apache.org/jira/browse/MINIFICPP-659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16667811#comment-16667811
]
ASF GitHub Bot commented on MINIFICPP-659:
--
Github user arpadboda commented on a diff in the pull request:
https://github.com/apache/nifi-minifi-cpp/pull/430#discussion_r229112853
--- Diff: thirdparty/rocksdb/CMakeLists.txt ---
@@ -592,16 +592,16 @@ if(WIN32)
set(LIBS ${ROCKSDB_STATIC_LIB} ${THIRDPARTY_LIBS} ${SYSTEM_LIBS})
else()
set(SYSTEM_LIBS ${CMAKE_THREAD_LIBS_INIT})
- set(LIBS ${ROCKSDB_SHARED_LIB} ${THIRDPARTY_LIBS} ${SYSTEM_LIBS})
- add_library(${ROCKSDB_SHARED_LIB} SHARED ${SOURCES})
+ set(LIBS ${ROCKSDB_STATIC_LIB} ${THIRDPARTY_LIBS} ${SYSTEM_LIBS})
+ #add_library(${ROCKSDB_SHARED_LIB} SHARED ${SOURCES})
# target_link_libraries(${ROCKSDB_SHARED_LIB}
#${THIRDPARTY_LIBS} ${SYSTEM_LIBS})
- set_target_properties(${ROCKSDB_SHARED_LIB} PROPERTIES
-LINKER_LANGUAGE CXX
-VERSION ${ROCKSDB_VERSION}
-SOVERSION ${ROCKSDB_VERSION_MAJOR}
-CXX_STANDARD 11
-OUTPUT_NAME "rocksdb")
+# set_target_properties(${ROCKSDB_SHARED_LIB} PROPERTIES
--- End diff --
Why commented?
If we comment it, even temporary, let's add some comment that justifies.
> Move CAPI code out of libminifi
> ---
>
> Key: MINIFICPP-659
> URL: https://issues.apache.org/jira/browse/MINIFICPP-659
> Project: NiFi MiNiFi C++
> Issue Type: Improvement
>Reporter: Mr TheSegfault
>Assignee: Mr TheSegfault
>Priority: Major
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (MINIFICPP-659) Move CAPI code out of libminifi
[
https://issues.apache.org/jira/browse/MINIFICPP-659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16667810#comment-16667810
]
ASF GitHub Bot commented on MINIFICPP-659:
--
Github user arpadboda commented on a diff in the pull request:
https://github.com/apache/nifi-minifi-cpp/pull/430#discussion_r229111455
--- Diff: nanofi/examples/CMakeLists.txt ---
@@ -61,18 +61,18 @@ add_executable(generate_flow generate_flow.c)
add_executable(terminate_handler terminate_handler.c)
-target_link_libraries(generate_flow capi ${CMAKE_THREAD_LIBS_INIT}
${LINK_FLAGS} minifi ${LINK_END_FLAGS})
+target_link_libraries(generate_flow nanofi ${CMAKE_THREAD_LIBS_INIT})
-target_link_libraries(terminate_handler capi ${CMAKE_THREAD_LIBS_INIT}
${LINK_FLAGS} minifi ${LINK_END_FLAGS})
+target_link_libraries(terminate_handler nanofi ${CMAKE_THREAD_LIBS_INIT})
if (NOT WIN32)
add_executable(transmit_flow transmit_flow.c)
-target_link_libraries(transmit_flow capi ${CMAKE_THREAD_LIBS_INIT}
${LINK_FLAGS} minifi ${LINK_END_FLAGS})
+target_link_libraries(transmit_flow nanofi ${CMAKE_THREAD_LIBS_INIT})
add_executable(monitor_directory monitor_directory.c)
-target_link_libraries(monitor_directory capi ${CMAKE_THREAD_LIBS_INIT}
${LINK_FLAGS} minifi ${LINK_END_FLAGS})
+target_link_libraries(monitor_directory nanofi ${CMAKE_THREAD_LIBS_INIT})
--- End diff --
Are you sure this is fine?
I see that nanofi is linked with whole-archive flag to make sure it
contains all symbols from minifi, however when linking happens here and
monitor_directory binary is created, the linker is still free to drop unused
symbols.
Fix me, I would also be happy if we could skip doing that ugly flag-magic
here. :)
> Move CAPI code out of libminifi
> ---
>
> Key: MINIFICPP-659
> URL: https://issues.apache.org/jira/browse/MINIFICPP-659
> Project: NiFi MiNiFi C++
> Issue Type: Improvement
>Reporter: Mr TheSegfault
>Assignee: Mr TheSegfault
>Priority: Major
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] nifi-minifi-cpp issue #168: MINIFICPP-280 Move extension tests and test head...
Github user arpadboda commented on the issue: https://github.com/apache/nifi-minifi-cpp/pull/168 This PR looks abandoned and seems to require a lot of effort to rebase, @phrocker do you think it worth to be kept open? ---
[jira] [Commented] (MINIFICPP-280) Move extension tests and test headers, lint extensions
[ https://issues.apache.org/jira/browse/MINIFICPP-280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16667814#comment-16667814 ] ASF GitHub Bot commented on MINIFICPP-280: -- Github user arpadboda commented on the issue: https://github.com/apache/nifi-minifi-cpp/pull/168 This PR looks abandoned and seems to require a lot of effort to rebase, @phrocker do you think it worth to be kept open? > Move extension tests and test headers, lint extensions > -- > > Key: MINIFICPP-280 > URL: https://issues.apache.org/jira/browse/MINIFICPP-280 > Project: NiFi MiNiFi C++ > Issue Type: Improvement >Reporter: Caleb Johnson >Assignee: Caleb Johnson >Priority: Minor > > * move extension tests into their respective folders > * separate source and header files > * remove unnecessary or nonexisting include directories > * run linter on extension source files as part of linter target > * clean up extensions according to linter > * add ability to specify more than one include and source folder for linter -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi issue #3111: NIFI-5757 AvroRecordSetWriter - Fix for slow synchronized ...
Github user arkadius commented on the issue: https://github.com/apache/nifi/pull/3111 @markap14 what do you think about using https://github.com/ben-manes/caffeine instead? They have benchmarks showing performance difference from current approach (named `LinkedHashMap_Lru`). Take a look at probably the most important _Read 100%_ scenario. If you want, I can also rewrite other `LinkedHashMap` usages for cache? `grep removeEldestEntry` shows 10 other places. ---
[jira] [Commented] (NIFI-5757) AvroRecordSetWriter synchronize every access to compiledAvroSchemaCache
[
https://issues.apache.org/jira/browse/NIFI-5757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16667822#comment-16667822
]
ASF GitHub Bot commented on NIFI-5757:
--
Github user arkadius commented on the issue:
https://github.com/apache/nifi/pull/3111
@markap14 what do you think about using
https://github.com/ben-manes/caffeine instead? They have benchmarks showing
performance difference from current approach (named `LinkedHashMap_Lru`). Take
a look at probably the most important _Read 100%_ scenario. If you want, I can
also rewrite other `LinkedHashMap` usages for cache? `grep removeEldestEntry`
shows 10 other places.
> AvroRecordSetWriter synchronize every access to compiledAvroSchemaCache
> ---
>
> Key: NIFI-5757
> URL: https://issues.apache.org/jira/browse/NIFI-5757
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Core Framework
>Affects Versions: 1.7.1
>Reporter: Arek Burdach
>Priority: Major
>
> Avro record serialization is a quite expensive operation.
> This stack trace I very often see in thread dumps:
> {noformat}
> Thread 48583: (state = BLOCKED)
> -
> org.apache.nifi.avro.AvroRecordSetWriter.compileAvroSchema(java.lang.String)
> @bci=9, line=124 (Compiled frame)
> -
> org.apache.nifi.avro.AvroRecordSetWriter.createWriter(org.apache.nifi.logging.ComponentLog,
> org.apache.nifi.serialization.record.RecordSchema, java.io.OutputStream)
> @bci=96, line=92 (Compiled frame)
> - sun.reflect.GeneratedMethodAccessor183.invoke(java.lang.Object,
> java.lang.Object[]) @bci=56 (Compiled frame)
> - sun.reflect.DelegatingMethodAccessorImpl.invoke(java.lang.Object,
> java.lang.Object[]) @bci=6, line=43 (Compiled frame)
> - java.lang.reflect.Method.invoke(java.lang.Object, java.lang.Object[])
> @bci=56, line=498 (Compiled frame)
> -
> org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler.invoke(java.lang.Object,
> java.lang.reflect.Method, java.lang.Object[]) @bci=309, line=89 (Compiled
> frame)
> - com.sun.proxy.$Proxy100.createWriter(org.apache.nifi.logging.ComponentLog,
> org.apache.nifi.serialization.record.RecordSchema, java.io.OutputStream)
> @bci=24 (Compiled frame)
> -
> org.apache.nifi.processors.kafka.pubsub.PublisherLease.publish(org.apache.nifi.flowfile.FlowFile,
> org.apache.nifi.serialization.record.RecordSet,
> org.apache.nifi.serialization.RecordSetWriterFactory,
> org.apache.nifi.serialization.record.RecordSchema, java.lang.String,
> java.lang.String) @bci=71, line=169 (Compiled frame)
> -
> org.apache.nifi.processors.kafka.pubsub.PublishKafkaRecord_1_0$1.process(java.io.InputStream)
> @bci=94, line=412 (Compiled frame)
> {noformat}
> The reason why it happens is because {{AvroRecordSetWriter}} synchronizing
> every access to cache of compiled schemas.
> I've prepared PR that is fixing this issue by using {{ConcurrentHashMap}}
> instead: https://github.com/apache/nifi/pull/3111
> It is not a perfect fix because it removes cache size limitation which BTW
> was hardcoded to {{20}}. Services can be reusable by many flows so such a
> hard limit is not a good choice.
> What do you think about such an improvement?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] nifi-minifi-cpp issue #168: MINIFICPP-280 Move extension tests and test head...
Github user phrocker commented on the issue: https://github.com/apache/nifi-minifi-cpp/pull/168 @arpadboda I started a branch as of my last comment -- but it's only a small part of my work, so it will take a bit of time. I don't think there is any harm in keeping it open ---
[jira] [Commented] (MINIFICPP-280) Move extension tests and test headers, lint extensions
[ https://issues.apache.org/jira/browse/MINIFICPP-280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16667895#comment-16667895 ] ASF GitHub Bot commented on MINIFICPP-280: -- Github user phrocker commented on the issue: https://github.com/apache/nifi-minifi-cpp/pull/168 @arpadboda I started a branch as of my last comment -- but it's only a small part of my work, so it will take a bit of time. I don't think there is any harm in keeping it open > Move extension tests and test headers, lint extensions > -- > > Key: MINIFICPP-280 > URL: https://issues.apache.org/jira/browse/MINIFICPP-280 > Project: NiFi MiNiFi C++ > Issue Type: Improvement >Reporter: Caleb Johnson >Assignee: Caleb Johnson >Priority: Minor > > * move extension tests into their respective folders > * separate source and header files > * remove unnecessary or nonexisting include directories > * run linter on extension source files as part of linter target > * clean up extensions according to linter > * add ability to specify more than one include and source folder for linter -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi-minifi-cpp pull request #430: MINIFICPP-659: Break out CAPI into nanofi
Github user phrocker commented on a diff in the pull request:
https://github.com/apache/nifi-minifi-cpp/pull/430#discussion_r229138444
--- Diff: nanofi/examples/CMakeLists.txt ---
@@ -61,18 +61,18 @@ add_executable(generate_flow generate_flow.c)
add_executable(terminate_handler terminate_handler.c)
-target_link_libraries(generate_flow capi ${CMAKE_THREAD_LIBS_INIT}
${LINK_FLAGS} minifi ${LINK_END_FLAGS})
+target_link_libraries(generate_flow nanofi ${CMAKE_THREAD_LIBS_INIT})
-target_link_libraries(terminate_handler capi ${CMAKE_THREAD_LIBS_INIT}
${LINK_FLAGS} minifi ${LINK_END_FLAGS})
+target_link_libraries(terminate_handler nanofi ${CMAKE_THREAD_LIBS_INIT})
if (NOT WIN32)
add_executable(transmit_flow transmit_flow.c)
-target_link_libraries(transmit_flow capi ${CMAKE_THREAD_LIBS_INIT}
${LINK_FLAGS} minifi ${LINK_END_FLAGS})
+target_link_libraries(transmit_flow nanofi ${CMAKE_THREAD_LIBS_INIT})
add_executable(monitor_directory monitor_directory.c)
-target_link_libraries(monitor_directory capi ${CMAKE_THREAD_LIBS_INIT}
${LINK_FLAGS} minifi ${LINK_END_FLAGS})
+target_link_libraries(monitor_directory nanofi ${CMAKE_THREAD_LIBS_INIT})
--- End diff --
Cmake implants those target libs in place. Therefore, I get
/usr/bin/c++ -std=c++11 -DOPENSSL_SUPPORT -std=c++11 -std=c++11
-rdynamic CMakeFiles/monitor_directory.dir/monitor_directory.c.o -o
monitor_directory
-Wl,-rpath,/home/marc/dev/nifi-minifi-cpp/build/thirdparty/uuid:/home/marc/dev/nifi-minifi-cpp/build/thirdparty/yaml-cpp-yaml-cpp-20171024
../libnanofi.a -lpthread -Wl,--whole-archive ../../libminifi/libminifi.a
../../libminifi/libcore-minifi.a -ldl ../../thirdparty/uuid/libuuid.so
../../thirdparty/yaml-cpp-yaml-cpp-20171024/libyaml-cpp.so.0.5.3
../../extensions/expression-language/noop/libminifi-expression-language-extensions.a
-lz -lssl -lcrypto -Wl,--no-whole-archive
Even though I had the above build statement. This is because of the way
CMAKE injects the targets, so there isn't the need for the extraneous target
protections. On my system monitror_directory works appropriately after
testing, but this does mean that MINIFICPP-660 should probably be changed to
enable better testing these examples in docker images. I will take a look at
that since I've already started 660
---
[jira] [Commented] (MINIFICPP-659) Move CAPI code out of libminifi
[
https://issues.apache.org/jira/browse/MINIFICPP-659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16667908#comment-16667908
]
ASF GitHub Bot commented on MINIFICPP-659:
--
Github user phrocker commented on a diff in the pull request:
https://github.com/apache/nifi-minifi-cpp/pull/430#discussion_r229138444
--- Diff: nanofi/examples/CMakeLists.txt ---
@@ -61,18 +61,18 @@ add_executable(generate_flow generate_flow.c)
add_executable(terminate_handler terminate_handler.c)
-target_link_libraries(generate_flow capi ${CMAKE_THREAD_LIBS_INIT}
${LINK_FLAGS} minifi ${LINK_END_FLAGS})
+target_link_libraries(generate_flow nanofi ${CMAKE_THREAD_LIBS_INIT})
-target_link_libraries(terminate_handler capi ${CMAKE_THREAD_LIBS_INIT}
${LINK_FLAGS} minifi ${LINK_END_FLAGS})
+target_link_libraries(terminate_handler nanofi ${CMAKE_THREAD_LIBS_INIT})
if (NOT WIN32)
add_executable(transmit_flow transmit_flow.c)
-target_link_libraries(transmit_flow capi ${CMAKE_THREAD_LIBS_INIT}
${LINK_FLAGS} minifi ${LINK_END_FLAGS})
+target_link_libraries(transmit_flow nanofi ${CMAKE_THREAD_LIBS_INIT})
add_executable(monitor_directory monitor_directory.c)
-target_link_libraries(monitor_directory capi ${CMAKE_THREAD_LIBS_INIT}
${LINK_FLAGS} minifi ${LINK_END_FLAGS})
+target_link_libraries(monitor_directory nanofi ${CMAKE_THREAD_LIBS_INIT})
--- End diff --
Cmake implants those target libs in place. Therefore, I get
/usr/bin/c++ -std=c++11 -DOPENSSL_SUPPORT -std=c++11 -std=c++11
-rdynamic CMakeFiles/monitor_directory.dir/monitor_directory.c.o -o
monitor_directory
-Wl,-rpath,/home/marc/dev/nifi-minifi-cpp/build/thirdparty/uuid:/home/marc/dev/nifi-minifi-cpp/build/thirdparty/yaml-cpp-yaml-cpp-20171024
../libnanofi.a -lpthread -Wl,--whole-archive ../../libminifi/libminifi.a
../../libminifi/libcore-minifi.a -ldl ../../thirdparty/uuid/libuuid.so
../../thirdparty/yaml-cpp-yaml-cpp-20171024/libyaml-cpp.so.0.5.3
../../extensions/expression-language/noop/libminifi-expression-language-extensions.a
-lz -lssl -lcrypto -Wl,--no-whole-archive
Even though I had the above build statement. This is because of the way
CMAKE injects the targets, so there isn't the need for the extraneous target
protections. On my system monitror_directory works appropriately after
testing, but this does mean that MINIFICPP-660 should probably be changed to
enable better testing these examples in docker images. I will take a look at
that since I've already started 660
> Move CAPI code out of libminifi
> ---
>
> Key: MINIFICPP-659
> URL: https://issues.apache.org/jira/browse/MINIFICPP-659
> Project: NiFi MiNiFi C++
> Issue Type: Improvement
>Reporter: Mr TheSegfault
>Assignee: Mr TheSegfault
>Priority: Major
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] nifi-minifi-cpp pull request #430: MINIFICPP-659: Break out CAPI into nanofi
Github user phrocker commented on a diff in the pull request:
https://github.com/apache/nifi-minifi-cpp/pull/430#discussion_r229138685
--- Diff: thirdparty/rocksdb/CMakeLists.txt ---
@@ -592,16 +592,16 @@ if(WIN32)
set(LIBS ${ROCKSDB_STATIC_LIB} ${THIRDPARTY_LIBS} ${SYSTEM_LIBS})
else()
set(SYSTEM_LIBS ${CMAKE_THREAD_LIBS_INIT})
- set(LIBS ${ROCKSDB_SHARED_LIB} ${THIRDPARTY_LIBS} ${SYSTEM_LIBS})
- add_library(${ROCKSDB_SHARED_LIB} SHARED ${SOURCES})
+ set(LIBS ${ROCKSDB_STATIC_LIB} ${THIRDPARTY_LIBS} ${SYSTEM_LIBS})
+ #add_library(${ROCKSDB_SHARED_LIB} SHARED ${SOURCES})
# target_link_libraries(${ROCKSDB_SHARED_LIB}
#${THIRDPARTY_LIBS} ${SYSTEM_LIBS})
- set_target_properties(${ROCKSDB_SHARED_LIB} PROPERTIES
-LINKER_LANGUAGE CXX
-VERSION ${ROCKSDB_VERSION}
-SOVERSION ${ROCKSDB_VERSION_MAJOR}
-CXX_STANDARD 11
-OUTPUT_NAME "rocksdb")
+# set_target_properties(${ROCKSDB_SHARED_LIB} PROPERTIES
--- End diff --
My OSX machine is having some issues so I left this here and remove once
OSX passes. Was hoping I could do a quick change but it seems that OSX is
failing. It will be removed with a comment telling maintainers why the removal
happened.
---
[jira] [Commented] (MINIFICPP-659) Move CAPI code out of libminifi
[
https://issues.apache.org/jira/browse/MINIFICPP-659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16667911#comment-16667911
]
ASF GitHub Bot commented on MINIFICPP-659:
--
Github user phrocker commented on a diff in the pull request:
https://github.com/apache/nifi-minifi-cpp/pull/430#discussion_r229138685
--- Diff: thirdparty/rocksdb/CMakeLists.txt ---
@@ -592,16 +592,16 @@ if(WIN32)
set(LIBS ${ROCKSDB_STATIC_LIB} ${THIRDPARTY_LIBS} ${SYSTEM_LIBS})
else()
set(SYSTEM_LIBS ${CMAKE_THREAD_LIBS_INIT})
- set(LIBS ${ROCKSDB_SHARED_LIB} ${THIRDPARTY_LIBS} ${SYSTEM_LIBS})
- add_library(${ROCKSDB_SHARED_LIB} SHARED ${SOURCES})
+ set(LIBS ${ROCKSDB_STATIC_LIB} ${THIRDPARTY_LIBS} ${SYSTEM_LIBS})
+ #add_library(${ROCKSDB_SHARED_LIB} SHARED ${SOURCES})
# target_link_libraries(${ROCKSDB_SHARED_LIB}
#${THIRDPARTY_LIBS} ${SYSTEM_LIBS})
- set_target_properties(${ROCKSDB_SHARED_LIB} PROPERTIES
-LINKER_LANGUAGE CXX
-VERSION ${ROCKSDB_VERSION}
-SOVERSION ${ROCKSDB_VERSION_MAJOR}
-CXX_STANDARD 11
-OUTPUT_NAME "rocksdb")
+# set_target_properties(${ROCKSDB_SHARED_LIB} PROPERTIES
--- End diff --
My OSX machine is having some issues so I left this here and remove once
OSX passes. Was hoping I could do a quick change but it seems that OSX is
failing. It will be removed with a comment telling maintainers why the removal
happened.
> Move CAPI code out of libminifi
> ---
>
> Key: MINIFICPP-659
> URL: https://issues.apache.org/jira/browse/MINIFICPP-659
> Project: NiFi MiNiFi C++
> Issue Type: Improvement
>Reporter: Mr TheSegfault
>Assignee: Mr TheSegfault
>Priority: Major
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-5764) Allow ListSftp connection parameter
[
https://issues.apache.org/jira/browse/NIFI-5764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16667931#comment-16667931
]
Koji Kawamura commented on NIFI-5764:
-
For reference about what ControlMaster is, from $ man ssh_config:
{quote}
ControlMaster
Enables the sharing of multiple sessions over a single network connection.
When set to yes, ssh(1) will listen for connections on a control socket
specified using the ControlPath argument. Additional sessions can connect to
this socket using the same ControlPath with ControlMaster set to no (the
default). These sessions will try to reuse the master instance's network
connection rather than initiating new ones, but will fall back to connecting
normally if the control socket does not exist, or is not listening.
Setting this to ask will cause ssh(1) to listen for control connections, but
require confirmation using ssh-askpass(1). If the ControlPath cannot be
opened, ssh(1) will continue without connecting to a master instance.
X11 and ssh-agent(1) forwarding is supported over these multiplexed
connections, however the display and agent forwarded will be the one belonging
to the master connec- tion i.e. it is not possible to forward multiple displays
or agents.
Two additional options allow for opportunistic multiplexing: try to use a
master connection but fall back to creating a new one if one does not already
exist. These options are: auto and autoask. The latter requires confirmation
like the ask option.
{quote}
> Allow ListSftp connection parameter
> ---
>
> Key: NIFI-5764
> URL: https://issues.apache.org/jira/browse/NIFI-5764
> Project: Apache NiFi
> Issue Type: Wish
> Components: Extensions
>Affects Versions: 1.6.0
>Reporter: dav
>Priority: Critical
> Labels: SFTP, customization, sftp
>
> ListSftp and other Sftp processors should be able to add parameters
> (like [-B buffer_size] [-b batchfile] [-c cipher]
> [-D sftp_server_path] [-F ssh_config] [-i identity_file] [-l limit]
> [-o ssh_option] [-P port] [-R num_requests] [-S program]
> [-s subsystem | sftp_server] host
> sftp [user@]host[:file ...]
> sftp [user@]host[:dir[/]]
> sftp -b batchfile [user@]host)
> in order to edit the type of connection on Sftp Server.
> For instance, I have this error on nifi:
> 2018-10-29 11:06:09,462 ERROR [Timer-Driven Process Thread-5]
> SimpleProcessLogger.java:254
> ListSFTP[id=766ac418-27ce-335a-5b13-52abe3495592] Failed to perform listing
> on remote host due to java.io.IOException: Failed to obtain connection to
> remote host due to com.jcraft.jsch.JSchException: Auth fail: {}
> java.io.IOException: Failed to obtain connection to remote host due to
> com.jcraft.jsch.JSchException: Auth fail
> at
> org.apache.nifi.processors.standard.util.SFTPTransfer.getChannel(SFTPTransfer.java:468)
> at
> org.apache.nifi.processors.standard.util.SFTPTransfer.getListing(SFTPTransfer.java:192)
> at
> org.apache.nifi.processors.standard.util.SFTPTransfer.getListing(SFTPTransfer.java:156)
> at
> org.apache.nifi.processors.standard.ListFileTransfer.performListing(ListFileTransfer.java:105)
> at
> org.apache.nifi.processor.util.list.AbstractListProcessor.onTrigger(AbstractListProcessor.java:401)
> at
> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
> at
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1147)
> at
> org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:175)
> at
> org.apache.nifi.controller.scheduling.QuartzSchedulingAgent$2.run(QuartzSchedulingAgent.java:140)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: com.jcraft.jsch.JSchException: Auth fail
> at com.jcraft.jsch.Session.connect(Session.java:519)
> at com.jcraft.jsch.Session.connect(Session.java:183)
> at
> org.apache.nifi.processors.standard.util.SFTPTransfer.getChannel(SFTPTransfer.java:448)
> ... 15 common frames omitted
> This can be avoided by connect to Sftp server with this string:
> *sftp -o “controlmaster auto” username@sftp_server*
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-5761) ReplaceText processor can stop processing data if it evaluates invalid expressions
[ https://issues.apache.org/jira/browse/NIFI-5761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16667944#comment-16667944 ] ASF GitHub Bot commented on NIFI-5761: -- Github user ijokarumawak commented on a diff in the pull request: https://github.com/apache/nifi/pull/3112#discussion_r229144580 --- Diff: nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/pom.xml --- @@ -24,6 +24,11 @@ nifi-api provided + +org.apache.nifi +nifi-expression-language +provided --- End diff -- Please remove the `provided` scope. The `nifi-expression-language` needs to be bundled in nifi-standard-processors.nar. Causing following exception: ``` java.util.ServiceConfigurationError: org.apache.nifi.processor.Processor: Provider org.apache.nifi.processors.standard.ReplaceText could not be instantiated at java.util.ServiceLoader.fail(ServiceLoader.java:232) at java.util.ServiceLoader.access$100(ServiceLoader.java:185) at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:384) at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404) at java.util.ServiceLoader$1.next(ServiceLoader.java:480) at org.apache.nifi.nar.ExtensionManager.loadExtensions(ExtensionManager.java:148) at org.apache.nifi.nar.ExtensionManager.discoverExtensions(ExtensionManager.java:123) at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:838) at org.apache.nifi.NiFi.(NiFi.java:157) at org.apache.nifi.NiFi.(NiFi.java:71) at org.apache.nifi.NiFi.main(NiFi.java:296) Caused by: java.lang.NoClassDefFoundError: org/apache/nifi/attribute/expression/language/exception/IllegalAttributeException at java.lang.Class.getDeclaredConstructors0(Native Method) at java.lang.Class.privateGetDeclaredConstructors(Class.java:2671) at java.lang.Class.getConstructor0(Class.java:3075) at java.lang.Class.newInstance(Class.java:412) at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:380) ... 8 common frames omitted Caused by: java.lang.ClassNotFoundException: org.apache.nifi.attribute.expression.language.exception.IllegalAttributeException at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ... 13 common frames omitted ``` > ReplaceText processor can stop processing data if it evaluates invalid > expressions > -- > > Key: NIFI-5761 > URL: https://issues.apache.org/jira/browse/NIFI-5761 > Project: Apache NiFi > Issue Type: Bug > Components: Extensions >Affects Versions: 1.5.0, 1.7.1 > Environment: ALL >Reporter: Gardella Juan Pablo >Assignee: Gardella Juan Pablo >Priority: Major > Original Estimate: 3h > Remaining Estimate: 3h > > Given a flowfile body with nifi expression, when _ReplaceText_ processor > evaluates it and the expression throws an exception, the processor will > rollback the flowfile and keep trying to evaluate instead of send the > flowfile to _failure_ relationshipt. > Discussion Thread: > http://apache-nifi-users-list.2361937.n4.nabble.com/ReplaceText-cannot-consume-messages-if-Regex-does-not-match-td5986.html -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi pull request #3112: NIFI-5761 ReplaceText processor can stop processing...
Github user ijokarumawak commented on a diff in the pull request: https://github.com/apache/nifi/pull/3112#discussion_r229144580 --- Diff: nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/pom.xml --- @@ -24,6 +24,11 @@ nifi-api provided + +org.apache.nifi +nifi-expression-language +provided --- End diff -- Please remove the `provided` scope. The `nifi-expression-language` needs to be bundled in nifi-standard-processors.nar. Causing following exception: ``` java.util.ServiceConfigurationError: org.apache.nifi.processor.Processor: Provider org.apache.nifi.processors.standard.ReplaceText could not be instantiated at java.util.ServiceLoader.fail(ServiceLoader.java:232) at java.util.ServiceLoader.access$100(ServiceLoader.java:185) at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:384) at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404) at java.util.ServiceLoader$1.next(ServiceLoader.java:480) at org.apache.nifi.nar.ExtensionManager.loadExtensions(ExtensionManager.java:148) at org.apache.nifi.nar.ExtensionManager.discoverExtensions(ExtensionManager.java:123) at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:838) at org.apache.nifi.NiFi.(NiFi.java:157) at org.apache.nifi.NiFi.(NiFi.java:71) at org.apache.nifi.NiFi.main(NiFi.java:296) Caused by: java.lang.NoClassDefFoundError: org/apache/nifi/attribute/expression/language/exception/IllegalAttributeException at java.lang.Class.getDeclaredConstructors0(Native Method) at java.lang.Class.privateGetDeclaredConstructors(Class.java:2671) at java.lang.Class.getConstructor0(Class.java:3075) at java.lang.Class.newInstance(Class.java:412) at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:380) ... 8 common frames omitted Caused by: java.lang.ClassNotFoundException: org.apache.nifi.attribute.expression.language.exception.IllegalAttributeException at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ... 13 common frames omitted ``` ---
[GitHub] nifi issue #3114: NIFI-5765 Fixing WriteJsonResult to use chosenDataType whe...
Github user ijokarumawak commented on the issue: https://github.com/apache/nifi/pull/3114 Reviewing... ---
[jira] [Commented] (NIFI-5765) WriteJsonResult fails with Class Cast Exception when Choice data type resolves to Array
[ https://issues.apache.org/jira/browse/NIFI-5765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16668013#comment-16668013 ] ASF GitHub Bot commented on NIFI-5765: -- Github user ijokarumawak commented on the issue: https://github.com/apache/nifi/pull/3114 Reviewing... > WriteJsonResult fails with Class Cast Exception when Choice data type > resolves to Array > --- > > Key: NIFI-5765 > URL: https://issues.apache.org/jira/browse/NIFI-5765 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.8.0 >Reporter: Joseph Percivall >Assignee: Joseph Percivall >Priority: Major > Attachments: Screen Shot 2018-10-29 at 2.35.50 PM.png, > WriteJsonResult_Choice_Array_example.xml > > > The problem is this line[1]. For the casting, it uses the passed in value > instead of the chosen data type. > A template demonstrating the problem is attached. The corner case is hit when > there is a choice of data types, and the data type chosen is Array. It > properly does everything else but fails when casting it due to: > org.apache.nifi.serialization.record.type.ChoiceDataType cannot be cast to > org.apache.nifi.serialization.record.type.ArrayDataType. > [1] > https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-standard-services/nifi-record-serialization-services-bundle/nifi-record-serialization-services/src/main/java/org/apache/nifi/json/WriteJsonResult.java#L379 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi pull request #3112: NIFI-5761 ReplaceText processor can stop processing...
Github user gardellajuanpablo commented on a diff in the pull request: https://github.com/apache/nifi/pull/3112#discussion_r229159256 --- Diff: nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/pom.xml --- @@ -24,6 +24,11 @@ nifi-api provided + +org.apache.nifi +nifi-expression-language +provided --- End diff -- Thanks, updated. ---
[jira] [Commented] (NIFI-5761) ReplaceText processor can stop processing data if it evaluates invalid expressions
[ https://issues.apache.org/jira/browse/NIFI-5761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16668027#comment-16668027 ] ASF GitHub Bot commented on NIFI-5761: -- Github user gardellajuanpablo commented on a diff in the pull request: https://github.com/apache/nifi/pull/3112#discussion_r229159256 --- Diff: nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/pom.xml --- @@ -24,6 +24,11 @@ nifi-api provided + +org.apache.nifi +nifi-expression-language +provided --- End diff -- Thanks, updated. > ReplaceText processor can stop processing data if it evaluates invalid > expressions > -- > > Key: NIFI-5761 > URL: https://issues.apache.org/jira/browse/NIFI-5761 > Project: Apache NiFi > Issue Type: Bug > Components: Extensions >Affects Versions: 1.5.0, 1.7.1 > Environment: ALL >Reporter: Gardella Juan Pablo >Assignee: Gardella Juan Pablo >Priority: Major > Original Estimate: 3h > Remaining Estimate: 3h > > Given a flowfile body with nifi expression, when _ReplaceText_ processor > evaluates it and the expression throws an exception, the processor will > rollback the flowfile and keep trying to evaluate instead of send the > flowfile to _failure_ relationshipt. > Discussion Thread: > http://apache-nifi-users-list.2361937.n4.nabble.com/ReplaceText-cannot-consume-messages-if-Regex-does-not-match-td5986.html -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi pull request #3114: NIFI-5765 Fixing WriteJsonResult to use chosenDataT...
Github user ijokarumawak commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3114#discussion_r229158995
--- Diff:
nifi-nar-bundles/nifi-standard-services/nifi-record-serialization-services-bundle/nifi-record-serialization-services/src/test/java/org/apache/nifi/json/TestWriteJsonResult.java
---
@@ -472,4 +472,34 @@ public void testOnelineOutput() throws IOException {
final String output = new String(data, StandardCharsets.UTF_8);
assertEquals(expected, output);
}
+
+@Test
+public void testChoiceArray() throws IOException {
+final List fields = new ArrayList<>();
+fields.add(new RecordField("path",
RecordFieldType.CHOICE.getChoiceDataType(RecordFieldType.ARRAY.getArrayDataType(RecordFieldType.STRING.getDataType();
+//fields.add(new RecordField("path",
RecordFieldType.ARRAY.getArrayDataType(RecordFieldType.STRING.getDataType(;
--- End diff --
I'll remove this comment line when I merge this.
---
