[jira] [Created] (MINIFICPP-2006) saving state for python processors
Kondakov Artem created MINIFICPP-2006: - Summary: saving state for python processors Key: MINIFICPP-2006 URL: https://issues.apache.org/jira/browse/MINIFICPP-2006 Project: Apache NiFi MiNiFi C++ Issue Type: New Feature Affects Versions: 0.12.0 Reporter: Kondakov Artem Fix For: 0.14.0 Hello. I'm using InvokeHTTP to periodically fetch data from the server. The data is received and in the response I can get the ID of the last returned record. Then through I save the data in a DB. How can I save the value of the identifier of this last record in order to form a new request to the server based on it. I know that the QueryDatabaseTable processor can save the ID of the last record received from the database. How to make an analogue of the same functionality? Is it possible to save the state of the data for the python processor in the same way as for QueryDatabaseTable? Is it possible to get the state of one processor from another (for example, in one Python processor I save the state of the id of the last record), and in another I take the identifier from the state of the first one, and form a new request? -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (MINIFICPP-1999) Persistent state of processors
Kondakov Artem created MINIFICPP-1999:
-
Summary: Persistent state of processors
Key: MINIFICPP-1999
URL: https://issues.apache.org/jira/browse/MINIFICPP-1999
Project: Apache NiFi MiNiFi C++
Issue Type: New Feature
Affects Versions: 0.12.0
Reporter: Kondakov Artem
Fix For: 0.13.0
Hi
In apache/nifi each processor can be started and stopped. It is also possible
to activate and deactivate. In this case, the state of the processors is saved
when the application is restarted.
minifi-cpp also has the ability to start and stop individual processors using
the C2 protocol or minificontroller (by passing the command and the name of the
processor (it's strange that you can't pass its uid))
However, after restarting minifi all the same, all processors start, while the
permanent state store is configured
{color:#172b4d}nifi.state.management.provider.local.path=./repositories/componentstate{color}
{color:#00875a}## To make the default state storage persist every state change,
set this to true{color}
{color:#00875a}## this comes at a performance penalty, but makes sure no state
is lost even on unclean shutdowns{color}
{color:#172b4d}nifi.state.management.provider.local.always.persist=true{color}
{color:#00875a}## To change the frequency at which the default state storage is
persisted, modify the following{color}
{color:#172b4d}nifi.state.management.provider.local.auto.persistence.interval=1min{color}
I think it would be very helpful
- support state storage for processors (active/inactive)
- support changing and storing state also for other types of nodes (input
ports, output ports, funnels, process groups)
- adding the "enabled" property to the properties of processors in yaml (and
other types of nodes) by analogy with Apache / nifi,
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (MINIFICPP-1994) Missing "failure", "retry" relationships for PutSql processor (and other processors)
[ https://issues.apache.org/jira/browse/MINIFICPP-1994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17640717#comment-17640717 ] Kondakov Artem edited comment on MINIFICPP-1994 at 11/29/22 1:51 PM: - Hello, [~aboda] "In case a processor has "INPUT_FORBIDDEN" requirement, that means that the given processor is a source (usually the ones that match list* or get* or consume*), so those don't need failure relationship. They just create a flowfile based on the data they receive, no flowfile is processed, so the processing can't go wrong. " Then how should resource availability errors be handled in this case? For example, I am trying to get a list of files using org.apache.nifi.minifi.processors.ListSFTP by cron schedule. If the remote server is not available at the moment, how do I handle this case I think for some other processors, a completely unforeseen situation may also arise when something went wrong, and at the same time I cannot make an alert, because no flow-file will be formed was (Author: JIRAUSER287304): Hello, [~aboda] "In case a processor has "INPUT_FORBIDDEN" requirement, that means that the given processor is a source (usually the ones that match list* or get* or consume*), so those don't need failure relationship. They just create a flowfile based on the data they receive, no flowfile is processed, so the processing can't go wrong. " Then how should resource availability errors be handled in this case? For example, I am trying to get a list of files using org.apache.nifi.minifi.processors.ListSFTP by cron schedule. If the remote server is not available at the moment, how do I handle this case > Missing "failure", "retry" relationships for PutSql processor (and other > processors) > > > Key: MINIFICPP-1994 > URL: https://issues.apache.org/jira/browse/MINIFICPP-1994 > Project: Apache NiFi MiNiFi C++ > Issue Type: New Feature >Reporter: Kondakov Artem >Assignee: Adam Debreceni >Priority: Major > > According to the manifest, sent by minifi, PutSql processors (and many > others) do not have a failure relationship, while in the apache / nifi > project similar processors have it. > I think maintaining these relationships is important. > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (MINIFICPP-1994) Missing "failure", "retry" relationships for PutSql processor (and other processors)
[ https://issues.apache.org/jira/browse/MINIFICPP-1994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17640717#comment-17640717 ] Kondakov Artem commented on MINIFICPP-1994: --- Hello, [~aboda] "In case a processor has "INPUT_FORBIDDEN" requirement, that means that the given processor is a source (usually the ones that match list* or get* or consume*), so those don't need failure relationship. They just create a flowfile based on the data they receive, no flowfile is processed, so the processing can't go wrong. " Then how should resource availability errors be handled in this case? For example, I am trying to get a list of files using org.apache.nifi.minifi.processors.ListSFTP by cron schedule. If the remote server is not available at the moment, how do I handle this case > Missing "failure", "retry" relationships for PutSql processor (and other > processors) > > > Key: MINIFICPP-1994 > URL: https://issues.apache.org/jira/browse/MINIFICPP-1994 > Project: Apache NiFi MiNiFi C++ > Issue Type: New Feature >Reporter: Kondakov Artem >Assignee: Adam Debreceni >Priority: Major > > According to the manifest, sent by minifi, PutSql processors (and many > others) do not have a failure relationship, while in the apache / nifi > project similar processors have it. > I think maintaining these relationships is important. > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (MINIFICPP-1995) Add configuring path for flowfile_checkpoint directory
[ https://issues.apache.org/jira/browse/MINIFICPP-1995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kondakov Artem updated MINIFICPP-1995: -- Description: In the minifi.properties file, there is no way to set the directory path for flowfile_checkpoint, similar to * nifi.provenance.repository.directory.default * nifi.flowfile.repository.directory.default * nifi.state.management.provider.local.path was: In the minifi.properties file, there is no way to set the directory path for flowfile_checkpoint, similar to -nifi.provenance.repository.directory.default - nifi.flowfile.repository.directory.default -nifi.state.management.provider.local.path > Add configuring path for flowfile_checkpoint directory > -- > > Key: MINIFICPP-1995 > URL: https://issues.apache.org/jira/browse/MINIFICPP-1995 > Project: Apache NiFi MiNiFi C++ > Issue Type: New Feature >Affects Versions: 0.12.0 >Reporter: Kondakov Artem >Priority: Major > > In the minifi.properties file, there is no way to set the directory path for > flowfile_checkpoint, similar to > * nifi.provenance.repository.directory.default > * nifi.flowfile.repository.directory.default > * nifi.state.management.provider.local.path -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (MINIFICPP-1995) Add configuring path for flowfile_checkpoint directory
Kondakov Artem created MINIFICPP-1995: - Summary: Add configuring path for flowfile_checkpoint directory Key: MINIFICPP-1995 URL: https://issues.apache.org/jira/browse/MINIFICPP-1995 Project: Apache NiFi MiNiFi C++ Issue Type: New Feature Affects Versions: 0.12.0 Reporter: Kondakov Artem In the minifi.properties file, there is no way to set the directory path for flowfile_checkpoint, similar to -nifi.provenance.repository.directory.default - nifi.flowfile.repository.directory.default -nifi.state.management.provider.local.path -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (MINIFICPP-1990) Transferring FlowFiles in Python Processors
[ https://issues.apache.org/jira/browse/MINIFICPP-1990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kondakov Artem updated MINIFICPP-1990: -- Description: Hello. There is a problem with the processing of flow-files in python processors, which are connected using the setting nifi.python.processor.dir=./minifi-python/ If you need to split one file into several, then you have to create new files in the Python processor and be sure to transfer the original one to REL_FAILURE. Otherwise, if I do not pass the original file to REL_FAILURE, then the processing result is rolled back and nothing happens It seems to me that it is necessary to make it possible for Python processors to add various relations, and not just REL_FAILURE REL_SUCCESS. Or add an additional possible relationship REL_ORIGIN to which you can pass the original file That. if you look at the metrics for these processors, you see TransferredToFaillure > 0, although the processors processed without errors Another additional possibility is to add the session.remove (flowfile ) method (as it is done for example in the processor ExecuteSQL). Then it will be possible to exclude incoming files from further processing !image-2022-11-23-20-03-53-653.png! was: Hello. There is a problem with the processing of flow-files in python processors, which are connected using the setting nifi.python.processor.dir=./minifi-python/ If you need to split one file into several, then you have to create new files in the Python processor and be sure to transfer the original one to REL_FAILURE. Otherwise, if I do not pass the original file to REL_FAILURE, then the processing result is rolled back and nothing happens It seems to me that it is necessary to make it possible for Python processors to add various relations, and not just REL_FAILURE REL_SUCCESS. Or add an additional possible relationship REL_ORIGIN to which you can pass the original file That. if you look at the metrics for these processors, you see TransferredToFaillure > 0, although the processors processed without errors > Transferring FlowFiles in Python Processors > --- > > Key: MINIFICPP-1990 > URL: https://issues.apache.org/jira/browse/MINIFICPP-1990 > Project: Apache NiFi MiNiFi C++ > Issue Type: Bug >Affects Versions: 0.12.0 >Reporter: Kondakov Artem >Priority: Major > Fix For: 0.13.0 > > Attachments: image-2022-11-23-20-03-09-400.png, > image-2022-11-23-20-03-53-653.png > > > Hello. > There is a problem with the processing of flow-files in python processors, > which are connected using the setting > nifi.python.processor.dir=./minifi-python/ > If you need to split one file into several, then you have to create new files > in the Python processor and be sure to transfer the original one to > REL_FAILURE. > Otherwise, if I do not pass the original file to REL_FAILURE, then the > processing result is rolled back and nothing happens > It seems to me that it is necessary to make it possible for Python processors > to add various relations, and not just REL_FAILURE REL_SUCCESS. > Or add an additional possible relationship REL_ORIGIN to which you can pass > the original file > That. if you look at the metrics for these processors, you see > TransferredToFaillure > 0, although the processors processed without errors > Another additional possibility is to add the session.remove (flowfile ) > method (as it is done for example in the processor ExecuteSQL). Then it will > be possible to exclude incoming files from further processing > !image-2022-11-23-20-03-53-653.png! > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (MINIFICPP-1990) Transferring FlowFiles in Python Processors
[ https://issues.apache.org/jira/browse/MINIFICPP-1990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kondakov Artem updated MINIFICPP-1990: -- Attachment: image-2022-11-23-20-03-09-400.png > Transferring FlowFiles in Python Processors > --- > > Key: MINIFICPP-1990 > URL: https://issues.apache.org/jira/browse/MINIFICPP-1990 > Project: Apache NiFi MiNiFi C++ > Issue Type: Bug >Affects Versions: 0.12.0 >Reporter: Kondakov Artem >Priority: Major > Fix For: 0.13.0 > > Attachments: image-2022-11-23-20-03-09-400.png, > image-2022-11-23-20-03-53-653.png > > > Hello. > There is a problem with the processing of flow-files in python processors, > which are connected using the setting > nifi.python.processor.dir=./minifi-python/ > If you need to split one file into several, then you have to create new files > in the Python processor and be sure to transfer the original one to > REL_FAILURE. > Otherwise, if I do not pass the original file to REL_FAILURE, then the > processing result is rolled back and nothing happens > It seems to me that it is necessary to make it possible for Python processors > to add various relations, and not just REL_FAILURE REL_SUCCESS. > Or add an additional possible relationship REL_ORIGIN to which you can pass > the original file > That. if you look at the metrics for these processors, you see > TransferredToFaillure > 0, although the processors processed without errors -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (MINIFICPP-1990) Transferring FlowFiles in Python Processors
[ https://issues.apache.org/jira/browse/MINIFICPP-1990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kondakov Artem updated MINIFICPP-1990: -- Attachment: image-2022-11-23-20-03-53-653.png > Transferring FlowFiles in Python Processors > --- > > Key: MINIFICPP-1990 > URL: https://issues.apache.org/jira/browse/MINIFICPP-1990 > Project: Apache NiFi MiNiFi C++ > Issue Type: Bug >Affects Versions: 0.12.0 >Reporter: Kondakov Artem >Priority: Major > Fix For: 0.13.0 > > Attachments: image-2022-11-23-20-03-09-400.png, > image-2022-11-23-20-03-53-653.png > > > Hello. > There is a problem with the processing of flow-files in python processors, > which are connected using the setting > nifi.python.processor.dir=./minifi-python/ > If you need to split one file into several, then you have to create new files > in the Python processor and be sure to transfer the original one to > REL_FAILURE. > Otherwise, if I do not pass the original file to REL_FAILURE, then the > processing result is rolled back and nothing happens > It seems to me that it is necessary to make it possible for Python processors > to add various relations, and not just REL_FAILURE REL_SUCCESS. > Or add an additional possible relationship REL_ORIGIN to which you can pass > the original file > That. if you look at the metrics for these processors, you see > TransferredToFaillure > 0, although the processors processed without errors -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (MINIFICPP-1994) Missing "failure", "retry" relationships for PutSql processor (and other processors)
[ https://issues.apache.org/jira/browse/MINIFICPP-1994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17637872#comment-17637872 ] Kondakov Artem commented on MINIFICPP-1994: --- At least these 3 processors definitely need such a relationship org.apache.nifi.minifi.processors.ExecuteSQL org.apache.nifi.minifi.processors.QueryDatabaseTable org.apache.nifi.minifi.processors.PutSQL > Missing "failure", "retry" relationships for PutSql processor (and other > processors) > > > Key: MINIFICPP-1994 > URL: https://issues.apache.org/jira/browse/MINIFICPP-1994 > Project: Apache NiFi MiNiFi C++ > Issue Type: New Feature >Reporter: Kondakov Artem >Assignee: Adam Debreceni >Priority: Major > > According to the manifest, sent by minifi, PutSql processors (and many > others) do not have a failure relationship, while in the apache / nifi > project similar processors have it. > I think maintaining these relationships is important. > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (MINIFICPP-1994) Missing "failure", "retry" relationships for PutSql processor (and other processors)
[ https://issues.apache.org/jira/browse/MINIFICPP-1994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17637868#comment-17637868 ] Kondakov Artem commented on MINIFICPP-1994: --- Hello [~aboda]. A simple script on the manifest produces the following list. (I'm not exactly sure that everyone needs a failure relationship) org.apache.nifi.minifi.processors.ListenHTTP org.apache.nifi.minifi.processors.FocusArchiveEntry org.apache.nifi.minifi.processors.UnfocusArchiveEntry org.apache.nifi.minifi.processors.ConsumeKafka org.apache.nifi.minifi.processors.ConsumeMQTT org.apache.nifi.minifi.processors.CapturePacket org.apache.nifi.minifi.extensions.procfs.ProcFsMonitor org.apache.nifi.minifi.processors.FetchSFTP org.apache.nifi.minifi.processors.ListSFTP org.apache.nifi.minifi.processors.ExecuteSQL org.apache.nifi.minifi.processors.QueryDatabaseTable org.apache.nifi.minifi.processors.PutSQL org.apache.nifi.minifi.processors.AppendHostInfo org.apache.nifi.minifi.processors.AttributesToJSON org.apache.nifi.minifi.processors.ExecuteProcess org.apache.nifi.minifi.processors.ExtractText org.apache.nifi.minifi.processors.GenerateFlowFile org.apache.nifi.minifi.processors.GetFile org.apache.nifi.minifi.processors.GetTCP org.apache.nifi.minifi.processors.ListFile org.apache.nifi.minifi.processors.ListenSyslog org.apache.nifi.minifi.processors.ListenTCP org.apache.nifi.minifi.processors.LogAttribute org.apache.nifi.minifi.processors.RouteText org.apache.nifi.minifi.processors.TailFile org.apache.nifi.minifi.extensions.systemd.ConsumeJournald > Missing "failure", "retry" relationships for PutSql processor (and other > processors) > > > Key: MINIFICPP-1994 > URL: https://issues.apache.org/jira/browse/MINIFICPP-1994 > Project: Apache NiFi MiNiFi C++ > Issue Type: New Feature >Reporter: Kondakov Artem >Assignee: Adam Debreceni >Priority: Major > > According to the manifest, sent by minifi, PutSql processors (and many > others) do not have a failure relationship, while in the apache / nifi > project similar processors have it. > I think maintaining these relationships is important. > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (MINIFICPP-1994) Missing "failure", "retry" relationships for PutSql processor (and other processors)
Kondakov Artem created MINIFICPP-1994: - Summary: Missing "failure", "retry" relationships for PutSql processor (and other processors) Key: MINIFICPP-1994 URL: https://issues.apache.org/jira/browse/MINIFICPP-1994 Project: Apache NiFi MiNiFi C++ Issue Type: New Feature Reporter: Kondakov Artem According to the manifest, sent by minifi, PutSql processors (and many others) do not have a failure relationship, while in the apache / nifi project similar processors have it. I think maintaining these relationships is important. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (MINIFICPP-1993) Minifi crashing while getting flow-file content in python processor
Kondakov Artem created MINIFICPP-1993: - Summary: Minifi crashing while getting flow-file content in python processor Key: MINIFICPP-1993 URL: https://issues.apache.org/jira/browse/MINIFICPP-1993 Project: Apache NiFi MiNiFi C++ Issue Type: Bug Affects Versions: 0.12.0 Reporter: Kondakov Artem Attachments: conf.zip, image-2022-11-22-19-46-11-969.png, image-2022-11-22-19-48-08-443.png, minifi-python.zip Hello. We built version 0.12.0 with the ability to execute python scripts as processors Made a simple process (see attachment) - calling and getting data using invokehttp get. After that, a custom python processor is executed (see attachment). Inside this python script, there is an attempt to get the contents of flow-file. At this point, an error occurs that stops the entire minifi application. At this code line !image-2022-11-22-19-46-11-969.png! There is a log file where you can see that the error occurs on this line of python code. !image-2022-11-22-19-48-08-443.png! At the same time, wrapping the python code in try / except does not give anything, apparently the fall occurs somewhere outside the python runtime After some time debugging, it turned out that this behavior only occurs when a flow-file is passed from the InvokeHttp processor through the relationship type success In this case, the transmitted content is empty (you can connect PutFile and make sure that an empty file will be saved) If the relationship type is set to response, then everything works as expected without minifi crashing. The expected behavior is that minifi will not suddenly stop if an attempt is made to get the contents of an empty flow-file -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (MINIFICPP-1990) Transferring FlowFiles in Python Processors
Kondakov Artem created MINIFICPP-1990: - Summary: Transferring FlowFiles in Python Processors Key: MINIFICPP-1990 URL: https://issues.apache.org/jira/browse/MINIFICPP-1990 Project: Apache NiFi MiNiFi C++ Issue Type: Bug Affects Versions: 0.12.0 Reporter: Kondakov Artem Fix For: 0.13.0 Hello. There is a problem with the processing of flow-files in python processors, which are connected using the setting nifi.python.processor.dir=./minifi-python/ If you need to split one file into several, then you have to create new files in the Python processor and be sure to transfer the original one to REL_FAILURE. Otherwise, if I do not pass the original file to REL_FAILURE, then the processing result is rolled back and nothing happens It seems to me that it is necessary to make it possible for Python processors to add various relations, and not just REL_FAILURE REL_SUCCESS. Or add an additional possible relationship REL_ORIGIN to which you can pass the original file That. if you look at the metrics for these processors, you see TransferredToFaillure > 0, although the processors processed without errors -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (MINIFICPP-1987) Configuring a metrics for Processors for C2 Protocol
[ https://issues.apache.org/jira/browse/MINIFICPP-1987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kondakov Artem updated MINIFICPP-1987: -- Summary: Configuring a metrics for Processors for C2 Protocol (was: Metrics for Processors via C2 Protocol) > Configuring a metrics for Processors for C2 Protocol > > > Key: MINIFICPP-1987 > URL: https://issues.apache.org/jira/browse/MINIFICPP-1987 > Project: Apache NiFi MiNiFi C++ > Issue Type: Improvement >Affects Versions: 0.12.0 >Reporter: Kondakov Artem >Priority: Major > Fix For: 0.13.0 > > > Hello. > I tried to setup receiving metrics using the C2 protocol for my processors. > The documentation describes that you must explicitly set the metric classes, > for example like this > nifi.c2.root.class.definitions.metrics.metrics.processorMetrics.classes=ListenHTTPMetrics,PostElasticsearchMetrics,InvokeHTTPMetrics,BinFilesMetrics,MergeContentMetrics... > It turns out a complex configuration, if I want to receive data on all my > processors, I need to update the list of these classes every time if I add a > new or remove type of processor. > If I try to configure classes for all possible processors, then I get 50+ > errors in the logs, , because not all processor types are involved, for > example > "No metric defined for PublishKafkaMetrics" > "No metric defined for ConsumeKafkaMetrics" > etc... > It seems to me that it is worth making one universal metric class for all > processes, or lower the error logging level when searching for a metric class > to warning -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (MINIFICPP-1987) Metrics for Processors via C2 Protocol
Kondakov Artem created MINIFICPP-1987: - Summary: Metrics for Processors via C2 Protocol Key: MINIFICPP-1987 URL: https://issues.apache.org/jira/browse/MINIFICPP-1987 Project: Apache NiFi MiNiFi C++ Issue Type: Improvement Affects Versions: 0.12.0 Reporter: Kondakov Artem Fix For: 0.13.0 Hello. I tried to setup receiving metrics using the C2 protocol for my processors. The documentation describes that you must explicitly set the metric classes, for example like this nifi.c2.root.class.definitions.metrics.metrics.processorMetrics.classes=ListenHTTPMetrics,PostElasticsearchMetrics,InvokeHTTPMetrics,BinFilesMetrics,MergeContentMetrics... It turns out a complex configuration, if I want to receive data on all my processors, I need to update the list of these classes every time if I add a new or remove type of processor. If I try to configure classes for all possible processors, then I get 50+ errors in the logs, , because not all processor types are involved, for example "No metric defined for PublishKafkaMetrics" "No metric defined for ConsumeKafkaMetrics" etc... It seems to me that it is worth making one universal metric class for all processes, or lower the error logging level when searching for a metric class to warning -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (MINIFICPP-1948) Possibility to display in the logs the uid of the component to which the logging relates
[ https://issues.apache.org/jira/browse/MINIFICPP-1948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17634205#comment-17634205 ] Kondakov Artem commented on MINIFICPP-1948: --- HI [~szaszm] The number of logs is huge, and if there are processing errors in several processors of the same class (especially if it is ExecutePython), then it is difficult to debug it. If it were possible to mark logging lines with processor identifiers or separate log files by processor instances, this would greatly simplify debugging > Possibility to display in the logs the uid of the component to which the > logging relates > > > Key: MINIFICPP-1948 > URL: https://issues.apache.org/jira/browse/MINIFICPP-1948 > Project: Apache NiFi MiNiFi C++ > Issue Type: Wish >Affects Versions: 0.12.0 >Reporter: Kondakov Artem >Priority: Major > Fix For: 0.13.0 > > > Is it possible, when logging to spdlog, to add the output of the identifier > of the component by which logging occurs, for example like this > spdlog.pattern=[%Y-%m-%d %H:%M:%S.%e] [%n] [%l] [%uid] %v > where [%uid] is the output [45b07291-7c57-46f0-ab48-6d4c88386f7f] of the > Processor ID or Controller Services ID or Connection ID or Flow Controller ID -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (MINIFICPP-1962) Error using process groups, input and output ports
[ https://issues.apache.org/jira/browse/MINIFICPP-1962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17634192#comment-17634192 ] Kondakov Artem commented on MINIFICPP-1962: --- Hi [~lordgamez], tested, works great > Error using process groups, input and output ports > -- > > Key: MINIFICPP-1962 > URL: https://issues.apache.org/jira/browse/MINIFICPP-1962 > Project: Apache NiFi MiNiFi C++ > Issue Type: New Feature >Affects Versions: 0.12.0 >Reporter: Kondakov Artem >Assignee: Gábor Gyimesi >Priority: Major > Fix For: 0.13.0 > > Attachments: config.yml > > Time Spent: 10m > Remaining Estimate: 0h > > Good afternoon. > I have created an simple example process for getting, sorting and storing > data in a database. > Everything is working. > After that, I tried to create a process group and move some of the processors > there, create the interaction with the main controller using incoming and > outgoing ports > The following errors began to appear. Connections cannot find matching schema > nodes and file processing stops. > in core::ProcessGroup > Cannot find the source processor with id > '92bcadbb-4500-4b4e-90bb-f6ddce2a8cef' for the connection [name = 'Node > 871d8dff-b14a-4eec-b635-3e6689e0fb7d/RouteOnAttribute', id = > 'c2c6b3e9-3a37-4ea0-b747-f15e003d812d'] > in core::ProcessGroup > Cannot find the destination processor with id > '92bcadbb-4500-4b4e-90bb-f6ddce2a8cef' for the connection [name = > 'SplitJson/Node 92bcadbb-4500-4b4e-90bb-f6ddce2a8cef', id = > 'b075e238-46ed-46a9-9fdf-894ba5a04fc1'] > > These is my config of minifi > [^config.yml] -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (MINIFICPP-1962) Error using process groups, input and output ports
Kondakov Artem created MINIFICPP-1962: - Summary: Error using process groups, input and output ports Key: MINIFICPP-1962 URL: https://issues.apache.org/jira/browse/MINIFICPP-1962 Project: Apache NiFi MiNiFi C++ Issue Type: Bug Affects Versions: 0.12.0 Reporter: Kondakov Artem Fix For: 0.13.0 Attachments: config.yml Good afternoon. I have created an simple example process for getting, sorting and storing data in a database. Everything is working. After that, I tried to create a process group and move some of the processors there, create the interaction with the main controller using incoming and outgoing ports The following errors began to appear. Connections cannot find matching schema nodes and file processing stops. in core::ProcessGroup Cannot find the source processor with id '92bcadbb-4500-4b4e-90bb-f6ddce2a8cef' for the connection [name = 'Node 871d8dff-b14a-4eec-b635-3e6689e0fb7d/RouteOnAttribute', id = 'c2c6b3e9-3a37-4ea0-b747-f15e003d812d'] in core::ProcessGroup Cannot find the destination processor with id '92bcadbb-4500-4b4e-90bb-f6ddce2a8cef' for the connection [name = 'SplitJson/Node 92bcadbb-4500-4b4e-90bb-f6ddce2a8cef', id = 'b075e238-46ed-46a9-9fdf-894ba5a04fc1'] These is my config of minifi [^config.yml] -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (MINIFICPP-1948) Possibility to display in the logs the uid of the component to which the logging relates
Kondakov Artem created MINIFICPP-1948: - Summary: Possibility to display in the logs the uid of the component to which the logging relates Key: MINIFICPP-1948 URL: https://issues.apache.org/jira/browse/MINIFICPP-1948 Project: Apache NiFi MiNiFi C++ Issue Type: Wish Affects Versions: 0.12.0 Reporter: Kondakov Artem Fix For: 0.13.0 Is it possible, when logging to spdlog, to add the output of the identifier of the component by which logging occurs, for example like this spdlog.pattern=[%Y-%m-%d %H:%M:%S.%e] [%n] [%l] [%uid] %v where [%uid] is the output [45b07291-7c57-46f0-ab48-6d4c88386f7f] of the Processor ID or Controller Services ID or Connection ID or Flow Controller ID -- This message was sent by Atlassian Jira (v8.20.10#820010)
