[jira] [Assigned] (NIFI-10955) In JASN1Reader allow preprocess of ASN files to reconcile unsupported features
[
https://issues.apache.org/jira/browse/NIFI-10955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Handermann reassigned NIFI-10955:
---
Assignee: Tamas Palfy
> In JASN1Reader allow preprocess of ASN files to reconcile unsupported features
> --
>
> Key: NIFI-10955
> URL: https://issues.apache.org/jira/browse/NIFI-10955
> Project: Apache NiFi
> Issue Type: Improvement
>Reporter: Tamas Palfy
>Assignee: Tamas Palfy
>Priority: Major
> Time Spent: 3h 50m
> Remaining Estimate: 0h
>
> The ASN specification allows the creation of valid ASN files that has
> features unrecognized by the asn1bean library we are using in JASN1Reader.
> We can add a preprocessing step that creates modified versions of the
> provided ASN files (leaving the originals intact) that removes unsupported
> features in a way that makes them less strict but otherwise should still be
> compatible with incoming data.
> Identified unsupported features:
> * Certain constraint types
> ** E.g.
> {code:java}
> SomeType ::= INTEGER (ALL EXCEPT (0..15))
> {code}
> * Extension marker (a.k.a "ellipsis" or
> {code:java}
> ...
> {code}
> )
> ** Can occur in constraints (e.g.
> {code:java}
> SomeType ::= INTEGER (0..8,...,100..200)
> {code}
> although
> {code:java}
> SomeType ::= INTEGER (0..8,...)
> {code}
> works)
> ** or in type definitions (e.g.
> {code:java}
> RootType::= SEQUENCE {
> field1 INTEGER,
> field2 INTEGER,
> ...,
> field3 INTEGER
> }
> {code}
> but this seems to work as well).
> * Version brackets e.g.
> {code:java}
> SomeType ::= SEQUENCE {
> integerField1 INTEGER,
> integerField2 INTEGER,
> ...,
> [[ -- from version 2
> integerField3 INTEGER,
> integerField4 INTEGER ]]
> }
> {code}
> Seems to require extension marker as well.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (NIFI-10955) In JASN1Reader allow preprocess of ASN files to reconcile unsupported features
[
https://issues.apache.org/jira/browse/NIFI-10955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Handermann resolved NIFI-10955.
-
Fix Version/s: 2.0.0
1.22.0
Resolution: Fixed
> In JASN1Reader allow preprocess of ASN files to reconcile unsupported features
> --
>
> Key: NIFI-10955
> URL: https://issues.apache.org/jira/browse/NIFI-10955
> Project: Apache NiFi
> Issue Type: Improvement
>Reporter: Tamas Palfy
>Assignee: Tamas Palfy
>Priority: Major
> Fix For: 2.0.0, 1.22.0
>
> Time Spent: 3h 50m
> Remaining Estimate: 0h
>
> The ASN specification allows the creation of valid ASN files that has
> features unrecognized by the asn1bean library we are using in JASN1Reader.
> We can add a preprocessing step that creates modified versions of the
> provided ASN files (leaving the originals intact) that removes unsupported
> features in a way that makes them less strict but otherwise should still be
> compatible with incoming data.
> Identified unsupported features:
> * Certain constraint types
> ** E.g.
> {code:java}
> SomeType ::= INTEGER (ALL EXCEPT (0..15))
> {code}
> * Extension marker (a.k.a "ellipsis" or
> {code:java}
> ...
> {code}
> )
> ** Can occur in constraints (e.g.
> {code:java}
> SomeType ::= INTEGER (0..8,...,100..200)
> {code}
> although
> {code:java}
> SomeType ::= INTEGER (0..8,...)
> {code}
> works)
> ** or in type definitions (e.g.
> {code:java}
> RootType::= SEQUENCE {
> field1 INTEGER,
> field2 INTEGER,
> ...,
> field3 INTEGER
> }
> {code}
> but this seems to work as well).
> * Version brackets e.g.
> {code:java}
> SomeType ::= SEQUENCE {
> integerField1 INTEGER,
> integerField2 INTEGER,
> ...,
> [[ -- from version 2
> integerField3 INTEGER,
> integerField4 INTEGER ]]
> }
> {code}
> Seems to require extension marker as well.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (NIFI-10955) In JASN1Reader allow preprocess of ASN files to reconcile unsupported features
[
https://issues.apache.org/jira/browse/NIFI-10955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17711178#comment-17711178
]
ASF subversion and git services commented on NIFI-10955:
Commit 6037af86e898aeec8e9a922d4abe82d655af259e in nifi's branch
refs/heads/support/nifi-1.x from Tamas Palfy
[ https://gitbox.apache.org/repos/asf?p=nifi.git;h=6037af86e8 ]
NIFI-10955 Added optional preprocessing to JASN1Reader
This close #6769
Signed-off-by: David Handermann
> In JASN1Reader allow preprocess of ASN files to reconcile unsupported features
> --
>
> Key: NIFI-10955
> URL: https://issues.apache.org/jira/browse/NIFI-10955
> Project: Apache NiFi
> Issue Type: Improvement
>Reporter: Tamas Palfy
>Priority: Major
> Time Spent: 3h 50m
> Remaining Estimate: 0h
>
> The ASN specification allows the creation of valid ASN files that has
> features unrecognized by the asn1bean library we are using in JASN1Reader.
> We can add a preprocessing step that creates modified versions of the
> provided ASN files (leaving the originals intact) that removes unsupported
> features in a way that makes them less strict but otherwise should still be
> compatible with incoming data.
> Identified unsupported features:
> * Certain constraint types
> ** E.g.
> {code:java}
> SomeType ::= INTEGER (ALL EXCEPT (0..15))
> {code}
> * Extension marker (a.k.a "ellipsis" or
> {code:java}
> ...
> {code}
> )
> ** Can occur in constraints (e.g.
> {code:java}
> SomeType ::= INTEGER (0..8,...,100..200)
> {code}
> although
> {code:java}
> SomeType ::= INTEGER (0..8,...)
> {code}
> works)
> ** or in type definitions (e.g.
> {code:java}
> RootType::= SEQUENCE {
> field1 INTEGER,
> field2 INTEGER,
> ...,
> field3 INTEGER
> }
> {code}
> but this seems to work as well).
> * Version brackets e.g.
> {code:java}
> SomeType ::= SEQUENCE {
> integerField1 INTEGER,
> integerField2 INTEGER,
> ...,
> [[ -- from version 2
> integerField3 INTEGER,
> integerField4 INTEGER ]]
> }
> {code}
> Seems to require extension marker as well.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (NIFI-10955) In JASN1Reader allow preprocess of ASN files to reconcile unsupported features
[
https://issues.apache.org/jira/browse/NIFI-10955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17711177#comment-17711177
]
ASF subversion and git services commented on NIFI-10955:
Commit 3322ad7a20c99dec01ee0c3f530c0566acd13258 in nifi's branch
refs/heads/main from Tamas Palfy
[ https://gitbox.apache.org/repos/asf?p=nifi.git;h=3322ad7a20 ]
NIFI-10955 Added optional preprocessing to JASN1Reader
This close #6769
Signed-off-by: David Handermann
> In JASN1Reader allow preprocess of ASN files to reconcile unsupported features
> --
>
> Key: NIFI-10955
> URL: https://issues.apache.org/jira/browse/NIFI-10955
> Project: Apache NiFi
> Issue Type: Improvement
>Reporter: Tamas Palfy
>Priority: Major
> Time Spent: 3h 50m
> Remaining Estimate: 0h
>
> The ASN specification allows the creation of valid ASN files that has
> features unrecognized by the asn1bean library we are using in JASN1Reader.
> We can add a preprocessing step that creates modified versions of the
> provided ASN files (leaving the originals intact) that removes unsupported
> features in a way that makes them less strict but otherwise should still be
> compatible with incoming data.
> Identified unsupported features:
> * Certain constraint types
> ** E.g.
> {code:java}
> SomeType ::= INTEGER (ALL EXCEPT (0..15))
> {code}
> * Extension marker (a.k.a "ellipsis" or
> {code:java}
> ...
> {code}
> )
> ** Can occur in constraints (e.g.
> {code:java}
> SomeType ::= INTEGER (0..8,...,100..200)
> {code}
> although
> {code:java}
> SomeType ::= INTEGER (0..8,...)
> {code}
> works)
> ** or in type definitions (e.g.
> {code:java}
> RootType::= SEQUENCE {
> field1 INTEGER,
> field2 INTEGER,
> ...,
> field3 INTEGER
> }
> {code}
> but this seems to work as well).
> * Version brackets e.g.
> {code:java}
> SomeType ::= SEQUENCE {
> integerField1 INTEGER,
> integerField2 INTEGER,
> ...,
> [[ -- from version 2
> integerField3 INTEGER,
> integerField4 INTEGER ]]
> }
> {code}
> Seems to require extension marker as well.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[GitHub] [nifi] exceptionfactory closed pull request #6769: NIFI-10955 - Added JASN1Reader the ability to try to adjust for unsupported ASN features
exceptionfactory closed pull request #6769: NIFI-10955 - Added JASN1Reader the ability to try to adjust for unsupported ASN features URL: https://github.com/apache/nifi/pull/6769 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (NIFI-11434) Error with UnpackContent
[
https://issues.apache.org/jira/browse/NIFI-11434?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rafael Fracasso Gomes updated NIFI-11434:
-
Description:
I posted in NiFi discussion group about a error that occurred in UnpackContent
processor, I will paste it here:
{panel:title=Error with UnpackContent}
I'm trying to unzip some files with NiFi and getting this errors:
{code:java}
UnpackContent[id=5c38660b-0187-1000-52a2-0b71f8e868c0] Unable to unpack
FlowFile[filename=58174fd0-750e-4e45-89bf-c43d9ab01b98]; routing to failure:
org.apache.nifi.processor.exception.ProcessException: IOException thrown from
UnpackContent[id=5c38660b-0187-1000-52a2-0b71f8e868c0]:
org.apache.commons.compress.archivers.zip.UnsupportedZipFeatureException:
Unsupported feature data descriptor used in entry
files/20230228_131431_84_24c8_3B_udm2_clip.tif - Caused by:
org.apache.commons.compress.archivers.zip.UnsupportedZipFeatureException:
Unsupported feature data descriptor used in entry
files/20230228_131431_84_24c8_3B_udm2_clip.tif{code}
I found some workaround in the link below but it doesn't suits me because i
need to keep my attributes in the flowfiles:
[https://community.cloudera.com/t5/Support-Questions/Unzip-file-containing-log-gz-files-with-nifi/m-p/304042]
And found some help in stackoverflow about this problem but it doesn't fit on
NiFi processor: (
[https://stackoverflow.com/questions/15738312/how-to-fix-org-apache-commons-compress-archivers-zip-unsupportedzipfeatureexcept]
)
I attached a sample file that I downloaded from nifi content.{panel}
David Handermann replyed me:
{panel}
As indicated by the error message, the Zip file in question includes a
specialized data descriptor feature that is not supported in the default
configuration of the UnpackContent Processor.
The UnpackContent Processor relies on Apache Commons Compress for extraction,
which has optional support for handling this feature. The feature is disabled
by default, which is the reason for the error.
Apache Commons Compress issue COMPRESS-555 [1] provides some additional
background on why this feature is disabled by default, but it sounds like
something that could be evaluated for adjustment in Apache NiFi.
[1] [https://issues.apache.org/jira/projects/COMPRESS/issues/COMPRESS-555]
{panel}
was:
I posted in NiFi discussion group about a error that occurred in UnpackContent
processor, I will paste it here:
{panel:title=Error with UnpackContent}
I'm trying to unzip some files with NiFi and getting this errors:
{code:java}
UnpackContent[id=5c38660b-0187-1000-52a2-0b71f8e868c0] Unable to unpack
FlowFile[filename=58174fd0-750e-4e45-89bf-c43d9ab01b98]; routing to failure:
org.apache.nifi.processor.exception.ProcessException: IOException thrown from
UnpackContent[id=5c38660b-0187-1000-52a2-0b71f8e868c0]:
org.apache.commons.compress.archivers.zip.UnsupportedZipFeatureException:
Unsupported feature data descriptor used in entry
files/20230228_131431_84_24c8_3B_udm2_clip.tif - Caused by:
org.apache.commons.compress.archivers.zip.UnsupportedZipFeatureException:
Unsupported feature data descriptor used in entry
files/20230228_131431_84_24c8_3B_udm2_clip.tif{code}
I found some workaround in the link below but it doesn't suits me because i
need to keep my attributes in the flowfiles:
[https://community.cloudera.com/t5/Support-Questions/Unzip-file-containing-log-gz-files-with-nifi/m-p/304042]
And found some help in stackoverflow about this problem but it doesn't fit on
NiFi processor: (
[https://stackoverflow.com/questions/15738312/how-to-fix-org-apache-commons-compress-archivers-zip-unsupportedzipfeatureexcept]
){panel}
David Handermann replyed me:
{panel}
As indicated by the error message, the Zip file in question includes a
specialized data descriptor feature that is not supported in the default
configuration of the UnpackContent Processor.
The UnpackContent Processor relies on Apache Commons Compress for extraction,
which has optional support for handling this feature. The feature is disabled
by default, which is the reason for the error.
Apache Commons Compress issue COMPRESS-555 [1] provides some additional
background on why this feature is disabled by default, but it sounds like
something that could be evaluated for adjustment in Apache NiFi.
[1]
[https://issues.apache.org/jira/projects/COMPRESS/issues/COMPRESS-555]{panel}
> Error with UnpackContent
>
>
> Key: NIFI-11434
> URL: https://issues.apache.org/jira/browse/NIFI-11434
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
>Affects Versions: 1.18.0
> Environment: OS Ubuntu 20.04.5, running in a K8S cluster but as
> standalone node, limited to 8gb, free on cpu usage.
>Reporter: Rafael Fracasso Gomes
>Priority: Blocker
>
[jira] [Created] (NIFI-11434) Error with UnpackContent
Rafael Fracasso Gomes created NIFI-11434:
Summary: Error with UnpackContent
Key: NIFI-11434
URL: https://issues.apache.org/jira/browse/NIFI-11434
Project: Apache NiFi
Issue Type: Bug
Components: Extensions
Affects Versions: 1.18.0
Environment: OS Ubuntu 20.04.5, running in a K8S cluster but as
standalone node, limited to 8gb, free on cpu usage.
Reporter: Rafael Fracasso Gomes
Attachments: 58174fd0-750e-4e45-89bf-c43d9ab01b98.zip
I posted in NiFi discussion group about a error that occurred in UnpackContent
processor, I will paste it here:
{panel:title=Error with UnpackContent}
I'm trying to unzip some files with NiFi and getting this errors:
{code:java}
UnpackContent[id=5c38660b-0187-1000-52a2-0b71f8e868c0] Unable to unpack
FlowFile[filename=58174fd0-750e-4e45-89bf-c43d9ab01b98]; routing to failure:
org.apache.nifi.processor.exception.ProcessException: IOException thrown from
UnpackContent[id=5c38660b-0187-1000-52a2-0b71f8e868c0]:
org.apache.commons.compress.archivers.zip.UnsupportedZipFeatureException:
Unsupported feature data descriptor used in entry
files/20230228_131431_84_24c8_3B_udm2_clip.tif - Caused by:
org.apache.commons.compress.archivers.zip.UnsupportedZipFeatureException:
Unsupported feature data descriptor used in entry
files/20230228_131431_84_24c8_3B_udm2_clip.tif{code}
I found some workaround in the link below but it doesn't suits me because i
need to keep my attributes in the flowfiles:
[https://community.cloudera.com/t5/Support-Questions/Unzip-file-containing-log-gz-files-with-nifi/m-p/304042]
And found some help in stackoverflow about this problem but it doesn't fit on
NiFi processor: (
[https://stackoverflow.com/questions/15738312/how-to-fix-org-apache-commons-compress-archivers-zip-unsupportedzipfeatureexcept]
){panel}
David Handermann replyed me:
{panel}
As indicated by the error message, the Zip file in question includes a
specialized data descriptor feature that is not supported in the default
configuration of the UnpackContent Processor.
The UnpackContent Processor relies on Apache Commons Compress for extraction,
which has optional support for handling this feature. The feature is disabled
by default, which is the reason for the error.
Apache Commons Compress issue COMPRESS-555 [1] provides some additional
background on why this feature is disabled by default, but it sounds like
something that could be evaluated for adjustment in Apache NiFi.
[1]
[https://issues.apache.org/jira/projects/COMPRESS/issues/COMPRESS-555]{panel}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[GitHub] [nifi] scottyaslan commented on pull request #7162: NIFI-11432 Upgrade Angular Version to 1.8.3
scottyaslan commented on PR #7162: URL: https://github.com/apache/nifi/pull/7162#issuecomment-1503977587 Reviewing... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] markap14 commented on a diff in pull request #7003: NIFI-11241: Initial implementation of Python-based Processor API with…
markap14 commented on code in PR #7003:
URL: https://github.com/apache/nifi/pull/7003#discussion_r1163227701
##
nifi-nar-bundles/nifi-py4j-bundle/nifi-py4j-bridge/src/main/java/org/apache/nifi/py4j/client/PythonProxyInvocationHandler.java:
##
@@ -0,0 +1,174 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.py4j.client;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import py4j.Protocol;
+import py4j.Py4JException;
+import py4j.StringUtil;
+import py4j.reflection.MethodInvoker;
+import py4j.reflection.ReflectionEngine;
+import py4j.reflection.TypeConverter;
+
+import java.lang.reflect.InvocationHandler;
+import java.lang.reflect.Method;
+import java.math.BigDecimal;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+public class PythonProxyInvocationHandler implements InvocationHandler {
+public static final String VOID_ARGUMENT_LINE =
String.valueOf(Protocol.VOID);
+public static final String METHOD_INVOCATION_START_CLAUSE = "c\n";
+public static final String METHOD_INVOCATION_END_CLAUSE = "e\n";
+private static final Logger logger =
LoggerFactory.getLogger(PythonProxyInvocationHandler.class);
+
+private final String objectId;
+private final NiFiPythonGateway gateway;
+private final JavaObjectBindings bindings;
+
+public PythonProxyInvocationHandler(final NiFiPythonGateway gateway, final
String objectId) {
+this.objectId = objectId;
+this.gateway = gateway;
+this.bindings = gateway.getObjectBindings();
+}
+
+@Override
+public Object invoke(final Object proxy, final Method method, final
Object[] args) throws Throwable {
+if (args == null && method.getName().equals("toString")) {
+return "PythonProxy[targetObjectId=" + objectId + "]";
+}
+
+final CommandBuilder commandBuilder = new CommandBuilder(bindings,
objectId, method.getName());
+final String command = commandBuilder.buildCommand(args);
+
+if (logger.isDebugEnabled()) {
+final List argList = args == null ?
Collections.emptyList() : Arrays.asList(args);
+logger.debug("Invoking {} on {} with args {} using command {}",
method, proxy, argList, command);
+}
+
+gateway.beginInvocation(this.objectId, method, args);
+
+final String response =
gateway.getCallbackClient().sendCommand(command);
+final Object output = Protocol.getReturnValue(response, gateway);
+final Object convertedOutput = convertOutput(method, output);
+
+// TODO: While we're waiting for this call to return, the Python side
may call back into the Java side and create objects.
+// When that happens we need to throw those onto the stack also!
+if (gateway.isUnbind(method)) {
+commandBuilder.getBoundIds().forEach(bindings::unbind);
+commandBuilder.getBoundIds().forEach(i -> logger.debug("For method
invocation {} unbound {} (from command builder)", method.getName(), i));
+} else {
+commandBuilder.getBoundIds().forEach(i -> logger.debug("For method
invocation {} will not unbind {} (from command builder) because arguments of
this method are not to be unbound",
+method.getName(), i));
+}
+
+gateway.endInvocation(this.objectId, method, args);
+
+return convertedOutput;
+}
+
+
+private String buildInvocationCommand(final String methodName, final
Object[] args) {
+final StringBuilder sb = new StringBuilder();
+sb.append(METHOD_INVOCATION_START_CLAUSE);
+sb.append(objectId).append("\n");
+sb.append(methodName).append("\n");
+
+if (args != null) {
+for (final Object arg : args) {
+final String argumentLine = getArgumentLine(arg);
+sb.append(argumentLine).append("\n");
+}
+}
+
+sb.append(METHOD_INVOCATION_END_CLAUSE);
+return sb.toString();
+}
+
+private String getArgumentLine(final Object arg) {
Review Comment:
[GitHub] [nifi] markap14 commented on a diff in pull request #7003: NIFI-11241: Initial implementation of Python-based Processor API with…
markap14 commented on code in PR #7003: URL: https://github.com/apache/nifi/pull/7003#discussion_r1163227225 ## nifi-nar-bundles/nifi-py4j-bundle/nifi-py4j-bridge/src/main/java/org/apache/nifi/py4j/client/NiFiPythonGateway.java: ## @@ -0,0 +1,203 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.py4j.client; + +import org.apache.nifi.python.processor.PreserveJavaBinding; +import org.apache.nifi.python.processor.PythonProcessor; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; +import py4j.CallbackClient; +import py4j.Gateway; +import py4j.reflection.PythonProxyHandler; + +import java.lang.reflect.Method; +import java.net.InetAddress; +import java.util.ArrayList; +import java.util.Arrays; +import java.util.List; +import java.util.Map; +import java.util.Objects; +import java.util.Stack; +import java.util.concurrent.ConcurrentHashMap; + + +// TODO: Switch to thread-local instead of a ConcurrentHashMap... +// TODO: Document what this is / why it exists Review Comment: Yeah I think this can be cleaned up. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] markap14 commented on a diff in pull request #7003: NIFI-11241: Initial implementation of Python-based Processor API with…
markap14 commented on code in PR #7003: URL: https://github.com/apache/nifi/pull/7003#discussion_r1163226996 ## nifi-nar-bundles/nifi-py4j-bundle/nifi-py4j-bridge/src/main/java/org/apache/nifi/py4j/PythonProcess.java: ## @@ -0,0 +1,293 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.py4j; + +import org.apache.nifi.py4j.client.JavaObjectBindings; +import org.apache.nifi.py4j.client.NiFiPythonGateway; +import org.apache.nifi.py4j.client.StandardPythonClient; +import org.apache.nifi.py4j.server.NiFiGatewayServer; +import org.apache.nifi.python.ControllerServiceTypeLookup; +import org.apache.nifi.python.PythonController; +import org.apache.nifi.python.PythonProcessConfig; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; +import py4j.CallbackClient; +import py4j.GatewayServer; + +import javax.net.ServerSocketFactory; +import javax.net.SocketFactory; +import java.io.File; +import java.io.IOException; +import java.nio.file.Files; +import java.util.Collections; +import java.util.Map; +import java.util.concurrent.ConcurrentHashMap; +import java.util.concurrent.TimeUnit; + +// TODO / Figure Out for MVP: +// MUST DO: +// - Documentation +// - Admin Guide +// - JavaDocs +// - Developer Guide +// - Explain how communication between Java & Python work. +// - Java is preferred, Python is slower and more expensive b/c of network +// - Different Extension Points (FlowFileTransform, RecordTransform) +// - What the API Looks like, Links to JavaDocs for ProcessContext, etc. +// - Example Code +// - Exposing properties +// - Relationships +// - Controller Services +// - Need to update docs to show the interfaces that are exposed, explain how to get these... +// - Design Doc +// - Setup proper logging on the Python side: https://docs.python.org/2/howto/logging-cookbook.html#using-file-rotation +// - For FlowFileTransform, allow the result to contain either a byte array or a String. If a String, just convert in the parent class. +// - Figure out how to deal with Python Packaging +// - Need to figure out how to deal with additionalDetails.html, docs directory in python project typically? +// - Understand how to deal with versioning +// - Look at performance improvements for Py4J - socket comms appear to be INCREDIBLY slow. +// - Create test that calls Python 1M times. Just returns 'hello'. See how long it takes +// - Create test that calls Python 1M times. Returns .toString() and see how long it takes. +// - Will help to understand if it's the call from Java to Python that's slow, Python to Java, or both. +// - Performance concern for TransformRecord +// - Currently, triggering the transform() method is pretty fast. But then the Result object comes back and we have to call into the Python side to call the getters +//over and over. Need to look into instead serializing the entire response as JSON and sending that back. +// - Also, since this is heavy JSON processing, might want to consider ORJSON or something like that instead of inbuilt JSON parser/generator +// - Test pip install nifi-my-proc, does nifi pick it up? +// - When ran DetectObjectInImage with multiple threads, Python died. Need to figure out why. +// - If Python Process dies, need to create a new process and need to then create all of the Processors that were in that Process and initialize them. +//- Milestone 2 or 3, not Milestone 1. +// - Remove test-pypi usage from ExtensionManager.py +// - Additional Interfaces beyond just FlowFileTransform +// - FlowFileSource +// - Restructure Maven projects +// - Should this all go under Framework? +// +// +// CONSIDER: +// - Clustering: Ensure component on all nodes? +// - Consider "pip freeze" type of thing to ensure that python dependencies are same across nodes when joining cluster. +// - Update python code
[GitHub] [nifi] markap14 commented on a diff in pull request #7003: NIFI-11241: Initial implementation of Python-based Processor API with…
markap14 commented on code in PR #7003: URL: https://github.com/apache/nifi/pull/7003#discussion_r1163223551 ## nifi-nar-bundles/nifi-py4j-bundle/nifi-py4j-bridge/src/main/java/org/apache/nifi/py4j/PythonProcess.java: ## @@ -0,0 +1,293 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.py4j; + +import org.apache.nifi.py4j.client.JavaObjectBindings; +import org.apache.nifi.py4j.client.NiFiPythonGateway; +import org.apache.nifi.py4j.client.StandardPythonClient; +import org.apache.nifi.py4j.server.NiFiGatewayServer; +import org.apache.nifi.python.ControllerServiceTypeLookup; +import org.apache.nifi.python.PythonController; +import org.apache.nifi.python.PythonProcessConfig; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; +import py4j.CallbackClient; +import py4j.GatewayServer; + +import javax.net.ServerSocketFactory; +import javax.net.SocketFactory; +import java.io.File; +import java.io.IOException; +import java.nio.file.Files; +import java.util.Collections; +import java.util.Map; +import java.util.concurrent.ConcurrentHashMap; +import java.util.concurrent.TimeUnit; + +// TODO / Figure Out for MVP: +// MUST DO: +// - Documentation +// - Admin Guide +// - JavaDocs +// - Developer Guide +// - Explain how communication between Java & Python work. +// - Java is preferred, Python is slower and more expensive b/c of network +// - Different Extension Points (FlowFileTransform, RecordTransform) +// - What the API Looks like, Links to JavaDocs for ProcessContext, etc. +// - Example Code +// - Exposing properties +// - Relationships +// - Controller Services +// - Need to update docs to show the interfaces that are exposed, explain how to get these... +// - Design Doc +// - Setup proper logging on the Python side: https://docs.python.org/2/howto/logging-cookbook.html#using-file-rotation +// - For FlowFileTransform, allow the result to contain either a byte array or a String. If a String, just convert in the parent class. +// - Figure out how to deal with Python Packaging +// - Need to figure out how to deal with additionalDetails.html, docs directory in python project typically? +// - Understand how to deal with versioning +// - Look at performance improvements for Py4J - socket comms appear to be INCREDIBLY slow. +// - Create test that calls Python 1M times. Just returns 'hello'. See how long it takes +// - Create test that calls Python 1M times. Returns .toString() and see how long it takes. +// - Will help to understand if it's the call from Java to Python that's slow, Python to Java, or both. +// - Performance concern for TransformRecord +// - Currently, triggering the transform() method is pretty fast. But then the Result object comes back and we have to call into the Python side to call the getters +//over and over. Need to look into instead serializing the entire response as JSON and sending that back. +// - Also, since this is heavy JSON processing, might want to consider ORJSON or something like that instead of inbuilt JSON parser/generator +// - Test pip install nifi-my-proc, does nifi pick it up? +// - When ran DetectObjectInImage with multiple threads, Python died. Need to figure out why. +// - If Python Process dies, need to create a new process and need to then create all of the Processors that were in that Process and initialize them. +//- Milestone 2 or 3, not Milestone 1. +// - Remove test-pypi usage from ExtensionManager.py +// - Additional Interfaces beyond just FlowFileTransform +// - FlowFileSource +// - Restructure Maven projects +// - Should this all go under Framework? +// +// +// CONSIDER: +// - Clustering: Ensure component on all nodes? +// - Consider "pip freeze" type of thing to ensure that python dependencies are same across nodes when joining cluster. +// - Update python code
[GitHub] [nifi] markap14 commented on a diff in pull request #7003: NIFI-11241: Initial implementation of Python-based Processor API with…
markap14 commented on code in PR #7003: URL: https://github.com/apache/nifi/pull/7003#discussion_r1163222378 ## nifi-nar-bundles/nifi-py4j-bundle/nifi-py4j-bridge/src/main/java/org/apache/nifi/py4j/PythonProcess.java: ## @@ -0,0 +1,293 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.py4j; + +import org.apache.nifi.py4j.client.JavaObjectBindings; +import org.apache.nifi.py4j.client.NiFiPythonGateway; +import org.apache.nifi.py4j.client.StandardPythonClient; +import org.apache.nifi.py4j.server.NiFiGatewayServer; +import org.apache.nifi.python.ControllerServiceTypeLookup; +import org.apache.nifi.python.PythonController; +import org.apache.nifi.python.PythonProcessConfig; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; +import py4j.CallbackClient; +import py4j.GatewayServer; + +import javax.net.ServerSocketFactory; +import javax.net.SocketFactory; +import java.io.File; +import java.io.IOException; +import java.nio.file.Files; +import java.util.Collections; +import java.util.Map; +import java.util.concurrent.ConcurrentHashMap; +import java.util.concurrent.TimeUnit; + +// TODO / Figure Out for MVP: +// MUST DO: +// - Documentation +// - Admin Guide +// - JavaDocs +// - Developer Guide +// - Explain how communication between Java & Python work. +// - Java is preferred, Python is slower and more expensive b/c of network +// - Different Extension Points (FlowFileTransform, RecordTransform) +// - What the API Looks like, Links to JavaDocs for ProcessContext, etc. +// - Example Code +// - Exposing properties +// - Relationships +// - Controller Services +// - Need to update docs to show the interfaces that are exposed, explain how to get these... +// - Design Doc +// - Setup proper logging on the Python side: https://docs.python.org/2/howto/logging-cookbook.html#using-file-rotation +// - For FlowFileTransform, allow the result to contain either a byte array or a String. If a String, just convert in the parent class. +// - Figure out how to deal with Python Packaging +// - Need to figure out how to deal with additionalDetails.html, docs directory in python project typically? +// - Understand how to deal with versioning +// - Look at performance improvements for Py4J - socket comms appear to be INCREDIBLY slow. +// - Create test that calls Python 1M times. Just returns 'hello'. See how long it takes +// - Create test that calls Python 1M times. Returns .toString() and see how long it takes. +// - Will help to understand if it's the call from Java to Python that's slow, Python to Java, or both. +// - Performance concern for TransformRecord +// - Currently, triggering the transform() method is pretty fast. But then the Result object comes back and we have to call into the Python side to call the getters +//over and over. Need to look into instead serializing the entire response as JSON and sending that back. +// - Also, since this is heavy JSON processing, might want to consider ORJSON or something like that instead of inbuilt JSON parser/generator +// - Test pip install nifi-my-proc, does nifi pick it up? +// - When ran DetectObjectInImage with multiple threads, Python died. Need to figure out why. +// - If Python Process dies, need to create a new process and need to then create all of the Processors that were in that Process and initialize them. +//- Milestone 2 or 3, not Milestone 1. +// - Remove test-pypi usage from ExtensionManager.py +// - Additional Interfaces beyond just FlowFileTransform +// - FlowFileSource +// - Restructure Maven projects +// - Should this all go under Framework? +// +// +// CONSIDER: +// - Clustering: Ensure component on all nodes? +// - Consider "pip freeze" type of thing to ensure that python dependencies are same across nodes when joining cluster. +// - Update python code
[GitHub] [nifi] markap14 commented on a diff in pull request #7003: NIFI-11241: Initial implementation of Python-based Processor API with…
markap14 commented on code in PR #7003: URL: https://github.com/apache/nifi/pull/7003#discussion_r1163221951 ## nifi-nar-bundles/nifi-py4j-bundle/nifi-py4j-bridge/src/main/java/org/apache/nifi/py4j/PythonProcess.java: ## @@ -0,0 +1,293 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.py4j; + +import org.apache.nifi.py4j.client.JavaObjectBindings; +import org.apache.nifi.py4j.client.NiFiPythonGateway; +import org.apache.nifi.py4j.client.StandardPythonClient; +import org.apache.nifi.py4j.server.NiFiGatewayServer; +import org.apache.nifi.python.ControllerServiceTypeLookup; +import org.apache.nifi.python.PythonController; +import org.apache.nifi.python.PythonProcessConfig; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; +import py4j.CallbackClient; +import py4j.GatewayServer; + +import javax.net.ServerSocketFactory; +import javax.net.SocketFactory; +import java.io.File; +import java.io.IOException; +import java.nio.file.Files; +import java.util.Collections; +import java.util.Map; +import java.util.concurrent.ConcurrentHashMap; +import java.util.concurrent.TimeUnit; + +// TODO / Figure Out for MVP: +// MUST DO: +// - Documentation +// - Admin Guide +// - JavaDocs +// - Developer Guide +// - Explain how communication between Java & Python work. +// - Java is preferred, Python is slower and more expensive b/c of network +// - Different Extension Points (FlowFileTransform, RecordTransform) +// - What the API Looks like, Links to JavaDocs for ProcessContext, etc. +// - Example Code +// - Exposing properties +// - Relationships +// - Controller Services +// - Need to update docs to show the interfaces that are exposed, explain how to get these... +// - Design Doc +// - Setup proper logging on the Python side: https://docs.python.org/2/howto/logging-cookbook.html#using-file-rotation +// - For FlowFileTransform, allow the result to contain either a byte array or a String. If a String, just convert in the parent class. +// - Figure out how to deal with Python Packaging +// - Need to figure out how to deal with additionalDetails.html, docs directory in python project typically? +// - Understand how to deal with versioning +// - Look at performance improvements for Py4J - socket comms appear to be INCREDIBLY slow. +// - Create test that calls Python 1M times. Just returns 'hello'. See how long it takes +// - Create test that calls Python 1M times. Returns .toString() and see how long it takes. +// - Will help to understand if it's the call from Java to Python that's slow, Python to Java, or both. +// - Performance concern for TransformRecord +// - Currently, triggering the transform() method is pretty fast. But then the Result object comes back and we have to call into the Python side to call the getters +//over and over. Need to look into instead serializing the entire response as JSON and sending that back. +// - Also, since this is heavy JSON processing, might want to consider ORJSON or something like that instead of inbuilt JSON parser/generator +// - Test pip install nifi-my-proc, does nifi pick it up? +// - When ran DetectObjectInImage with multiple threads, Python died. Need to figure out why. +// - If Python Process dies, need to create a new process and need to then create all of the Processors that were in that Process and initialize them. +//- Milestone 2 or 3, not Milestone 1. +// - Remove test-pypi usage from ExtensionManager.py +// - Additional Interfaces beyond just FlowFileTransform +// - FlowFileSource +// - Restructure Maven projects +// - Should this all go under Framework? +// +// +// CONSIDER: +// - Clustering: Ensure component on all nodes? +// - Consider "pip freeze" type of thing to ensure that python dependencies are same across nodes when joining cluster. +// - Update python code
[GitHub] [nifi] exceptionfactory commented on pull request #4773: NIFI-8161 NiFi EL: migration from SimpleDateFormat to DateTimeFormatter
exceptionfactory commented on PR #4773: URL: https://github.com/apache/nifi/pull/4773#issuecomment-1503885247 Thanks for revisiting this pull request @arkadius! Yes, with the main branch version now set to 2.0.0-SNAPSHOT, this is a good time to rebase the pull request and move forward with these changes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] arkadius commented on pull request #4773: NIFI-8161 NiFi EL: migration from SimpleDateFormat to DateTimeFormatter
arkadius commented on PR #4773: URL: https://github.com/apache/nifi/pull/4773#issuecomment-1503816326 I heard that you are planning to release 2.0 version. Are you interested in merge'ing this performance improvement into the new version. I could rebase this branch if you want. What do you think @turcsanyip ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] exceptionfactory commented on pull request #7163: Nifi 11430 - Fixed empty file being produced when grouping by query and fixed _source and _meta extraction by query
exceptionfactory commented on PR #7163: URL: https://github.com/apache/nifi/pull/7163#issuecomment-1503603605 Thanks for the contribution @r-vandenbos! One procedural note, it looks like the username on this pull request @r-vandenbos does not match the username on the Git commit, which is @vandenbos. In general, these usernames should match for proper attribution of both the PR and the commits. It should be possible to update the author on the commit to match. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] tpalfy opened a new pull request, #7164: NIFI-11386 Add "resource" and "audience" support for StandardOauth2AccessTokenProvider
tpalfy opened a new pull request, #7164: URL: https://github.com/apache/nifi/pull/7164 # Summary [NIFI-11386](https://issues.apache.org/jira/browse/NIFI-11386) # Tracking Please complete the following tracking steps prior to pull request creation. ### Issue Tracking - [ ] [Apache NiFi Jira](https://issues.apache.org/jira/browse/NIFI) issue created ### Pull Request Tracking - [ ] Pull Request title starts with Apache NiFi Jira issue number, such as `NIFI-0` - [ ] Pull Request commit message starts with Apache NiFi Jira issue number, as such `NIFI-0` ### Pull Request Formatting - [ ] Pull Request based on current revision of the `main` branch - [ ] Pull Request refers to a feature branch with one commit containing changes # Verification Please indicate the verification steps performed prior to pull request creation. ### Build - [ ] Build completed using `mvn clean install -P contrib-check` - [ ] JDK 11 - [ ] JDK 17 ### Licensing - [ ] New dependencies are compatible with the [Apache License 2.0](https://apache.org/licenses/LICENSE-2.0) according to the [License Policy](https://www.apache.org/legal/resolved.html) - [ ] New dependencies are documented in applicable `LICENSE` and `NOTICE` files ### Documentation - [ ] Documentation formatting appears as expected in rendered files -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] szaszm commented on pull request #1549: MINIFICPP-2093 Update default features/extensions
szaszm commented on PR #1549: URL: https://github.com/apache/nifi-minifi-cpp/pull/1549#issuecomment-1503569610 The windows CI failure is unrelated, due to some ODBC env issue. Do you think we can keep OPC-UA? @fgerlits @martinzink -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] r-vandenbos opened a new pull request, #7163: Nifi 11430 - Fixed empty file being produced when grouping by query and fixed _source and _meta extraction by query
r-vandenbos opened a new pull request, #7163: URL: https://github.com/apache/nifi/pull/7163 # Summary [NIFI-11430](https://issues.apache.org/jira/browse/NIFI-11430) # Tracking Please complete the following tracking steps prior to pull request creation. ### Issue Tracking - [x] [Apache NiFi Jira](https://issues.apache.org/jira/browse/NIFI) issue created ### Pull Request Tracking - [x] Pull Request title starts with Apache NiFi Jira issue number, such as `NIFI-0` - [x] Pull Request commit message starts with Apache NiFi Jira issue number, as such `NIFI-0` ### Pull Request Formatting - [x] Pull Request based on current revision of the `main` branch - [x] Pull Request refers to a feature branch with one commit containing changes # Verification Please indicate the verification steps performed prior to pull request creation. ### Build - [ ] Build completed using `mvn clean install -P contrib-check` - [x] JDK 11 - [ ] JDK 17 ### Licensing - [x] New dependencies are compatible with the [Apache License 2.0](https://apache.org/licenses/LICENSE-2.0) according to the [License Policy](https://www.apache.org/legal/resolved.html) - [x] New dependencies are documented in applicable `LICENSE` and `NOTICE` files ### Documentation - [x] Documentation formatting appears as expected in rendered files -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Assigned] (NIFI-11432) Upgrade Angular Version to 1.8.3
[ https://issues.apache.org/jira/browse/NIFI-11432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mike R reassigned NIFI-11432: - Assignee: Mike R > Upgrade Angular Version to 1.8.3 > > > Key: NIFI-11432 > URL: https://issues.apache.org/jira/browse/NIFI-11432 > Project: Apache NiFi > Issue Type: Improvement >Affects Versions: 1.21.0 >Reporter: Mike R >Assignee: Mike R >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > Upgrade Angular Version to 1.8.3 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [nifi-minifi-cpp] martinzink commented on a diff in pull request #1552: MINIFICPP-2089 prefix EventData in flat JSON output so it doesnt need t…
martinzink commented on code in PR #1552:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1552#discussion_r1162822085
##
CMakeLists.txt:
##
@@ -288,6 +288,7 @@ target_include_directories(concurrentqueue SYSTEM INTERFACE
"${CMAKE_CURRENT_SOU
# RapidJSON
add_library(RapidJSON INTERFACE)
target_include_directories(RapidJSON SYSTEM INTERFACE
"${CMAKE_CURRENT_SOURCE_DIR}/thirdparty/rapidjson-48fbd8cd202ca54031fe799db2ad44ffa8e77c13/include")
+target_compile_definitions(RapidJSON INTERFACE RAPIDJSON_HAS_STDSTRING)
Review Comment:
Rapidjson is header only, and this change only enables helper functions so
its a bit more user friendly with strings
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi] mr1716 opened a new pull request, #7162: NIFI-11432 Upgrade Angular Version to 1.8.3
mr1716 opened a new pull request, #7162: URL: https://github.com/apache/nifi/pull/7162 # Summary [NIFI-11432](https://issues.apache.org/jira/browse/NIFI-11432) # Tracking Please complete the following tracking steps prior to pull request creation. ### Issue Tracking - [X] [Apache NiFi Jira](https://issues.apache.org/jira/browse/NIFI-11432) issue created ### Pull Request Tracking - [X] Pull Request title starts with Apache NiFi Jira issue number, such as `NIFI-0` - [X] Pull Request commit message starts with Apache NiFi Jira issue number, as such `NIFI-0` ### Pull Request Formatting - [X] Pull Request based on current revision of the `main` branch - [X] Pull Request refers to a feature branch with one commit containing changes # Verification Please indicate the verification steps performed prior to pull request creation. ### Build - [ ] Build completed using `mvn clean install -P contrib-check` - [ ] JDK 11 - [ ] JDK 17 ### Licensing - [ ] New dependencies are compatible with the [Apache License 2.0](https://apache.org/licenses/LICENSE-2.0) according to the [License Policy](https://www.apache.org/legal/resolved.html) - [ ] New dependencies are documented in applicable `LICENSE` and `NOTICE` files ### Documentation - [ ] Documentation formatting appears as expected in rendered files -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (NIFI-11433) Upgrade moment to 2.29.4
Mike R created NIFI-11433: - Summary: Upgrade moment to 2.29.4 Key: NIFI-11433 URL: https://issues.apache.org/jira/browse/NIFI-11433 Project: Apache NiFi Issue Type: Improvement Affects Versions: 1.21.0 Reporter: Mike R Upgrade moment to 2.29.4 to remediate vulnerability [Regular Expression Denial of Service (ReDoS) in moment | CVE-2022-31129 | Snyk|https://security.snyk.io/vuln/SNYK-JS-MOMENT-2944238] -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [nifi-minifi-cpp] szaszm commented on a diff in pull request #1552: MINIFICPP-2089 prefix EventData in flat JSON output so it doesnt need t…
szaszm commented on code in PR #1552:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1552#discussion_r1162810078
##
extensions/windows-event-log/wel/JSONUtils.cpp:
##
@@ -135,28 +157,37 @@ rapidjson::Document toJSONImpl(const pugi::xml_node&
root, bool flatten) {
{
auto eventData_xml = event_xml.child("EventData");
-// create EventData subarray even if flatten requested
-doc.AddMember("EventData", rapidjson::kArrayType, doc.GetAllocator());
-for (const auto& data : eventData_xml.children()) {
- auto name_attr = data.attribute("Name");
- rapidjson::Value item(rapidjson::kObjectType);
- item.AddMember("Name", rapidjson::StringRef(name_attr.value()),
doc.GetAllocator());
- item.AddMember("Content", rapidjson::StringRef(data.text().get()),
doc.GetAllocator());
- item.AddMember("Type", rapidjson::StringRef(data.name()),
doc.GetAllocator());
- // we need to query EventData because a reference to it wouldn't be
stable, as we
- // possibly add members to its parent which could result in reallocation
- doc["EventData"].PushBack(item, doc.GetAllocator());
- // check collision
- if (flatten && !name_attr.empty() && !doc.HasMember(name_attr.value())) {
-doc.AddMember(rapidjson::StringRef(name_attr.value()),
rapidjson::StringRef(data.text().get()), doc.GetAllocator());
+if (flatten) {
+ for (const auto& event_data_child : eventData_xml.children()) {
+std::string key = "EventData";
+if (auto name_attr = event_data_child.attribute("Name").value();
strlen(name_attr)) {
+ key = utils::StringUtils::join_pack(key, ".", name_attr);
+}
+
+if (auto type = event_data_child.name(); strlen(type) > 0) {
+ key = utils::StringUtils::join_pack(key, ".", type);
+}
+
+doc.AddMember(rapidjson::Value(createUniqueKey(key, doc),
doc.GetAllocator()).Move(),
rapidjson::StringRef(event_data_child.text().get()), doc.GetAllocator());
+ }
+} else {
+ auto& event_data = doc.AddMember("EventData", rapidjson::kArrayType,
doc.GetAllocator());
+ for (const auto& event_data_child : eventData_xml.children()) {
+auto name_attr = event_data_child.attribute("Name");
+rapidjson::Value item(rapidjson::kObjectType);
+item.AddMember("Name", rapidjson::StringRef(name_attr.value()),
doc.GetAllocator());
+item.AddMember("Content",
rapidjson::StringRef(event_data_child.text().get()), doc.GetAllocator());
+item.AddMember("Type", rapidjson::StringRef(event_data_child.name()),
doc.GetAllocator());
+doc["EventData"].PushBack(item, doc.GetAllocator());
Review Comment:
Could we change the format to avoid a subobject for all custom values? So
instead of this:
```json
"EventData": [{
"Name": "Foobar",
"Content": "Lorem ipsum",
"Type": "Text"
}]
```
something like this:
```json
"EventData": {
"Foobar": "Lorem ipsum"
}
```
##
CMakeLists.txt:
##
@@ -288,6 +288,7 @@ target_include_directories(concurrentqueue SYSTEM INTERFACE
"${CMAKE_CURRENT_SOU
# RapidJSON
add_library(RapidJSON INTERFACE)
target_include_directories(RapidJSON SYSTEM INTERFACE
"${CMAKE_CURRENT_SOURCE_DIR}/thirdparty/rapidjson-48fbd8cd202ca54031fe799db2ad44ffa8e77c13/include")
+target_compile_definitions(RapidJSON INTERFACE RAPIDJSON_HAS_STDSTRING)
Review Comment:
Is this an interface-only change, or does it require RapidJSON
recompilation? (e.g. missing related symbols)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Created] (NIFI-11432) Upgrade Angular Version to 1.8.3
Mike R created NIFI-11432: - Summary: Upgrade Angular Version to 1.8.3 Key: NIFI-11432 URL: https://issues.apache.org/jira/browse/NIFI-11432 Project: Apache NiFi Issue Type: Improvement Affects Versions: 1.21.0 Reporter: Mike R Upgrade Angular Version to 1.8.3 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-11428) Upgrade groovy to 3.0.17 and spock-core to 2.3
[ https://issues.apache.org/jira/browse/NIFI-11428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Handermann updated NIFI-11428: Fix Version/s: 2.0.0 1.22.0 Resolution: Fixed Status: Resolved (was: Patch Available) > Upgrade groovy to 3.0.17 and spock-core to 2.3 > -- > > Key: NIFI-11428 > URL: https://issues.apache.org/jira/browse/NIFI-11428 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: Pierre Villard >Assignee: Pierre Villard >Priority: Major > Fix For: 2.0.0, 1.22.0 > > Time Spent: 20m > Remaining Estimate: 0h > > Some dependencies already on Groovy 3 but not on 3.0.17. > Also upgrading spock-core to use 2.3-groovy-3.0 > https://spockframework.org/spock/docs/2.3/release_notes.html -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-11426) Upgrade jline to 3.23.0
[ https://issues.apache.org/jira/browse/NIFI-11426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Handermann updated NIFI-11426: Fix Version/s: 2.0.0 1.22.0 Resolution: Fixed Status: Resolved (was: Patch Available) > Upgrade jline to 3.23.0 > --- > > Key: NIFI-11426 > URL: https://issues.apache.org/jira/browse/NIFI-11426 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Pierre Villard >Assignee: Pierre Villard >Priority: Major > Fix For: 2.0.0, 1.22.0 > > Time Spent: 20m > Remaining Estimate: 0h > > Upgrade jline to 3.23.0 > https://github.com/jline/jline3/blob/master/changelog.md -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (NIFI-11427) Upgrade Atlas to 2.3.0
[ https://issues.apache.org/jira/browse/NIFI-11427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17710940#comment-17710940 ] ASF subversion and git services commented on NIFI-11427: Commit b8b508c1a3e104928856f5e2d3e116736fd8 in nifi's branch refs/heads/support/nifi-1.x from Pierre Villard [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=b8b508c1a3 ] NIFI-11427 Upgraded Atlas from 2.2.0 to 2.3.0 This closes #7158 Signed-off-by: David Handermann (cherry picked from commit cbca499070404914dc3e31b14262936f26ba9a41) > Upgrade Atlas to 2.3.0 > -- > > Key: NIFI-11427 > URL: https://issues.apache.org/jira/browse/NIFI-11427 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: Pierre Villard >Assignee: Pierre Villard >Priority: Major > Time Spent: 20m > Remaining Estimate: 0h > > Upgrade Atlas to 2.3.0. > * org.apache.atlas:atlas-client-v2 from 2.2.0 to 2.3.0 > * org.apache.atlas:atlas-notification from 2.2.0 to 2.3.0 > [https://atlas.apache.org/#/WhatsNew-2.3] -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-11427) Upgrade Atlas to 2.3.0
[ https://issues.apache.org/jira/browse/NIFI-11427?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Handermann updated NIFI-11427: Fix Version/s: 2.0.0 1.22.0 Resolution: Fixed Status: Resolved (was: Patch Available) > Upgrade Atlas to 2.3.0 > -- > > Key: NIFI-11427 > URL: https://issues.apache.org/jira/browse/NIFI-11427 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: Pierre Villard >Assignee: Pierre Villard >Priority: Major > Fix For: 2.0.0, 1.22.0 > > Time Spent: 20m > Remaining Estimate: 0h > > Upgrade Atlas to 2.3.0. > * org.apache.atlas:atlas-client-v2 from 2.2.0 to 2.3.0 > * org.apache.atlas:atlas-notification from 2.2.0 to 2.3.0 > [https://atlas.apache.org/#/WhatsNew-2.3] -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (NIFI-11426) Upgrade jline to 3.23.0
[ https://issues.apache.org/jira/browse/NIFI-11426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17710939#comment-17710939 ] ASF subversion and git services commented on NIFI-11426: Commit d105ed6165c680d9ae0e4983f4bcc2a74716948e in nifi's branch refs/heads/support/nifi-1.x from Pierre Villard [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=d105ed6165 ] NIFI-11426 Upgraded JLine from 3.22.0 to 3.23.0 This closes #7157 Signed-off-by: David Handermann (cherry picked from commit 3b216ced924c8d4e6316fe4b0b2185ae816079cd) > Upgrade jline to 3.23.0 > --- > > Key: NIFI-11426 > URL: https://issues.apache.org/jira/browse/NIFI-11426 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Pierre Villard >Assignee: Pierre Villard >Priority: Major > Time Spent: 20m > Remaining Estimate: 0h > > Upgrade jline to 3.23.0 > https://github.com/jline/jline3/blob/master/changelog.md -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (NIFI-11428) Upgrade groovy to 3.0.17 and spock-core to 2.3
[ https://issues.apache.org/jira/browse/NIFI-11428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17710941#comment-17710941 ] ASF subversion and git services commented on NIFI-11428: Commit 605bf150fa1cbd0817608a0758399edbde4824a0 in nifi's branch refs/heads/support/nifi-1.x from Pierre Villard [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=605bf150fa ] NIFI-11428 Upgraded Groovy to 3.0.17 and spock-core to 2.3 This closes #7159 Signed-off-by: David Handermann (cherry picked from commit 594e45cf9a19dfee31ec76d306d91314b165aa6a) > Upgrade groovy to 3.0.17 and spock-core to 2.3 > -- > > Key: NIFI-11428 > URL: https://issues.apache.org/jira/browse/NIFI-11428 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: Pierre Villard >Assignee: Pierre Villard >Priority: Major > Time Spent: 20m > Remaining Estimate: 0h > > Some dependencies already on Groovy 3 but not on 3.0.17. > Also upgrading spock-core to use 2.3-groovy-3.0 > https://spockframework.org/spock/docs/2.3/release_notes.html -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (NIFI-11426) Upgrade jline to 3.23.0
[ https://issues.apache.org/jira/browse/NIFI-11426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17710933#comment-17710933 ] ASF subversion and git services commented on NIFI-11426: Commit 3b216ced924c8d4e6316fe4b0b2185ae816079cd in nifi's branch refs/heads/main from Pierre Villard [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=3b216ced92 ] NIFI-11426 Upgraded JLine from 3.22.0 to 3.23.0 This closes #7157 Signed-off-by: David Handermann > Upgrade jline to 3.23.0 > --- > > Key: NIFI-11426 > URL: https://issues.apache.org/jira/browse/NIFI-11426 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Pierre Villard >Assignee: Pierre Villard >Priority: Major > Time Spent: 20m > Remaining Estimate: 0h > > Upgrade jline to 3.23.0 > https://github.com/jline/jline3/blob/master/changelog.md -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (NIFI-11427) Upgrade Atlas to 2.3.0
[ https://issues.apache.org/jira/browse/NIFI-11427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17710934#comment-17710934 ] ASF subversion and git services commented on NIFI-11427: Commit cbca499070404914dc3e31b14262936f26ba9a41 in nifi's branch refs/heads/main from Pierre Villard [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=cbca499070 ] NIFI-11427 Upgraded Atlas from 2.2.0 to 2.3.0 This closes #7158 Signed-off-by: David Handermann > Upgrade Atlas to 2.3.0 > -- > > Key: NIFI-11427 > URL: https://issues.apache.org/jira/browse/NIFI-11427 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: Pierre Villard >Assignee: Pierre Villard >Priority: Major > Time Spent: 20m > Remaining Estimate: 0h > > Upgrade Atlas to 2.3.0. > * org.apache.atlas:atlas-client-v2 from 2.2.0 to 2.3.0 > * org.apache.atlas:atlas-notification from 2.2.0 to 2.3.0 > [https://atlas.apache.org/#/WhatsNew-2.3] -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (NIFI-11428) Upgrade groovy to 3.0.17 and spock-core to 2.3
[ https://issues.apache.org/jira/browse/NIFI-11428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17710935#comment-17710935 ] ASF subversion and git services commented on NIFI-11428: Commit 594e45cf9a19dfee31ec76d306d91314b165aa6a in nifi's branch refs/heads/main from Pierre Villard [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=594e45cf9a ] NIFI-11428 Upgraded Groovy to 3.0.17 and spock-core to 2.3 This closes #7159 Signed-off-by: David Handermann > Upgrade groovy to 3.0.17 and spock-core to 2.3 > -- > > Key: NIFI-11428 > URL: https://issues.apache.org/jira/browse/NIFI-11428 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: Pierre Villard >Assignee: Pierre Villard >Priority: Major > Time Spent: 20m > Remaining Estimate: 0h > > Some dependencies already on Groovy 3 but not on 3.0.17. > Also upgrading spock-core to use 2.3-groovy-3.0 > https://spockframework.org/spock/docs/2.3/release_notes.html -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [nifi] exceptionfactory closed pull request #7157: NIFI-11426 - Upgrade jline to 3.23.0
exceptionfactory closed pull request #7157: NIFI-11426 - Upgrade jline to 3.23.0 URL: https://github.com/apache/nifi/pull/7157 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] exceptionfactory closed pull request #7158: NIFI-11427 - Upgrade Atlas to 2.3.0
exceptionfactory closed pull request #7158: NIFI-11427 - Upgrade Atlas to 2.3.0 URL: https://github.com/apache/nifi/pull/7158 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] exceptionfactory closed pull request #7159: NIFI-11428 - Upgrade groovy to 3.0.17 and spock-core to 2.3
exceptionfactory closed pull request #7159: NIFI-11428 - Upgrade groovy to 3.0.17 and spock-core to 2.3 URL: https://github.com/apache/nifi/pull/7159 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] exceptionfactory commented on pull request #7160: NIFI-11429 - Upgrade Gremlin to 3.6.2
exceptionfactory commented on PR #7160: URL: https://github.com/apache/nifi/pull/7160#issuecomment-1503282414 @mattyb149 Can you take a closer look at this particular upgrade? There was an issue with some graph databases in past versions, but this is a good opportunity to move to the latest version of the Gremlin libraries. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (NIFI-11430) Elasticsearch Paginated Query processor empty hits and extraction issues
[ https://issues.apache.org/jira/browse/NIFI-11430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ryan updated NIFI-11430: Description: If you use the PaginatedJsonQueryProcessor with the following settings |el-rest-split-up-hits|PER_QUERY| |el-rest-pagination-type|POINT_IN_TIME !https://nifi.apache.org/docs/nifi-docs/html/images/iconInfo.png!| |el-rest-output-no-hits|true| If you run the processor and hits are returned it outputs 2 flow files, 1 with the query hits and the other that is empty which is incorrect, it should return only 1 flow file that contains hits. --- If you use the PaginatedJsonQueryProcessor with the following settings |el-rest-split-up-hits|PER_QUERY| |el-rest-pagination-type|POINT_IN_TIME | |el-rest-output-no-hits|true| |el-rest-format-hits|SOURCE_ONLY| or |el-rest-split-up-hits|PER_QUERY| |el-rest-pagination-type|POINT_IN_TIME | |el-rest-output-no-hits|true| |el-rest-format-hits|METADATA_ONLY| If you run the processor and hits are returned it does not extract either _source or _metadata was: If you use the PaginatedJsonQueryProcessor with the following settings |el-rest-split-up-hits|PER_QUERY| |el-rest-pagination-type|POINT_IN_TIME !https://nifi.apache.org/docs/nifi-docs/html/images/iconInfo.png!| |el-rest-output-no-hits|true| If you run the processor and hits are returned it outputs 2 flow files, 1 with the query hits and the other that is empty which is incorrect, it should return only 1 flow file that contains hits. Summary: Elasticsearch Paginated Query processor empty hits and extraction issues (was: Elasticsearch Paginated Query processor outputs an empty file in error if you set it the group results per query) > Elasticsearch Paginated Query processor empty hits and extraction issues > > > Key: NIFI-11430 > URL: https://issues.apache.org/jira/browse/NIFI-11430 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.20.0 >Reporter: Ryan >Priority: Major > > If you use the PaginatedJsonQueryProcessor with the following settings > |el-rest-split-up-hits|PER_QUERY| > |el-rest-pagination-type|POINT_IN_TIME > !https://nifi.apache.org/docs/nifi-docs/html/images/iconInfo.png!| > |el-rest-output-no-hits|true| > If you run the processor and hits are returned it outputs 2 flow files, 1 > with the query hits and the other that is empty which is incorrect, it should > return only 1 flow file that contains hits. > > --- > > If you use the PaginatedJsonQueryProcessor with the following settings > |el-rest-split-up-hits|PER_QUERY| > |el-rest-pagination-type|POINT_IN_TIME | > |el-rest-output-no-hits|true| > |el-rest-format-hits|SOURCE_ONLY| > or > |el-rest-split-up-hits|PER_QUERY| > |el-rest-pagination-type|POINT_IN_TIME | > |el-rest-output-no-hits|true| > |el-rest-format-hits|METADATA_ONLY| > If you run the processor and hits are returned it does not extract either > _source or _metadata -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (NIFI-11431) Elasticsearch Paginated Query processor does not extract _source or _meta if you group the results by query
[ https://issues.apache.org/jira/browse/NIFI-11431?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ryan resolved NIFI-11431. - Resolution: Duplicate > Elasticsearch Paginated Query processor does not extract _source or _meta if > you group the results by query > --- > > Key: NIFI-11431 > URL: https://issues.apache.org/jira/browse/NIFI-11431 > Project: Apache NiFi > Issue Type: Bug >Reporter: Ryan >Priority: Major > > If you use the PaginatedJsonQueryProcessor with the following settings > |el-rest-split-up-hits|PER_QUERY| > |el-rest-pagination-type|POINT_IN_TIME | > |el-rest-output-no-hits|true| > |el-rest-format-hits|SOURCE_ONLY| > or > |el-rest-split-up-hits|PER_QUERY| > |el-rest-pagination-type|POINT_IN_TIME | > |el-rest-output-no-hits|true| > |el-rest-format-hits|METADATA_ONLY| > If you run the processor and hits are returned it does not extract either > _source or _metadata -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (NIFI-11431) Elasticsearch Paginated Query processor does not extract _source or _meta if you group the results by query
Ryan created NIFI-11431: --- Summary: Elasticsearch Paginated Query processor does not extract _source or _meta if you group the results by query Key: NIFI-11431 URL: https://issues.apache.org/jira/browse/NIFI-11431 Project: Apache NiFi Issue Type: Bug Reporter: Ryan If you use the PaginatedJsonQueryProcessor with the following settings |el-rest-split-up-hits|PER_QUERY| |el-rest-pagination-type|POINT_IN_TIME | |el-rest-output-no-hits|true| |el-rest-format-hits|SOURCE_ONLY| or |el-rest-split-up-hits|PER_QUERY| |el-rest-pagination-type|POINT_IN_TIME | |el-rest-output-no-hits|true| |el-rest-format-hits|METADATA_ONLY| If you run the processor and hits are returned it does not extract either _source or _metadata -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (NIFI-11430) Elasticsearch Paginated Query processor outputs an empty file in error if you set it the group results per query
Ryan created NIFI-11430: --- Summary: Elasticsearch Paginated Query processor outputs an empty file in error if you set it the group results per query Key: NIFI-11430 URL: https://issues.apache.org/jira/browse/NIFI-11430 Project: Apache NiFi Issue Type: Bug Affects Versions: 1.20.0 Reporter: Ryan If you use the PaginatedJsonQueryProcessor with the following settings |el-rest-split-up-hits|PER_QUERY| |el-rest-pagination-type|POINT_IN_TIME !https://nifi.apache.org/docs/nifi-docs/html/images/iconInfo.png!| |el-rest-output-no-hits|true| If you run the processor and hits are returned it outputs 2 flow files, 1 with the query hits and the other that is empty which is incorrect, it should return only 1 flow file that contains hits. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (NIFI-11366) MiNiFi/C2 - Support access via LB / Proxy
[ https://issues.apache.org/jira/browse/NIFI-11366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17710869#comment-17710869 ] ASF subversion and git services commented on NIFI-11366: Commit a98ddf3e01ee0bf3c0bd4d7a846674a2fb7908f1 in nifi's branch refs/heads/support/nifi-1.x from Ferenc Kis [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=a98ddf3e01 ] NIFI-11366 Proxy aware C2 communication Signed-off-by: Ferenc Erdei This closes #7125 (cherry picked from commit d84ce83) > MiNiFi/C2 - Support access via LB / Proxy > - > > Key: NIFI-11366 > URL: https://issues.apache.org/jira/browse/NIFI-11366 > Project: Apache NiFi > Issue Type: New Feature > Components: C2, MiNiFi >Reporter: Ferenc Kis >Assignee: Ferenc Kis >Priority: Major > Labels: minifi-java > Fix For: 2.0.0, 1.22.0 > > Time Spent: 2h 40m > Remaining Estimate: 0h > > While the Heartbeat and Acknowledge communication from the agents can be > configured via reverse proxy / load balancer currently the agents are not > aware of the host > For flow and asset download full urls are generated in the operation requests > by the C2 server so the actual download happens via direct node access > bypassing the proxy / LB. > For backwards compatibility we are not touching the existing configuration. > We could introduce 3 new properties: > c2.rest.path.base=[http://c2host/c2-server/api|http://unsecure.cemcldr.link/] > c2.rest.path.heartbeat=/heartbeat > c2.rest.path.acknowledge=/acknowledge > If these are set then use them if not then fall back to the old properties > and parse the host from there > h4. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-11366) MiNiFi/C2 - Support access via LB / Proxy
[ https://issues.apache.org/jira/browse/NIFI-11366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ferenc Erdei updated NIFI-11366: Fix Version/s: 2.0.0 1.22.0 > MiNiFi/C2 - Support access via LB / Proxy > - > > Key: NIFI-11366 > URL: https://issues.apache.org/jira/browse/NIFI-11366 > Project: Apache NiFi > Issue Type: New Feature > Components: C2, MiNiFi >Reporter: Ferenc Kis >Assignee: Ferenc Kis >Priority: Major > Labels: minifi-java > Fix For: 2.0.0, 1.22.0 > > Time Spent: 2h 40m > Remaining Estimate: 0h > > While the Heartbeat and Acknowledge communication from the agents can be > configured via reverse proxy / load balancer currently the agents are not > aware of the host > For flow and asset download full urls are generated in the operation requests > by the C2 server so the actual download happens via direct node access > bypassing the proxy / LB. > For backwards compatibility we are not touching the existing configuration. > We could introduce 3 new properties: > c2.rest.path.base=[http://c2host/c2-server/api|http://unsecure.cemcldr.link/] > c2.rest.path.heartbeat=/heartbeat > c2.rest.path.acknowledge=/acknowledge > If these are set then use them if not then fall back to the old properties > and parse the host from there > h4. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-11366) MiNiFi/C2 - Support access via LB / Proxy
[ https://issues.apache.org/jira/browse/NIFI-11366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ferenc Erdei updated NIFI-11366: Description: While the Heartbeat and Acknowledge communication from the agents can be configured via reverse proxy / load balancer currently the agents are not aware of the host For flow and asset download full urls are generated in the operation requests by the C2 server so the actual download happens via direct node access bypassing the proxy / LB. For backwards compatibility we are not touching the existing configuration. We could introduce 3 new properties: c2.rest.path.base=[http://c2host/c2-server/api|http://unsecure.cemcldr.link/] c2.rest.path.heartbeat=/heartbeat c2.rest.path.acknowledge=/acknowledge If these are set then use them if not then fall back to the old properties and parse the host from there h4. was: While the Heartbeat and Acknowledge communication from the agents can be configured via reverse proxy / load balancer currently the agents are not aware of the host For flow and asset download full urls are generated in the operation requests by the C2 server so the actual download happens via direct node access bypassing the proxy / LB. For backwards compatibility we are not touching the existing configuration. We could introduce 3 new properties: c2.rest.api=[http://c2host/c2-server/api|http://unsecure.cemcldr.link/] c2.rest.path.hb=/heartbeat c2.rest.path.ack=//acknowledge If these are set then use them if not then fall back to the old properties and parse the host from there h4. > MiNiFi/C2 - Support access via LB / Proxy > - > > Key: NIFI-11366 > URL: https://issues.apache.org/jira/browse/NIFI-11366 > Project: Apache NiFi > Issue Type: New Feature > Components: C2, MiNiFi >Reporter: Ferenc Kis >Assignee: Ferenc Kis >Priority: Major > Labels: minifi-java > Time Spent: 2h 40m > Remaining Estimate: 0h > > While the Heartbeat and Acknowledge communication from the agents can be > configured via reverse proxy / load balancer currently the agents are not > aware of the host > For flow and asset download full urls are generated in the operation requests > by the C2 server so the actual download happens via direct node access > bypassing the proxy / LB. > For backwards compatibility we are not touching the existing configuration. > We could introduce 3 new properties: > c2.rest.path.base=[http://c2host/c2-server/api|http://unsecure.cemcldr.link/] > c2.rest.path.heartbeat=/heartbeat > c2.rest.path.acknowledge=/acknowledge > If these are set then use them if not then fall back to the old properties > and parse the host from there > h4. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (NIFI-11366) MiNiFi/C2 - Support access via LB / Proxy
[ https://issues.apache.org/jira/browse/NIFI-11366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17710861#comment-17710861 ] ASF subversion and git services commented on NIFI-11366: Commit d84ce836541266ce930bf12f32df2ecb035ac23f in nifi's branch refs/heads/main from Ferenc Kis [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=d84ce83654 ] NIFI-11366 Proxy aware C2 communication Signed-off-by: Ferenc Erdei This closes #7125 > MiNiFi/C2 - Support access via LB / Proxy > - > > Key: NIFI-11366 > URL: https://issues.apache.org/jira/browse/NIFI-11366 > Project: Apache NiFi > Issue Type: New Feature > Components: C2, MiNiFi >Reporter: Ferenc Kis >Assignee: Ferenc Kis >Priority: Major > Labels: minifi-java > Time Spent: 2h 40m > Remaining Estimate: 0h > > While the Heartbeat and Acknowledge communication from the agents can be > configured via reverse proxy / load balancer currently the agents are not > aware of the host > For flow and asset download full urls are generated in the operation requests > by the C2 server so the actual download happens via direct node access > bypassing the proxy / LB. > For backwards compatibility we are not touching the existing configuration. > We could introduce 3 new properties: > c2.rest.api=[http://c2host/c2-server/api|http://unsecure.cemcldr.link/] > c2.rest.path.hb=/heartbeat > c2.rest.path.ack=//acknowledge > If these are set then use them if not then fall back to the old properties > and parse the host from there > h4. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [nifi] ferencerdei closed pull request #7125: NIFI-11366 Proxy aware C2 communication
ferencerdei closed pull request #7125: NIFI-11366 Proxy aware C2 communication URL: https://github.com/apache/nifi/pull/7125 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] ferencerdei commented on pull request #7125: NIFI-11366 Proxy aware C2 communication
ferencerdei commented on PR #7125: URL: https://github.com/apache/nifi/pull/7125#issuecomment-1502973153 Tested the change with the Debug and Update Configuration commands. It works properly. +1 Merging... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (MINIFICPP-2095) Inconsistent naming in C2 DeviceInfoNode::systemInfo::machinearch
Martin Zink created MINIFICPP-2095: -- Summary: Inconsistent naming in C2 DeviceInfoNode::systemInfo::machinearch Key: MINIFICPP-2095 URL: https://issues.apache.org/jira/browse/MINIFICPP-2095 Project: Apache NiFi MiNiFi C++ Issue Type: Bug Reporter: Martin Zink Assignee: Martin Zink Nifi and minifi java uses machineArch (the other properties also use camelCase), whilst minifi cpp uses machinearch (this makes it hard to create C2 servers that can be used simultaneously with java and cpp agents. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [nifi] bejancsaba commented on a diff in pull request #7125: NIFI-11366 Proxy aware C2 communication
bejancsaba commented on code in PR #7125:
URL: https://github.com/apache/nifi/pull/7125#discussion_r1162473024
##
c2/c2-client-bundle/c2-client-http/src/test/java/org/apache/nifi/c2/client/http/url/C2UrlProviderFactoryTest.java:
##
@@ -0,0 +1,96 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.c2.client.http.url;
+
+import static org.junit.jupiter.api.Assertions.assertInstanceOf;
+import static org.junit.jupiter.api.Assertions.assertThrowsExactly;
+import static org.mockito.Mockito.lenient;
+import static org.mockito.Mockito.when;
+
+import org.apache.nifi.c2.client.C2ClientConfig;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.extension.ExtendWith;
+import org.mockito.InjectMocks;
+import org.mockito.Mock;
+import org.mockito.junit.jupiter.MockitoExtension;
+
+@ExtendWith(MockitoExtension.class)
+public class C2UrlProviderFactoryTest {
+
+private static final String C2_REST_BASE = "https://host:8080/c2/api;;
+private static final String HEARTBEAT_PATH = "/heartbeat";
+private static final String ACKNOWLEDGE_PATH = "/acknowledge";
+
+@Mock
+private C2ClientConfig clientConfig;
+@InjectMocks
+private C2UrlProviderFactory testC2UrlProviderFactory;
+
+@Test
+public void testProxyAwareC2UrlProviderIsCreated() {
+// given
+when(clientConfig.getC2RestPathBase()).thenReturn(C2_REST_BASE);
+when(clientConfig.getC2RestPathHeartbeat()).thenReturn(HEARTBEAT_PATH);
+
when(clientConfig.getC2RestPathAcknowledge()).thenReturn(ACKNOWLEDGE_PATH);
+
+// when
+C2UrlProvider c2UrlProvider = testC2UrlProviderFactory.create();
+
+// then
+assertInstanceOf(ProxyAwareC2UrlProvider.class, c2UrlProvider);
+}
+
+@Test
+public void testLegacyC2UrlProviderIsCreated() {
+// given
+when(clientConfig.getC2Url()).thenReturn(C2_REST_BASE +
HEARTBEAT_PATH);
+when(clientConfig.getC2AckUrl()).thenReturn(C2_REST_BASE +
ACKNOWLEDGE_PATH);
+
+// when
+C2UrlProvider c2UrlProvider = testC2UrlProviderFactory.create();
+
+// then
+assertInstanceOf(LegacyC2UrlProvider.class, c2UrlProvider);
+}
+
+@Test
+public void testProxyAwareProviderTakesPrecedenceOverLegacy() {
Review Comment:
I was looking for this test, thank you!
##
c2/c2-client-bundle/c2-client-api/src/main/java/org/apache/nifi/c2/client/api/C2Client.java:
##
@@ -57,4 +57,14 @@ public interface C2Client {
* @return optional error message if any issues occurred
*/
Optional uploadBundle(String callbackUrl, byte[] bundle);
+
+/**
+ * Creates a callback URL according to proxy aware C2 settings
+ *
+ * @param absoluteUrl absolute url sent by C2 server
+ * @param relativeUrl relative url sent by C2 server
+ * @return finalised callback url
+ * @throws Exception when the callback url can not be created as per the
current configuration and parameters
+ */
+String getCallbackUrl(String absoluteUrl, String relativeUrl) throws
Exception;
Review Comment:
Thanks looks good
##
c2/c2-client-bundle/c2-client-http/src/main/java/org/apache/nifi/c2/client/http/url/ProxyAwareC2UrlProvider.java:
##
@@ -0,0 +1,66 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.c2.client.http.url;
+
+import static
[GitHub] [nifi] mark-bathori opened a new pull request, #7161: NIFI-11334: PutIceberg processor instance interference due same class loader usage
mark-bathori opened a new pull request, #7161: URL: https://github.com/apache/nifi/pull/7161 # Summary [NIFI-11334](https://issues.apache.org/jira/browse/NIFI-11334) # Tracking Please complete the following tracking steps prior to pull request creation. ### Issue Tracking - [x] [Apache NiFi Jira](https://issues.apache.org/jira/browse/NIFI) issue created ### Pull Request Tracking - [ ] Pull Request title starts with Apache NiFi Jira issue number, such as `NIFI-0` - [ ] Pull Request commit message starts with Apache NiFi Jira issue number, as such `NIFI-0` ### Pull Request Formatting - [ ] Pull Request based on current revision of the `main` branch - [ ] Pull Request refers to a feature branch with one commit containing changes # Verification Please indicate the verification steps performed prior to pull request creation. ### Build - [x] Build completed using `mvn clean install -P contrib-check` - [x] JDK 11 - [ ] JDK 17 ### Licensing - [ ] New dependencies are compatible with the [Apache License 2.0](https://apache.org/licenses/LICENSE-2.0) according to the [License Policy](https://www.apache.org/legal/resolved.html) - [ ] New dependencies are documented in applicable `LICENSE` and `NOTICE` files ### Documentation - [ ] Documentation formatting appears as expected in rendered files -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (MINIFICPP-2089) Revisit CWEL EventData format in flattened JSON output
[ https://issues.apache.org/jira/browse/MINIFICPP-2089?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Martin Zink updated MINIFICPP-2089: --- Status: Patch Available (was: In Progress) https://github.com/apache/nifi-minifi-cpp/pull/1552 > Revisit CWEL EventData format in flattened JSON output > -- > > Key: MINIFICPP-2089 > URL: https://issues.apache.org/jira/browse/MINIFICPP-2089 > Project: Apache NiFi MiNiFi C++ > Issue Type: Improvement >Reporter: Marton Szasz >Assignee: Martin Zink >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > In flattened JSON, the data in an EventData tag is duplicated: Once in the > normal flattened output in the main object, and once in a list of subobjects > in a slightly different format. The reason for this was to avoid losing data, > even if the inner keys of EventData are clashing with keys of the outer > System object, which happens in practice, but there may be a better way to > ensure this. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [nifi-minifi-cpp] martinzink opened a new pull request, #1552: MINIFICPP-2089 prefix EventData in flatten output so it doesnt need t…
martinzink opened a new pull request, #1552: URL: https://github.com/apache/nifi-minifi-cpp/pull/1552 …o be included twice Thank you for submitting a contribution to Apache NiFi - MiNiFi C++. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [x] Does your PR title start with MINIFICPP- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically main)? - [x] Is your initial contribution a single, squashed commit? ### For code changes: - [x] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [x] If applicable, have you updated the LICENSE file? - [x] If applicable, have you updated the NOTICE file? ### For documentation related changes: - [x] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check GitHub Actions CI results for build issues and submit an update to your PR as soon as possible. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (MINIFICPP-2094) Refactoring: validators should not be shared_ptr's
Ferenc Gerlits created MINIFICPP-2094: - Summary: Refactoring: validators should not be shared_ptr's Key: MINIFICPP-2094 URL: https://issues.apache.org/jira/browse/MINIFICPP-2094 Project: Apache NiFi MiNiFi C++ Issue Type: Improvement Reporter: Ferenc Gerlits Assignee: Ferenc Gerlits -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (MINIFICPP-1825) Create Properties at compile time
[
https://issues.apache.org/jira/browse/MINIFICPP-1825?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ferenc Gerlits reassigned MINIFICPP-1825:
-
Assignee: Ferenc Gerlits
> Create Properties at compile time
> -
>
> Key: MINIFICPP-1825
> URL: https://issues.apache.org/jira/browse/MINIFICPP-1825
> Project: Apache NiFi MiNiFi C++
> Issue Type: Improvement
>Reporter: Ferenc Gerlits
>Assignee: Ferenc Gerlits
>Priority: Minor
>
> Properties and Relationships of Processors and Controller Services should be
> {{{}constexpr{}}}, so they get created at compile time. This would speed up
> the startup, as well as avoid the problem of the unpredictable order of
> initialization of these Properties.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
