[jira] [Created] (NIFI-12842) InvokeHTTP version wrong encoding of % in URL
WojciechWitos created NIFI-12842: Summary: InvokeHTTP version wrong encoding of % in URL Key: NIFI-12842 URL: https://issues.apache.org/jira/browse/NIFI-12842 Project: Apache NiFi Issue Type: Bug Affects Versions: 1.25.0 Environment: RHEL 7 Reporter: WojciechWitos Attachments: image-2024-02-26-08-10-12-657.png, image-2024-02-26-08-11-25-213.png, image-2024-02-26-08-13-14-199.png, image-2024-02-26-08-16-36-309.png Hi! I've encountered on the version 1.25 issue with encoding of % in invokehttp processor. It changes every % into %25, which causes error 403 and most of the flows don't work. On the version 1.24 everything works properly with this processor. Here are the screenshots of 1.25: !image-2024-02-26-08-10-12-657.png! And request: !image-2024-02-26-08-11-25-213.png! Where on version 1.24 this issue doesn't persist: !image-2024-02-26-08-13-14-199.png! !image-2024-02-26-08-16-36-309.png! Please investigate this issue, it is the blocker of upgrading the environment to this version. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (NIFI-12841) Introduce RemoveXYZ type of processors
[
https://issues.apache.org/jira/browse/NIFI-12841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17820519#comment-17820519
]
endzeit commented on NIFI-12841:
Thank you for your feedback [~markap14], that makes perfect sense. Somehow I
must've overlooked those processors. ._.'
What are you thoughts on adding a dedicated {{DeleteFile}} processor?
> Introduce RemoveXYZ type of processors
> --
>
> Key: NIFI-12841
> URL: https://issues.apache.org/jira/browse/NIFI-12841
> Project: Apache NiFi
> Issue Type: Improvement
>Reporter: endzeit
>Priority: Minor
>
> There is the notion of "families" or "types" of processors in the standard
> distribution of NiFi.
> Among others, these are {{ListXYZ}}, {{GetXYZ}}, {{FetchXYZ}}, {{UpdateXYZ}},
> and {{PutXYZ}}.
> The following examples will be based on files on the local filesystem.
> However, the same principle applies to other types of resources, e.g. files
> on a SFTP server.
> The existing {{GetFile}} and {{FetchFile}} processors support the removal of
> the resource from the source after successful transfer into the content of a
> FlowFile.
> However, in some scenarios it might be undesired to remove the resource until
> it has been processed successfully and the transformation result be stored,
> e.g. to a secure network storage.
> This cannot be achieved with a {{GetXYZ}} or {{FetchXYZ}} processor on its
> own.
> As of now, one of the scripting processors or even a full-fledged custom
> processor can be used to achieve this.
> However, these might get relatively involved due to session handling or other
> concerns.
> This issue proposes the introduction of an additional such processor "type",
> namely {{RemoveXYZ}} which removes a resource.
> The base processor should have two properties, namely {{path}} and
> {{filename}}, by default retrieving their values from the respective core
> FlowFile attributes. Implementations may add protocol specific properties,
> e.g. for authentication.
> There should be three outgoing relationships at least:
> - "success" for FlowFiles, where the resource was removed from the source,
> - "not exists" for FlowFiles, where the resource did (no longer) exist on the
> source,
> - "failure" for FlowFiles, where the resource couldn't be removed from the
> source, e.g. due to network errors or missing permissions.
> An initial implementation should provide {{RemoveXYZ}} for one of the
> existing resources types, e.g. File, FTP, SFTP...
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (NIFI-12841) Introduce RemoveXYZ type of processors
[
https://issues.apache.org/jira/browse/NIFI-12841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17820514#comment-17820514
]
Mark Payne commented on NIFI-12841:
---
Hey [~EndzeitBegins] thanks for reaching out about this.
In general, the naming convention used in NiFi for such a thing is DeleteXYZ,
rather than RemoveXYZ. Several of these components exist. For example:
DeleteMongo, DeleteHDFS, DeleteDynamoDB, DeleteSQS, DeleteS3Object,
DeleteGCSObject.
It is important to note that these Processors should not share a base class.
There is no significant code reuse that would be gained by sharing a base
class, but doing so would constraint the extensibility of the Processors. For
example, DeleteS3Object is likely to extend from an AbstractS3Processor, etc.
Typically, the Processor will have both a "success" and a "failure"
relationship. As for a "does not exist" relationship, it depends on the
Processor. Some Processors may provide such a relationship while others do not.
It should be documented how each Processor behaves in such a condition -
whether it's a specific relationship, or the FlowFile goes to failure (because
it failed to delete the file), or the FlowFile goes to success (because the
file no longer exists), etc. It would be good to ensure that we are consistent,
but given that several Delete* Processors already exist, it may not make sense
to start changing the behavior. It would take some investigation there.
It is also important to note that detecting whether or not a given file exists
may also even have significant performance considerations. For something like
an SQS Processor it may be expensive to make the request for every single
message to detect whether or not it exists.
> Introduce RemoveXYZ type of processors
> --
>
> Key: NIFI-12841
> URL: https://issues.apache.org/jira/browse/NIFI-12841
> Project: Apache NiFi
> Issue Type: Improvement
>Reporter: endzeit
>Priority: Minor
>
> There is the notion of "families" or "types" of processors in the standard
> distribution of NiFi.
> Among others, these are {{ListXYZ}}, {{GetXYZ}}, {{FetchXYZ}}, {{UpdateXYZ}},
> and {{PutXYZ}}.
> The following examples will be based on files on the local filesystem.
> However, the same principle applies to other types of resources, e.g. files
> on a SFTP server.
> The existing {{GetFile}} and {{FetchFile}} processors support the removal of
> the resource from the source after successful transfer into the content of a
> FlowFile.
> However, in some scenarios it might be undesired to remove the resource until
> it has been processed successfully and the transformation result be stored,
> e.g. to a secure network storage.
> This cannot be achieved with a {{GetXYZ}} or {{FetchXYZ}} processor on its
> own.
> As of now, one of the scripting processors or even a full-fledged custom
> processor can be used to achieve this.
> However, these might get relatively involved due to session handling or other
> concerns.
> This issue proposes the introduction of an additional such processor "type",
> namely {{RemoveXYZ}} which removes a resource.
> The base processor should have two properties, namely {{path}} and
> {{filename}}, by default retrieving their values from the respective core
> FlowFile attributes. Implementations may add protocol specific properties,
> e.g. for authentication.
> There should be three outgoing relationships at least:
> - "success" for FlowFiles, where the resource was removed from the source,
> - "not exists" for FlowFiles, where the resource did (no longer) exist on the
> source,
> - "failure" for FlowFiles, where the resource couldn't be removed from the
> source, e.g. due to network errors or missing permissions.
> An initial implementation should provide {{RemoveXYZ}} for one of the
> existing resources types, e.g. File, FTP, SFTP...
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [PR] NIFI-12614: Create record reader service for Protobuf messages [nifi]
mark-bathori commented on code in PR #8250:
URL: https://github.com/apache/nifi/pull/8250#discussion_r1501870173
##
nifi-nar-bundles/nifi-protobuf-bundle/nifi-protobuf-services/src/main/java/org/apache/nifi/services/protobuf/converter/ProtobufDataConverter.java:
##
@@ -0,0 +1,380 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.services.protobuf.converter;
+
+import com.google.protobuf.ByteString;
+import com.google.protobuf.CodedInputStream;
+import com.google.protobuf.InvalidProtocolBufferException;
+import com.google.protobuf.UnknownFieldSet;
+import com.squareup.wire.schema.EnumType;
+import com.squareup.wire.schema.Field;
+import com.squareup.wire.schema.MessageType;
+import com.squareup.wire.schema.OneOf;
+import com.squareup.wire.schema.ProtoType;
+import com.squareup.wire.schema.Schema;
+import org.apache.nifi.serialization.record.DataType;
+import org.apache.nifi.serialization.record.MapRecord;
+import org.apache.nifi.serialization.record.RecordField;
+import org.apache.nifi.serialization.record.RecordSchema;
+import org.apache.nifi.serialization.record.type.RecordDataType;
+import org.apache.nifi.serialization.record.util.DataTypeUtils;
+import org.apache.nifi.services.protobuf.FieldType;
+import org.apache.nifi.services.protobuf.schema.ProtoSchemaParser;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.math.BigInteger;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Optional;
+import java.util.function.Function;
+
+import static com.google.protobuf.CodedInputStream.decodeZigZag32;
+import static com.google.protobuf.TextFormat.unsignedToString;
+
+/**
+ * The class is responsible for creating Record by mapping the provided proto
schema fields with the list of Unknown fields parsed from encoded proto data.
+ */
+public class ProtobufDataConverter {
+
+public static final String MAP_KEY_FIELD_NAME = "key";
+public static final String MAP_VALUE_FIELD_NAME = "value";
+public static final String ANY_TYPE_URL_FIELD_NAME = "type_url";
+public static final String ANY_VALUE_FIELD_NAME = "value";
+public static final String ANY_MESSAGE_TYPE = "google.protobuf.Any";
+
+private final Schema schema;
+private final String message;
+private final RecordSchema rootRecordSchema;
+private final boolean coerceTypes;
+private final boolean dropUnknownFields;
+
+private boolean containsAnyField = false;
+
+public ProtobufDataConverter(Schema schema, String message, RecordSchema
recordSchema, boolean coerceTypes, boolean dropUnknownFields) {
+this.schema = schema;
+this.message = message;
+this.rootRecordSchema = recordSchema;
+this.coerceTypes = coerceTypes;
+this.dropUnknownFields = dropUnknownFields;
+}
+
+/**
+ * Creates a record from the root message.
+ *
+ * @return created record
+ * @throws IOException failed to read input stream
+ */
+public MapRecord createRecord(InputStream data) throws IOException {
+final MessageType rootType = (MessageType) schema.getType(message);
+Objects.requireNonNull(rootType, String.format("Message with name [%s]
not found in the provided proto files", message));
+
+MapRecord record = createRecord(rootType, ByteString.readFrom(data),
rootRecordSchema);
+if (containsAnyField) {
+record.regenerateSchema();
+}
+
+return record;
+}
+
+/**
+ * Creates a record for the provided message.
+ *
+ * @param messageType message to create a record from
+ * @param data proto message data
+ * @param recordSchema record schema for the created record
+ * @return created record
+ * @throws InvalidProtocolBufferException failed to parse input data
+ */
+private MapRecord createRecord(MessageType messageType, ByteString data,
RecordSchema recordSchema) throws InvalidProtocolBufferException {
+final UnknownFieldSet unknownFieldSet =
UnknownFieldSet.parseFrom(data);
+
+if ((ANY_MESSAGE_TYPE).equals(messageType.getType().toString())) {
+containsAnyF
Re: [PR] NIFI-12614: Create record reader service for Protobuf messages [nifi]
mark-bathori commented on code in PR #8250:
URL: https://github.com/apache/nifi/pull/8250#discussion_r1501869967
##
nifi-nar-bundles/nifi-protobuf-bundle/nifi-protobuf-services/src/main/java/org/apache/nifi/services/protobuf/ProtobufReader.java:
##
@@ -0,0 +1,136 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.services.protobuf;
+
+import com.squareup.wire.schema.CoreLoaderKt;
+import com.squareup.wire.schema.Location;
+import com.squareup.wire.schema.Schema;
+import com.squareup.wire.schema.SchemaLoader;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnEnabled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.context.PropertyContext;
+import org.apache.nifi.controller.ConfigurationContext;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.schema.access.SchemaAccessStrategy;
+import org.apache.nifi.schema.access.SchemaAccessUtils;
+import org.apache.nifi.schema.access.SchemaNotFoundException;
+import org.apache.nifi.schemaregistry.services.SchemaRegistry;
+import org.apache.nifi.serialization.RecordReader;
+import org.apache.nifi.serialization.RecordReaderFactory;
+import org.apache.nifi.serialization.SchemaRegistryService;
+import org.apache.nifi.services.protobuf.schema.ProtoSchemaStrategy;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.file.FileSystems;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+
+@Tags({"protobuf", "record", "reader", "parser"})
+@CapabilityDescription("Parses a Protocol Buffers message from binary format.")
+public class ProtobufReader extends SchemaRegistryService implements
RecordReaderFactory {
+
+private static final String ANY_PROTO = "google/protobuf/any.proto";
+private static final String DURATION_PROTO =
"google/protobuf/duration.proto";
+private static final String EMPTY_PROTO = "google/protobuf/empty.proto";
+private static final String STRUCT_PROTO = "google/protobuf/struct.proto";
+private static final String TIMESTAMP_PROTO =
"google/protobuf/timestamp.proto";
+private static final String WRAPPERS_PROTO =
"google/protobuf/wrappers.proto";
+
+private static final AllowableValue GENERATE_FROM_PROTO_FILE = new
AllowableValue("generate-from-proto-file",
+"Generate from Proto file", "The record schema is generated from
the provided proto file");
+
+private String message;
+private Schema protoSchema;
+
+public static final PropertyDescriptor PROTOBUF_DIRECTORY = new
PropertyDescriptor.Builder()
+.name("Proto Directory")
+.displayName("Proto Directory")
+.description("Directory containing Protocol Buffers message
definition (.proto) file(s).")
+.required(true)
+
.addValidator(StandardValidators.createDirectoryExistsValidator(true, false))
+.expressionLanguageSupported(ExpressionLanguageScope.ENVIRONMENT)
+.build();
+
+public static final PropertyDescriptor MESSAGE_TYPE = new
PropertyDescriptor.Builder()
+.name("Message Type")
+.displayName("Message Type")
+.description("Fully qualified name of the Protocol Buffers message
type including its package (eg. mypackage.MyMessage). " +
+"The .proto files configured in '" +
PROTOBUF_DIRECTORY.getDisplayName() + "' must contain the definition of this
message type.")
+.required(true)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.expressionLanguageSupported(ExpressionLanguageScope.ENVIRONMENT)
+.build();
+
+@Override
+protected List getSupportedPropertyDescriptors() {
+final List properties = new
ArrayList<>(super.getSupportedPropertyDescriptors());
+properties.add(PROTOBUF_DIRECTORY);
+
[jira] [Updated] (NIFI-12841) Introduce RemoveXYZ type of processors
[
https://issues.apache.org/jira/browse/NIFI-12841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
endzeit updated NIFI-12841:
---
Description:
There is the notion of "families" or "types" of processors in the standard
distribution of NiFi.
Among others, these are {{ListXYZ}}, {{GetXYZ}}, {{FetchXYZ}}, {{UpdateXYZ}},
and {{PutXYZ}}.
The following examples will be based on files on the local filesystem. However,
the same principle applies to other types of resources, e.g. files on a SFTP
server.
The existing {{GetFile}} and {{FetchFile}} processors support the removal of
the resource from the source after successful transfer into the content of a
FlowFile.
However, in some scenarios it might be undesired to remove the resource until
it has been processed successfully and the transformation result be stored,
e.g. to a secure network storage.
This cannot be achieved with a {{GetXYZ}} or {{FetchXYZ}} processor on its own.
As of now, one of the scripting processors or even a full-fledged custom
processor can be used to achieve this.
However, these might get relatively involved due to session handling or other
concerns.
This issue proposes the introduction of an additional such processor "type",
namely {{RemoveXYZ}} which removes a resource.
The base processor should have two properties, namely {{path}} and
{{filename}}, by default retrieving their values from the respective core
FlowFile attributes. Implementations may add protocol specific properties, e.g.
for authentication.
There should be three outgoing relationships at least:
- "success" for FlowFiles, where the resource was removed from the source,
- "not exists" for FlowFiles, where the resource did (no longer) exist on the
source,
- "failure" for FlowFiles, where the resource couldn't be removed from the
source, e.g. due to network errors or missing permissions.
An initial implementation should provide {{RemoveXYZ}} for one of the existing
resources types, e.g. File, FTP, SFTP...
was:
There is the notion of "families" or "types" of processors in the standard
distribution of NiFi.
Among others, these are {{ListXYZ}}, {{GetXYZ}}, {{FetchXYZ}}, {{UpdateXYZ}},
and {{PutXYZ}}.
The following examples will be based on files on the local filesystem. However,
the same principle applies to other types of resources, e.g. files on a SFTP
server.
The existing {{GetFile}} and {{FetchFile}} processors support the removal of
the resource from the source after successful transfer into the content of a
FlowFile.
However, in some scenarios it might be undesired to remove the resource until
it has been processed successfully and the transformation result be stored,
e.g. to a secure network storage.
This cannot be achieved with a {{GetXYZ}} or {{FetchXYZ}} processor on its own.
As of now, one of the scripting processors or even a full-fledged custom
processor can be used to achieve this.
This issue proposes the introduction of an additional such processor "type",
namely {{RemoveXYZ}} which removes a resource.
The base processor should have two properties, namely {{path}} and
{{filename}}, by default retrieving their values from the respective core
FlowFile attributes. Implementations may add protocol specific properties, e.g.
for authentication.
There should be three outgoing relationships at least:
- "success" for FlowFiles, where the resource was removed from the source,
- "not exists" for FlowFiles, where the resource did (no longer) exist on the
source,
- "failure" for FlowFiles, where the resource couldn't be removed from the
source, e.g. due to network errors or missing permissions.
An initial implementation should provide {{RemoveXYZ}} for one of the existing
resources types, e.g. File, FTP, SFTP...
> Introduce RemoveXYZ type of processors
> --
>
> Key: NIFI-12841
> URL: https://issues.apache.org/jira/browse/NIFI-12841
> Project: Apache NiFi
> Issue Type: Improvement
>Reporter: endzeit
>Priority: Minor
>
> There is the notion of "families" or "types" of processors in the standard
> distribution of NiFi.
> Among others, these are {{ListXYZ}}, {{GetXYZ}}, {{FetchXYZ}}, {{UpdateXYZ}},
> and {{PutXYZ}}.
> The following examples will be based on files on the local filesystem.
> However, the same principle applies to other types of resources, e.g. files
> on a SFTP server.
> The existing {{GetFile}} and {{FetchFile}} processors support the removal of
> the resource from the source after successful transfer into the content of a
> FlowFile.
> However, in some scenarios it might be undesired to remove the resource until
> it has been processed successfully and the transformation result be stored,
> e.g. to a secure network storage.
> This cannot be achieved with a {{GetXYZ}} or {{FetchXYZ}} processor on its
> own.
> As of now, one of
[jira] [Updated] (NIFI-12841) Introduce RemoveXYZ type of processors
[
https://issues.apache.org/jira/browse/NIFI-12841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
endzeit updated NIFI-12841:
---
Description:
There is the notion of "families" or "types" of processors in the standard
distribution of NiFi.
Among others, these are {{ListXYZ}}, {{GetXYZ}}, {{FetchXYZ}}, {{UpdateXYZ}},
and {{PutXYZ}}.
The following examples will be based on files on the local filesystem. However,
the same principle applies to other types of resources, e.g. files on a SFTP
server.
The existing {{GetFile}} and {{FetchFile}} processors support the removal of
the resource from the source after successful transfer into the content of a
FlowFile.
However, in some scenarios it might be undesired to remove the resource until
it has been processed successfully and the transformation result be stored,
e.g. to a secure network storage.
This cannot be achieved with a {{GetXYZ}} or {{FetchXYZ}} processor on its own.
As of now, one of the scripting processors or even a full-fledged custom
processor can be used to achieve this.
This issue proposes the introduction of an additional such processor "type",
namely {{RemoveXYZ}} which removes a resource.
The base processor should have two properties, namely {{path}} and
{{filename}}, by default retrieving their values from the respective core
FlowFile attributes. Implementations may add protocol specific properties, e.g.
for authentication.
There should be three outgoing relationships at least:
- "success" for FlowFiles, where the resource was removed from the source,
- "not exists" for FlowFiles, where the resource did (no longer) exist on the
source,
- "failure" for FlowFiles, where the resource couldn't be removed from the
source, e.g. due to network errors or missing permissions.
An initial implementation should provide {{RemoveXYZ}} for one of the existing
resources types, e.g. File, FTP, SFTP...
was:
There is the notion of "families" or "types" of processors in the standard
distribution of NiFi.
Among others, these are {{ListXYZ}}, {{GetXYZ}}, {{FetchXYZ}}, {{UpdateXYZ}},
and {{PutXYZ}}.
The following examples will be based on files on the local filesystem. However,
the same principle applies to other types of resources, e.g. files on a SFTP
server.
The existing {{GetFile}} and {{FetchFile}} processors support the removal of
the resource from the source after successful transfer into the content of a
FlowFile.
However, in some scenarios it might be undesired to remove the resource until
it has been processed successfully and the transformation result be stored,
e.g. to a secure network storage.
This cannot be achieved with the {{GetXYZ}} or {{FetchXYZ}} processor on its
own.
As of now, one of the scripting processors or even a full-fledged custom
processor can be used to achieve this.
This issue proposes the introduction of an additional such processor "type",
namely {{RemoveXYZ}} which removes a resource.
The base processor should have two properties, namely {{path}} and
{{filename}}, by default retrieving their values from the respective core
FlowFile attributes. Implementations may add protocol specific properties, e.g.
for authentication.
There should be three outgoing relationships at least:
- "success" for FlowFiles, where the resource was removed from the source,
- "not exists" for FlowFiles, where the resource did (no longer) exist on the
source,
- "failure" for FlowFiles, where the resource couldn't be removed from the
source, e.g. due to network errors or missing permissions.
An initial implementation should provide {{RemoveXYZ}} for one of the existing
resources types, e.g. File, FTP, SFTP...
> Introduce RemoveXYZ type of processors
> --
>
> Key: NIFI-12841
> URL: https://issues.apache.org/jira/browse/NIFI-12841
> Project: Apache NiFi
> Issue Type: Improvement
>Reporter: endzeit
>Priority: Minor
>
> There is the notion of "families" or "types" of processors in the standard
> distribution of NiFi.
> Among others, these are {{ListXYZ}}, {{GetXYZ}}, {{FetchXYZ}}, {{UpdateXYZ}},
> and {{PutXYZ}}.
> The following examples will be based on files on the local filesystem.
> However, the same principle applies to other types of resources, e.g. files
> on a SFTP server.
> The existing {{GetFile}} and {{FetchFile}} processors support the removal of
> the resource from the source after successful transfer into the content of a
> FlowFile.
> However, in some scenarios it might be undesired to remove the resource until
> it has been processed successfully and the transformation result be stored,
> e.g. to a secure network storage.
> This cannot be achieved with a {{GetXYZ}} or {{FetchXYZ}} processor on its
> own.
> As of now, one of the scripting processors or even a full-fledged custom
> processor can be used to
[jira] [Updated] (NIFI-12841) Introduce RemoveXYZ type of processors
[
https://issues.apache.org/jira/browse/NIFI-12841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
endzeit updated NIFI-12841:
---
Description:
There is the notion of "families" or "types" of processors in the standard
distribution of NiFi.
Among others, these are {{ListXYZ}}, {{GetXYZ}}, {{FetchXYZ}}, {{UpdateXYZ}},
and {{PutXYZ}}.
The following examples will be based on files on the local filesystem. However,
the same principle applies to other types of resources, e.g. files on a SFTP
server.
The existing {{GetFile}} and {{FetchFile}} processors support the removal of
the resource from the source after successful transfer into the content of a
FlowFile.
However, in some scenarios it might be undesired to remove the resource until
it has been processed successfully and the transformation result be stored,
e.g. to a secure network storage.
This cannot be achieved with the {{GetXYZ}} or {{FetchXYZ}} processor on its
own.
As of now, one of the scripting processors or even a full-fledged custom
processor can be used to achieve this.
This issue proposes the introduction of an additional such processor "type",
namely {{RemoveXYZ}} which removes a resource.
The base processor should have two properties, namely {{path}} and
{{filename}}, by default retrieving their values from the respective core
FlowFile attributes. Implementations may add protocol specific properties, e.g.
for authentication.
There should be three outgoing relationships at least:
- "success" for FlowFiles, where the resource was removed from the source,
- "not exists" for FlowFiles, where the resource did (no longer) exist on the
source,
- "failure" for FlowFiles, where the resource couldn't be removed from the
source, e.g. due to network errors or missing permissions.
An initial implementation should provide {{RemoveXYZ}} for one of the existing
resources types, e.g. File, FTP, SFTP...
was:
There is the notion of "families" or "types" of processors in the standard
distribution of NiFi.
Among others, these are {{ListXYZ}}, {{GetXYZ}}, {{FetchXYZ}}, {{UpdateXYZ}},
and {{PutXYZ}}.
This issue proposes the introduction of an additional such "type", namely
{{RemoveXYZ}} which removes a resource.
The following examples will be based on files on the local filesystem. However,
the same principle applies to other types of resources, e.g. files on a SFTP
server.
TODO ..
> Introduce RemoveXYZ type of processors
> --
>
> Key: NIFI-12841
> URL: https://issues.apache.org/jira/browse/NIFI-12841
> Project: Apache NiFi
> Issue Type: Improvement
>Reporter: endzeit
>Priority: Minor
>
> There is the notion of "families" or "types" of processors in the standard
> distribution of NiFi.
> Among others, these are {{ListXYZ}}, {{GetXYZ}}, {{FetchXYZ}}, {{UpdateXYZ}},
> and {{PutXYZ}}.
> The following examples will be based on files on the local filesystem.
> However, the same principle applies to other types of resources, e.g. files
> on a SFTP server.
> The existing {{GetFile}} and {{FetchFile}} processors support the removal of
> the resource from the source after successful transfer into the content of a
> FlowFile.
> However, in some scenarios it might be undesired to remove the resource until
> it has been processed successfully and the transformation result be stored,
> e.g. to a secure network storage.
> This cannot be achieved with the {{GetXYZ}} or {{FetchXYZ}} processor on its
> own.
> As of now, one of the scripting processors or even a full-fledged custom
> processor can be used to achieve this.
> This issue proposes the introduction of an additional such processor "type",
> namely {{RemoveXYZ}} which removes a resource.
> The base processor should have two properties, namely {{path}} and
> {{filename}}, by default retrieving their values from the respective core
> FlowFile attributes. Implementations may add protocol specific properties,
> e.g. for authentication.
> There should be three outgoing relationships at least:
> - "success" for FlowFiles, where the resource was removed from the source,
> - "not exists" for FlowFiles, where the resource did (no longer) exist on the
> source,
> - "failure" for FlowFiles, where the resource couldn't be removed from the
> source, e.g. due to network errors or missing permissions.
> An initial implementation should provide {{RemoveXYZ}} for one of the
> existing resources types, e.g. File, FTP, SFTP...
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (NIFI-12841) Introduce RemoveXYZ type of processors
[
https://issues.apache.org/jira/browse/NIFI-12841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
endzeit updated NIFI-12841:
---
Description:
There is the notion of "families" or "types" of processors in the standard
distribution of NiFi.
Among others, these are {{ListXYZ}}, {{GetXYZ}}, {{FetchXYZ}}, {{UpdateXYZ}},
and {{PutXYZ}}.
This issue proposes the introduction of an additional such "type", namely
{{RemoveXYZ}} which removes a resource.
The following examples will be based on files on the local filesystem. However,
the same principle applies to other types of resources, e.g. files on a SFTP
server.
TODO ..
was:
There is the notion of "families" or "types" of processors in the standard
distribution of NiFi.
Among others, these are {{ListXYZ}}, {{GetXYZ}}, {{FetchXYZ}}, {{UpdateXYZ}},
and {{PutXYZ}}.
This issue propose the introduction of an additional such "type", namely
{{RemoveXYZ}}.
TODO ...
> Introduce RemoveXYZ type of processors
> --
>
> Key: NIFI-12841
> URL: https://issues.apache.org/jira/browse/NIFI-12841
> Project: Apache NiFi
> Issue Type: Improvement
>Reporter: endzeit
>Priority: Minor
>
> There is the notion of "families" or "types" of processors in the standard
> distribution of NiFi.
> Among others, these are {{ListXYZ}}, {{GetXYZ}}, {{FetchXYZ}}, {{UpdateXYZ}},
> and {{PutXYZ}}.
> This issue proposes the introduction of an additional such "type", namely
> {{RemoveXYZ}} which removes a resource.
> The following examples will be based on files on the local filesystem.
> However, the same principle applies to other types of resources, e.g. files
> on a SFTP server.
> TODO ..
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (NIFI-12841) Introduce RemoveXYZ type of processors
[
https://issues.apache.org/jira/browse/NIFI-12841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
endzeit updated NIFI-12841:
---
Description:
There is the notion of "families" or "types" of processors in the standard
distribution of NiFi.
Among others, these are {{ListXYZ}}, {{GetXYZ}}, {{FetchXYZ}}, {{UpdateXYZ}},
and {{PutXYZ}}.
This issue propose the introduction of an additional such "type", namely
{{RemoveXYZ}}.
TODO ...
was:There exist several "families" of processor types already, to name a few
\{{ListXYZ}}
> Introduce RemoveXYZ type of processors
> --
>
> Key: NIFI-12841
> URL: https://issues.apache.org/jira/browse/NIFI-12841
> Project: Apache NiFi
> Issue Type: Improvement
>Reporter: endzeit
>Priority: Minor
>
> There is the notion of "families" or "types" of processors in the standard
> distribution of NiFi.
> Among others, these are {{ListXYZ}}, {{GetXYZ}}, {{FetchXYZ}}, {{UpdateXYZ}},
> and {{PutXYZ}}.
> This issue propose the introduction of an additional such "type", namely
> {{RemoveXYZ}}.
> TODO ...
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (NIFI-12841) Introduce RemoveXYZ type of processors
[
https://issues.apache.org/jira/browse/NIFI-12841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
endzeit updated NIFI-12841:
---
Summary: Introduce RemoveXYZ type of processors (was: Introduce RemoveX
type of processors)
> Introduce RemoveXYZ type of processors
> --
>
> Key: NIFI-12841
> URL: https://issues.apache.org/jira/browse/NIFI-12841
> Project: Apache NiFi
> Issue Type: Improvement
>Reporter: endzeit
>Priority: Minor
>
> There exist several "families" of processor types already, to name a few
> \{{ListXYZ}}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (NIFI-12841) Introduce RemoveX type of processors
[
https://issues.apache.org/jira/browse/NIFI-12841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
endzeit updated NIFI-12841:
---
Description: There exist several "families" of processor types already, to
name a few \{{ListXYZ}} (was: TODO)
> Introduce RemoveX type of processors
>
>
> Key: NIFI-12841
> URL: https://issues.apache.org/jira/browse/NIFI-12841
> Project: Apache NiFi
> Issue Type: Improvement
>Reporter: endzeit
>Priority: Minor
>
> There exist several "families" of processor types already, to name a few
> \{{ListXYZ}}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (NIFI-12841) Introduce RemoveX type of processors
endzeit created NIFI-12841: -- Summary: Introduce RemoveX type of processors Key: NIFI-12841 URL: https://issues.apache.org/jira/browse/NIFI-12841 Project: Apache NiFi Issue Type: Improvement Reporter: endzeit TODO -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (NIFI-12498) The Prioritization description in the User Guide is different from the actual source code implementation.
[
https://issues.apache.org/jira/browse/NIFI-12498?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
endzeit reassigned NIFI-12498:
--
Assignee: endzeit
> The Prioritization description in the User Guide is different from the actual
> source code implementation.
> -
>
> Key: NIFI-12498
> URL: https://issues.apache.org/jira/browse/NIFI-12498
> Project: Apache NiFi
> Issue Type: Bug
> Components: Documentation & Website
>Reporter: Doin Cha
>Assignee: endzeit
>Priority: Minor
> Time Spent: 10m
> Remaining Estimate: 0h
>
> In the prioritization explanation of the User Guide, it is stated that
> *OldestFlowFileFirstPrioritizer* is the _"default scheme that is used if no
> prioritizers are selected."_
> _([https://nifi.apache.org/docs/nifi-docs/html/user-guide.html#prioritization)|https://nifi.apache.org/docs/nifi-docs/html/user-guide.html#prioritization]_
>
>
> However, in the actual source code implementation, {color:#ff}*there is
> no automatic default setting when prioritizers are not selected.* {color}
> In such cases, the sorting is done by comparing the *ContentClaim* *of
> FlowFiles.*
> _([https://github.com/apache/nifi/blob/9a5ec83baa1b3593031f0917659a69e7a36bb0be/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/queue/QueuePrioritizer.java#L39-L90])_
>
>
> It looks like the user guide needs to be revised.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (NIFI-12498) The Prioritization description in the User Guide is different from the actual source code implementation.
[
https://issues.apache.org/jira/browse/NIFI-12498?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
endzeit updated NIFI-12498:
---
Affects Version/s: 2.0.0-M2
1.25.0
> The Prioritization description in the User Guide is different from the actual
> source code implementation.
> -
>
> Key: NIFI-12498
> URL: https://issues.apache.org/jira/browse/NIFI-12498
> Project: Apache NiFi
> Issue Type: Bug
> Components: Documentation & Website
>Affects Versions: 1.25.0, 2.0.0-M2
>Reporter: Doin Cha
>Assignee: endzeit
>Priority: Minor
> Time Spent: 10m
> Remaining Estimate: 0h
>
> In the prioritization explanation of the User Guide, it is stated that
> *OldestFlowFileFirstPrioritizer* is the _"default scheme that is used if no
> prioritizers are selected."_
> _([https://nifi.apache.org/docs/nifi-docs/html/user-guide.html#prioritization)|https://nifi.apache.org/docs/nifi-docs/html/user-guide.html#prioritization]_
>
>
> However, in the actual source code implementation, {color:#ff}*there is
> no automatic default setting when prioritizers are not selected.* {color}
> In such cases, the sorting is done by comparing the *ContentClaim* *of
> FlowFiles.*
> _([https://github.com/apache/nifi/blob/9a5ec83baa1b3593031f0917659a69e7a36bb0be/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/queue/QueuePrioritizer.java#L39-L90])_
>
>
> It looks like the user guide needs to be revised.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[PR] NIFI-12498 Adjust docs on default behaviour of prioritizers [nifi]
EndzeitBegins opened a new pull request, #8451: URL: https://github.com/apache/nifi/pull/8451 # Summary As outlined in [NIFI-12498](https://issues.apache.org/jira/browse/NIFI-12498), the documentation falsely claims that `OldestFlowFileFirstPrioritizer` is used whenever no `FlowFilePrioritizer` is specified explicitly. The PR adjusts the documentation to remove the long standing claim. # Tracking Please complete the following tracking steps prior to pull request creation. ### Issue Tracking - [x] [Apache NiFi Jira](https://issues.apache.org/jira/browse/NIFI) issue created ### Pull Request Tracking - [x] Pull Request title starts with Apache NiFi Jira issue number, such as `NIFI-0` - [x] Pull Request commit message starts with Apache NiFi Jira issue number, as such `NIFI-0` ### Pull Request Formatting - [x] Pull Request based on current revision of the `main` branch - [x] Pull Request refers to a feature branch with one commit containing changes # Verification Please indicate the verification steps performed prior to pull request creation. ### Build - [] Build completed using `mvn clean install -P contrib-check` - [] JDK 21 ### Licensing - [x] New dependencies are compatible with the [Apache License 2.0](https://apache.org/licenses/LICENSE-2.0) according to the [License Policy](https://www.apache.org/legal/resolved.html) - [x] New dependencies are documented in applicable `LICENSE` and `NOTICE` files ### Documentation - [x] Documentation formatting appears as expected in rendered files -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (NIFI-12704) Record Path function escapeJson raises NPE when given root "/" as argument
[
https://issues.apache.org/jira/browse/NIFI-12704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
endzeit updated NIFI-12704:
---
Affects Version/s: 2.0.0-M2
> Record Path function escapeJson raises NPE when given root "/" as argument
> --
>
> Key: NIFI-12704
> URL: https://issues.apache.org/jira/browse/NIFI-12704
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
>Affects Versions: 1.24.0, 2.0.0-M2
> Environment: Docker, RedHat 8
>Reporter: Stephen Jeffrey Hindmarch
>Assignee: endzeit
>Priority: Minor
> Time Spent: 10m
> Remaining Estimate: 0h
>
> My use case is I want to create a field called "original" which is the
> escaped string of my whole record. This will preserve the original contents
> of the message before I start trnasforming it.
> My expectation is that I should be able to use an UpdateRecord processor to
> create the field using RecordPath language.
> What actually happens is that when I use
> {code:java}
> escapeJson(/)
> {code}
> as the function the result is that the processor throws a Null Pointer
> Exception (NPE).
> Detail:
> For any input flow file containing JSON records, define an UpdateRecord
> processor with these settings.
> * Replacement Value Strategy = Record Path Value
> * /original = escapeJson(/)
> When the flow file is presented to the processor, the processor generates an
> NPE.
> For example, I present this record.
> {noformat}
> [{"hello":"world","record":{"key":"one","value":"hello","subrecord":{"key":"two","value":"bob"}},"array":[0,1,2]}]{noformat}
> If the escapeJson function is offered an existing field to escape then it
> works as expected. Stack trace as follows.
> {noformat}
> 2024-01-31 12:17:15 2024-01-31 12:17:14,741 ERROR [Timer-Driven Process
> Thread-8] o.a.n.processors.standard.UpdateRecord
> UpdateRecord[id=5f6b498d-018d-1000--323c6b89] Failed to process
> StandardFlowFileRecord[uuid=2d4a3eab-e752-4117-93d3-3e5515b6f3f8,claim=StandardContentClaim
> [resourceClaim=StandardResourceClaim[id=1706703050563-1, container=default,
> section=1], offset=114,
> length=114],offset=0,name=2d4a3eab-e752-4117-93d3-3e5515b6f3f8,size=114];
> will route to failure
> 2024-01-31 12:17:15
> org.apache.nifi.record.path.exception.RecordPathException:
> java.lang.NullPointerException
> 2024-01-31 12:17:15 at
> org.apache.nifi.record.path.RecordPath.compile(RecordPath.java:105)
> 2024-01-31 12:17:15 at
> com.github.benmanes.caffeine.cache.LocalLoadingCache.lambda$newMappingFunction$2(LocalLoadingCache.java:145)
> 2024-01-31 12:17:15 at
> com.github.benmanes.caffeine.cache.BoundedLocalCache.lambda$doComputeIfAbsent$14(BoundedLocalCache.java:2406)
> 2024-01-31 12:17:15 at
> java.base/java.util.concurrent.ConcurrentHashMap.compute(Unknown Source)
> 2024-01-31 12:17:15 at
> com.github.benmanes.caffeine.cache.BoundedLocalCache.doComputeIfAbsent(BoundedLocalCache.java:2404)
> 2024-01-31 12:17:15 at
> com.github.benmanes.caffeine.cache.BoundedLocalCache.computeIfAbsent(BoundedLocalCache.java:2387)
> 2024-01-31 12:17:15 at
> com.github.benmanes.caffeine.cache.LocalCache.computeIfAbsent(LocalCache.java:108)
> 2024-01-31 12:17:15 at
> com.github.benmanes.caffeine.cache.LocalLoadingCache.get(LocalLoadingCache.java:56)
> 2024-01-31 12:17:15 at
> org.apache.nifi.record.path.util.RecordPathCache.getCompiled(RecordPathCache.java:34)
> 2024-01-31 12:17:15 at
> org.apache.nifi.processors.standard.UpdateRecord.process(UpdateRecord.java:166)
> 2024-01-31 12:17:15 at
> org.apache.nifi.processors.standard.AbstractRecordProcessor$1.process(AbstractRecordProcessor.java:147)
> 2024-01-31 12:17:15 at
> org.apache.nifi.controller.repository.StandardProcessSession.write(StandardProcessSession.java:3432)
> 2024-01-31 12:17:15 at
> org.apache.nifi.processors.standard.AbstractRecordProcessor.onTrigger(AbstractRecordProcessor.java:122)
> 2024-01-31 12:17:15 at
> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
> 2024-01-31 12:17:15 at
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1361)
> 2024-01-31 12:17:15 at
> org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:247)
> 2024-01-31 12:17:15 at
> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:102)
> 2024-01-31 12:17:15 at
> org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
> 2024-01-31 12:17:15 at
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
> 2024-01-31 12:17:15 at
> java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source)
> 2024-01-31 12:17:15 at
> java.
[PR] NIFI-12704 Support record root reference in function arguments [nifi]
EndzeitBegins opened a new pull request, #8450: URL: https://github.com/apache/nifi/pull/8450 # Summary Add support for references to the record root in functions. [NIFI-12704](https://issues.apache.org/jira/browse/NIFI-12704) # Tracking Please complete the following tracking steps prior to pull request creation. ### Issue Tracking - [x] [Apache NiFi Jira](https://issues.apache.org/jira/browse/NIFI) issue created ### Pull Request Tracking - [x] Pull Request title starts with Apache NiFi Jira issue number, such as `NIFI-0` - [x] Pull Request commit message starts with Apache NiFi Jira issue number, as such `NIFI-0` ### Pull Request Formatting - [x] Pull Request based on current revision of the `main` branch - [x] Pull Request refers to a feature branch with ~~one commit~~ two commits containing changes - I've split the actual changes and refactoring in the tests up into two separate commits # Verification Please indicate the verification steps performed prior to pull request creation. ### Build - [ ] Build completed using `mvn clean install -P contrib-check` - [ ] JDK 21 ### Licensing - [x] New dependencies are compatible with the [Apache License 2.0](https://apache.org/licenses/LICENSE-2.0) according to the [License Policy](https://www.apache.org/legal/resolved.html) - [x] New dependencies are documented in applicable `LICENSE` and `NOTICE` files ### Documentation - [x] Documentation formatting appears as expected in rendered files -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] NIFI-12648 Support record root reference in function arguments [nifi]
EndzeitBegins closed pull request #8449: NIFI-12648 Support record root reference in function arguments URL: https://github.com/apache/nifi/pull/8449 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Assigned] (NIFI-12704) Record Path function escapeJson raises NPE when given root "/" as argument
[
https://issues.apache.org/jira/browse/NIFI-12704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
endzeit reassigned NIFI-12704:
--
Assignee: endzeit
> Record Path function escapeJson raises NPE when given root "/" as argument
> --
>
> Key: NIFI-12704
> URL: https://issues.apache.org/jira/browse/NIFI-12704
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
>Affects Versions: 1.24.0
> Environment: Docker, RedHat 8
>Reporter: Stephen Jeffrey Hindmarch
>Assignee: endzeit
>Priority: Minor
>
> My use case is I want to create a field called "original" which is the
> escaped string of my whole record. This will preserve the original contents
> of the message before I start trnasforming it.
> My expectation is that I should be able to use an UpdateRecord processor to
> create the field using RecordPath language.
> What actually happens is that when I use
> {code:java}
> escapeJson(/)
> {code}
> as the function the result is that the processor throws a Null Pointer
> Exception (NPE).
> Detail:
> For any input flow file containing JSON records, define an UpdateRecord
> processor with these settings.
> * Replacement Value Strategy = Record Path Value
> * /original = escapeJson(/)
> When the flow file is presented to the processor, the processor generates an
> NPE.
> For example, I present this record.
> {noformat}
> [{"hello":"world","record":{"key":"one","value":"hello","subrecord":{"key":"two","value":"bob"}},"array":[0,1,2]}]{noformat}
> If the escapeJson function is offered an existing field to escape then it
> works as expected. Stack trace as follows.
> {noformat}
> 2024-01-31 12:17:15 2024-01-31 12:17:14,741 ERROR [Timer-Driven Process
> Thread-8] o.a.n.processors.standard.UpdateRecord
> UpdateRecord[id=5f6b498d-018d-1000--323c6b89] Failed to process
> StandardFlowFileRecord[uuid=2d4a3eab-e752-4117-93d3-3e5515b6f3f8,claim=StandardContentClaim
> [resourceClaim=StandardResourceClaim[id=1706703050563-1, container=default,
> section=1], offset=114,
> length=114],offset=0,name=2d4a3eab-e752-4117-93d3-3e5515b6f3f8,size=114];
> will route to failure
> 2024-01-31 12:17:15
> org.apache.nifi.record.path.exception.RecordPathException:
> java.lang.NullPointerException
> 2024-01-31 12:17:15 at
> org.apache.nifi.record.path.RecordPath.compile(RecordPath.java:105)
> 2024-01-31 12:17:15 at
> com.github.benmanes.caffeine.cache.LocalLoadingCache.lambda$newMappingFunction$2(LocalLoadingCache.java:145)
> 2024-01-31 12:17:15 at
> com.github.benmanes.caffeine.cache.BoundedLocalCache.lambda$doComputeIfAbsent$14(BoundedLocalCache.java:2406)
> 2024-01-31 12:17:15 at
> java.base/java.util.concurrent.ConcurrentHashMap.compute(Unknown Source)
> 2024-01-31 12:17:15 at
> com.github.benmanes.caffeine.cache.BoundedLocalCache.doComputeIfAbsent(BoundedLocalCache.java:2404)
> 2024-01-31 12:17:15 at
> com.github.benmanes.caffeine.cache.BoundedLocalCache.computeIfAbsent(BoundedLocalCache.java:2387)
> 2024-01-31 12:17:15 at
> com.github.benmanes.caffeine.cache.LocalCache.computeIfAbsent(LocalCache.java:108)
> 2024-01-31 12:17:15 at
> com.github.benmanes.caffeine.cache.LocalLoadingCache.get(LocalLoadingCache.java:56)
> 2024-01-31 12:17:15 at
> org.apache.nifi.record.path.util.RecordPathCache.getCompiled(RecordPathCache.java:34)
> 2024-01-31 12:17:15 at
> org.apache.nifi.processors.standard.UpdateRecord.process(UpdateRecord.java:166)
> 2024-01-31 12:17:15 at
> org.apache.nifi.processors.standard.AbstractRecordProcessor$1.process(AbstractRecordProcessor.java:147)
> 2024-01-31 12:17:15 at
> org.apache.nifi.controller.repository.StandardProcessSession.write(StandardProcessSession.java:3432)
> 2024-01-31 12:17:15 at
> org.apache.nifi.processors.standard.AbstractRecordProcessor.onTrigger(AbstractRecordProcessor.java:122)
> 2024-01-31 12:17:15 at
> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
> 2024-01-31 12:17:15 at
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1361)
> 2024-01-31 12:17:15 at
> org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:247)
> 2024-01-31 12:17:15 at
> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:102)
> 2024-01-31 12:17:15 at
> org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
> 2024-01-31 12:17:15 at
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
> 2024-01-31 12:17:15 at
> java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source)
> 2024-01-31 12:17:15 at
> java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFut
[PR] NIFI-12648 Support record root reference in function arguments [nifi]
EndzeitBegins opened a new pull request, #8449: URL: https://github.com/apache/nifi/pull/8449 # Summary Add support for references to the record root in functions. [NIFI-12648](https://issues.apache.org/jira/browse/NIFI-12648) # Tracking Please complete the following tracking steps prior to pull request creation. ### Issue Tracking - [x] [Apache NiFi Jira](https://issues.apache.org/jira/browse/NIFI) issue created ### Pull Request Tracking - [x] Pull Request title starts with Apache NiFi Jira issue number, such as `NIFI-0` - [x] Pull Request commit message starts with Apache NiFi Jira issue number, as such `NIFI-0` ### Pull Request Formatting - [x] Pull Request based on current revision of the `main` branch - [x] Pull Request refers to a feature branch with ~~one commit~~ two commits containing changes - I've split the actual changes and refactoring in the tests up into two separate commits # Verification Please indicate the verification steps performed prior to pull request creation. ### Build - [ ] Build completed using `mvn clean install -P contrib-check` - [ ] JDK 21 ### Licensing - [x] New dependencies are compatible with the [Apache License 2.0](https://apache.org/licenses/LICENSE-2.0) according to the [License Policy](https://www.apache.org/legal/resolved.html) - [x] New dependencies are documented in applicable `LICENSE` and `NOTICE` files ### Documentation - [x] Documentation formatting appears as expected in rendered files -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
