[nifi] branch master updated: NIFI-3988: Add fragment attributes to SplitRecord
This is an automated email from the ASF dual-hosted git repository.
ijokarumawak pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/nifi.git
The following commit(s) were added to refs/heads/master by this push:
new 7548d7c NIFI-3988: Add fragment attributes to SplitRecord
7548d7c is described below
commit 7548d7c85914d9c64efeccce62d61de522170286
Author: Matthew Burgess
AuthorDate: Thu Dec 13 14:02:14 2018 -0500
NIFI-3988: Add fragment attributes to SplitRecord
This closes #3217.
Signed-off-by: Koji Kawamura
---
.../nifi/processors/standard/SplitRecord.java | 25 --
.../nifi/processors/standard/TestSplitRecord.java | 15 -
2 files changed, 37 insertions(+), 3 deletions(-)
diff --git
a/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/SplitRecord.java
b/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/SplitRecord.java
index 2a5679d..d9e7bb5 100644
---
a/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/SplitRecord.java
+++
b/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/SplitRecord.java
@@ -26,6 +26,7 @@ import java.util.HashSet;
import java.util.List;
import java.util.Map;
import java.util.Set;
+import java.util.UUID;
import org.apache.nifi.annotation.behavior.EventDriven;
import org.apache.nifi.annotation.behavior.InputRequirement;
@@ -40,6 +41,7 @@ import org.apache.nifi.components.PropertyDescriptor;
import org.apache.nifi.expression.ExpressionLanguageScope;
import org.apache.nifi.flowfile.FlowFile;
import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.flowfile.attributes.FragmentAttributes;
import org.apache.nifi.processor.AbstractProcessor;
import org.apache.nifi.processor.ProcessContext;
import org.apache.nifi.processor.ProcessSession;
@@ -66,11 +68,20 @@ import org.apache.nifi.serialization.record.RecordSet;
@Tags({"split", "generic", "schema", "json", "csv", "avro", "log", "logs",
"freeform", "text"})
@WritesAttributes({
@WritesAttribute(attribute = "mime.type", description = "Sets the
mime.type attribute to the MIME Type specified by the Record Writer for the
FlowFiles routed to the 'splits' Relationship."),
-@WritesAttribute(attribute = "record.count", description = "The number of
records in the FlowFile. This is added to FlowFiles that are routed to the
'splits' Relationship.")
+@WritesAttribute(attribute = "record.count", description = "The number of
records in the FlowFile. This is added to FlowFiles that are routed to the
'splits' Relationship."),
+@WritesAttribute(attribute = "fragment.identifier", description = "All
split FlowFiles produced from the same parent FlowFile will have the same
randomly generated UUID added for this attribute"),
+@WritesAttribute(attribute = "fragment.index", description = "A one-up
number that indicates the ordering of the split FlowFiles that were created
from a single parent FlowFile"),
+@WritesAttribute(attribute = "fragment.count", description = "The number
of split FlowFiles generated from the parent FlowFile"),
+@WritesAttribute(attribute = "segment.original.filename ", description =
"The filename of the parent FlowFile")
})
@CapabilityDescription("Splits up an input FlowFile that is in a
record-oriented data format into multiple smaller FlowFiles")
public class SplitRecord extends AbstractProcessor {
+public static final String FRAGMENT_ID =
FragmentAttributes.FRAGMENT_ID.key();
+public static final String FRAGMENT_INDEX =
FragmentAttributes.FRAGMENT_INDEX.key();
+public static final String FRAGMENT_COUNT =
FragmentAttributes.FRAGMENT_COUNT.key();
+public static final String SEGMENT_ORIGINAL_FILENAME =
FragmentAttributes.SEGMENT_ORIGINAL_FILENAME.key();
+
static final PropertyDescriptor RECORD_READER = new
PropertyDescriptor.Builder()
.name("Record Reader")
.description("Specifies the Controller Service to use for reading
incoming data")
@@ -125,7 +136,7 @@ public class SplitRecord extends AbstractProcessor {
@Override
public void onTrigger(final ProcessContext context, final ProcessSession
session) throws ProcessException {
-FlowFile original = session.get();
+final FlowFile original = session.get();
if (original == null) {
return;
}
@@ -137,6 +148,7 @@ public class SplitRecord extends AbstractProcessor {
final List splits = new ArrayList<>();
final Map originalAttributes =
original.getAttributes();
+final String fragmentId = UUID.randomUUID().toString();
try {
session.read(original, new InputStreamCallback() {
@Override
@@ -148,6 +1
[nifi] branch master updated: NIFI-5748 Improved Proxy Header Support
This is an automated email from the ASF dual-hosted git repository.
kdoran pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/nifi.git
The following commit(s) were added to refs/heads/master by this push:
new cc47a8c NIFI-5748 Improved Proxy Header Support
cc47a8c is described below
commit cc47a8c0e1762b17f3889bf8e36b92cf51a8e7af
Author: Jeff Storck
AuthorDate: Mon Oct 29 13:29:28 2018 -0400
NIFI-5748 Improved Proxy Header Support
- Fixed proxy header support to use X-Forwarded-Host instead of
X-ForwardedServer
- Added support for the context path header used by Traefik when proxying a
service (X-Forwarded-Prefix)
- Added tests to ApplicationResourceTest for X-Forwarded-Context and
X-Forwarded-Prefix
- Updated administration doc to include X-Forwarded-Prefix
- Added NIFI_WEB_PROXY_CONTEXT_PATH env var to dockerhub and dockermaven
start.sh scripts
- Added documentation for NIFI_WEB_PROXY_CONTEXT_PATH to dockerhub README.md
- Updated ApplicationResource to handle a port specified in X-ProxyPort and
X-Forwarded-Port headers
This closes #3129.
Signed-off-by: Kevin Doran
---
.../java/org/apache/nifi/web/util/WebUtils.java| 19 ++--
.../org/apache/nifi/web/util/WebUtilsTest.groovy | 43 ++---
nifi-docker/dockerhub/README.md| 5 +-
nifi-docker/dockerhub/sh/start.sh | 1 +
nifi-docker/dockermaven/sh/start.sh| 1 +
.../src/main/asciidoc/administration-guide.adoc| 6 +-
.../apache/nifi/web/api/ApplicationResource.java | 50 +-
.../nifi/web/api/ApplicationResourceTest.groovy| 101 +++--
.../apache/nifi/web/ContentViewerController.java | 3 +-
9 files changed, 195 insertions(+), 34 deletions(-)
diff --git
a/nifi-commons/nifi-web-utils/src/main/java/org/apache/nifi/web/util/WebUtils.java
b/nifi-commons/nifi-web-utils/src/main/java/org/apache/nifi/web/util/WebUtils.java
index 90a83a9..fbf5c19 100644
---
a/nifi-commons/nifi-web-utils/src/main/java/org/apache/nifi/web/util/WebUtils.java
+++
b/nifi-commons/nifi-web-utils/src/main/java/org/apache/nifi/web/util/WebUtils.java
@@ -21,6 +21,7 @@ import java.util.Arrays;
import java.util.List;
import java.util.concurrent.locks.ReadWriteLock;
import java.util.concurrent.locks.ReentrantReadWriteLock;
+import java.util.stream.Stream;
import javax.net.ssl.SSLContext;
import javax.servlet.ServletRequest;
import javax.servlet.http.HttpServletRequest;
@@ -45,6 +46,7 @@ public final class WebUtils {
private static final String PROXY_CONTEXT_PATH_HTTP_HEADER =
"X-ProxyContextPath";
private static final String FORWARDED_CONTEXT_HTTP_HEADER =
"X-Forwarded-Context";
+private static final String FORWARDED_PREFIX_HTTP_HEADER =
"X-Forwarded-Prefix";
private WebUtils() {
}
@@ -199,7 +201,8 @@ public final class WebUtils {
}
/**
- * Determines the context path if populated in {@code X-ProxyContextPath}
or {@code X-ForwardContext} headers. If not populated, returns an empty string.
+ * Determines the context path if populated in {@code X-ProxyContextPath},
{@code X-ForwardContext},
+ * or {@code X-Forwarded-Prefix} headers. If not populated, returns an
empty string.
*
* @param request the HTTP request
* @return the provided context path or an empty string
@@ -208,18 +211,20 @@ public final class WebUtils {
String contextPath = request.getContextPath();
String proxyContextPath =
request.getHeader(PROXY_CONTEXT_PATH_HTTP_HEADER);
String forwardedContext =
request.getHeader(FORWARDED_CONTEXT_HTTP_HEADER);
+String prefix = request.getHeader(FORWARDED_PREFIX_HTTP_HEADER);
logger.debug("Context path: " + contextPath);
String determinedContextPath = "";
-// If either header is set, log both
-if (anyNotBlank(proxyContextPath, forwardedContext)) {
+// If a context path header is set, log each
+if (anyNotBlank(proxyContextPath, forwardedContext, prefix)) {
logger.debug(String.format("On the request, the following context
paths were parsed" +
-" from headers:\n\t X-ProxyContextPath:
%s\n\tX-Forwarded-Context: %s",
-proxyContextPath, forwardedContext));
+" from headers:\n\t X-ProxyContextPath:
%s\n\tX-Forwarded-Context: %s\n\tX-Forwarded-Prefix: %s",
+proxyContextPath, forwardedContext, prefix));
-// Implementing preferred order here: PCP, FCP
-determinedContextPath = StringUtils.isNotBlank(proxyContextPath) ?
proxyContextPath : forwardedContext;
+// Implementing preferred order here: PCP, FC, FP
+determinedContextPath = Stream.of(proxyContextPath,
forwardedContext, prefix)
+.filter(StringUtils::isNotBlank).findFirst().orElse("");
[nifi] 01/01: Merge pull request #3220 from zenfenan/NIFI-5898
This is an automated email from the ASF dual-hosted git repository. alopresto pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/nifi.git commit 7c9617a0ec56649370b17f9f276a660119c94f05 Merge: 0efddf4 10e29ee Author: Andy LoPresto AuthorDate: Mon Dec 17 11:39:09 2018 -0800 Merge pull request #3220 from zenfenan/NIFI-5898 NIFI-5898: Updated the display name for ACCESS_KEY & SECRET_KEY Signed-off-by: Andy LoPresto .../credentials/provider/factory/CredentialPropertyDescriptors.java | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
[nifi] branch master updated (0efddf4 -> 7c9617a)
This is an automated email from the ASF dual-hosted git repository. alopresto pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/nifi.git. from 0efddf4 NIFI-4579: Fix ValidateRecord type coercing add 10e29ee NIFI-5898: Updated the display name for ACCESS_KEY & SECRET_KEY new 7c9617a Merge pull request #3220 from zenfenan/NIFI-5898 The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../credentials/provider/factory/CredentialPropertyDescriptors.java | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
