[nifi] branch master updated: NIFI-4735: ParseEVTX only outputs one event per chunk
This is an automated email from the ASF dual-hosted git repository.
mattyb149 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/nifi.git
The following commit(s) were added to refs/heads/master by this push:
new 48a6c81 NIFI-4735: ParseEVTX only outputs one event per chunk
48a6c81 is described below
commit 48a6c81fa261339c645773a3124ce0ee351346c4
Author: Ferenc Szabó
AuthorDate: Wed Mar 20 12:28:08 2019 +0100
NIFI-4735: ParseEVTX only outputs one event per chunk
This change is based on https://github.com/apache/nifi/pull/2489
I have reproduced the issue with some additional test cases and test files
then applied the original fix.
commit message from the original change:
Updated the EVTX FileHeader class to correctly check if there are more
chunks in the file. Previously this would not process the last chunk.
Updated the EVTX ChunkHeader class to correctly check if there are
additional records in the chunk. Previously this would only process the first
record of each chunk. It was using the fileLastRecordNumber where it should
have been using the logLastRecordNumber value.
Updated the EVTX unit tests to have the correct expected number of events
and use the logLastRecordNumber.
refactoring duplicated code and magic numbers
Signed-off-by: Matthew Burgess
This closes #2489
This closes #3379
---
.../nifi/processors/evtx/parser/ChunkHeader.java | 2 +-
.../nifi/processors/evtx/parser/FileHeader.java| 4 +-
.../apache/nifi/processors/evtx/ParseEvtxTest.java | 43 ++---
.../processors/evtx/parser/ChunkHeaderTest.java| 4 +-
.../src/test/resources/1344_events.evtx| Bin 0 -> 1118208 bytes
.../test/resources/3778_events_not_exported.evtx | Bin 0 -> 1052672 bytes
6 files changed, 43 insertions(+), 10 deletions(-)
diff --git
a/nifi-nar-bundles/nifi-evtx-bundle/nifi-evtx-processors/src/main/java/org/apache/nifi/processors/evtx/parser/ChunkHeader.java
b/nifi-nar-bundles/nifi-evtx-bundle/nifi-evtx-processors/src/main/java/org/apache/nifi/processors/evtx/parser/ChunkHeader.java
index 7f01adf..bb4e4d7 100644
---
a/nifi-nar-bundles/nifi-evtx-bundle/nifi-evtx-processors/src/main/java/org/apache/nifi/processors/evtx/parser/ChunkHeader.java
+++
b/nifi-nar-bundles/nifi-evtx-bundle/nifi-evtx-processors/src/main/java/org/apache/nifi/processors/evtx/parser/ChunkHeader.java
@@ -158,7 +158,7 @@ public class ChunkHeader extends Block {
}
public boolean hasNext() {
-return fileLastRecordNumber.compareTo(recordNumber) > 0;
+return logLastRecordNumber.compareTo(recordNumber) > 0;
}
public String getString(int offset) {
diff --git
a/nifi-nar-bundles/nifi-evtx-bundle/nifi-evtx-processors/src/main/java/org/apache/nifi/processors/evtx/parser/FileHeader.java
b/nifi-nar-bundles/nifi-evtx-bundle/nifi-evtx-processors/src/main/java/org/apache/nifi/processors/evtx/parser/FileHeader.java
index 8610fe9..914d518 100644
---
a/nifi-nar-bundles/nifi-evtx-bundle/nifi-evtx-processors/src/main/java/org/apache/nifi/processors/evtx/parser/FileHeader.java
+++
b/nifi-nar-bundles/nifi-evtx-bundle/nifi-evtx-processors/src/main/java/org/apache/nifi/processors/evtx/parser/FileHeader.java
@@ -141,10 +141,10 @@ public class FileHeader extends Block {
/**
* Tests whether there are more chunks
- * @return true iff there are chunks left
+ * @return true if there are chunks left
*/
public boolean hasNext() {
-return count < chunkCount;
+return count <= chunkCount;
}
/**
diff --git
a/nifi-nar-bundles/nifi-evtx-bundle/nifi-evtx-processors/src/test/java/org/apache/nifi/processors/evtx/ParseEvtxTest.java
b/nifi-nar-bundles/nifi-evtx-bundle/nifi-evtx-processors/src/test/java/org/apache/nifi/processors/evtx/ParseEvtxTest.java
index 2e5e90d..260869d 100644
---
a/nifi-nar-bundles/nifi-evtx-bundle/nifi-evtx-processors/src/test/java/org/apache/nifi/processors/evtx/ParseEvtxTest.java
+++
b/nifi-nar-bundles/nifi-evtx-bundle/nifi-evtx-processors/src/test/java/org/apache/nifi/processors/evtx/ParseEvtxTest.java
@@ -74,6 +74,7 @@ public class ParseEvtxTest {
public static final String USER_DATA = "UserData";
public static final String EVENT_DATA = "EventData";
public static final Set DATA_TAGS = new
HashSet<>(Arrays.asList(EVENT_DATA, USER_DATA));
+public static final int EXPECTED_SUCCESSFUL_EVENT_COUNT = 1053;
@Mock
FileHeaderFactory fileHeaderFactory;
@@ -366,7 +367,7 @@ public class ParseEvtxTest {
assertEquals(1, failureFlowFiles.size());
validateFlowFiles(failureFlowFiles);
// We expect the same number of records to come out no matter the
granularity
-assertEquals(960, validateFlowFiles(failureFlowFiles));
+assertEquals(EXPECTED_SUCCESSFUL_EVENT_COUNT,
validateFlowFiles(failureFlowFiles));
// Whole
[nifi] branch master updated: NIFI-6027
This is an automated email from the ASF dual-hosted git repository.
kdoran pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/nifi.git
The following commit(s) were added to refs/heads/master by this push:
new d35d15c NIFI-6027
d35d15c is described below
commit d35d15cdda32c2aedd5bfd4124662f522461c159
Author: Matt Gilman
AuthorDate: Mon Mar 18 13:01:15 2019 -0400
NIFI-6027
- Allowing user or group existence enforcement to be parameterized.
- Fixing error handling when loading user groups which may have resulted in
stack trace leaking.
This closes #3377.
Signed-off-by: Kevin Doran
---
.../apache/nifi/web/StandardNiFiServiceFacade.java | 70 +-
.../web/security/NiFiAuthenticationFilter.java | 19 --
2 files changed, 58 insertions(+), 31 deletions(-)
diff --git
a/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/StandardNiFiServiceFacade.java
b/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/StandardNiFiServiceFacade.java
index a960dbf..fbce19b 100644
---
a/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/StandardNiFiServiceFacade.java
+++
b/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/StandardNiFiServiceFacade.java
@@ -39,8 +39,8 @@ import org.apache.nifi.authorization.User;
import org.apache.nifi.authorization.UserContextKeys;
import org.apache.nifi.authorization.resource.Authorizable;
import
org.apache.nifi.authorization.resource.EnforcePolicyPermissionsThroughBaseResource;
-import org.apache.nifi.authorization.resource.ResourceFactory;
import org.apache.nifi.authorization.resource.OperationAuthorizable;
+import org.apache.nifi.authorization.resource.ResourceFactory;
import org.apache.nifi.authorization.user.NiFiUser;
import org.apache.nifi.authorization.user.NiFiUserUtils;
import org.apache.nifi.bundle.BundleCoordinate;
@@ -48,10 +48,10 @@ import
org.apache.nifi.cluster.coordination.ClusterCoordinator;
import org.apache.nifi.cluster.coordination.heartbeat.HeartbeatMonitor;
import org.apache.nifi.cluster.coordination.heartbeat.NodeHeartbeat;
import org.apache.nifi.cluster.coordination.node.ClusterRoles;
-import org.apache.nifi.cluster.coordination.node.OffloadCode;
import org.apache.nifi.cluster.coordination.node.DisconnectionCode;
import org.apache.nifi.cluster.coordination.node.NodeConnectionState;
import org.apache.nifi.cluster.coordination.node.NodeConnectionStatus;
+import org.apache.nifi.cluster.coordination.node.OffloadCode;
import org.apache.nifi.cluster.event.NodeEvent;
import org.apache.nifi.cluster.manager.exception.IllegalNodeDeletionException;
import org.apache.nifi.cluster.manager.exception.UnknownNodeException;
@@ -599,8 +599,8 @@ public class StandardNiFiServiceFacade implements
NiFiServiceFacade {
authorizable,
() -> accessPolicyDAO.updateAccessPolicy(accessPolicyDTO),
accessPolicy -> {
-final Set users =
accessPolicy.getUsers().stream().map(mapUserIdToTenantEntity()).collect(Collectors.toSet());
-final Set userGroups =
accessPolicy.getGroups().stream().map(mapUserGroupIdToTenantEntity()).collect(Collectors.toSet());
+final Set users =
accessPolicy.getUsers().stream().map(mapUserIdToTenantEntity(false)).collect(Collectors.toSet());
+final Set userGroups =
accessPolicy.getGroups().stream().map(mapUserGroupIdToTenantEntity(false)).collect(Collectors.toSet());
final ComponentReferenceEntity componentReference =
createComponentReferenceEntity(accessPolicy.getResource());
return dtoFactory.createAccessPolicyDto(accessPolicy,
userGroups, users, componentReference);
});
@@ -618,7 +618,7 @@ public class StandardNiFiServiceFacade implements
NiFiServiceFacade {
usersAuthorizable,
() -> userDAO.updateUser(userDTO),
user -> {
-final Set tenantEntities =
groups.stream().map(g ->
g.getIdentifier()).map(mapUserGroupIdToTenantEntity()).collect(Collectors.toSet());
+final Set tenantEntities =
groups.stream().map(g ->
g.getIdentifier()).map(mapUserGroupIdToTenantEntity(false)).collect(Collectors.toSet());
final Set policyEntities =
policies.stream().map(ap ->
createAccessPolicySummaryEntity(ap)).collect(Collectors.toSet());
return dtoFactory.createUserDto(user, tenantEntities,
policyEntities);
});
@@ -635,7 +635,7 @@ public class StandardNiFiServiceFacade implements
NiFiServiceFacade {
userGroupsAuthorizable,
() -> userGroupDAO.updateUserGroup(userGro
[jira] [Assigned] (MINIFI-496) Multiple RPGs to the same NiFi instance will preclude successful transformation
[ https://issues.apache.org/jira/browse/MINIFI-496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aldrin Piri reassigned MINIFI-496: -- Assignee: Aldrin Piri > Multiple RPGs to the same NiFi instance will preclude successful > transformation > --- > > Key: MINIFI-496 > URL: https://issues.apache.org/jira/browse/MINIFI-496 > Project: Apache NiFi MiNiFi > Issue Type: Bug > Components: Processing Configuration >Reporter: Aldrin Piri >Assignee: Aldrin Piri >Priority: Major > Fix For: 0.6.0 > > > If a template references the same NiFi instance within more than one RPGs, > the transformer will be nonplussed over the reuse of a unique id. While we > need certain unique constraints for components, this is a special case that > needs separate handling. Will supply a template that exhibits this issue. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (MINIFI-496) Multiple RPGs to the same NiFi instance will preclude successful transformation
Aldrin Piri created MINIFI-496: -- Summary: Multiple RPGs to the same NiFi instance will preclude successful transformation Key: MINIFI-496 URL: https://issues.apache.org/jira/browse/MINIFI-496 Project: Apache NiFi MiNiFi Issue Type: Bug Components: Processing Configuration Reporter: Aldrin Piri Fix For: 0.6.0 If a template references the same NiFi instance within more than one RPGs, the transformer will be nonplussed over the reuse of a unique id. While we need certain unique constraints for components, this is a special case that needs separate handling. Will supply a template that exhibits this issue. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
