MINIFI-115 Bump NiFi versions to 1.0.0 MINIFI-115 Remove provenance reporting bundle to use the NiFi equivalent.
Add dependency on nifi-framework-api and change AuthorityProvider to Authorizer. work in progress removing references to provenance reporting bundle. this should default to the bundle introduced in NiFi Adjust provenance repository to new API. Adjusting ConfigMain to use ComponentDTO. Adjusting handling of remote authorization failures for RPGs and updating status code. Completed all code changes to support libraries. Now issues at runtime with needed dependencies. Provide nifi-framework-api to provide FlowFilePrioritizer interface. Adjusting classloading and adjusting Persistent Prov repo. Include httpclient as it is needed for Site to Site. Creating a configexception to allow removal of ZooKeeper dependency. Excluding lucene-analyzers-common. Removing reliance on LuceneUtil. Including lucene core needed for utility methods. Removing extraneous logging message. Adding site-to-site-reporting nar to assembly. Removing unused imports causing contrib-check to fail. Adding comment and SuppressWarning for unused QuorumPeerConfig Removing remaining references to authorization issues in docs and associated status classes. Revert "MINIFI-115 Remove provenance reporting bundle to use the NiFi equivalent." This reverts commit a2ffa018d4b8c80d35f8cf18532843d63e19a7a9. Reverting back to including a separate minifi-provenance-reporting nar that makes use of the nifi-site-to-site reporting jar. Marking dependencies provided that are provided by lib. Creating custom bundles to reuse libs already provided at the root instead of duplicates with each NAR. Removing duplicated libraries and reusing bootstrap libraries where possible/compatible. Migrating shared libraries out of bootstrap into lib. Adjusting dependencies. Converting NiFiComponentDTO to ComponentDTO as part of 1.0 API change. Adjusting dependency management and assembly inclusions. Removing extraneous provenance reporting files as we are using the standard NiFi reporter. Signed-off-by: Joseph Percivall <jperciv...@apache.org> Project: http://git-wip-us.apache.org/repos/asf/nifi-minifi/repo Commit: http://git-wip-us.apache.org/repos/asf/nifi-minifi/commit/52888507 Tree: http://git-wip-us.apache.org/repos/asf/nifi-minifi/tree/52888507 Diff: http://git-wip-us.apache.org/repos/asf/nifi-minifi/diff/52888507 Branch: refs/heads/master Commit: 52888507f8ace399e6cfd0872a39f7b677cbfb9a Parents: c999f07 Author: Aldrin Piri <ald...@apache.org> Authored: Wed Oct 5 16:47:14 2016 -0400 Committer: Joseph Percivall <jperciv...@apache.org> Committed: Sun Nov 27 12:53:32 2016 -0500 ---------------------------------------------------------------------- minifi-api/pom.xml | 4 + minifi-assembly/pom.xml | 46 +- .../src/main/assembly/dependencies.xml | 26 +- .../status/rpg/RemoteProcessGroupHealth.java | 12 - .../rpg/RemoteProcessGroupStatusBean.java | 15 +- .../status/util/StatusReportPopulator.java | 6 +- .../minifi/commons/status/TestStatusReport.java | 4 +- .../src/main/markdown/System_Admin_Guide.md | 5 +- .../minifi-framework-nar/pom.xml | 60 +- .../minifi-framework-core/pom.xml | 42 ++ .../org/apache/nifi/minifi/MiNiFiServer.java | 56 +- .../nifi/minifi/status/StatusRequestParser.java | 5 - .../server/quorum/QuorumPeerConfig.java | 41 ++ .../minifi/status/TestStatusConfigReporter.java | 36 +- .../minifi-framework/minifi-nar-utils/pom.xml | 7 +- .../org/apache/nifi/nar/ExtensionManager.java | 31 +- .../org/apache/nifi/nar/NarClassLoaders.java | 131 ++-- .../java/org/apache/nifi/nar/NarCloseable.java | 51 +- .../nifi/nar/NarThreadContextClassLoader.java | 39 +- .../org/apache/nifi/nar/NarUnpackerTest.java | 56 +- .../src/main/resources/bin/minifi.sh | 11 +- .../java/org/apache/nifi/minifi/MiNiFi.java | 8 +- .../minifi-framework-bundle/pom.xml | 1 - .../minifi-provenance-reporting-nar/pom.xml | 40 -- .../src/main/resources/META-INF/NOTICE | 15 - .../minifi-provenance-reporting-task/pom.xml | 82 --- .../reporting/ProvenanceReportingTask.java | 462 ------------- .../org.apache.nifi.reporting.ReportingTask | 16 - .../reporting/TestProvenanceReportingTask.java | 186 ------ .../minifi-provenance-reporting-bundle/pom.xml | 46 -- .../MiNiFiPersistentProvenanceRepository.java | 123 ++-- ....apache.nifi.provenance.ProvenanceRepository | 15 + ...estMiNiFiPersistentProvenanceRepository.java | 28 +- .../org/apache/nifi/provenance/TestUtil.java | 15 +- .../minifi-provenance-repository-nar/pom.xml | 2 - .../minifi-ssl-context-service-nar/pom.xml | 61 ++ .../src/main/resources/META-INF/LICENSE | 337 ++++++++++ .../src/main/resources/META-INF/NOTICE | 196 ++++++ minifi-nar-bundles/minifi-standard-nar/pom.xml | 52 +- .../minifi-standard-services-api-nar/pom.xml | 79 +++ .../src/main/resources/META-INF/LICENSE | 337 ++++++++++ .../src/main/resources/META-INF/NOTICE | 196 ++++++ minifi-nar-bundles/pom.xml | 3 +- .../toolkit/configuration/ConfigMain.java | 8 +- pom.xml | 652 ++++++++++++++++++- 45 files changed, 2471 insertions(+), 1173 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/nifi-minifi/blob/52888507/minifi-api/pom.xml ---------------------------------------------------------------------- diff --git a/minifi-api/pom.xml b/minifi-api/pom.xml index 111144b..98a78af 100644 --- a/minifi-api/pom.xml +++ b/minifi-api/pom.xml @@ -30,5 +30,9 @@ limitations under the License. <groupId>org.apache.nifi</groupId> <artifactId>nifi-api</artifactId> </dependency> + <dependency> + <groupId>org.apache.nifi</groupId> + <artifactId>nifi-framework-api</artifactId> + </dependency> </dependencies> </project> \ No newline at end of file http://git-wip-us.apache.org/repos/asf/nifi-minifi/blob/52888507/minifi-assembly/pom.xml ---------------------------------------------------------------------- diff --git a/minifi-assembly/pom.xml b/minifi-assembly/pom.xml index ac5ed31..89ed9e4 100644 --- a/minifi-assembly/pom.xml +++ b/minifi-assembly/pom.xml @@ -113,12 +113,11 @@ limitations under the License. </dependency> <dependency> <groupId>org.apache.nifi.minifi</groupId> - <artifactId>minifi-framework-nar</artifactId> - <type>nar</type> + <artifactId>minifi-framework-core</artifactId> </dependency> <dependency> <groupId>org.apache.nifi.minifi</groupId> - <artifactId>minifi-provenance-reporting-nar</artifactId> + <artifactId>minifi-framework-nar</artifactId> <type>nar</type> </dependency> <dependency> @@ -150,16 +149,24 @@ limitations under the License. </dependency> <dependency> <groupId>org.apache.nifi</groupId> - <artifactId>nifi-framework-core-api</artifactId> + <artifactId>nifi-ssl-context-service-api</artifactId> </dependency> <dependency> <groupId>org.apache.nifi</groupId> - <artifactId>nifi-standard-services-api-nar</artifactId> - <type>nar</type> + <artifactId>nifi-framework-api</artifactId> </dependency> <dependency> <groupId>org.apache.nifi</groupId> - <artifactId>nifi-ssl-context-service-nar</artifactId> + <artifactId>nifi-framework-core-api</artifactId> + </dependency> + <dependency> + <groupId>org.apache.nifi.minifi</groupId> + <artifactId>minifi-standard-services-api-nar</artifactId> + <type>nar</type> + </dependency> + <dependency> + <groupId>org.apache.nifi.minifi</groupId> + <artifactId>minifi-ssl-context-service-nar</artifactId> <type>nar</type> </dependency> <dependency> @@ -177,23 +184,40 @@ limitations under the License. <!-- Provided in NiFi so must include here too --> <dependency> + <groupId>org.bouncycastle</groupId> + <artifactId>bcprov-jdk15on</artifactId> + <scope>compile</scope> + </dependency> + <dependency> <groupId>org.eclipse.jetty</groupId> - <artifactId>jetty-servlet</artifactId> + <artifactId>jetty-util</artifactId> <scope>compile</scope> </dependency> <dependency> + <groupId>org.apache.commons</groupId> + <artifactId>commons-lang3</artifactId> + </dependency> + <dependency> + <groupId>org.apache.httpcomponents</groupId> + <artifactId>httpclient</artifactId> + </dependency> + <dependency> + <groupId>com.google.guava</groupId> + <artifactId>guava</artifactId> + </dependency> + <dependency> <groupId>org.eclipse.jetty</groupId> - <artifactId>jetty-webapp</artifactId> + <artifactId>jetty-servlet</artifactId> <scope>compile</scope> </dependency> <dependency> <groupId>org.eclipse.jetty</groupId> - <artifactId>jetty-servlets</artifactId> + <artifactId>jetty-webapp</artifactId> <scope>compile</scope> </dependency> <dependency> <groupId>org.eclipse.jetty</groupId> - <artifactId>jetty-jsp</artifactId> + <artifactId>jetty-servlets</artifactId> <scope>compile</scope> </dependency> <dependency> http://git-wip-us.apache.org/repos/asf/nifi-minifi/blob/52888507/minifi-assembly/src/main/assembly/dependencies.xml ---------------------------------------------------------------------- diff --git a/minifi-assembly/src/main/assembly/dependencies.xml b/minifi-assembly/src/main/assembly/dependencies.xml index a774e49..d77b098 100644 --- a/minifi-assembly/src/main/assembly/dependencies.xml +++ b/minifi-assembly/src/main/assembly/dependencies.xml @@ -33,24 +33,24 @@ <fileMode>0660</fileMode> <useTransitiveFiltering>true</useTransitiveFiltering> <excludes> - <exclude>minifi-bootstrap</exclude> + <exclude>minifi-bootstrap</exclude> <exclude>minifi-resources</exclude> <!-- Filter items introduced via transitive dependencies that are provided in associated NARs --> + <exclude>zookeeper</exclude> <exclude>spring-aop</exclude> <exclude>spring-context</exclude> - <exclude>spring-security-core</exclude> <exclude>spring-beans</exclude> <exclude>swagger-annotations</exclude> - <exclude>slf4j-log4j12</exclude> <exclude>aspectjweaver</exclude> <exclude>h2</exclude> - <exclude>netty</exclude> <exclude>jaxb-impl</exclude> - <exclude>httpclient</exclude> <exclude>mail</exclude> <exclude>log4j</exclude> + <exclude>lucene-analyzers-common</exclude> <exclude>lucene-queryparser</exclude> <exclude>commons-net</exclude> + <exclude>spring-context</exclude> + <exclude>spring-security-core</exclude> </excludes> </dependencySet> @@ -63,18 +63,10 @@ <fileMode>0660</fileMode> <useTransitiveFiltering>true</useTransitiveFiltering> <includes> - <include>minifi-bootstrap</include> - <include>slf4j-api</include> - <include>logback-classic</include> - <include>nifi-api</include> + <include>minifi-bootstrap</include> + <include>minifi-utils</include> <include>nifi-utils</include> - <include>jetty-server</include> - <include>jetty-util</include> - <include>jetty-http</include> - <include>jetty-io</include> - <include>javax.servlet-api</include> <include>commons-io</include> - <include>okhttp</include> </includes> </dependencySet> @@ -87,7 +79,7 @@ <fileMode>0664</fileMode> <useTransitiveFiltering>true</useTransitiveFiltering> <includes> - <include>minifi-resources</include> + <include>minifi-resources</include> </includes> <unpack>true</unpack> <unpackOptions> @@ -113,7 +105,7 @@ <fileMode>0770</fileMode> <useTransitiveFiltering>true</useTransitiveFiltering> <includes> - <include>minifi-resources</include> + <include>minifi-resources</include> </includes> <unpack>true</unpack> <unpackOptions> http://git-wip-us.apache.org/repos/asf/nifi-minifi/blob/52888507/minifi-commons/minifi-utils/src/main/java/org/apache/nifi/minifi/commons/status/rpg/RemoteProcessGroupHealth.java ---------------------------------------------------------------------- diff --git a/minifi-commons/minifi-utils/src/main/java/org/apache/nifi/minifi/commons/status/rpg/RemoteProcessGroupHealth.java b/minifi-commons/minifi-utils/src/main/java/org/apache/nifi/minifi/commons/status/rpg/RemoteProcessGroupHealth.java index 0994b53..0853e92 100644 --- a/minifi-commons/minifi-utils/src/main/java/org/apache/nifi/minifi/commons/status/rpg/RemoteProcessGroupHealth.java +++ b/minifi-commons/minifi-utils/src/main/java/org/apache/nifi/minifi/commons/status/rpg/RemoteProcessGroupHealth.java @@ -20,7 +20,6 @@ package org.apache.nifi.minifi.commons.status.rpg; public class RemoteProcessGroupHealth implements java.io.Serializable { private String transmissionStatus; private boolean hasBulletins; - private boolean hasAuthorizationIssues; private int activePortCount; private int inactivePortCount; @@ -43,14 +42,6 @@ public class RemoteProcessGroupHealth implements java.io.Serializable { this.hasBulletins = hasBulletins; } - public boolean isHasAuthorizationIssues() { - return hasAuthorizationIssues; - } - - public void setHasAuthorizationIssues(boolean hasAuthorizationIssues) { - this.hasAuthorizationIssues = hasAuthorizationIssues; - } - public int getActivePortCount() { return activePortCount; } @@ -75,7 +66,6 @@ public class RemoteProcessGroupHealth implements java.io.Serializable { RemoteProcessGroupHealth that = (RemoteProcessGroupHealth) o; if (isHasBulletins() != that.isHasBulletins()) return false; - if (isHasAuthorizationIssues() != that.isHasAuthorizationIssues()) return false; if (getActivePortCount() != that.getActivePortCount()) return false; if (getInactivePortCount() != that.getInactivePortCount()) return false; return getTransmissionStatus() != null ? getTransmissionStatus().equals(that.getTransmissionStatus()) : that.getTransmissionStatus() == null; @@ -86,7 +76,6 @@ public class RemoteProcessGroupHealth implements java.io.Serializable { public int hashCode() { int result = getTransmissionStatus() != null ? getTransmissionStatus().hashCode() : 0; result = 31 * result + (isHasBulletins() ? 1 : 0); - result = 31 * result + (isHasAuthorizationIssues() ? 1 : 0); result = 31 * result + getActivePortCount(); result = 31 * result + getInactivePortCount(); return result; @@ -97,7 +86,6 @@ public class RemoteProcessGroupHealth implements java.io.Serializable { return "{" + "transmissionStatus='" + transmissionStatus + '\'' + ", hasBulletins=" + hasBulletins + - ", hasAuthorizationIssues=" + hasAuthorizationIssues + ", activePortCount=" + activePortCount + ", inactivePortCount=" + inactivePortCount + '}'; http://git-wip-us.apache.org/repos/asf/nifi-minifi/blob/52888507/minifi-commons/minifi-utils/src/main/java/org/apache/nifi/minifi/commons/status/rpg/RemoteProcessGroupStatusBean.java ---------------------------------------------------------------------- diff --git a/minifi-commons/minifi-utils/src/main/java/org/apache/nifi/minifi/commons/status/rpg/RemoteProcessGroupStatusBean.java b/minifi-commons/minifi-utils/src/main/java/org/apache/nifi/minifi/commons/status/rpg/RemoteProcessGroupStatusBean.java index b4ee5ce..3acedc0 100644 --- a/minifi-commons/minifi-utils/src/main/java/org/apache/nifi/minifi/commons/status/rpg/RemoteProcessGroupStatusBean.java +++ b/minifi-commons/minifi-utils/src/main/java/org/apache/nifi/minifi/commons/status/rpg/RemoteProcessGroupStatusBean.java @@ -19,13 +19,13 @@ package org.apache.nifi.minifi.commons.status.rpg; import org.apache.nifi.minifi.commons.status.common.BulletinStatus; +import java.io.Serializable; import java.util.List; -public class RemoteProcessGroupStatusBean implements java.io.Serializable { +public class RemoteProcessGroupStatusBean implements Serializable { private String name; private RemoteProcessGroupHealth remoteProcessGroupHealth; private List<BulletinStatus> bulletinList; - private List<String> authorizationIssues; private List<InputPortStatus> inputPortStatusList; private RemoteProcessGroupStats remoteProcessGroupStats; @@ -56,14 +56,6 @@ public class RemoteProcessGroupStatusBean implements java.io.Serializable { this.bulletinList = bulletinList; } - public List<String> getAuthorizationIssues() { - return authorizationIssues; - } - - public void setAuthorizationIssues(List<String> authorizationIssues) { - this.authorizationIssues = authorizationIssues; - } - public List<InputPortStatus> getInputPortStatusList() { return inputPortStatusList; } @@ -90,7 +82,6 @@ public class RemoteProcessGroupStatusBean implements java.io.Serializable { if (getName() != null ? !getName().equals(that.getName()) : that.getName() != null) return false; if (getRemoteProcessGroupHealth() != null ? !getRemoteProcessGroupHealth().equals(that.getRemoteProcessGroupHealth()) : that.getRemoteProcessGroupHealth() != null) return false; if (getBulletinList() != null ? !getBulletinList().equals(that.getBulletinList()) : that.getBulletinList() != null) return false; - if (getAuthorizationIssues() != null ? !getAuthorizationIssues().equals(that.getAuthorizationIssues()) : that.getAuthorizationIssues() != null) return false; if (getInputPortStatusList() != null ? !getInputPortStatusList().equals(that.getInputPortStatusList()) : that.getInputPortStatusList() != null) return false; return getRemoteProcessGroupStats() != null ? getRemoteProcessGroupStats().equals(that.getRemoteProcessGroupStats()) : that.getRemoteProcessGroupStats() == null; @@ -101,7 +92,6 @@ public class RemoteProcessGroupStatusBean implements java.io.Serializable { int result = getName() != null ? getName().hashCode() : 0; result = 31 * result + (getRemoteProcessGroupHealth() != null ? getRemoteProcessGroupHealth().hashCode() : 0); result = 31 * result + (getBulletinList() != null ? getBulletinList().hashCode() : 0); - result = 31 * result + (getAuthorizationIssues() != null ? getAuthorizationIssues().hashCode() : 0); result = 31 * result + (getInputPortStatusList() != null ? getInputPortStatusList().hashCode() : 0); result = 31 * result + (getRemoteProcessGroupStats() != null ? getRemoteProcessGroupStats().hashCode() : 0); return result; @@ -113,7 +103,6 @@ public class RemoteProcessGroupStatusBean implements java.io.Serializable { "name='" + name + '\'' + ", remoteProcessGroupHealth=" + remoteProcessGroupHealth + ", bulletinList=" + bulletinList + - ", authorizationIssues=" + authorizationIssues + ", inputPortStatusList=" + inputPortStatusList + ", remoteProcessGroupStats=" + remoteProcessGroupStats + '}'; http://git-wip-us.apache.org/repos/asf/nifi-minifi/blob/52888507/minifi-commons/minifi-utils/src/main/java/org/apache/nifi/minifi/commons/status/util/StatusReportPopulator.java ---------------------------------------------------------------------- diff --git a/minifi-commons/minifi-utils/src/main/java/org/apache/nifi/minifi/commons/status/util/StatusReportPopulator.java b/minifi-commons/minifi-utils/src/main/java/org/apache/nifi/minifi/commons/status/util/StatusReportPopulator.java index ff3d2cc..8bdee99 100644 --- a/minifi-commons/minifi-utils/src/main/java/org/apache/nifi/minifi/commons/status/util/StatusReportPopulator.java +++ b/minifi-commons/minifi-utils/src/main/java/org/apache/nifi/minifi/commons/status/util/StatusReportPopulator.java @@ -134,19 +134,15 @@ public class StatusReportPopulator { flowStatusReport.setConnectionStatusList(Collections.singletonList(expectedConnectionStatus)); } - public static void addExpectedRemoteProcessGroupStatus(FlowStatusReport flowStatusReport, boolean addHealth, boolean addAuthorizationIssues, boolean addInputPort, + public static void addExpectedRemoteProcessGroupStatus(FlowStatusReport flowStatusReport, boolean addHealth, boolean addInputPort, boolean addStats, boolean addBulletins, boolean populateBulletins) { RemoteProcessGroupStatusBean expectedRemoteProcessGroupStatus = new RemoteProcessGroupStatusBean(); expectedRemoteProcessGroupStatus.setName("rpg1"); - if (addAuthorizationIssues) { - expectedRemoteProcessGroupStatus.setAuthorizationIssues(Collections.singletonList("auth issue")); - } if (addHealth) { RemoteProcessGroupHealth remoteProcessGroupHealth = new RemoteProcessGroupHealth(); remoteProcessGroupHealth.setTransmissionStatus("Transmitting"); remoteProcessGroupHealth.setHasBulletins(populateBulletins); - remoteProcessGroupHealth.setHasAuthorizationIssues(addAuthorizationIssues); remoteProcessGroupHealth.setActivePortCount(1); remoteProcessGroupHealth.setInactivePortCount(2); expectedRemoteProcessGroupStatus.setRemoteProcessGroupHealth(remoteProcessGroupHealth); http://git-wip-us.apache.org/repos/asf/nifi-minifi/blob/52888507/minifi-commons/minifi-utils/src/test/java/org/apache/nifi/minifi/commons/status/TestStatusReport.java ---------------------------------------------------------------------- diff --git a/minifi-commons/minifi-utils/src/test/java/org/apache/nifi/minifi/commons/status/TestStatusReport.java b/minifi-commons/minifi-utils/src/test/java/org/apache/nifi/minifi/commons/status/TestStatusReport.java index fc697e3..cae5b3a 100644 --- a/minifi-commons/minifi-utils/src/test/java/org/apache/nifi/minifi/commons/status/TestStatusReport.java +++ b/minifi-commons/minifi-utils/src/test/java/org/apache/nifi/minifi/commons/status/TestStatusReport.java @@ -47,7 +47,7 @@ public class TestStatusReport { addReportingTaskStatus(original, true, true, true, true); addConnectionStatus(original, true, true); addProcessorStatus(original, true, true, true, true, true); - addExpectedRemoteProcessGroupStatus(original, true, true, true, true, true, true); + addExpectedRemoteProcessGroupStatus(original, true, true, true, true, true); byte[] byteArrayCopy = serialize(original); FlowStatusReport copy = unSerialize(byteArrayCopy, FlowStatusReport.class); @@ -63,7 +63,7 @@ public class TestStatusReport { addInstanceStatus(original, true, true, true, true); addSystemDiagnosticStatus(original, true, true, true, true, true); addProcessorStatus(original, true, true, true, true, true); - addExpectedRemoteProcessGroupStatus(original, true, true, true, true, true, true); + addExpectedRemoteProcessGroupStatus(original, true, true, true, true, true); byte[] byteArrayCopy = serialize(original); FlowStatusReport copy = unSerialize(byteArrayCopy, FlowStatusReport.class); http://git-wip-us.apache.org/repos/asf/nifi-minifi/blob/52888507/minifi-docs/src/main/markdown/System_Admin_Guide.md ---------------------------------------------------------------------- diff --git a/minifi-docs/src/main/markdown/System_Admin_Guide.md b/minifi-docs/src/main/markdown/System_Admin_Guide.md index eaa8697..361e601 100644 --- a/minifi-docs/src/main/markdown/System_Admin_Guide.md +++ b/minifi-docs/src/main/markdown/System_Admin_Guide.md @@ -229,14 +229,13 @@ Option | Description ------ | ----------- health | The connections's queued bytes and queued FlowFile count. bulletins | A list of all the current bulletins (if there are any). -authorizationIssues | A list of all the current authorization issues (if there are any). inputPorts | A list of every input port for this RPG and their status. Their status includes it's name, whether the target exit and whether it's currently running. stats | The current stats of the RPG. This includes the active threads, sent content size and count. -An example query to get the health, bulletins, authorization issues, input ports and stats of all the RPGS is below. +An example query to get the health, bulletins, input ports and stats of all the RPGS is below. ``` -minifi.sh flowStatus remoteprocessinggroup:all:health,bulletins,authorizationIssues,inputports,stats +minifi.sh flowStatus remoteprocessinggroup:all:health,bulletins,inputports,stats ``` ### Controller Services http://git-wip-us.apache.org/repos/asf/nifi-minifi/blob/52888507/minifi-nar-bundles/minifi-framework-bundle/minifi-framework-nar/pom.xml ---------------------------------------------------------------------- diff --git a/minifi-nar-bundles/minifi-framework-bundle/minifi-framework-nar/pom.xml b/minifi-nar-bundles/minifi-framework-bundle/minifi-framework-nar/pom.xml index b460043..3815952 100644 --- a/minifi-nar-bundles/minifi-framework-bundle/minifi-framework-nar/pom.xml +++ b/minifi-nar-bundles/minifi-framework-bundle/minifi-framework-nar/pom.xml @@ -25,7 +25,6 @@ limitations under the License. <groupId>org.apache.nifi.minifi</groupId> <artifactId>minifi-framework-nar</artifactId> - <version>0.1.0-SNAPSHOT</version> <packaging>nar</packaging> <description>MiNiFi: Framework NAR</description> @@ -34,51 +33,70 @@ limitations under the License. <source.skip>true</source.skip> </properties> <dependencies> - <!-- mark these nifi artifacts as provided since it is included in the lib --> + <dependency> + <groupId>org.apache.nifi.minifi</groupId> + <artifactId>minifi-framework-core</artifactId> + <scope>provided</scope> + </dependency> + <dependency> + <groupId>org.apache.nifi.minifi</groupId> + <artifactId>minifi-api</artifactId> + <scope>provided</scope> + </dependency> + + <!-- NiFi Framework level dependencies --> <dependency> <groupId>org.apache.nifi</groupId> <artifactId>nifi-api</artifactId> - <scope>provided</scope> + <scope>compile</scope> </dependency> <dependency> <groupId>org.apache.nifi</groupId> <artifactId>nifi-runtime</artifactId> - <scope>provided</scope> + <scope>compile</scope> </dependency> <dependency> <groupId>org.apache.nifi</groupId> <artifactId>nifi-nar-utils</artifactId> - <scope>provided</scope> + <scope>compile</scope> </dependency> <dependency> <groupId>org.apache.nifi</groupId> <artifactId>nifi-properties</artifactId> - <scope>provided</scope> + <scope>compile</scope> </dependency> <dependency> - <groupId>org.apache.nifi.minifi</groupId> - <artifactId>minifi-framework-core</artifactId> - <scope>provided</scope> + <groupId>org.apache.nifi</groupId> + <artifactId>nifi-security</artifactId> + <scope>compile</scope> + <version>1.0.0</version> </dependency> <dependency> - <groupId>org.apache.nifi</groupId> - <artifactId>nifi-administration</artifactId> - <scope>provided</scope> + <groupId>org.eclipse.jetty</groupId> + <artifactId>jetty-server</artifactId> </dependency> <dependency> - <groupId>org.apache.nifi.minifi</groupId> - <artifactId>minifi-api</artifactId> - <scope>provided</scope> + <groupId>org.eclipse.jetty</groupId> + <artifactId>jetty-servlet</artifactId> </dependency> <dependency> - <groupId>org.apache.nifi</groupId> - <artifactId>nifi-framework-core</artifactId> - <scope>provided</scope> + <groupId>org.eclipse.jetty</groupId> + <artifactId>jetty-webapp</artifactId> </dependency> <dependency> - <groupId>org.apache.nifi</groupId> - <artifactId>nifi-framework-core-api</artifactId> - <scope>provided</scope> + <groupId>org.eclipse.jetty</groupId> + <artifactId>jetty-annotations</artifactId> + </dependency> + <dependency> + <groupId>org.eclipse.jetty</groupId> + <artifactId>jetty-servlets</artifactId> </dependency> + + <dependency> + <groupId>cglib</groupId> + <artifactId>cglib-nodep</artifactId> + <scope>compile</scope> + </dependency> + </dependencies> </project> \ No newline at end of file http://git-wip-us.apache.org/repos/asf/nifi-minifi/blob/52888507/minifi-nar-bundles/minifi-framework-bundle/minifi-framework/minifi-framework-core/pom.xml ---------------------------------------------------------------------- diff --git a/minifi-nar-bundles/minifi-framework-bundle/minifi-framework/minifi-framework-core/pom.xml b/minifi-nar-bundles/minifi-framework-bundle/minifi-framework/minifi-framework-core/pom.xml index 1900ca7..d89f1c5 100644 --- a/minifi-nar-bundles/minifi-framework-bundle/minifi-framework/minifi-framework-core/pom.xml +++ b/minifi-nar-bundles/minifi-framework-bundle/minifi-framework/minifi-framework-core/pom.xml @@ -27,6 +27,48 @@ limitations under the License. <dependencies> <dependency> + <groupId>org.eclipse.jetty</groupId> + <artifactId>jetty-server</artifactId> + <version>9.3.9.v20160517</version> + + </dependency> + <dependency> + <groupId>org.eclipse.jetty</groupId> + <artifactId>jetty-servlet</artifactId> + <version>9.3.9.v20160517</version> + + </dependency> + <dependency> + <groupId>org.eclipse.jetty</groupId> + <artifactId>jetty-webapp</artifactId> + <version>9.3.9.v20160517</version> + + </dependency> + <dependency> + <groupId>org.eclipse.jetty</groupId> + <artifactId>jetty-servlets</artifactId> + <version>9.3.9.v20160517</version> + + </dependency> + <dependency> + <groupId>org.eclipse.jetty</groupId> + <artifactId>jetty-annotations</artifactId> + <version>9.3.9.v20160517</version> + + </dependency> + <dependency> + <groupId>org.eclipse.jetty</groupId> + <artifactId>apache-jsp</artifactId> + <version>9.3.9.v20160517</version> + + </dependency> + <dependency> + <groupId>org.eclipse.jetty</groupId> + <artifactId>apache-jstl</artifactId> + <version>9.3.9.v20160517</version> + + </dependency> + <dependency> <groupId>org.apache.nifi.minifi</groupId> <artifactId>minifi-nar-utils</artifactId> </dependency> http://git-wip-us.apache.org/repos/asf/nifi-minifi/blob/52888507/minifi-nar-bundles/minifi-framework-bundle/minifi-framework/minifi-framework-core/src/main/java/org/apache/nifi/minifi/MiNiFiServer.java ---------------------------------------------------------------------- diff --git a/minifi-nar-bundles/minifi-framework-bundle/minifi-framework/minifi-framework-core/src/main/java/org/apache/nifi/minifi/MiNiFiServer.java b/minifi-nar-bundles/minifi-framework-bundle/minifi-framework/minifi-framework-core/src/main/java/org/apache/nifi/minifi/MiNiFiServer.java index ba22826..ad82039 100644 --- a/minifi-nar-bundles/minifi-framework-bundle/minifi-framework/minifi-framework-core/src/main/java/org/apache/nifi/minifi/MiNiFiServer.java +++ b/minifi-nar-bundles/minifi-framework-bundle/minifi-framework/minifi-framework-core/src/main/java/org/apache/nifi/minifi/MiNiFiServer.java @@ -17,18 +17,28 @@ package org.apache.nifi.minifi; import org.apache.nifi.admin.service.AuditService; -import org.apache.nifi.admin.service.UserService; import org.apache.nifi.admin.service.impl.StandardAuditService; -import org.apache.nifi.admin.service.impl.StandardUserService; +import org.apache.nifi.authorization.AuthorizationRequest; +import org.apache.nifi.authorization.AuthorizationResult; +import org.apache.nifi.authorization.Authorizer; +import org.apache.nifi.authorization.AuthorizerConfigurationContext; +import org.apache.nifi.authorization.AuthorizerInitializationContext; +import org.apache.nifi.authorization.exception.AuthorizationAccessException; +import org.apache.nifi.authorization.exception.AuthorizerCreationException; +import org.apache.nifi.authorization.exception.AuthorizerDestructionException; import org.apache.nifi.controller.FlowController; import org.apache.nifi.controller.StandardFlowService; import org.apache.nifi.controller.repository.FlowFileEventRepository; import org.apache.nifi.controller.repository.RingBufferEventRepository; import org.apache.nifi.encrypt.StringEncryptor; +import org.apache.nifi.events.VolatileBulletinRepository; import org.apache.nifi.minifi.commons.status.FlowStatusReport; import org.apache.nifi.minifi.status.StatusConfigReporter; import org.apache.nifi.minifi.status.StatusRequestException; +import org.apache.nifi.registry.VariableRegistry; +import org.apache.nifi.reporting.BulletinRepository; import org.apache.nifi.services.FlowService; +import org.apache.nifi.util.FileBasedVariableRegistry; import org.apache.nifi.util.NiFiProperties; import org.slf4j.Logger; import org.slf4j.LoggerFactory; @@ -40,6 +50,7 @@ public class MiNiFiServer { private static final Logger logger = LoggerFactory.getLogger(MiNiFiServer.class); private final NiFiProperties props; private FlowService flowService; + private FlowController flowController; /** * @@ -55,20 +66,47 @@ public class MiNiFiServer { FlowFileEventRepository flowFileEventRepository = new RingBufferEventRepository(5); AuditService auditService = new StandardAuditService(); - UserService userService = new StandardUserService(); - StringEncryptor encryptor = StringEncryptor.createEncryptor(); + Authorizer authorizer = new Authorizer() { + @Override + public AuthorizationResult authorize(AuthorizationRequest request) throws AuthorizationAccessException { + return AuthorizationResult.approved(); + } + + @Override + public void initialize(AuthorizerInitializationContext initializationContext) throws AuthorizerCreationException { + // do nothing + } + + @Override + public void onConfigured(AuthorizerConfigurationContext configurationContext) throws AuthorizerCreationException { + // do nothing + } + + @Override + public void preDestruction() throws AuthorizerDestructionException { + // do nothing + } + }; + StringEncryptor encryptor = StringEncryptor.createEncryptor(props); + VariableRegistry variableRegistry = new FileBasedVariableRegistry(props.getVariableRegistryPropertiesPaths()); + BulletinRepository bulletinRepository = new VolatileBulletinRepository(); FlowController flowController = FlowController.createStandaloneInstance( flowFileEventRepository, props, - userService, + authorizer, auditService, - encryptor); + encryptor, + bulletinRepository, + variableRegistry + ); flowService = StandardFlowService.createStandaloneInstance( flowController, props, - encryptor); + encryptor, + null, // revision manager + authorizer); // start and load the flow flowService.start(); @@ -76,6 +114,8 @@ public class MiNiFiServer { flowController.onFlowInitialized(true); flowController.getGroup(flowController.getRootGroupId()).startProcessing(); + this.flowController = flowController; + logger.info("Flow loaded successfully."); } catch (Exception e) { // ensure the flow service is terminated @@ -107,6 +147,6 @@ public class MiNiFiServer { } public FlowStatusReport getStatusReport(String requestString) throws StatusRequestException { - return StatusConfigReporter.getStatus(flowService.getController(), requestString, logger); + return StatusConfigReporter.getStatus(this.flowController, requestString, logger); } } http://git-wip-us.apache.org/repos/asf/nifi-minifi/blob/52888507/minifi-nar-bundles/minifi-framework-bundle/minifi-framework/minifi-framework-core/src/main/java/org/apache/nifi/minifi/status/StatusRequestParser.java ---------------------------------------------------------------------- diff --git a/minifi-nar-bundles/minifi-framework-bundle/minifi-framework/minifi-framework-core/src/main/java/org/apache/nifi/minifi/status/StatusRequestParser.java b/minifi-nar-bundles/minifi-framework-bundle/minifi-framework/minifi-framework-core/src/main/java/org/apache/nifi/minifi/status/StatusRequestParser.java index 061120c..74d3b1f 100644 --- a/minifi-nar-bundles/minifi-framework-bundle/minifi-framework/minifi-framework-core/src/main/java/org/apache/nifi/minifi/status/StatusRequestParser.java +++ b/minifi-nar-bundles/minifi-framework-bundle/minifi-framework/minifi-framework-core/src/main/java/org/apache/nifi/minifi/status/StatusRequestParser.java @@ -122,7 +122,6 @@ public final class StatusRequestParser { new BulletinQuery.Builder() .sourceIdMatches(inputRemoteProcessGroupStatus.getId()) .build()); - List<String> authorizationIssues = inputRemoteProcessGroupStatus.getAuthorizationIssues(); for (String statusType : statusSplits) { switch (statusType.toLowerCase().trim()) { @@ -130,7 +129,6 @@ public final class StatusRequestParser { RemoteProcessGroupHealth remoteProcessGroupHealth = new RemoteProcessGroupHealth(); remoteProcessGroupHealth.setTransmissionStatus(inputRemoteProcessGroupStatus.getTransmissionStatus().name()); - remoteProcessGroupHealth.setHasAuthorizationIssues(!authorizationIssues.isEmpty()); remoteProcessGroupHealth.setActivePortCount(inputRemoteProcessGroupStatus.getActiveRemotePortCount()); remoteProcessGroupHealth.setInactivePortCount(inputRemoteProcessGroupStatus.getInactiveRemotePortCount()); remoteProcessGroupHealth.setHasBulletins(!bulletinList.isEmpty()); @@ -140,9 +138,6 @@ public final class StatusRequestParser { case "bulletins": remoteProcessGroupStatusBean.setBulletinList(transformBulletins(bulletinList)); break; - case "authorizationissues": - remoteProcessGroupStatusBean.setAuthorizationIssues(authorizationIssues); - break; case "inputports": List<InputPortStatus> inputPortStatusList = new LinkedList<>(); RemoteProcessGroup remoteProcessGroup = flowController.getGroup(rootGroupId).getRemoteProcessGroup(inputRemoteProcessGroupStatus.getId()); http://git-wip-us.apache.org/repos/asf/nifi-minifi/blob/52888507/minifi-nar-bundles/minifi-framework-bundle/minifi-framework/minifi-framework-core/src/main/java/org/apache/zookeeper/server/quorum/QuorumPeerConfig.java ---------------------------------------------------------------------- diff --git a/minifi-nar-bundles/minifi-framework-bundle/minifi-framework/minifi-framework-core/src/main/java/org/apache/zookeeper/server/quorum/QuorumPeerConfig.java b/minifi-nar-bundles/minifi-framework-bundle/minifi-framework/minifi-framework-core/src/main/java/org/apache/zookeeper/server/quorum/QuorumPeerConfig.java new file mode 100644 index 0000000..294b66f --- /dev/null +++ b/minifi-nar-bundles/minifi-framework-bundle/minifi-framework/minifi-framework-core/src/main/java/org/apache/zookeeper/server/quorum/QuorumPeerConfig.java @@ -0,0 +1,41 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.zookeeper.server.quorum; + + +/** + * This class is provided such that the ConfigException can be provided to the FlowController currently reused from NiFi. + * By creating the class within the MiNiFi codebase it is possible to remove the dependency on ZooKeeper. + * + * This totality of this class is eligible for removal when a FlowController explicitly for MiNiFi is introduced. + */ +@SuppressWarnings(value = "unused") +public class QuorumPeerConfig { + + @SuppressWarnings("serial") + public static class ConfigException extends Exception { + public ConfigException(String msg) { + super(msg); + } + public ConfigException(String msg, Exception e) { + super(msg, e); + } + } + +} http://git-wip-us.apache.org/repos/asf/nifi-minifi/blob/52888507/minifi-nar-bundles/minifi-framework-bundle/minifi-framework/minifi-framework-core/src/test/java/org/apache/nifi/minifi/status/TestStatusConfigReporter.java ---------------------------------------------------------------------- diff --git a/minifi-nar-bundles/minifi-framework-bundle/minifi-framework/minifi-framework-core/src/test/java/org/apache/nifi/minifi/status/TestStatusConfigReporter.java b/minifi-nar-bundles/minifi-framework-bundle/minifi-framework/minifi-framework-core/src/test/java/org/apache/nifi/minifi/status/TestStatusConfigReporter.java index 027d685..37483af 100644 --- a/minifi-nar-bundles/minifi-framework-bundle/minifi-framework/minifi-framework-core/src/test/java/org/apache/nifi/minifi/status/TestStatusConfigReporter.java +++ b/minifi-nar-bundles/minifi-framework-bundle/minifi-framework/minifi-framework-core/src/test/java/org/apache/nifi/minifi/status/TestStatusConfigReporter.java @@ -435,7 +435,7 @@ public class TestStatusConfigReporter { FlowStatusReport expected = new FlowStatusReport(); expected.setErrorsGeneratingReport(Collections.EMPTY_LIST); - addExpectedRemoteProcessGroupStatus(expected, true, false, false, false, false, false); + addExpectedRemoteProcessGroupStatus(expected, true, false, false, false, false); assertEquals(expected, actual); } @@ -450,22 +450,7 @@ public class TestStatusConfigReporter { FlowStatusReport expected = new FlowStatusReport(); expected.setErrorsGeneratingReport(Collections.EMPTY_LIST); - addExpectedRemoteProcessGroupStatus(expected, false, false, false, false, true, true); - - assertEquals(expected, actual); - } - - @Test - public void remoteProcessGroupStatusAuthorizationIssues() throws Exception { - populateRemoteProcessGroup(false, true); - - String statusRequest = "remoteProcessGroup:all:authorizationissues"; - FlowStatusReport actual = StatusConfigReporter.getStatus(mockFlowController, statusRequest, LoggerFactory.getLogger(TestStatusConfigReporter.class)); - - FlowStatusReport expected = new FlowStatusReport(); - expected.setErrorsGeneratingReport(Collections.EMPTY_LIST); - - addExpectedRemoteProcessGroupStatus(expected, false, true, false, false, false, false); + addExpectedRemoteProcessGroupStatus(expected, false, false, false, true, true); assertEquals(expected, actual); } @@ -480,7 +465,7 @@ public class TestStatusConfigReporter { FlowStatusReport expected = new FlowStatusReport(); expected.setErrorsGeneratingReport(Collections.EMPTY_LIST); - addExpectedRemoteProcessGroupStatus(expected, false, false, true, false, false, false); + addExpectedRemoteProcessGroupStatus(expected, false, true, false, false, false); assertEquals(expected, actual); } @@ -495,7 +480,7 @@ public class TestStatusConfigReporter { FlowStatusReport expected = new FlowStatusReport(); expected.setErrorsGeneratingReport(Collections.EMPTY_LIST); - addExpectedRemoteProcessGroupStatus(expected, false, false, false, true, false, false); + addExpectedRemoteProcessGroupStatus(expected, false, false, true, false, false); assertEquals(expected, actual); } @@ -505,13 +490,13 @@ public class TestStatusConfigReporter { public void remoteProcessGroupStatusAll() throws Exception { populateRemoteProcessGroup(true, true); - String statusRequest = "remoteProcessGroup:all:health, authorizationissues, bulletins, inputPorts, stats"; + String statusRequest = "remoteProcessGroup:all:health, bulletins, inputPorts, stats"; FlowStatusReport actual = StatusConfigReporter.getStatus(mockFlowController, statusRequest, LoggerFactory.getLogger(TestStatusConfigReporter.class)); FlowStatusReport expected = new FlowStatusReport(); expected.setErrorsGeneratingReport(Collections.EMPTY_LIST); - addExpectedRemoteProcessGroupStatus(expected, true, true, true, true, true, true); + addExpectedRemoteProcessGroupStatus(expected, true, true, true, true, true); assertEquals(expected, actual); } @@ -530,7 +515,7 @@ public class TestStatusConfigReporter { String statusRequest = "controllerServices:bulletins,health; processor:all:health,stats,bulletins; instance:bulletins,health,stats ; systemDiagnostics:garbagecollection, heap, " + "processorstats, contentrepositoryusage, flowfilerepositoryusage; connection:all:health,stats; provenanceReporting:health,bulletins; remoteProcessGroup:all:health, " + - "authorizationissues, bulletins, inputPorts, stats"; + "bulletins, inputPorts, stats"; FlowStatusReport actual = StatusConfigReporter.getStatus(mockFlowController, statusRequest, LoggerFactory.getLogger(TestStatusConfigReporter.class)); @@ -543,7 +528,7 @@ public class TestStatusConfigReporter { addReportingTaskStatus(expected, true, true, true, false); addConnectionStatus(expected, true, true); addProcessorStatus(expected, true, true, true, true, false); - addExpectedRemoteProcessGroupStatus(expected, true, true, true, true, true, false); + addExpectedRemoteProcessGroupStatus(expected, true, true, true, true, false); assertEquals(expected, actual); } @@ -714,11 +699,6 @@ public class TestStatusConfigReporter { RemoteProcessGroupStatus remoteProcessGroupStatus = new RemoteProcessGroupStatus(); addRemoteProcessGroupStatus(remoteProcessGroupStatus); - if (addAuthIssues) { - remoteProcessGroupStatus.setAuthorizationIssues(Collections.singletonList("auth issue")); - } else { - remoteProcessGroupStatus.setAuthorizationIssues(Collections.EMPTY_LIST); - } if (addBulletins) { addBulletins("Bulletin message", remoteProcessGroupStatus.getId()); } http://git-wip-us.apache.org/repos/asf/nifi-minifi/blob/52888507/minifi-nar-bundles/minifi-framework-bundle/minifi-framework/minifi-nar-utils/pom.xml ---------------------------------------------------------------------- diff --git a/minifi-nar-bundles/minifi-framework-bundle/minifi-framework/minifi-nar-utils/pom.xml b/minifi-nar-bundles/minifi-framework-bundle/minifi-framework/minifi-nar-utils/pom.xml index ad9cf8d..50fdeb5 100644 --- a/minifi-nar-bundles/minifi-framework-bundle/minifi-framework/minifi-nar-utils/pom.xml +++ b/minifi-nar-bundles/minifi-framework-bundle/minifi-framework/minifi-nar-utils/pom.xml @@ -15,7 +15,8 @@ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. --> -<project xmlns="http://maven.apache.org/POM/4.0.0"; xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd";> +<project xmlns="http://maven.apache.org/POM/4.0.0"; xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; + xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd";> <parent> <artifactId>minifi-framework</artifactId> <groupId>org.apache.nifi.minifi</groupId> @@ -35,6 +36,10 @@ limitations under the License. <groupId>org.apache.nifi</groupId> <artifactId>nifi-api</artifactId> </dependency> + <dependency> + <groupId>org.apache.nifi</groupId> + <artifactId>nifi-framework-api</artifactId> + </dependency> </dependencies> </project> \ No newline at end of file http://git-wip-us.apache.org/repos/asf/nifi-minifi/blob/52888507/minifi-nar-bundles/minifi-framework-bundle/minifi-framework/minifi-nar-utils/src/main/java/org/apache/nifi/nar/ExtensionManager.java ---------------------------------------------------------------------- diff --git a/minifi-nar-bundles/minifi-framework-bundle/minifi-framework/minifi-nar-utils/src/main/java/org/apache/nifi/nar/ExtensionManager.java b/minifi-nar-bundles/minifi-framework-bundle/minifi-framework/minifi-nar-utils/src/main/java/org/apache/nifi/nar/ExtensionManager.java index db0b35e..787fb3c 100644 --- a/minifi-nar-bundles/minifi-framework-bundle/minifi-framework/minifi-nar-utils/src/main/java/org/apache/nifi/nar/ExtensionManager.java +++ b/minifi-nar-bundles/minifi-framework-bundle/minifi-framework/minifi-nar-utils/src/main/java/org/apache/nifi/nar/ExtensionManager.java @@ -24,7 +24,7 @@ import java.util.ServiceLoader; import java.util.Set; import org.apache.nifi.authentication.LoginIdentityProvider; -import org.apache.nifi.authorization.AuthorityProvider; +import org.apache.nifi.authorization.Authorizer; import org.apache.nifi.controller.ControllerService; import org.apache.nifi.controller.repository.ContentRepository; import org.apache.nifi.controller.repository.FlowFileRepository; @@ -32,7 +32,7 @@ import org.apache.nifi.controller.repository.FlowFileSwapManager; import org.apache.nifi.controller.status.history.ComponentStatusRepository; import org.apache.nifi.flowfile.FlowFilePrioritizer; import org.apache.nifi.processor.Processor; -import org.apache.nifi.provenance.ProvenanceEventRepository; +import org.apache.nifi.provenance.ProvenanceRepository; import org.apache.nifi.reporting.ReportingTask; import org.slf4j.Logger; @@ -54,23 +54,24 @@ public class ExtensionManager { private static final Map<String, ClassLoader> extensionClassloaderLookup = new HashMap<>(); static { - definitionMap.put(Processor.class, new HashSet<Class>()); - definitionMap.put(FlowFilePrioritizer.class, new HashSet<Class>()); - definitionMap.put(ReportingTask.class, new HashSet<Class>()); - definitionMap.put(ControllerService.class, new HashSet<Class>()); - definitionMap.put(AuthorityProvider.class, new HashSet<Class>()); - definitionMap.put(LoginIdentityProvider.class, new HashSet<Class>()); - definitionMap.put(ProvenanceEventRepository.class, new HashSet<Class>()); - definitionMap.put(ComponentStatusRepository.class, new HashSet<Class>()); - definitionMap.put(FlowFileRepository.class, new HashSet<Class>()); - definitionMap.put(FlowFileSwapManager.class, new HashSet<Class>()); - definitionMap.put(ContentRepository.class, new HashSet<Class>()); + definitionMap.put(Processor.class, new HashSet<>()); + definitionMap.put(FlowFilePrioritizer.class, new HashSet<>()); + definitionMap.put(ReportingTask.class, new HashSet<>()); + definitionMap.put(ControllerService.class, new HashSet<>()); + definitionMap.put(Authorizer.class, new HashSet<>()); + definitionMap.put(LoginIdentityProvider.class, new HashSet<>()); + definitionMap.put(ProvenanceRepository.class, new HashSet<>()); + definitionMap.put(ComponentStatusRepository.class, new HashSet<>()); + definitionMap.put(FlowFileRepository.class, new HashSet<>()); + definitionMap.put(FlowFileSwapManager.class, new HashSet<>()); + definitionMap.put(ContentRepository.class, new HashSet<>()); } /** * Loads all FlowFileProcessor, FlowFileComparator, ReportingTask class types that can be found on the bootstrap classloader and by creating classloaders for all NARs found within the classpath. + * @param extensionLoaders the loaders to scan through in search of extensions */ - public static void discoverExtensions() { + public static void discoverExtensions(final Set<ClassLoader> extensionLoaders) { final ClassLoader systemClassLoader = ClassLoader.getSystemClassLoader(); // get the current context class loader @@ -80,7 +81,7 @@ public class ExtensionManager { loadExtensions(systemClassLoader); // consider each nar class loader - for (final ClassLoader ncl : NarClassLoaders.getExtensionClassLoaders()) { + for (final ClassLoader ncl : extensionLoaders) { // Must set the context class loader to the nar classloader itself // so that static initialization techniques that depend on the context class loader will work properly http://git-wip-us.apache.org/repos/asf/nifi-minifi/blob/52888507/minifi-nar-bundles/minifi-framework-bundle/minifi-framework/minifi-nar-utils/src/main/java/org/apache/nifi/nar/NarClassLoaders.java ---------------------------------------------------------------------- diff --git a/minifi-nar-bundles/minifi-framework-bundle/minifi-framework/minifi-nar-utils/src/main/java/org/apache/nifi/nar/NarClassLoaders.java b/minifi-nar-bundles/minifi-framework-bundle/minifi-framework/minifi-nar-utils/src/main/java/org/apache/nifi/nar/NarClassLoaders.java index a4525b0..9219721 100644 --- a/minifi-nar-bundles/minifi-framework-bundle/minifi-framework/minifi-nar-utils/src/main/java/org/apache/nifi/nar/NarClassLoaders.java +++ b/minifi-nar-bundles/minifi-framework-bundle/minifi-framework/minifi-nar-utils/src/main/java/org/apache/nifi/nar/NarClassLoaders.java @@ -28,12 +28,9 @@ import java.util.LinkedHashSet; import java.util.List; import java.util.Map; import java.util.Set; -import java.util.concurrent.atomic.AtomicBoolean; -import java.util.concurrent.atomic.AtomicReference; import java.util.jar.Attributes; import java.util.jar.Manifest; import org.apache.nifi.util.FileUtils; -import org.apache.nifi.util.NiFiProperties; import org.slf4j.Logger; import org.slf4j.LoggerFactory; @@ -44,28 +41,85 @@ public final class NarClassLoaders { public static final String FRAMEWORK_NAR_ID = "minifi-framework-nar"; + private static volatile NarClassLoaders ncl; + private volatile InitContext initContext; private static final Logger logger = LoggerFactory.getLogger(NarClassLoaders.class); - private static final AtomicBoolean initialized = new AtomicBoolean(false); - private static final AtomicReference<Map<String, ClassLoader>> extensionClassLoaders = new AtomicReference<>(); - private static final AtomicReference<ClassLoader> frameworkClassLoader = new AtomicReference<>(); + + private final static class InitContext { + + private final File frameworkWorkingDir; + private final File extensionWorkingDir; + private final ClassLoader frameworkClassLoader; + private final Map<String, ClassLoader> extensionClassLoaders; + + private InitContext( + final File frameworkDir, + final File extensionDir, + final ClassLoader frameworkClassloader, + final Map<String, ClassLoader> extensionClassLoaders) { + this.frameworkWorkingDir = frameworkDir; + this.extensionWorkingDir = extensionDir; + this.frameworkClassLoader = frameworkClassloader; + this.extensionClassLoaders = extensionClassLoaders; + } + } + + private NarClassLoaders() { + } /** - * Loads the extensions class loaders from the specified working directory. - * Loading is only performed during the initial invocation of load. - * Subsequent attempts will be ignored. - * + * @return The singleton instance of the NarClassLoaders + */ + public static NarClassLoaders getInstance() { + NarClassLoaders result = ncl; + if (result == null) { + synchronized (NarClassLoaders.class) { + result = ncl; + if (result == null) { + ncl = result = new NarClassLoaders(); + } + } + } + return result; + } + + /** + * Initializes and loads the NarClassLoaders. This method must be called + * before the rest of the methods to access the classloaders are called and + * it can be safely called any number of times provided the same framework + * and extension working dirs are used. * - * @param properties properties object to initialize with - * @throws java.io.IOException ioe - * @throws java.lang.ClassNotFoundException cfne - * @throws IllegalStateException if the class loaders have already been - * created + * @param frameworkWorkingDir where to find framework artifacts + * @param extensionsWorkingDir where to find extension artifacts + * @throws java.io.IOException if any issue occurs while exploding nar working directories. + * @throws java.lang.ClassNotFoundException if unable to load class definition + * @throws IllegalStateException already initialized with a given pair of + * directories cannot reinitialize or use a different pair of directories. */ - public static void load(final NiFiProperties properties) throws IOException, ClassNotFoundException { - if (initialized.getAndSet(true)) { - throw new IllegalStateException("Extensions class loaders have already been loaded."); + public void init(final File frameworkWorkingDir, final File extensionsWorkingDir) throws IOException, ClassNotFoundException { + if (frameworkWorkingDir == null || extensionsWorkingDir == null) { + throw new NullPointerException("cannot have empty arguments"); } + InitContext ic = initContext; + if (ic == null) { + synchronized (this) { + ic = initContext; + if (ic == null) { + initContext = ic = load(frameworkWorkingDir, extensionsWorkingDir); + } + } + } + boolean matching = initContext.extensionWorkingDir.equals(extensionsWorkingDir) + && initContext.frameworkWorkingDir.equals(frameworkWorkingDir); + if (!matching) { + throw new IllegalStateException("Cannot reinitialize and extension/framework directories cannot change"); + } + } + /** + * Should be called at most once. + */ + private InitContext load(final File frameworkWorkingDir, final File extensionsWorkingDir) throws IOException, ClassNotFoundException { // get the system classloader final ClassLoader systemClassLoader = ClassLoader.getSystemClassLoader(); @@ -76,19 +130,16 @@ public final class NarClassLoaders { final Map<String, ClassLoader> extensionDirectoryClassLoaderLookup = new LinkedHashMap<>(); final Map<String, ClassLoader> narIdClassLoaderLookup = new HashMap<>(); - final File frameworkWorkingDirectory = properties.getFrameworkWorkingDirectory(); - final File extensionsWorkingDirectory = properties.getExtensionsWorkingDirectory(); - // make sure the nar directory is there and accessible - FileUtils.ensureDirectoryExistAndCanAccess(frameworkWorkingDirectory); - FileUtils.ensureDirectoryExistAndCanAccess(extensionsWorkingDirectory); + FileUtils.ensureDirectoryExistAndCanAccess(frameworkWorkingDir); + FileUtils.ensureDirectoryExistAndCanAccess(extensionsWorkingDir); final List<File> narWorkingDirContents = new ArrayList<>(); - final File[] frameworkWorkingDirContents = frameworkWorkingDirectory.listFiles(); + final File[] frameworkWorkingDirContents = frameworkWorkingDir.listFiles(); if (frameworkWorkingDirContents != null) { narWorkingDirContents.addAll(Arrays.asList(frameworkWorkingDirContents)); } - final File[] extensionsWorkingDirContents = extensionsWorkingDirectory.listFiles(); + final File[] extensionsWorkingDirContents = extensionsWorkingDir.listFiles(); if (extensionsWorkingDirContents != null) { narWorkingDirContents.addAll(Arrays.asList(extensionsWorkingDirContents)); } @@ -145,11 +196,7 @@ public final class NarClassLoaders { } } - // set the framework class loader - frameworkClassLoader.set(narIdClassLoaderLookup.get(FRAMEWORK_NAR_ID)); - - // set the extensions class loader map - extensionClassLoaders.set(new LinkedHashMap<>(extensionDirectoryClassLoaderLookup)); + return new InitContext(frameworkWorkingDir, extensionsWorkingDir, narIdClassLoaderLookup.get(FRAMEWORK_NAR_ID), new LinkedHashMap<>(extensionDirectoryClassLoaderLookup)); } /** @@ -199,12 +246,12 @@ public final class NarClassLoaders { * @throws IllegalStateException if the frame class loader has not been * loaded */ - public static ClassLoader getFrameworkClassLoader() { - if (!initialized.get()) { + public ClassLoader getFrameworkClassLoader() { + if (initContext == null) { throw new IllegalStateException("Framework class loader has not been loaded."); } - return frameworkClassLoader.get(); + return initContext.frameworkClassLoader; } /** @@ -213,14 +260,17 @@ public final class NarClassLoaders { * null when no class loader exists for the specified working directory * @throws IllegalStateException if the class loaders have not been loaded */ - public static ClassLoader getExtensionClassLoader(final File extensionWorkingDirectory) { - if (!initialized.get()) { + public ClassLoader getExtensionClassLoader(final File extensionWorkingDirectory) { + if (initContext == null) { throw new IllegalStateException("Extensions class loaders have not been loaded."); } try { - return extensionClassLoaders.get().get(extensionWorkingDirectory.getCanonicalPath()); + return initContext.extensionClassLoaders.get(extensionWorkingDirectory.getCanonicalPath()); } catch (final IOException ioe) { + if(logger.isDebugEnabled()){ + logger.debug("Unable to get extension classloader for working directory '{}'", extensionWorkingDirectory); + } return null; } } @@ -229,12 +279,12 @@ public final class NarClassLoaders { * @return the extension class loaders * @throws IllegalStateException if the class loaders have not been loaded */ - public static Set<ClassLoader> getExtensionClassLoaders() { - if (!initialized.get()) { + public Set<ClassLoader> getExtensionClassLoaders() { + if (initContext == null) { throw new IllegalStateException("Extensions class loaders have not been loaded."); } - return new LinkedHashSet<>(extensionClassLoaders.get().values()); + return new LinkedHashSet<>(initContext.extensionClassLoaders.values()); } private static class NarDetails { @@ -267,7 +317,4 @@ public final class NarClassLoaders { this.narWorkingDirectory = narWorkingDirectory; } } - - private NarClassLoaders() { - } } http://git-wip-us.apache.org/repos/asf/nifi-minifi/blob/52888507/minifi-nar-bundles/minifi-framework-bundle/minifi-framework/minifi-nar-utils/src/main/java/org/apache/nifi/nar/NarCloseable.java ---------------------------------------------------------------------- diff --git a/minifi-nar-bundles/minifi-framework-bundle/minifi-framework/minifi-nar-utils/src/main/java/org/apache/nifi/nar/NarCloseable.java b/minifi-nar-bundles/minifi-framework-bundle/minifi-framework/minifi-nar-utils/src/main/java/org/apache/nifi/nar/NarCloseable.java index c0b43dc..639d032 100644 --- a/minifi-nar-bundles/minifi-framework-bundle/minifi-framework/minifi-nar-utils/src/main/java/org/apache/nifi/nar/NarCloseable.java +++ b/minifi-nar-bundles/minifi-framework-bundle/minifi-framework/minifi-nar-utils/src/main/java/org/apache/nifi/nar/NarCloseable.java @@ -16,6 +16,9 @@ */ package org.apache.nifi.nar; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + import java.io.Closeable; /** @@ -23,10 +26,54 @@ import java.io.Closeable; */ public class NarCloseable implements Closeable { - public static org.apache.nifi.nar.NarCloseable withNarLoader() { + private static final Logger logger = LoggerFactory.getLogger(NarCloseable.class); + + public static NarCloseable withNarLoader() { final ClassLoader current = Thread.currentThread().getContextClassLoader(); Thread.currentThread().setContextClassLoader(NarThreadContextClassLoader.getInstance()); - return new org.apache.nifi.nar.NarCloseable(current); + return new NarCloseable(current); + } + + /** + * Sets the current thread context class loader to the specific appropriate + * Nar class loader for the given configurable component. Restores to the + * previous classloader once complete. If the given class is not assignable + * from ConfigurableComponent then the NarThreadContextClassLoader is used. + * + * @param componentClass componentClass + * @return NarCloseable with current thread context classloader jailed to + * the nar of the component + */ + public static NarCloseable withComponentNarLoader(final Class componentClass) { + final ClassLoader current = Thread.currentThread().getContextClassLoader(); + Thread.currentThread().setContextClassLoader(componentClass.getClassLoader()); + return new NarCloseable(current); + } + + /** + * Creates a Closeable object that can be used to to switch to current class + * loader to the framework class loader and will automatically set the + * ClassLoader back to the previous class loader when closed + * + * @return a NarCloseable + */ + public static NarCloseable withFrameworkNar() { + final ClassLoader frameworkClassLoader; + try { + frameworkClassLoader = NarClassLoaders.getInstance().getFrameworkClassLoader(); + } catch (final Exception e) { + // This should never happen in a running instance, but it will occur in unit tests + logger.error("Unable to access Framework ClassLoader due to " + e + ". Will continue without changing ClassLoaders."); + if (logger.isDebugEnabled()) { + logger.error("", e); + } + + return new NarCloseable(null); + } + + final ClassLoader current = Thread.currentThread().getContextClassLoader(); + Thread.currentThread().setContextClassLoader(frameworkClassLoader); + return new NarCloseable(current); } private final ClassLoader toSet; http://git-wip-us.apache.org/repos/asf/nifi-minifi/blob/52888507/minifi-nar-bundles/minifi-framework-bundle/minifi-framework/minifi-nar-utils/src/main/java/org/apache/nifi/nar/NarThreadContextClassLoader.java ---------------------------------------------------------------------- diff --git a/minifi-nar-bundles/minifi-framework-bundle/minifi-framework/minifi-nar-utils/src/main/java/org/apache/nifi/nar/NarThreadContextClassLoader.java b/minifi-nar-bundles/minifi-framework-bundle/minifi-framework/minifi-nar-utils/src/main/java/org/apache/nifi/nar/NarThreadContextClassLoader.java index c3b17fd..5c70927 100644 --- a/minifi-nar-bundles/minifi-framework-bundle/minifi-framework/minifi-nar-utils/src/main/java/org/apache/nifi/nar/NarThreadContextClassLoader.java +++ b/minifi-nar-bundles/minifi-framework-bundle/minifi-framework/minifi-nar-utils/src/main/java/org/apache/nifi/nar/NarThreadContextClassLoader.java @@ -17,7 +17,7 @@ package org.apache.nifi.nar; import org.apache.nifi.authentication.LoginIdentityProvider; -import org.apache.nifi.authorization.AuthorityProvider; +import org.apache.nifi.authorization.Authorizer; import org.apache.nifi.components.Validator; import org.apache.nifi.controller.ControllerService; import org.apache.nifi.controller.repository.ContentRepository; @@ -29,11 +29,14 @@ import org.apache.nifi.processor.Processor; import org.apache.nifi.processor.io.InputStreamCallback; import org.apache.nifi.processor.io.OutputStreamCallback; import org.apache.nifi.processor.io.StreamCallback; -import org.apache.nifi.provenance.ProvenanceEventRepository; +import org.apache.nifi.provenance.ProvenanceRepository; import org.apache.nifi.reporting.ReportingTask; +import org.apache.nifi.util.NiFiProperties; import java.io.IOException; import java.io.InputStream; +import java.lang.reflect.Constructor; +import java.lang.reflect.InvocationTargetException; import java.net.URL; import java.net.URLClassLoader; import java.util.ArrayList; @@ -58,9 +61,9 @@ public class NarThreadContextClassLoader extends URLClassLoader { narSpecificClasses.add(OutputStreamCallback.class); narSpecificClasses.add(StreamCallback.class); narSpecificClasses.add(ControllerService.class); - narSpecificClasses.add(AuthorityProvider.class); + narSpecificClasses.add(Authorizer.class); narSpecificClasses.add(LoginIdentityProvider.class); - narSpecificClasses.add(ProvenanceEventRepository.class); + narSpecificClasses.add(ProvenanceRepository.class); narSpecificClasses.add(ComponentStatusRepository.class); narSpecificClasses.add(FlowFileRepository.class); narSpecificClasses.add(FlowFileSwapManager.class); @@ -165,7 +168,22 @@ public class NarThreadContextClassLoader extends URLClassLoader { } } - public static <T> T createInstance(final String implementationClassName, final Class<T> typeDefinition) throws InstantiationException, IllegalAccessException, ClassNotFoundException { + /** + * Constructs an instance of the given type using either default no args + * constructor or a constructor which takes a NiFiProperties object + * (preferred). + * + * @param <T> the type to create an instance for + * @param implementationClassName the implementation class name + * @param typeDefinition the type definition + * @param nifiProperties the NiFiProperties instance + * @return constructed instance + * @throws InstantiationException if there is an error instantiating the class + * @throws IllegalAccessException if there is an error accessing the type + * @throws ClassNotFoundException if the class cannot be found + */ + public static <T> T createInstance(final String implementationClassName, final Class<T> typeDefinition, final NiFiProperties nifiProperties) + throws InstantiationException, IllegalAccessException, ClassNotFoundException { final ClassLoader originalClassLoader = Thread.currentThread().getContextClassLoader(); Thread.currentThread().setContextClassLoader(NarThreadContextClassLoader.getInstance()); try { @@ -181,7 +199,16 @@ public class NarThreadContextClassLoader extends URLClassLoader { Thread.currentThread().setContextClassLoader(detectedClassLoaderForType); final Class<?> desiredClass = rawClass.asSubclass(typeDefinition); - return typeDefinition.cast(desiredClass.newInstance()); + if(nifiProperties == null){ + return typeDefinition.cast(desiredClass.newInstance()); + } + Constructor<?> constructor = null; + try { + constructor = desiredClass.getConstructor(NiFiProperties.class); + return typeDefinition.cast(constructor.newInstance(nifiProperties)); + } catch (final NoSuchMethodException | InvocationTargetException ex) { + return typeDefinition.cast(desiredClass.newInstance()); + } } finally { Thread.currentThread().setContextClassLoader(originalClassLoader); } http://git-wip-us.apache.org/repos/asf/nifi-minifi/blob/52888507/minifi-nar-bundles/minifi-framework-bundle/minifi-framework/minifi-nar-utils/src/test/java/org/apache/nifi/nar/NarUnpackerTest.java ---------------------------------------------------------------------- diff --git a/minifi-nar-bundles/minifi-framework-bundle/minifi-framework/minifi-nar-utils/src/test/java/org/apache/nifi/nar/NarUnpackerTest.java b/minifi-nar-bundles/minifi-framework-bundle/minifi-framework/minifi-nar-utils/src/test/java/org/apache/nifi/nar/NarUnpackerTest.java index 882c8c6..4587e5e 100644 --- a/minifi-nar-bundles/minifi-framework-bundle/minifi-framework/minifi-nar-utils/src/test/java/org/apache/nifi/nar/NarUnpackerTest.java +++ b/minifi-nar-bundles/minifi-framework-bundle/minifi-framework/minifi-nar-utils/src/test/java/org/apache/nifi/nar/NarUnpackerTest.java @@ -21,11 +21,8 @@ import org.junit.Assert; import org.junit.BeforeClass; import org.junit.Test; -import java.io.BufferedInputStream; import java.io.File; -import java.io.FileInputStream; import java.io.IOException; -import java.io.InputStream; import java.net.URISyntaxException; import java.nio.file.FileVisitResult; import java.nio.file.Files; @@ -33,7 +30,10 @@ import java.nio.file.Path; import java.nio.file.Paths; import java.nio.file.SimpleFileVisitor; import java.nio.file.attribute.BasicFileAttributes; +import java.util.Collections; +import java.util.HashMap; import java.util.HashSet; +import java.util.Map; import java.util.Set; import static java.nio.file.StandardCopyOption.REPLACE_EXISTING; @@ -81,7 +81,7 @@ public class NarUnpackerTest { @Test public void testUnpackNars() { - NiFiProperties properties = loadSpecifiedProperties("/NarUnpacker/conf/nifi.properties"); + NiFiProperties properties = loadSpecifiedProperties("/NarUnpacker/conf/nifi.properties", Collections.EMPTY_MAP); assertEquals("./target/NarUnpacker/lib/", properties.getProperty("nifi.nar.library.directory")); @@ -112,14 +112,14 @@ public class NarUnpackerTest { @Test public void testUnpackNarsFromEmptyDir() throws IOException { - NiFiProperties properties = loadSpecifiedProperties("/NarUnpacker/conf/nifi.properties"); - final File emptyDir = new File("./target/empty/dir"); emptyDir.delete(); emptyDir.deleteOnExit(); assertTrue(emptyDir.mkdirs()); - properties.setProperty("nifi.nar.library.directory.alt", emptyDir.toString()); + final Map<String, String> others = new HashMap<>(); + others.put("nifi.nar.library.directory.alt", emptyDir.toString()); + NiFiProperties properties = loadSpecifiedProperties("/NarUnpacker/conf/nifi.properties", others); final ExtensionMapping extensionMapping = NarUnpacker.unpackNars(properties); @@ -141,8 +141,9 @@ public class NarUnpackerTest { nonExistantDir.delete(); nonExistantDir.deleteOnExit(); - NiFiProperties properties = loadSpecifiedProperties("/NarUnpacker/conf/nifi.properties"); - properties.setProperty("nifi.nar.library.directory.alt", nonExistantDir.toString()); + final Map<String, String> others = new HashMap<>(); + others.put("nifi.nar.library.directory.alt", nonExistantDir.toString()); + NiFiProperties properties = loadSpecifiedProperties("/NarUnpacker/conf/nifi.properties", others); final ExtensionMapping extensionMapping = NarUnpacker.unpackNars(properties); @@ -165,15 +166,16 @@ public class NarUnpackerTest { nonDir.createNewFile(); nonDir.deleteOnExit(); - NiFiProperties properties = loadSpecifiedProperties("/NarUnpacker/conf/nifi.properties"); - properties.setProperty("nifi.nar.library.directory.alt", nonDir.toString()); + final Map<String, String> others = new HashMap<>(); + others.put("nifi.nar.library.directory.alt", nonDir.toString()); + NiFiProperties properties = loadSpecifiedProperties("/NarUnpacker/conf/nifi.properties", others); final ExtensionMapping extensionMapping = NarUnpacker.unpackNars(properties); assertNull(extensionMapping); } - private NiFiProperties loadSpecifiedProperties(String propertiesFile) { + private NiFiProperties loadSpecifiedProperties(final String propertiesFile, final Map<String, String> others) { String filePath; try { filePath = NarUnpackerTest.class.getResource(propertiesFile).toURI().getPath(); @@ -181,34 +183,6 @@ public class NarUnpackerTest { throw new RuntimeException("Cannot load properties file due to " + ex.getLocalizedMessage(), ex); } - System.setProperty(NiFiProperties.PROPERTIES_FILE_PATH, filePath); - - NiFiProperties properties = NiFiProperties.getInstance(); - - // clear out existing properties - for (String prop : properties.stringPropertyNames()) { - properties.remove(prop); - } - - InputStream inStream = null; - try { - inStream = new BufferedInputStream(new FileInputStream(filePath)); - properties.load(inStream); - } catch (final Exception ex) { - throw new RuntimeException("Cannot load properties file due to " - + ex.getLocalizedMessage(), ex); - } finally { - if (null != inStream) { - try { - inStream.close(); - } catch (final Exception ex) { - /** - * do nothing * - */ - } - } - } - - return properties; + return NiFiProperties.createBasicNiFiProperties(filePath, others); } } \ No newline at end of file http://git-wip-us.apache.org/repos/asf/nifi-minifi/blob/52888507/minifi-nar-bundles/minifi-framework-bundle/minifi-framework/minifi-resources/src/main/resources/bin/minifi.sh ---------------------------------------------------------------------- diff --git a/minifi-nar-bundles/minifi-framework-bundle/minifi-framework/minifi-resources/src/main/resources/bin/minifi.sh b/minifi-nar-bundles/minifi-framework-bundle/minifi-framework/minifi-resources/src/main/resources/bin/minifi.sh index 0482472..45fabc7 100755 --- a/minifi-nar-bundles/minifi-framework-bundle/minifi-framework/minifi-resources/src/main/resources/bin/minifi.sh +++ b/minifi-nar-bundles/minifi-framework-bundle/minifi-framework/minifi-resources/src/main/resources/bin/minifi.sh @@ -162,6 +162,7 @@ install() { run() { BOOTSTRAP_CONF_DIR="${MINIFI_HOME}/conf" BOOTSTRAP_CONF="${BOOTSTRAP_CONF_DIR}/bootstrap.conf"; + MINIFI_LIBS="${MINIFI_HOME}/lib/*" BOOTSTRAP_LIBS="${MINIFI_HOME}/lib/bootstrap/*" run_as=$(grep run.as "${BOOTSTRAP_CONF}" | cut -d'=' -f2) @@ -181,10 +182,11 @@ run() { BOOTSTRAP_CONF=$(cygpath --path --windows "${BOOTSTRAP_CONF}") BOOTSTRAP_CONF_DIR=$(cygpath --path --windows "${BOOTSTRAP_CONF_DIR}") BOOTSTRAP_LIBS=$(cygpath --path --windows "${BOOTSTRAP_LIBS}") - BOOTSTRAP_CLASSPATH="${BOOTSTRAP_CONF_DIR};${BOOTSTRAP_LIBS}" + MINIFI_LIBS=$(cygpath --path --windows "${MINIFI_LIBS}") + BOOTSTRAP_CLASSPATH="${BOOTSTRAP_CONF_DIR};${BOOTSTRAP_LIBS};${MINIFI_LIBS}" if [ -n "${TOOLS_JAR}" ]; then TOOLS_JAR=$(cygpath --path --windows "${TOOLS_JAR}") - BOOTSTRAP_CLASSPATH="${TOOLS_JAR};${BOOTSTRAP_CLASSPATH}" + BOOTSTRAP_CLASSPATH="${TOOLS_JAR};${BOOTSTRAP_CLASSPATH};${MINIFI_LIBS}" fi else if [ -n "${run_as}" ]; then @@ -195,13 +197,14 @@ run() { exit 1 fi fi; - BOOTSTRAP_CLASSPATH="${BOOTSTRAP_CONF_DIR}:${BOOTSTRAP_LIBS}" + BOOTSTRAP_CLASSPATH="${BOOTSTRAP_CONF_DIR}:${BOOTSTRAP_LIBS}:${MINIFI_LIBS}" if [ -n "${TOOLS_JAR}" ]; then - BOOTSTRAP_CLASSPATH="${TOOLS_JAR}:${BOOTSTRAP_CLASSPATH}" + BOOTSTRAP_CLASSPATH="${TOOLS_JAR}:${BOOTSTRAP_CLASSPATH}:${MINIFI_LIBS}" fi fi echo + echo "Bootstrap Classpath: ${BOOTSTRAP_CLASSPATH}" echo "Java home: ${JAVA_HOME}" echo "MiNiFi home: ${MINIFI_HOME}" echo http://git-wip-us.apache.org/repos/asf/nifi-minifi/blob/52888507/minifi-nar-bundles/minifi-framework-bundle/minifi-framework/minifi-runtime/src/main/java/org/apache/nifi/minifi/MiNiFi.java ---------------------------------------------------------------------- diff --git a/minifi-nar-bundles/minifi-framework-bundle/minifi-framework/minifi-runtime/src/main/java/org/apache/nifi/minifi/MiNiFi.java b/minifi-nar-bundles/minifi-framework-bundle/minifi-framework/minifi-runtime/src/main/java/org/apache/nifi/minifi/MiNiFi.java index b2b025b..573b139 100644 --- a/minifi-nar-bundles/minifi-framework-bundle/minifi-framework/minifi-runtime/src/main/java/org/apache/nifi/minifi/MiNiFi.java +++ b/minifi-nar-bundles/minifi-framework-bundle/minifi-framework/minifi-runtime/src/main/java/org/apache/nifi/minifi/MiNiFi.java @@ -108,16 +108,16 @@ public class MiNiFi { NarUnpacker.unpackNars(properties); // load the extensions classloaders - NarClassLoaders.load(properties); + NarClassLoaders.getInstance().init(properties.getFrameworkWorkingDirectory(), properties.getExtensionsWorkingDirectory()); // load the framework classloader - final ClassLoader frameworkClassLoader = NarClassLoaders.getFrameworkClassLoader(); + final ClassLoader frameworkClassLoader = NarClassLoaders.getInstance().getFrameworkClassLoader(); if (frameworkClassLoader == null) { throw new IllegalStateException("Unable to find the framework NAR ClassLoader."); } // discover the extensions - ExtensionManager.discoverExtensions(); + ExtensionManager.discoverExtensions(NarClassLoaders.getInstance().getExtensionClassLoaders()); ExtensionManager.logClassLoaderMapping(); // load the server from the framework classloader @@ -229,7 +229,7 @@ public class MiNiFi { public static void main(String[] args) { logger.info("Launching MiNiFi..."); try { - NiFiProperties niFiProperties = NiFiProperties.getInstance(); + NiFiProperties niFiProperties = NiFiProperties.createBasicNiFiProperties(null, null); new MiNiFi(niFiProperties); } catch (final Throwable t) { logger.error("Failure to launch MiNiFi due to " + t, t); http://git-wip-us.apache.org/repos/asf/nifi-minifi/blob/52888507/minifi-nar-bundles/minifi-framework-bundle/pom.xml ---------------------------------------------------------------------- diff --git a/minifi-nar-bundles/minifi-framework-bundle/pom.xml b/minifi-nar-bundles/minifi-framework-bundle/pom.xml index 57fbc07..73a545c 100644 --- a/minifi-nar-bundles/minifi-framework-bundle/pom.xml +++ b/minifi-nar-bundles/minifi-framework-bundle/pom.xml @@ -26,7 +26,6 @@ limitations under the License. <groupId>org.apache.nifi.minifi</groupId> <artifactId>minifi-framework-bundle</artifactId> <packaging>pom</packaging> - <version>0.1.0-SNAPSHOT</version> <modules> <module>minifi-framework-nar</module> http://git-wip-us.apache.org/repos/asf/nifi-minifi/blob/52888507/minifi-nar-bundles/minifi-provenance-reporting-bundle/minifi-provenance-reporting-nar/pom.xml ---------------------------------------------------------------------- diff --git a/minifi-nar-bundles/minifi-provenance-reporting-bundle/minifi-provenance-reporting-nar/pom.xml b/minifi-nar-bundles/minifi-provenance-reporting-bundle/minifi-provenance-reporting-nar/pom.xml deleted file mode 100644 index 88b5b90..0000000 --- a/minifi-nar-bundles/minifi-provenance-reporting-bundle/minifi-provenance-reporting-nar/pom.xml +++ /dev/null @@ -1,40 +0,0 @@ -<project xmlns="http://maven.apache.org/POM/4.0.0"; xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd";> - <!-- - Licensed to the Apache Software Foundation (ASF) under one or more - contributor license agreements. See the NOTICE file distributed with - this work for additional information regarding copyright ownership. - The ASF licenses this file to You under the Apache License, Version 2.0 - (the "License"); you may not use this file except in compliance with - the License. You may obtain a copy of the License at - http://www.apache.org/licenses/LICENSE-2.0 - Unless required by applicable law or agreed to in writing, software - distributed under the License is distributed on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - See the License for the specific language governing permissions and - limitations under the License. ---> - <modelVersion>4.0.0</modelVersion> - <parent> - <groupId>org.apache.nifi.minifi</groupId> - <artifactId>minifi-provenance-reporting-bundle</artifactId> - <version>0.1.0-SNAPSHOT</version> - </parent> - - <artifactId>minifi-provenance-reporting-nar</artifactId> - <packaging>nar</packaging> - <properties> - <maven.javadoc.skip>true</maven.javadoc.skip> - <source.skip>true</source.skip> - </properties> - <dependencies> - <dependency> - <groupId>org.apache.nifi.minifi</groupId> - <artifactId>minifi-provenance-reporting-task</artifactId> - </dependency> - <dependency> - <groupId>org.apache.nifi</groupId> - <artifactId>nifi-standard-services-api-nar</artifactId> - <type>nar</type> - </dependency> - </dependencies> -</project>
