[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...
Github user ottobackwards closed the pull request at: https://github.com/apache/metron/pull/530 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
Re: [GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...
Eesh, sorry, please ignore. This was incorrect, due to reading two diff blocks as contiguous. On 8/16/17, 9:58 PM, "mattf-horton" wrote: Github user mattf-horton commented on a diff in the pull request: https://github.com/apache/metron/pull/530#discussion_r133623540 --- Diff: bundles-lib/src/main/java/org/apache/metron/bundles/BundleClassLoaders.java --- @@ -119,8 +120,10 @@ public static void init(final FileSystemManager fileSystemManager, final List
[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...
Github user mattf-horton commented on a diff in the pull request: https://github.com/apache/metron/pull/530#discussion_r133623540 --- Diff: bundles-lib/src/main/java/org/apache/metron/bundles/BundleClassLoaders.java --- @@ -119,8 +120,10 @@ public static void init(final FileSystemManager fileSystemManager, final List
[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/530#discussion_r133619506 --- Diff: bundles-lib/src/test/java/org/apache/metron/bundles/BundleThreadContextClassLoaderTest.java --- @@ -27,11 +27,21 @@ import org.apache.metron.bundles.bundle.Bundle; import org.apache.metron.bundles.util.BundleProperties; import org.apache.metron.bundles.util.FileSystemManagerFactory; +import org.junit.After; import org.junit.AfterClass; import org.junit.Test; public class BundleThreadContextClassLoaderTest { + @AfterClass + public static void after() { +BundleClassLoaders.reset(); + } --- End diff -- I hate this static stuff. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/530#discussion_r133619456 --- Diff: bundles-lib/src/main/java/org/apache/metron/bundles/BundleClassLoaders.java --- @@ -95,19 +91,11 @@ public static BundleClassLoaders getInstance() { */ public static void reset() { --- End diff -- ok --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/530#discussion_r133619481 --- Diff: bundles-lib/src/main/java/org/apache/metron/bundles/ExtensionManager.java --- @@ -103,61 +104,52 @@ private ExtensionManager(){} /** * @return The singleton instance of the ExtensionManager */ - public static ExtensionManager getInstance() { -ExtensionManager result = extensionManager; -if (result == null) { - synchronized (ExtensionManager.class) { -result = extensionManager; -if (result == null) { - extensionManager = new ExtensionManager(); - result = extensionManager; -} + public static ExtensionManager getInstance() throws NotInitializedException { +synchronized (ExtensionManager.class) { + if (extensionManager == null) { +throw new NotInitializedException("ExtensionManager not initialized"); } + return extensionManager; } -return result; } /** * Uninitializes the ExtensionManager. * TESTING ONLY */ + @VisibleForTesting public static void reset() { synchronized (ExtensionManager.class) { - getInstance().forgetExtensions(); + initContext = null; extensionManager = null; } } - private void forgetExtensions() { - initContext = null; - } - - /** * Loads all extension class types that can be found on the bootstrap classloader and by creating * classloaders for all BUNDLES found within the classpath. * * @param bundles the bundles to scan through in search of extensions */ - public void init(final List classes, final Bundle systemBundle, final Set bundles) + public static void init(final List classes, final Bundle systemBundle, final Set bundles) throws NotInitializedException { -InitContext ic = initContext; -if (ic == null) { - synchronized (this) { -ic = initContext; -if (ic == null) { - initContext = discoverExtensions(classes, systemBundle, bundles); - ic = initContext; -} +if (systemBundle == null) { + throw new IllegalArgumentException("systemBundle is required"); +} + +synchronized (ExtensionManager.class) { + if (extensionManager != null) { +throw new IllegalStateException("ExtensionManager already exists"); } + extensionManager = new ExtensionManager(); + initContext = extensionManager.discoverExtensions(classes, systemBundle, bundles); --- End diff -- ok --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/530#discussion_r133619468 --- Diff: bundles-lib/src/main/java/org/apache/metron/bundles/BundleClassLoaders.java --- @@ -121,33 +109,18 @@ private void unInit() { * @throws IllegalStateException when already initialized with a given set of extension directories * and extensionDirs does not match */ - public void init(final FileSystemManager fileSystemManager, final List extensionsDirs, + public static void init(final FileSystemManager fileSystemManager, final List extensionsDirs, BundleProperties props) throws FileSystemException, ClassNotFoundException, URISyntaxException { if (extensionsDirs == null || fileSystemManager == null) { throw new NullPointerException("cannot have empty arguments"); } -InitContext ic = initContext; -if (ic == null) { - synchronized (this) { -ic = initContext; -if (ic == null) { - initContext = load(fileSystemManager, extensionsDirs, props); - ic = initContext; -} - } -} else { - boolean matching = CollectionUtils - .isEqualCollection(initContext.extensionDirs, extensionsDirs); - if (!matching) { -throw new IllegalStateException( -"Cannot reinitialize and extension directories cannot change"); - } - matching = initContext.properties.match(props); - if (!matching) { -throw new IllegalStateException( -"Cannot re-initialize and properties cannot change"); +synchronized (BundleClassLoaders.class) { + if (bundleClassLoaders != null) { +throw new IllegalStateException("BundleClassloader already exists"); } + bundleClassLoaders = new BundleClassLoaders(); + initContext = bundleClassLoaders.load(fileSystemManager, extensionsDirs, props); --- End diff -- ok --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/530#discussion_r133619443 --- Diff: bundles-lib/src/main/java/org/apache/metron/bundles/BundleClassLoaders.java --- @@ -73,19 +73,15 @@ private BundleClassLoaders() { /** * @return The singleton instance of the BundleClassLoaders + * @throws NotInitializedException if BundleClassLoaders has not been init'd */ - public static BundleClassLoaders getInstance() { -BundleClassLoaders result = bundleClassLoaders; -if (result == null) { - synchronized (BundleClassLoaders.class) { -result = bundleClassLoaders; -if (result == null) { - bundleClassLoaders = new BundleClassLoaders(); - result = bundleClassLoaders; -} + public static BundleClassLoaders getInstance() throws NotInitializedException { +synchronized (BundleClassLoaders.class) { + if (bundleClassLoaders == null) { +throw new NotInitializedException("BundleClassLoaders not initialized"); } + return bundleClassLoaders; } -return result; } --- End diff -- ok --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...
Github user mattf-horton commented on a diff in the pull request: https://github.com/apache/metron/pull/530#discussion_r133530431 --- Diff: bundles-lib/src/main/java/org/apache/metron/bundles/ExtensionManager.java --- @@ -103,61 +104,52 @@ private ExtensionManager(){} /** * @return The singleton instance of the ExtensionManager */ - public static ExtensionManager getInstance() { -ExtensionManager result = extensionManager; -if (result == null) { - synchronized (ExtensionManager.class) { -result = extensionManager; -if (result == null) { - extensionManager = new ExtensionManager(); - result = extensionManager; -} + public static ExtensionManager getInstance() throws NotInitializedException { +synchronized (ExtensionManager.class) { + if (extensionManager == null) { +throw new NotInitializedException("ExtensionManager not initialized"); } + return extensionManager; } -return result; } /** * Uninitializes the ExtensionManager. * TESTING ONLY */ + @VisibleForTesting public static void reset() { synchronized (ExtensionManager.class) { - getInstance().forgetExtensions(); + initContext = null; extensionManager = null; } } - private void forgetExtensions() { - initContext = null; - } - - /** * Loads all extension class types that can be found on the bootstrap classloader and by creating * classloaders for all BUNDLES found within the classpath. * * @param bundles the bundles to scan through in search of extensions */ - public void init(final List classes, final Bundle systemBundle, final Set bundles) + public static void init(final List classes, final Bundle systemBundle, final Set bundles) throws NotInitializedException { -InitContext ic = initContext; -if (ic == null) { - synchronized (this) { -ic = initContext; -if (ic == null) { - initContext = discoverExtensions(classes, systemBundle, bundles); - ic = initContext; -} +if (systemBundle == null) { + throw new IllegalArgumentException("systemBundle is required"); +} + +synchronized (ExtensionManager.class) { + if (extensionManager != null) { +throw new IllegalStateException("ExtensionManager already exists"); } + extensionManager = new ExtensionManager(); + initContext = extensionManager.discoverExtensions(classes, systemBundle, bundles); --- End diff -- And to protect unsynchronized getInstance(): ```java ExtensionManager em = new ExtensionManager(); InitContext ic = em.discoverExtensions(classes, systemBundle, bundles); initContext = ic; extensionManager = em; ``` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...
Github user mattf-horton commented on a diff in the pull request: https://github.com/apache/metron/pull/530#discussion_r133533234 --- Diff: bundles-lib/src/test/java/org/apache/metron/bundles/BundleThreadContextClassLoaderTest.java --- @@ -27,11 +27,21 @@ import org.apache.metron.bundles.bundle.Bundle; import org.apache.metron.bundles.util.BundleProperties; import org.apache.metron.bundles.util.FileSystemManagerFactory; +import org.junit.After; import org.junit.AfterClass; import org.junit.Test; public class BundleThreadContextClassLoaderTest { + @AfterClass + public static void after() { +BundleClassLoaders.reset(); + } --- End diff -- Did you intend to do `ExtensionManager.reset();` here also? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...
Github user mattf-horton commented on a diff in the pull request: https://github.com/apache/metron/pull/530#discussion_r133527688 --- Diff: bundles-lib/src/main/java/org/apache/metron/bundles/BundleClassLoaders.java --- @@ -121,33 +109,18 @@ private void unInit() { * @throws IllegalStateException when already initialized with a given set of extension directories * and extensionDirs does not match */ - public void init(final FileSystemManager fileSystemManager, final List extensionsDirs, + public static void init(final FileSystemManager fileSystemManager, final List extensionsDirs, BundleProperties props) throws FileSystemException, ClassNotFoundException, URISyntaxException { if (extensionsDirs == null || fileSystemManager == null) { throw new NullPointerException("cannot have empty arguments"); } -InitContext ic = initContext; -if (ic == null) { - synchronized (this) { -ic = initContext; -if (ic == null) { - initContext = load(fileSystemManager, extensionsDirs, props); - ic = initContext; -} - } -} else { - boolean matching = CollectionUtils - .isEqualCollection(initContext.extensionDirs, extensionsDirs); - if (!matching) { -throw new IllegalStateException( -"Cannot reinitialize and extension directories cannot change"); - } - matching = initContext.properties.match(props); - if (!matching) { -throw new IllegalStateException( -"Cannot re-initialize and properties cannot change"); +synchronized (BundleClassLoaders.class) { + if (bundleClassLoaders != null) { +throw new IllegalStateException("BundleClassloader already exists"); } + bundleClassLoaders = new BundleClassLoaders(); + initContext = bundleClassLoaders.load(fileSystemManager, extensionsDirs, props); --- End diff -- So that getInstance() can safely not synchronize, do this: ```java BundleClassLoaders b = new BundleClassLoaders(); InitContext ic = b.load(fileSystemManager, extensionsDirs, props); initContext = ic; bundleClassLoaders = b; ``` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...
Github user mattf-horton commented on a diff in the pull request: https://github.com/apache/metron/pull/530#discussion_r133522636 --- Diff: bundles-lib/src/main/java/org/apache/metron/bundles/BundleClassLoaders.java --- @@ -73,19 +73,15 @@ private BundleClassLoaders() { /** * @return The singleton instance of the BundleClassLoaders + * @throws NotInitializedException if BundleClassLoaders has not been init'd */ - public static BundleClassLoaders getInstance() { -BundleClassLoaders result = bundleClassLoaders; -if (result == null) { - synchronized (BundleClassLoaders.class) { -result = bundleClassLoaders; -if (result == null) { - bundleClassLoaders = new BundleClassLoaders(); - result = bundleClassLoaders; -} + public static BundleClassLoaders getInstance() throws NotInitializedException { +synchronized (BundleClassLoaders.class) { + if (bundleClassLoaders == null) { +throw new NotInitializedException("BundleClassLoaders not initialized"); } + return bundleClassLoaders; } -return result; } --- End diff -- There's no longer a need for getInstance() to synchronize, if we deal with bundleClassLoader atomically. (See also init() method comment below): ```java public static BundleClassLoaders getInstance() throws NotInitializedException { BundleClassLoaders result = bundleClassLoaders; if (result == null) { throw new NotInitializedException("BundleClassLoaders not initialized"); } return result; } ``` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...
Github user mattf-horton commented on a diff in the pull request: https://github.com/apache/metron/pull/530#discussion_r133532210 --- Diff: bundles-lib/src/main/java/org/apache/metron/bundles/BundleClassLoaders.java --- @@ -95,19 +91,11 @@ public static BundleClassLoaders getInstance() { */ public static void reset() { --- End diff -- Please add @ VisibleForTesting annotation. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...
Github user mattf-horton commented on a diff in the pull request: https://github.com/apache/metron/pull/530#discussion_r133529473 --- Diff: bundles-lib/src/main/java/org/apache/metron/bundles/ExtensionManager.java --- @@ -103,61 +104,52 @@ private ExtensionManager(){} /** * @return The singleton instance of the ExtensionManager */ - public static ExtensionManager getInstance() { -ExtensionManager result = extensionManager; -if (result == null) { - synchronized (ExtensionManager.class) { -result = extensionManager; -if (result == null) { - extensionManager = new ExtensionManager(); - result = extensionManager; -} + public static ExtensionManager getInstance() throws NotInitializedException { +synchronized (ExtensionManager.class) { + if (extensionManager == null) { +throw new NotInitializedException("ExtensionManager not initialized"); } + return extensionManager; } -return result; } --- End diff -- As with BundleClassLoader, to deal with extensionManager atomically: ```java public static ExtensionManager getInstance() throws NotInitializedException { ExtensionManager result = extensionManager; if (result == null) { throw new NotInitializedException("ExtensionManager not initialized"); } return result; } ``` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/530#discussion_r133083369 --- Diff: bundles-lib/src/main/java/org/apache/metron/bundles/BundleSystem.java --- @@ -0,0 +1,182 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.metron.bundles; + +import java.lang.invoke.MethodHandles; +import java.net.URI; +import java.util.ArrayList; +import java.util.LinkedList; +import java.util.List; +import org.apache.commons.vfs2.FileObject; +import org.apache.commons.vfs2.FileSystemManager; +import org.apache.metron.bundles.bundle.Bundle; +import org.apache.metron.bundles.util.BundleProperties; +import org.apache.metron.bundles.util.FileSystemManagerFactory; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +/** + * High level interface to the Bundle System. While you may want to use the lower level classes it + * is not required, as BundleSystem provides the base required interface for initializing the system + * and instantiating classes + */ +public class BundleSystem { + + private static final Logger LOG = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass()); + + /** + * Builder for a BundleSystem. only {@link BundleProperties} are required. Beyond that, the + * BundleProperties, if they are the only parameter must have archive extension and bundle + * extension types properties present. + */ + public static class Builder { + +private BundleProperties properties; +private FileSystemManager fileSystemManager; +private List extensionClasses = new LinkedList<>(); +private Bundle systemBundle; + +/** + * The BundleProperties to use. Unless other builder parameters override options + * (withExtensionClasses ), they must have archive extension and bundle extensions types + * specified + * + * @param properties The BundleProperties + * @return Builder + */ +public Builder withBundleProperties(BundleProperties properties) { + this.properties = properties; + return this; +} + +/** + * Provide a {@link FileSystemManager} to overide the default + * + * @param fileSystemManager override + * @return Builder + */ +public Builder withFileSystemManager(FileSystemManager fileSystemManager) { + this.fileSystemManager = fileSystemManager; + return this; +} + +/** + * Provide Extension Classes. If not provided with this override then the classes will be + * configured from the BundleProperties. If provided, the properties file will not be used. + * + * @param extensionClasses override + * @return Builder + */ +public Builder withExtensionClasses(List extensionClasses) { + this.extensionClasses.addAll(extensionClasses); + return this; +} + +/** + * Provide a SystemBundle. If not provided with this override then the default SystemBundle + * will be created. + */ +public Builder withSystemBundle(Bundle systemBundle) { + this.systemBundle = systemBundle; + return this; +} + +/** + * Builds a new BundleSystem. + * + * @return BundleSystem + * @throws NotInitializedException if any errors happen during build + */ +public BundleSystem build() throws NotInitializedException { + if (this.properties == null) { +throw new IllegalArgumentException("BundleProperties are required"); --- End diff -- I will make it more clear they will not be used for the classes, but will be used for other things --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is ena
[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...
Github user mattf-horton commented on a diff in the pull request: https://github.com/apache/metron/pull/530#discussion_r133081975 --- Diff: bundles-lib/src/main/java/org/apache/metron/bundles/BundleSystem.java --- @@ -0,0 +1,182 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.metron.bundles; + +import java.lang.invoke.MethodHandles; +import java.net.URI; +import java.util.ArrayList; +import java.util.LinkedList; +import java.util.List; +import org.apache.commons.vfs2.FileObject; +import org.apache.commons.vfs2.FileSystemManager; +import org.apache.metron.bundles.bundle.Bundle; +import org.apache.metron.bundles.util.BundleProperties; +import org.apache.metron.bundles.util.FileSystemManagerFactory; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +/** + * High level interface to the Bundle System. While you may want to use the lower level classes it + * is not required, as BundleSystem provides the base required interface for initializing the system + * and instantiating classes + */ +public class BundleSystem { + + private static final Logger LOG = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass()); + + /** + * Builder for a BundleSystem. only {@link BundleProperties} are required. Beyond that, the + * BundleProperties, if they are the only parameter must have archive extension and bundle + * extension types properties present. + */ + public static class Builder { + +private BundleProperties properties; +private FileSystemManager fileSystemManager; +private List extensionClasses = new LinkedList<>(); +private Bundle systemBundle; + +/** + * The BundleProperties to use. Unless other builder parameters override options + * (withExtensionClasses ), they must have archive extension and bundle extensions types + * specified + * + * @param properties The BundleProperties + * @return Builder + */ +public Builder withBundleProperties(BundleProperties properties) { + this.properties = properties; + return this; +} + +/** + * Provide a {@link FileSystemManager} to overide the default + * + * @param fileSystemManager override + * @return Builder + */ +public Builder withFileSystemManager(FileSystemManager fileSystemManager) { + this.fileSystemManager = fileSystemManager; + return this; +} + +/** + * Provide Extension Classes. If not provided with this override then the classes will be + * configured from the BundleProperties. If provided, the properties file will not be used. + * + * @param extensionClasses override + * @return Builder + */ +public Builder withExtensionClasses(List extensionClasses) { + this.extensionClasses.addAll(extensionClasses); + return this; +} + +/** + * Provide a SystemBundle. If not provided with this override then the default SystemBundle + * will be created. + */ +public Builder withSystemBundle(Bundle systemBundle) { + this.systemBundle = systemBundle; + return this; +} + +/** + * Builds a new BundleSystem. + * + * @return BundleSystem + * @throws NotInitializedException if any errors happen during build + */ +public BundleSystem build() throws NotInitializedException { + if (this.properties == null) { +throw new IllegalArgumentException("BundleProperties are required"); --- End diff -- Ah, then please correct the above comment on withExtensionClasses() that says "If \[ExtensionClasses\] provided, the properties file will not be used.". --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this fe
[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...
Github user mattf-horton commented on a diff in the pull request: https://github.com/apache/metron/pull/530#discussion_r133081623 --- Diff: bundles-lib/src/main/java/org/apache/metron/bundles/util/ImmutableCollectionUtils.java --- @@ -0,0 +1,65 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.metron.bundles.util; + +import com.google.common.collect.ImmutableList; +import com.google.common.collect.ImmutableMap; +import com.google.common.collect.ImmutableSet; +import java.util.List; +import java.util.Map; +import java.util.Set; + +/** + * Utility class for creating immutable versions of collections, where the inner collection needs to + * be immutable as well. + * + * {@see com.google.common.collect} + */ +public class ImmutableCollectionUtils { --- End diff -- accepted. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...
Github user mattf-horton commented on a diff in the pull request: https://github.com/apache/metron/pull/530#discussion_r133081227 --- Diff: bundles-lib/src/main/java/org/apache/metron/bundles/ExtensionManager.java --- @@ -0,0 +1,540 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.metron.bundles; + + +import com.google.common.collect.ImmutableList; +import com.google.common.collect.ImmutableMap; +import com.google.common.collect.ImmutableSet; +import java.util.ArrayList; +import java.util.Collections; +import java.util.HashMap; +import java.util.HashSet; +import java.util.List; +import java.util.Map; +import java.util.Set; +import org.apache.commons.vfs2.FileObject; +import org.apache.commons.vfs2.FileSystemException; +import org.apache.commons.vfs2.FileSystemManager; +import org.apache.metron.bundles.bundle.Bundle; +import org.apache.metron.bundles.bundle.BundleCoordinates; +import org.apache.metron.bundles.bundle.BundleDetails; +import org.apache.metron.bundles.util.BundleProperties; +import org.apache.metron.bundles.util.DummyFileObject; +import org.apache.metron.bundles.util.FileUtils; +import org.apache.metron.bundles.util.ImmutableCollectionUtils; +import org.apache.metron.bundles.util.StringUtils; +import org.apache.metron.bundles.annotation.behavior.RequiresInstanceClassLoading; + +import org.atteo.classindex.ClassIndex; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.io.IOException; +import java.lang.reflect.Modifier; +import java.net.URI; +import java.net.URISyntaxException; +import java.net.URL; +import java.net.URLClassLoader; +import java.util.concurrent.ConcurrentHashMap; + +/** + * A Singleton class for scanning through the classpath to load all extension components using + * the ClassIndex and running through all classloaders (root, BUNDLEs). + * + * + * @ThreadSafe - is immutable + */ +@SuppressWarnings("rawtypes") +public class ExtensionManager { + + private static volatile ExtensionManager extensionManager; + private volatile InitContext initContext; + + private static final Logger logger = LoggerFactory.getLogger(ExtensionManager.class); + + public static final BundleCoordinates SYSTEM_BUNDLE_COORDINATE = new BundleCoordinates( + BundleCoordinates.DEFAULT_GROUP, "system", BundleCoordinates.DEFAULT_VERSION); + + private static final class InitContext { + +// Maps a service definition (interface) to those classes that implement the interface +private final Map> definitionMap; +private final Map> classNameBundleLookup; +private final Map bundleCoordinateBundleLookup; +private final Map classLoaderBundleLookup; +private final Set requiresInstanceClassLoading; +private final Map instanceClassloaderLookup; + +private InitContext(Map> definitionMap, +Map> classNameBundleLookup, +Map bundleCoordinateBundleLookup, +Map classLoaderBundleLookup, +Set requiresInstanceClassLoading, +Map instanceClassloaderLookup) { + + this.definitionMap = ImmutableCollectionUtils.immutableMapOfSets(definitionMap); + this.classNameBundleLookup = ImmutableCollectionUtils + .immutableMapOfLists(classNameBundleLookup); + this.bundleCoordinateBundleLookup = ImmutableMap.copyOf(bundleCoordinateBundleLookup); + this.classLoaderBundleLookup = ImmutableMap.copyOf(classLoaderBundleLookup); + this.requiresInstanceClassLoading = ImmutableSet.copyOf(requiresInstanceClassLoading); + this.instanceClassloaderLookup = new ConcurrentHashMap<>(instanceClassloaderLookup); +} --- End diff -- Ok --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the fe
[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/530#discussion_r133079554 --- Diff: bundles-lib/src/main/java/org/apache/metron/bundles/ExtensionManager.java --- @@ -0,0 +1,540 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.metron.bundles; + + +import com.google.common.collect.ImmutableList; +import com.google.common.collect.ImmutableMap; +import com.google.common.collect.ImmutableSet; +import java.util.ArrayList; +import java.util.Collections; +import java.util.HashMap; +import java.util.HashSet; +import java.util.List; +import java.util.Map; +import java.util.Set; +import org.apache.commons.vfs2.FileObject; +import org.apache.commons.vfs2.FileSystemException; +import org.apache.commons.vfs2.FileSystemManager; +import org.apache.metron.bundles.bundle.Bundle; +import org.apache.metron.bundles.bundle.BundleCoordinates; +import org.apache.metron.bundles.bundle.BundleDetails; +import org.apache.metron.bundles.util.BundleProperties; +import org.apache.metron.bundles.util.DummyFileObject; +import org.apache.metron.bundles.util.FileUtils; +import org.apache.metron.bundles.util.ImmutableCollectionUtils; +import org.apache.metron.bundles.util.StringUtils; +import org.apache.metron.bundles.annotation.behavior.RequiresInstanceClassLoading; + +import org.atteo.classindex.ClassIndex; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.io.IOException; +import java.lang.reflect.Modifier; +import java.net.URI; +import java.net.URISyntaxException; +import java.net.URL; +import java.net.URLClassLoader; +import java.util.concurrent.ConcurrentHashMap; + +/** + * A Singleton class for scanning through the classpath to load all extension components using + * the ClassIndex and running through all classloaders (root, BUNDLEs). + * + * + * @ThreadSafe - is immutable + */ +@SuppressWarnings("rawtypes") +public class ExtensionManager { + + private static volatile ExtensionManager extensionManager; + private volatile InitContext initContext; + + private static final Logger logger = LoggerFactory.getLogger(ExtensionManager.class); + + public static final BundleCoordinates SYSTEM_BUNDLE_COORDINATE = new BundleCoordinates( + BundleCoordinates.DEFAULT_GROUP, "system", BundleCoordinates.DEFAULT_VERSION); + + private static final class InitContext { + +// Maps a service definition (interface) to those classes that implement the interface +private final Map> definitionMap; +private final Map> classNameBundleLookup; +private final Map bundleCoordinateBundleLookup; +private final Map classLoaderBundleLookup; +private final Set requiresInstanceClassLoading; +private final Map instanceClassloaderLookup; + +private InitContext(Map> definitionMap, +Map> classNameBundleLookup, +Map bundleCoordinateBundleLookup, +Map classLoaderBundleLookup, +Set requiresInstanceClassLoading, +Map instanceClassloaderLookup) { + + this.definitionMap = ImmutableCollectionUtils.immutableMapOfSets(definitionMap); + this.classNameBundleLookup = ImmutableCollectionUtils + .immutableMapOfLists(classNameBundleLookup); + this.bundleCoordinateBundleLookup = ImmutableMap.copyOf(bundleCoordinateBundleLookup); + this.classLoaderBundleLookup = ImmutableMap.copyOf(classLoaderBundleLookup); + this.requiresInstanceClassLoading = ImmutableSet.copyOf(requiresInstanceClassLoading); + this.instanceClassloaderLookup = new ConcurrentHashMap<>(instanceClassloaderLookup); +} + } + + private ExtensionManager(){} + + /** + * @return The singleton instance of the ExtensionManager + */ + public static ExtensionManager getInstance() { +ExtensionManager
[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/530#discussion_r133079524 --- Diff: bundles-lib/src/main/java/org/apache/metron/bundles/ExtensionManager.java --- @@ -0,0 +1,540 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.metron.bundles; + + +import com.google.common.collect.ImmutableList; +import com.google.common.collect.ImmutableMap; +import com.google.common.collect.ImmutableSet; +import java.util.ArrayList; +import java.util.Collections; +import java.util.HashMap; +import java.util.HashSet; +import java.util.List; +import java.util.Map; +import java.util.Set; +import org.apache.commons.vfs2.FileObject; +import org.apache.commons.vfs2.FileSystemException; +import org.apache.commons.vfs2.FileSystemManager; +import org.apache.metron.bundles.bundle.Bundle; +import org.apache.metron.bundles.bundle.BundleCoordinates; +import org.apache.metron.bundles.bundle.BundleDetails; +import org.apache.metron.bundles.util.BundleProperties; +import org.apache.metron.bundles.util.DummyFileObject; +import org.apache.metron.bundles.util.FileUtils; +import org.apache.metron.bundles.util.ImmutableCollectionUtils; +import org.apache.metron.bundles.util.StringUtils; +import org.apache.metron.bundles.annotation.behavior.RequiresInstanceClassLoading; + +import org.atteo.classindex.ClassIndex; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.io.IOException; +import java.lang.reflect.Modifier; +import java.net.URI; +import java.net.URISyntaxException; +import java.net.URL; +import java.net.URLClassLoader; +import java.util.concurrent.ConcurrentHashMap; + +/** + * A Singleton class for scanning through the classpath to load all extension components using + * the ClassIndex and running through all classloaders (root, BUNDLEs). + * + * + * @ThreadSafe - is immutable + */ +@SuppressWarnings("rawtypes") +public class ExtensionManager { + + private static volatile ExtensionManager extensionManager; + private volatile InitContext initContext; + + private static final Logger logger = LoggerFactory.getLogger(ExtensionManager.class); + + public static final BundleCoordinates SYSTEM_BUNDLE_COORDINATE = new BundleCoordinates( + BundleCoordinates.DEFAULT_GROUP, "system", BundleCoordinates.DEFAULT_VERSION); + + private static final class InitContext { + +// Maps a service definition (interface) to those classes that implement the interface +private final Map> definitionMap; +private final Map> classNameBundleLookup; +private final Map bundleCoordinateBundleLookup; +private final Map classLoaderBundleLookup; +private final Set requiresInstanceClassLoading; +private final Map instanceClassloaderLookup; + +private InitContext(Map> definitionMap, +Map> classNameBundleLookup, +Map bundleCoordinateBundleLookup, +Map classLoaderBundleLookup, +Set requiresInstanceClassLoading, +Map instanceClassloaderLookup) { + + this.definitionMap = ImmutableCollectionUtils.immutableMapOfSets(definitionMap); + this.classNameBundleLookup = ImmutableCollectionUtils + .immutableMapOfLists(classNameBundleLookup); + this.bundleCoordinateBundleLookup = ImmutableMap.copyOf(bundleCoordinateBundleLookup); + this.classLoaderBundleLookup = ImmutableMap.copyOf(classLoaderBundleLookup); + this.requiresInstanceClassLoading = ImmutableSet.copyOf(requiresInstanceClassLoading); + this.instanceClassloaderLookup = new ConcurrentHashMap<>(instanceClassloaderLookup); +} + } + + private ExtensionManager(){} + + /** + * @return The singleton instance of the ExtensionManager + */ + public static ExtensionManager getInstance() { +ExtensionManager
[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/530#discussion_r133079584 --- Diff: bundles-lib/src/main/java/org/apache/metron/bundles/ExtensionMapping.java --- @@ -0,0 +1,152 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.metron.bundles; + +import org.apache.metron.bundles.bundle.BundleCoordinates; + +import java.util.*; +import java.util.function.BiFunction; + +/** + * The ExtensionMapping represents a mapping of the extensions available to the system. + * It is the product of the BundleMapper. + * + * It is NOT used at runtime for loading extensions, rather it may be used by a system to --- End diff -- done --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/530#discussion_r133079513 --- Diff: bundles-lib/src/main/java/org/apache/metron/bundles/ExtensionManager.java --- @@ -0,0 +1,540 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.metron.bundles; + + +import com.google.common.collect.ImmutableList; +import com.google.common.collect.ImmutableMap; +import com.google.common.collect.ImmutableSet; +import java.util.ArrayList; +import java.util.Collections; +import java.util.HashMap; +import java.util.HashSet; +import java.util.List; +import java.util.Map; +import java.util.Set; +import org.apache.commons.vfs2.FileObject; +import org.apache.commons.vfs2.FileSystemException; +import org.apache.commons.vfs2.FileSystemManager; +import org.apache.metron.bundles.bundle.Bundle; +import org.apache.metron.bundles.bundle.BundleCoordinates; +import org.apache.metron.bundles.bundle.BundleDetails; +import org.apache.metron.bundles.util.BundleProperties; +import org.apache.metron.bundles.util.DummyFileObject; +import org.apache.metron.bundles.util.FileUtils; +import org.apache.metron.bundles.util.ImmutableCollectionUtils; +import org.apache.metron.bundles.util.StringUtils; +import org.apache.metron.bundles.annotation.behavior.RequiresInstanceClassLoading; + +import org.atteo.classindex.ClassIndex; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.io.IOException; +import java.lang.reflect.Modifier; +import java.net.URI; +import java.net.URISyntaxException; +import java.net.URL; +import java.net.URLClassLoader; +import java.util.concurrent.ConcurrentHashMap; + +/** + * A Singleton class for scanning through the classpath to load all extension components using + * the ClassIndex and running through all classloaders (root, BUNDLEs). + * + * + * @ThreadSafe - is immutable + */ +@SuppressWarnings("rawtypes") +public class ExtensionManager { + + private static volatile ExtensionManager extensionManager; + private volatile InitContext initContext; + + private static final Logger logger = LoggerFactory.getLogger(ExtensionManager.class); + + public static final BundleCoordinates SYSTEM_BUNDLE_COORDINATE = new BundleCoordinates( + BundleCoordinates.DEFAULT_GROUP, "system", BundleCoordinates.DEFAULT_VERSION); + + private static final class InitContext { + +// Maps a service definition (interface) to those classes that implement the interface +private final Map> definitionMap; +private final Map> classNameBundleLookup; +private final Map bundleCoordinateBundleLookup; +private final Map classLoaderBundleLookup; +private final Set requiresInstanceClassLoading; +private final Map instanceClassloaderLookup; + +private InitContext(Map> definitionMap, +Map> classNameBundleLookup, +Map bundleCoordinateBundleLookup, +Map classLoaderBundleLookup, +Set requiresInstanceClassLoading, +Map instanceClassloaderLookup) { + + this.definitionMap = ImmutableCollectionUtils.immutableMapOfSets(definitionMap); + this.classNameBundleLookup = ImmutableCollectionUtils + .immutableMapOfLists(classNameBundleLookup); + this.bundleCoordinateBundleLookup = ImmutableMap.copyOf(bundleCoordinateBundleLookup); + this.classLoaderBundleLookup = ImmutableMap.copyOf(classLoaderBundleLookup); + this.requiresInstanceClassLoading = ImmutableSet.copyOf(requiresInstanceClassLoading); + this.instanceClassloaderLookup = new ConcurrentHashMap<>(instanceClassloaderLookup); +} + } + + private ExtensionManager(){} + + /** + * @return The singleton instance of the ExtensionManager + */ + public static ExtensionManager getInstance() { +ExtensionManager
[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/530#discussion_r133079462 --- Diff: bundles-lib/src/main/java/org/apache/metron/bundles/ExtensionManager.java --- @@ -0,0 +1,540 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.metron.bundles; + + +import com.google.common.collect.ImmutableList; +import com.google.common.collect.ImmutableMap; +import com.google.common.collect.ImmutableSet; +import java.util.ArrayList; +import java.util.Collections; +import java.util.HashMap; +import java.util.HashSet; +import java.util.List; +import java.util.Map; +import java.util.Set; +import org.apache.commons.vfs2.FileObject; +import org.apache.commons.vfs2.FileSystemException; +import org.apache.commons.vfs2.FileSystemManager; +import org.apache.metron.bundles.bundle.Bundle; +import org.apache.metron.bundles.bundle.BundleCoordinates; +import org.apache.metron.bundles.bundle.BundleDetails; +import org.apache.metron.bundles.util.BundleProperties; +import org.apache.metron.bundles.util.DummyFileObject; +import org.apache.metron.bundles.util.FileUtils; +import org.apache.metron.bundles.util.ImmutableCollectionUtils; +import org.apache.metron.bundles.util.StringUtils; +import org.apache.metron.bundles.annotation.behavior.RequiresInstanceClassLoading; + +import org.atteo.classindex.ClassIndex; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.io.IOException; +import java.lang.reflect.Modifier; +import java.net.URI; +import java.net.URISyntaxException; +import java.net.URL; +import java.net.URLClassLoader; +import java.util.concurrent.ConcurrentHashMap; + +/** + * A Singleton class for scanning through the classpath to load all extension components using + * the ClassIndex and running through all classloaders (root, BUNDLEs). + * + * + * @ThreadSafe - is immutable --- End diff -- done --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/530#discussion_r133079476 --- Diff: bundles-lib/src/main/java/org/apache/metron/bundles/ExtensionManager.java --- @@ -0,0 +1,540 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.metron.bundles; + + +import com.google.common.collect.ImmutableList; +import com.google.common.collect.ImmutableMap; +import com.google.common.collect.ImmutableSet; +import java.util.ArrayList; +import java.util.Collections; +import java.util.HashMap; +import java.util.HashSet; +import java.util.List; +import java.util.Map; +import java.util.Set; +import org.apache.commons.vfs2.FileObject; +import org.apache.commons.vfs2.FileSystemException; +import org.apache.commons.vfs2.FileSystemManager; +import org.apache.metron.bundles.bundle.Bundle; +import org.apache.metron.bundles.bundle.BundleCoordinates; +import org.apache.metron.bundles.bundle.BundleDetails; +import org.apache.metron.bundles.util.BundleProperties; +import org.apache.metron.bundles.util.DummyFileObject; +import org.apache.metron.bundles.util.FileUtils; +import org.apache.metron.bundles.util.ImmutableCollectionUtils; +import org.apache.metron.bundles.util.StringUtils; +import org.apache.metron.bundles.annotation.behavior.RequiresInstanceClassLoading; + +import org.atteo.classindex.ClassIndex; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.io.IOException; +import java.lang.reflect.Modifier; +import java.net.URI; +import java.net.URISyntaxException; +import java.net.URL; +import java.net.URLClassLoader; +import java.util.concurrent.ConcurrentHashMap; + +/** + * A Singleton class for scanning through the classpath to load all extension components using + * the ClassIndex and running through all classloaders (root, BUNDLEs). + * + * + * @ThreadSafe - is immutable + */ +@SuppressWarnings("rawtypes") +public class ExtensionManager { + + private static volatile ExtensionManager extensionManager; + private volatile InitContext initContext; --- End diff -- done --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/530#discussion_r133079502 --- Diff: bundles-lib/src/main/java/org/apache/metron/bundles/ExtensionManager.java --- @@ -0,0 +1,540 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.metron.bundles; + + +import com.google.common.collect.ImmutableList; +import com.google.common.collect.ImmutableMap; +import com.google.common.collect.ImmutableSet; +import java.util.ArrayList; +import java.util.Collections; +import java.util.HashMap; +import java.util.HashSet; +import java.util.List; +import java.util.Map; +import java.util.Set; +import org.apache.commons.vfs2.FileObject; +import org.apache.commons.vfs2.FileSystemException; +import org.apache.commons.vfs2.FileSystemManager; +import org.apache.metron.bundles.bundle.Bundle; +import org.apache.metron.bundles.bundle.BundleCoordinates; +import org.apache.metron.bundles.bundle.BundleDetails; +import org.apache.metron.bundles.util.BundleProperties; +import org.apache.metron.bundles.util.DummyFileObject; +import org.apache.metron.bundles.util.FileUtils; +import org.apache.metron.bundles.util.ImmutableCollectionUtils; +import org.apache.metron.bundles.util.StringUtils; +import org.apache.metron.bundles.annotation.behavior.RequiresInstanceClassLoading; + +import org.atteo.classindex.ClassIndex; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.io.IOException; +import java.lang.reflect.Modifier; +import java.net.URI; +import java.net.URISyntaxException; +import java.net.URL; +import java.net.URLClassLoader; +import java.util.concurrent.ConcurrentHashMap; + +/** + * A Singleton class for scanning through the classpath to load all extension components using + * the ClassIndex and running through all classloaders (root, BUNDLEs). + * + * + * @ThreadSafe - is immutable + */ +@SuppressWarnings("rawtypes") +public class ExtensionManager { + + private static volatile ExtensionManager extensionManager; + private volatile InitContext initContext; + + private static final Logger logger = LoggerFactory.getLogger(ExtensionManager.class); + + public static final BundleCoordinates SYSTEM_BUNDLE_COORDINATE = new BundleCoordinates( + BundleCoordinates.DEFAULT_GROUP, "system", BundleCoordinates.DEFAULT_VERSION); + + private static final class InitContext { + +// Maps a service definition (interface) to those classes that implement the interface +private final Map> definitionMap; +private final Map> classNameBundleLookup; +private final Map bundleCoordinateBundleLookup; +private final Map classLoaderBundleLookup; +private final Set requiresInstanceClassLoading; +private final Map instanceClassloaderLookup; + +private InitContext(Map> definitionMap, +Map> classNameBundleLookup, +Map bundleCoordinateBundleLookup, +Map classLoaderBundleLookup, +Set requiresInstanceClassLoading, +Map instanceClassloaderLookup) { + + this.definitionMap = ImmutableCollectionUtils.immutableMapOfSets(definitionMap); + this.classNameBundleLookup = ImmutableCollectionUtils + .immutableMapOfLists(classNameBundleLookup); + this.bundleCoordinateBundleLookup = ImmutableMap.copyOf(bundleCoordinateBundleLookup); + this.classLoaderBundleLookup = ImmutableMap.copyOf(classLoaderBundleLookup); + this.requiresInstanceClassLoading = ImmutableSet.copyOf(requiresInstanceClassLoading); + this.instanceClassloaderLookup = new ConcurrentHashMap<>(instanceClassloaderLookup); +} + } + + private ExtensionManager(){} + + /** + * @return The singleton instance of the ExtensionManager + */ + public static ExtensionManager getInstance() { +ExtensionManager
[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/530#discussion_r133079542 --- Diff: bundles-lib/src/main/java/org/apache/metron/bundles/ExtensionManager.java --- @@ -0,0 +1,540 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.metron.bundles; + + +import com.google.common.collect.ImmutableList; +import com.google.common.collect.ImmutableMap; +import com.google.common.collect.ImmutableSet; +import java.util.ArrayList; +import java.util.Collections; +import java.util.HashMap; +import java.util.HashSet; +import java.util.List; +import java.util.Map; +import java.util.Set; +import org.apache.commons.vfs2.FileObject; +import org.apache.commons.vfs2.FileSystemException; +import org.apache.commons.vfs2.FileSystemManager; +import org.apache.metron.bundles.bundle.Bundle; +import org.apache.metron.bundles.bundle.BundleCoordinates; +import org.apache.metron.bundles.bundle.BundleDetails; +import org.apache.metron.bundles.util.BundleProperties; +import org.apache.metron.bundles.util.DummyFileObject; +import org.apache.metron.bundles.util.FileUtils; +import org.apache.metron.bundles.util.ImmutableCollectionUtils; +import org.apache.metron.bundles.util.StringUtils; +import org.apache.metron.bundles.annotation.behavior.RequiresInstanceClassLoading; + +import org.atteo.classindex.ClassIndex; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.io.IOException; +import java.lang.reflect.Modifier; +import java.net.URI; +import java.net.URISyntaxException; +import java.net.URL; +import java.net.URLClassLoader; +import java.util.concurrent.ConcurrentHashMap; + +/** + * A Singleton class for scanning through the classpath to load all extension components using + * the ClassIndex and running through all classloaders (root, BUNDLEs). + * + * + * @ThreadSafe - is immutable + */ +@SuppressWarnings("rawtypes") +public class ExtensionManager { + + private static volatile ExtensionManager extensionManager; + private volatile InitContext initContext; + + private static final Logger logger = LoggerFactory.getLogger(ExtensionManager.class); + + public static final BundleCoordinates SYSTEM_BUNDLE_COORDINATE = new BundleCoordinates( + BundleCoordinates.DEFAULT_GROUP, "system", BundleCoordinates.DEFAULT_VERSION); + + private static final class InitContext { + +// Maps a service definition (interface) to those classes that implement the interface +private final Map> definitionMap; +private final Map> classNameBundleLookup; +private final Map bundleCoordinateBundleLookup; +private final Map classLoaderBundleLookup; +private final Set requiresInstanceClassLoading; +private final Map instanceClassloaderLookup; + +private InitContext(Map> definitionMap, +Map> classNameBundleLookup, +Map bundleCoordinateBundleLookup, +Map classLoaderBundleLookup, +Set requiresInstanceClassLoading, +Map instanceClassloaderLookup) { + + this.definitionMap = ImmutableCollectionUtils.immutableMapOfSets(definitionMap); + this.classNameBundleLookup = ImmutableCollectionUtils + .immutableMapOfLists(classNameBundleLookup); + this.bundleCoordinateBundleLookup = ImmutableMap.copyOf(bundleCoordinateBundleLookup); + this.classLoaderBundleLookup = ImmutableMap.copyOf(classLoaderBundleLookup); + this.requiresInstanceClassLoading = ImmutableSet.copyOf(requiresInstanceClassLoading); + this.instanceClassloaderLookup = new ConcurrentHashMap<>(instanceClassloaderLookup); +} + } + + private ExtensionManager(){} + + /** + * @return The singleton instance of the ExtensionManager + */ + public static ExtensionManager getInstance() { +ExtensionManager
[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/530#discussion_r133079324 --- Diff: bundles-lib/src/main/java/org/apache/metron/bundles/BundleClassLoaders.java --- @@ -0,0 +1,376 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.metron.bundles; + +import com.google.common.collect.ImmutableList; +import com.google.common.collect.ImmutableMap; +import java.net.URISyntaxException; +import java.util.*; + +import org.apache.commons.collections.CollectionUtils; +import org.apache.commons.vfs2.FileObject; +import org.apache.commons.vfs2.FileSystemException; +import org.apache.commons.vfs2.FileSystemManager; +import org.apache.metron.bundles.bundle.Bundle; +import org.apache.metron.bundles.bundle.BundleCoordinates; +import org.apache.metron.bundles.bundle.BundleDetails; +import org.apache.metron.bundles.util.BundleProperties; +import org.apache.metron.bundles.util.BundleSelector; +import org.apache.metron.bundles.util.FileUtils; +import org.apache.metron.bundles.util.BundleUtil; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +/** + * A singleton class used to initialize the extension and framework classloaders. + */ +public final class BundleClassLoaders { + + private static volatile BundleClassLoaders bundleClassLoaders; + private volatile InitContext initContext; + private static final Logger logger = LoggerFactory.getLogger(BundleClassLoaders.class); + + /** + * Holds the context from {@code BundleClassLoaders} initialization, + * being the coordinate to bundle mapping. + * + * After initialization these are not changed, and as such they + * are immutable. + * + */ + private final static class InitContext { + +private final List extensionDirs; +private final Map bundles; +private final BundleProperties properties; + +private InitContext( +final List extensionDirs, +final Map bundles, +final BundleProperties properties) { + this.extensionDirs = ImmutableList.copyOf(extensionDirs); + this.bundles = ImmutableMap.copyOf(bundles); + this.properties = properties; +} + } + + private BundleClassLoaders() { + } + + /** + * @return The singleton instance of the BundleClassLoaders + */ + public static BundleClassLoaders getInstance() { +BundleClassLoaders result = bundleClassLoaders; +if (result == null) { + synchronized (BundleClassLoaders.class) { +result = bundleClassLoaders; +if (result == null) { + bundleClassLoaders = new BundleClassLoaders(); + result = bundleClassLoaders; +} + } +} +return result; + } --- End diff -- done --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/530#discussion_r133079437 --- Diff: bundles-lib/src/main/java/org/apache/metron/bundles/BundleThreadContextClassLoader.java --- @@ -0,0 +1,213 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.metron.bundles; + +import java.io.IOException; +import java.io.InputStream; +import java.lang.reflect.Constructor; +import java.lang.reflect.InvocationTargetException; +import java.net.URL; +import java.net.URLClassLoader; +import java.util.ArrayList; +import java.util.Enumeration; +import java.util.List; +import java.util.concurrent.atomic.AtomicBoolean; + +import org.apache.metron.bundles.bundle.Bundle; +import org.apache.metron.bundles.util.BundleProperties; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +/** + * THREAD SAFE + */ +public class BundleThreadContextClassLoader extends URLClassLoader { +static final Logger LOG = LoggerFactory.getLogger(BundleThreadContextClassLoader.class); +static final ContextSecurityManager contextSecurityManager = new ContextSecurityManager(); +private final ClassLoader forward = ClassLoader.getSystemClassLoader(); + +private BundleThreadContextClassLoader() { +super(new URL[0]); +} + +@Override +public void clearAssertionStatus() { +lookupClassLoader().clearAssertionStatus(); +} + +@Override +public URL getResource(String name) { +return lookupClassLoader().getResource(name); +} + +@Override +public InputStream getResourceAsStream(String name) { +return lookupClassLoader().getResourceAsStream(name); +} + +@Override +public Enumeration getResources(String name) throws IOException { +return lookupClassLoader().getResources(name); +} + +@Override +public Class loadClass(String name) throws ClassNotFoundException { +return lookupClassLoader().loadClass(name); +} + +@Override +public void setClassAssertionStatus(String className, boolean enabled) { +lookupClassLoader().setClassAssertionStatus(className, enabled); +} + +@Override +public void setDefaultAssertionStatus(boolean enabled) { +lookupClassLoader().setDefaultAssertionStatus(enabled); +} + +@Override +public void setPackageAssertionStatus(String packageName, boolean enabled) { +lookupClassLoader().setPackageAssertionStatus(packageName, enabled); +} + +private ClassLoader lookupClassLoader() { +final Class[] classStack = contextSecurityManager.getExecutionStack(); + +for (Class currentClass : classStack) { +final Class bundleClass = findBundleClass(currentClass); +if (bundleClass != null) { +final ClassLoader desiredClassLoader = bundleClass.getClassLoader(); + +// When new Threads are created, the new Thread inherits the ClassLoaderContext of +// the caller. However, the call stack of that new Thread may not trace back to any app-specific +// code. Therefore, the BundleThreadContextClassLoader will be unable to find the appropriate Bundle +// ClassLoader. As a result, we want to set the ContextClassLoader to the Bundle ClassLoader that +// contains the class or resource that we are looking for. +// This locks the current Thread into the appropriate Bundle ClassLoader Context. The framework will change +// the ContextClassLoader back to the BundleThreadContextClassLoader as appropriate via the +// +// TL;DR +// We need to make sure the classloader for the thread is setup correctly to use the bundl
[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/530#discussion_r133079342 --- Diff: bundles-lib/src/main/java/org/apache/metron/bundles/BundleClassLoaders.java --- @@ -0,0 +1,376 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.metron.bundles; + +import com.google.common.collect.ImmutableList; +import com.google.common.collect.ImmutableMap; +import java.net.URISyntaxException; +import java.util.*; + +import org.apache.commons.collections.CollectionUtils; +import org.apache.commons.vfs2.FileObject; +import org.apache.commons.vfs2.FileSystemException; +import org.apache.commons.vfs2.FileSystemManager; +import org.apache.metron.bundles.bundle.Bundle; +import org.apache.metron.bundles.bundle.BundleCoordinates; +import org.apache.metron.bundles.bundle.BundleDetails; +import org.apache.metron.bundles.util.BundleProperties; +import org.apache.metron.bundles.util.BundleSelector; +import org.apache.metron.bundles.util.FileUtils; +import org.apache.metron.bundles.util.BundleUtil; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +/** + * A singleton class used to initialize the extension and framework classloaders. + */ +public final class BundleClassLoaders { + + private static volatile BundleClassLoaders bundleClassLoaders; + private volatile InitContext initContext; + private static final Logger logger = LoggerFactory.getLogger(BundleClassLoaders.class); + + /** + * Holds the context from {@code BundleClassLoaders} initialization, + * being the coordinate to bundle mapping. + * + * After initialization these are not changed, and as such they + * are immutable. + * + */ + private final static class InitContext { + +private final List extensionDirs; +private final Map bundles; +private final BundleProperties properties; + +private InitContext( +final List extensionDirs, +final Map bundles, +final BundleProperties properties) { + this.extensionDirs = ImmutableList.copyOf(extensionDirs); + this.bundles = ImmutableMap.copyOf(bundles); + this.properties = properties; +} + } + + private BundleClassLoaders() { + } + + /** + * @return The singleton instance of the BundleClassLoaders + */ + public static BundleClassLoaders getInstance() { +BundleClassLoaders result = bundleClassLoaders; +if (result == null) { + synchronized (BundleClassLoaders.class) { +result = bundleClassLoaders; +if (result == null) { + bundleClassLoaders = new BundleClassLoaders(); + result = bundleClassLoaders; +} + } +} +return result; + } + + /** + * Uninitializes the BundleClassloaders. After calling this init must be called + * before the rest of the methods are called afterwards. + * This is for TESTING ONLY at this time. Reset does not unload or clear any loaded classloaders. + */ + public static void reset() { +synchronized (BundleClassLoaders.class) { + getInstance().unInit(); + bundleClassLoaders = null; +} + } + + private void unInit() { +synchronized (this) { + if(initContext != null) { +initContext = null; + } --- End diff -- done --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/530#discussion_r133079379 --- Diff: bundles-lib/src/main/java/org/apache/metron/bundles/BundleThreadContextClassLoader.java --- @@ -0,0 +1,213 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.metron.bundles; + +import java.io.IOException; +import java.io.InputStream; +import java.lang.reflect.Constructor; +import java.lang.reflect.InvocationTargetException; +import java.net.URL; +import java.net.URLClassLoader; +import java.util.ArrayList; +import java.util.Enumeration; +import java.util.List; +import java.util.concurrent.atomic.AtomicBoolean; + +import org.apache.metron.bundles.bundle.Bundle; +import org.apache.metron.bundles.util.BundleProperties; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +/** + * THREAD SAFE --- End diff -- gone --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/530#discussion_r133079301 --- Diff: bundles-lib/src/main/java/org/apache/metron/bundles/BundleClassLoaders.java --- @@ -0,0 +1,376 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.metron.bundles; + +import com.google.common.collect.ImmutableList; +import com.google.common.collect.ImmutableMap; +import java.net.URISyntaxException; +import java.util.*; + +import org.apache.commons.collections.CollectionUtils; +import org.apache.commons.vfs2.FileObject; +import org.apache.commons.vfs2.FileSystemException; +import org.apache.commons.vfs2.FileSystemManager; +import org.apache.metron.bundles.bundle.Bundle; +import org.apache.metron.bundles.bundle.BundleCoordinates; +import org.apache.metron.bundles.bundle.BundleDetails; +import org.apache.metron.bundles.util.BundleProperties; +import org.apache.metron.bundles.util.BundleSelector; +import org.apache.metron.bundles.util.FileUtils; +import org.apache.metron.bundles.util.BundleUtil; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +/** + * A singleton class used to initialize the extension and framework classloaders. + */ +public final class BundleClassLoaders { + + private static volatile BundleClassLoaders bundleClassLoaders; + private volatile InitContext initContext; --- End diff -- done --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/530#discussion_r133079362 --- Diff: bundles-lib/src/main/java/org/apache/metron/bundles/BundleClassLoaders.java --- @@ -0,0 +1,376 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.metron.bundles; + +import com.google.common.collect.ImmutableList; +import com.google.common.collect.ImmutableMap; +import java.net.URISyntaxException; +import java.util.*; + +import org.apache.commons.collections.CollectionUtils; +import org.apache.commons.vfs2.FileObject; +import org.apache.commons.vfs2.FileSystemException; +import org.apache.commons.vfs2.FileSystemManager; +import org.apache.metron.bundles.bundle.Bundle; +import org.apache.metron.bundles.bundle.BundleCoordinates; +import org.apache.metron.bundles.bundle.BundleDetails; +import org.apache.metron.bundles.util.BundleProperties; +import org.apache.metron.bundles.util.BundleSelector; +import org.apache.metron.bundles.util.FileUtils; +import org.apache.metron.bundles.util.BundleUtil; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +/** + * A singleton class used to initialize the extension and framework classloaders. + */ +public final class BundleClassLoaders { + + private static volatile BundleClassLoaders bundleClassLoaders; + private volatile InitContext initContext; + private static final Logger logger = LoggerFactory.getLogger(BundleClassLoaders.class); + + /** + * Holds the context from {@code BundleClassLoaders} initialization, + * being the coordinate to bundle mapping. + * + * After initialization these are not changed, and as such they + * are immutable. + * + */ + private final static class InitContext { + +private final List extensionDirs; +private final Map bundles; +private final BundleProperties properties; + +private InitContext( +final List extensionDirs, +final Map bundles, +final BundleProperties properties) { + this.extensionDirs = ImmutableList.copyOf(extensionDirs); + this.bundles = ImmutableMap.copyOf(bundles); + this.properties = properties; +} + } + + private BundleClassLoaders() { + } + + /** + * @return The singleton instance of the BundleClassLoaders + */ + public static BundleClassLoaders getInstance() { +BundleClassLoaders result = bundleClassLoaders; +if (result == null) { + synchronized (BundleClassLoaders.class) { +result = bundleClassLoaders; +if (result == null) { + bundleClassLoaders = new BundleClassLoaders(); + result = bundleClassLoaders; +} + } +} +return result; + } + + /** + * Uninitializes the BundleClassloaders. After calling this init must be called + * before the rest of the methods are called afterwards. + * This is for TESTING ONLY at this time. Reset does not unload or clear any loaded classloaders. + */ + public static void reset() { +synchronized (BundleClassLoaders.class) { + getInstance().unInit(); + bundleClassLoaders = null; +} + } + + private void unInit() { +synchronized (this) { + if(initContext != null) { +initContext = null; + } +} + } + + /** + * Initializes and loads the BundleClassLoaders. This method must be called before the rest of the + * methods to access the classloaders are called. Multiple calls to this method will have no effect, + * unless a different set of extension directories is passed, which will result in an IllegaStateException + * + * @param fileSystemManager the FileSystemManager + * @param extensionsDirs where to find extension artifacts + * @param props BundleProperties +
[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/530#discussion_r133078904 --- Diff: bundles-lib/src/main/java/org/apache/metron/bundles/BundleSystem.java --- @@ -0,0 +1,182 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.metron.bundles; + +import java.lang.invoke.MethodHandles; +import java.net.URI; +import java.util.ArrayList; +import java.util.LinkedList; +import java.util.List; +import org.apache.commons.vfs2.FileObject; +import org.apache.commons.vfs2.FileSystemManager; +import org.apache.metron.bundles.bundle.Bundle; +import org.apache.metron.bundles.util.BundleProperties; +import org.apache.metron.bundles.util.FileSystemManagerFactory; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +/** + * High level interface to the Bundle System. While you may want to use the lower level classes it + * is not required, as BundleSystem provides the base required interface for initializing the system + * and instantiating classes + */ +public class BundleSystem { + + private static final Logger LOG = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass()); + + /** + * Builder for a BundleSystem. only {@link BundleProperties} are required. Beyond that, the + * BundleProperties, if they are the only parameter must have archive extension and bundle + * extension types properties present. + */ + public static class Builder { + +private BundleProperties properties; +private FileSystemManager fileSystemManager; +private List extensionClasses = new LinkedList<>(); +private Bundle systemBundle; + +/** + * The BundleProperties to use. Unless other builder parameters override options + * (withExtensionClasses ), they must have archive extension and bundle extensions types + * specified + * + * @param properties The BundleProperties + * @return Builder + */ +public Builder withBundleProperties(BundleProperties properties) { + this.properties = properties; + return this; +} + +/** + * Provide a {@link FileSystemManager} to overide the default + * + * @param fileSystemManager override + * @return Builder + */ +public Builder withFileSystemManager(FileSystemManager fileSystemManager) { + this.fileSystemManager = fileSystemManager; + return this; +} + +/** + * Provide Extension Classes. If not provided with this override then the classes will be + * configured from the BundleProperties. If provided, the properties file will not be used. + * + * @param extensionClasses override + * @return Builder + */ +public Builder withExtensionClasses(List extensionClasses) { + this.extensionClasses.addAll(extensionClasses); + return this; +} + +/** + * Provide a SystemBundle. If not provided with this override then the default SystemBundle + * will be created. + */ +public Builder withSystemBundle(Bundle systemBundle) { + this.systemBundle = systemBundle; + return this; +} + +/** + * Builds a new BundleSystem. + * + * @return BundleSystem + * @throws NotInitializedException if any errors happen during build + */ +public BundleSystem build() throws NotInitializedException { + if (this.properties == null) { +throw new IllegalArgumentException("BundleProperties are required"); --- End diff -- We need the properties for other things --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infra
[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/530#discussion_r133078755 --- Diff: bundles-lib/src/main/java/org/apache/metron/bundles/util/ImmutableCollectionUtils.java --- @@ -0,0 +1,65 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.metron.bundles.util; + +import com.google.common.collect.ImmutableList; +import com.google.common.collect.ImmutableMap; +import com.google.common.collect.ImmutableSet; +import java.util.List; +import java.util.Map; +import java.util.Set; + +/** + * Utility class for creating immutable versions of collections, where the inner collection needs to + * be immutable as well. + * + * {@see com.google.common.collect} + */ +public class ImmutableCollectionUtils { --- End diff -- I think it is more const correct --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/530#discussion_r133078228 --- Diff: bundles-lib/src/main/java/org/apache/metron/bundles/ExtensionManager.java --- @@ -0,0 +1,540 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.metron.bundles; + + +import com.google.common.collect.ImmutableList; +import com.google.common.collect.ImmutableMap; +import com.google.common.collect.ImmutableSet; +import java.util.ArrayList; +import java.util.Collections; +import java.util.HashMap; +import java.util.HashSet; +import java.util.List; +import java.util.Map; +import java.util.Set; +import org.apache.commons.vfs2.FileObject; +import org.apache.commons.vfs2.FileSystemException; +import org.apache.commons.vfs2.FileSystemManager; +import org.apache.metron.bundles.bundle.Bundle; +import org.apache.metron.bundles.bundle.BundleCoordinates; +import org.apache.metron.bundles.bundle.BundleDetails; +import org.apache.metron.bundles.util.BundleProperties; +import org.apache.metron.bundles.util.DummyFileObject; +import org.apache.metron.bundles.util.FileUtils; +import org.apache.metron.bundles.util.ImmutableCollectionUtils; +import org.apache.metron.bundles.util.StringUtils; +import org.apache.metron.bundles.annotation.behavior.RequiresInstanceClassLoading; + +import org.atteo.classindex.ClassIndex; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.io.IOException; +import java.lang.reflect.Modifier; +import java.net.URI; +import java.net.URISyntaxException; +import java.net.URL; +import java.net.URLClassLoader; +import java.util.concurrent.ConcurrentHashMap; + +/** + * A Singleton class for scanning through the classpath to load all extension components using + * the ClassIndex and running through all classloaders (root, BUNDLEs). + * + * + * @ThreadSafe - is immutable + */ +@SuppressWarnings("rawtypes") +public class ExtensionManager { + + private static volatile ExtensionManager extensionManager; + private volatile InitContext initContext; + + private static final Logger logger = LoggerFactory.getLogger(ExtensionManager.class); + + public static final BundleCoordinates SYSTEM_BUNDLE_COORDINATE = new BundleCoordinates( + BundleCoordinates.DEFAULT_GROUP, "system", BundleCoordinates.DEFAULT_VERSION); + + private static final class InitContext { + +// Maps a service definition (interface) to those classes that implement the interface +private final Map> definitionMap; +private final Map> classNameBundleLookup; +private final Map bundleCoordinateBundleLookup; +private final Map classLoaderBundleLookup; +private final Set requiresInstanceClassLoading; +private final Map instanceClassloaderLookup; + +private InitContext(Map> definitionMap, +Map> classNameBundleLookup, +Map bundleCoordinateBundleLookup, +Map classLoaderBundleLookup, +Set requiresInstanceClassLoading, +Map instanceClassloaderLookup) { + + this.definitionMap = ImmutableCollectionUtils.immutableMapOfSets(definitionMap); + this.classNameBundleLookup = ImmutableCollectionUtils + .immutableMapOfLists(classNameBundleLookup); + this.bundleCoordinateBundleLookup = ImmutableMap.copyOf(bundleCoordinateBundleLookup); + this.classLoaderBundleLookup = ImmutableMap.copyOf(classLoaderBundleLookup); + this.requiresInstanceClassLoading = ImmutableSet.copyOf(requiresInstanceClassLoading); + this.instanceClassloaderLookup = new ConcurrentHashMap<>(instanceClassloaderLookup); +} --- End diff -- Yes, there is a public create call. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature e
[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/530#discussion_r133077758 --- Diff: bundles-lib/src/main/java/org/apache/metron/bundles/BundleThreadContextClassLoader.java --- @@ -0,0 +1,213 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.metron.bundles; + +import java.io.IOException; +import java.io.InputStream; +import java.lang.reflect.Constructor; +import java.lang.reflect.InvocationTargetException; +import java.net.URL; +import java.net.URLClassLoader; +import java.util.ArrayList; +import java.util.Enumeration; +import java.util.List; +import java.util.concurrent.atomic.AtomicBoolean; + +import org.apache.metron.bundles.bundle.Bundle; +import org.apache.metron.bundles.util.BundleProperties; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +/** + * THREAD SAFE + */ +public class BundleThreadContextClassLoader extends URLClassLoader { +static final Logger LOG = LoggerFactory.getLogger(BundleThreadContextClassLoader.class); +static final ContextSecurityManager contextSecurityManager = new ContextSecurityManager(); +private final ClassLoader forward = ClassLoader.getSystemClassLoader(); + +private BundleThreadContextClassLoader() { +super(new URL[0]); +} + +@Override +public void clearAssertionStatus() { +lookupClassLoader().clearAssertionStatus(); +} + +@Override +public URL getResource(String name) { +return lookupClassLoader().getResource(name); +} + +@Override +public InputStream getResourceAsStream(String name) { +return lookupClassLoader().getResourceAsStream(name); +} + +@Override +public Enumeration getResources(String name) throws IOException { +return lookupClassLoader().getResources(name); +} + +@Override +public Class loadClass(String name) throws ClassNotFoundException { +return lookupClassLoader().loadClass(name); +} + +@Override +public void setClassAssertionStatus(String className, boolean enabled) { +lookupClassLoader().setClassAssertionStatus(className, enabled); +} + +@Override +public void setDefaultAssertionStatus(boolean enabled) { +lookupClassLoader().setDefaultAssertionStatus(enabled); +} + +@Override +public void setPackageAssertionStatus(String packageName, boolean enabled) { +lookupClassLoader().setPackageAssertionStatus(packageName, enabled); +} + +private ClassLoader lookupClassLoader() { +final Class[] classStack = contextSecurityManager.getExecutionStack(); + +for (Class currentClass : classStack) { +final Class bundleClass = findBundleClass(currentClass); +if (bundleClass != null) { +final ClassLoader desiredClassLoader = bundleClass.getClassLoader(); + +// When new Threads are created, the new Thread inherits the ClassLoaderContext of +// the caller. However, the call stack of that new Thread may not trace back to any app-specific +// code. Therefore, the BundleThreadContextClassLoader will be unable to find the appropriate Bundle +// ClassLoader. As a result, we want to set the ContextClassLoader to the Bundle ClassLoader that +// contains the class or resource that we are looking for. +// This locks the current Thread into the appropriate Bundle ClassLoader Context. The framework will change +// the ContextClassLoader back to the BundleThreadContextClassLoader as appropriate via the +// +// TL;DR +// We need to make sure the classloader for the thread is setup correctly to use the bundl
[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/530#discussion_r133076687 --- Diff: bundles-lib/src/main/java/org/apache/metron/bundles/BundleThreadContextClassLoader.java --- @@ -0,0 +1,213 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.metron.bundles; + +import java.io.IOException; +import java.io.InputStream; +import java.lang.reflect.Constructor; +import java.lang.reflect.InvocationTargetException; +import java.net.URL; +import java.net.URLClassLoader; +import java.util.ArrayList; +import java.util.Enumeration; +import java.util.List; +import java.util.concurrent.atomic.AtomicBoolean; + +import org.apache.metron.bundles.bundle.Bundle; +import org.apache.metron.bundles.util.BundleProperties; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +/** + * THREAD SAFE + */ +public class BundleThreadContextClassLoader extends URLClassLoader { +static final Logger LOG = LoggerFactory.getLogger(BundleThreadContextClassLoader.class); +static final ContextSecurityManager contextSecurityManager = new ContextSecurityManager(); +private final ClassLoader forward = ClassLoader.getSystemClassLoader(); + --- End diff -- If we can't get rid of all the static init, at least we can get rid of as much as possible. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...
Github user mattf-horton commented on a diff in the pull request: https://github.com/apache/metron/pull/530#discussion_r133051603 --- Diff: bundles-lib/src/main/java/org/apache/metron/bundles/ExtensionManager.java --- @@ -0,0 +1,540 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.metron.bundles; + + +import com.google.common.collect.ImmutableList; +import com.google.common.collect.ImmutableMap; +import com.google.common.collect.ImmutableSet; +import java.util.ArrayList; +import java.util.Collections; +import java.util.HashMap; +import java.util.HashSet; +import java.util.List; +import java.util.Map; +import java.util.Set; +import org.apache.commons.vfs2.FileObject; +import org.apache.commons.vfs2.FileSystemException; +import org.apache.commons.vfs2.FileSystemManager; +import org.apache.metron.bundles.bundle.Bundle; +import org.apache.metron.bundles.bundle.BundleCoordinates; +import org.apache.metron.bundles.bundle.BundleDetails; +import org.apache.metron.bundles.util.BundleProperties; +import org.apache.metron.bundles.util.DummyFileObject; +import org.apache.metron.bundles.util.FileUtils; +import org.apache.metron.bundles.util.ImmutableCollectionUtils; +import org.apache.metron.bundles.util.StringUtils; +import org.apache.metron.bundles.annotation.behavior.RequiresInstanceClassLoading; + +import org.atteo.classindex.ClassIndex; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.io.IOException; +import java.lang.reflect.Modifier; +import java.net.URI; +import java.net.URISyntaxException; +import java.net.URL; +import java.net.URLClassLoader; +import java.util.concurrent.ConcurrentHashMap; + +/** + * A Singleton class for scanning through the classpath to load all extension components using + * the ClassIndex and running through all classloaders (root, BUNDLEs). + * + * + * @ThreadSafe - is immutable + */ +@SuppressWarnings("rawtypes") +public class ExtensionManager { + + private static volatile ExtensionManager extensionManager; + private volatile InitContext initContext; + + private static final Logger logger = LoggerFactory.getLogger(ExtensionManager.class); + + public static final BundleCoordinates SYSTEM_BUNDLE_COORDINATE = new BundleCoordinates( + BundleCoordinates.DEFAULT_GROUP, "system", BundleCoordinates.DEFAULT_VERSION); + + private static final class InitContext { + +// Maps a service definition (interface) to those classes that implement the interface +private final Map> definitionMap; +private final Map> classNameBundleLookup; +private final Map bundleCoordinateBundleLookup; +private final Map classLoaderBundleLookup; +private final Set requiresInstanceClassLoading; +private final Map instanceClassloaderLookup; + +private InitContext(Map> definitionMap, +Map> classNameBundleLookup, +Map bundleCoordinateBundleLookup, +Map classLoaderBundleLookup, +Set requiresInstanceClassLoading, +Map instanceClassloaderLookup) { + + this.definitionMap = ImmutableCollectionUtils.immutableMapOfSets(definitionMap); + this.classNameBundleLookup = ImmutableCollectionUtils + .immutableMapOfLists(classNameBundleLookup); + this.bundleCoordinateBundleLookup = ImmutableMap.copyOf(bundleCoordinateBundleLookup); + this.classLoaderBundleLookup = ImmutableMap.copyOf(classLoaderBundleLookup); + this.requiresInstanceClassLoading = ImmutableSet.copyOf(requiresInstanceClassLoading); + this.instanceClassloaderLookup = new ConcurrentHashMap<>(instanceClassloaderLookup); +} + } + + private ExtensionManager(){} + + /** + * @return The singleton instance of the ExtensionManager + */ + public static ExtensionManager getInstance() { +ExtensionManager
[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...
Github user mattf-horton commented on a diff in the pull request: https://github.com/apache/metron/pull/530#discussion_r133033964 --- Diff: bundles-lib/src/main/java/org/apache/metron/bundles/BundleClassLoaders.java --- @@ -0,0 +1,376 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.metron.bundles; + +import com.google.common.collect.ImmutableList; +import com.google.common.collect.ImmutableMap; +import java.net.URISyntaxException; +import java.util.*; + +import org.apache.commons.collections.CollectionUtils; +import org.apache.commons.vfs2.FileObject; +import org.apache.commons.vfs2.FileSystemException; +import org.apache.commons.vfs2.FileSystemManager; +import org.apache.metron.bundles.bundle.Bundle; +import org.apache.metron.bundles.bundle.BundleCoordinates; +import org.apache.metron.bundles.bundle.BundleDetails; +import org.apache.metron.bundles.util.BundleProperties; +import org.apache.metron.bundles.util.BundleSelector; +import org.apache.metron.bundles.util.FileUtils; +import org.apache.metron.bundles.util.BundleUtil; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +/** + * A singleton class used to initialize the extension and framework classloaders. + */ +public final class BundleClassLoaders { + + private static volatile BundleClassLoaders bundleClassLoaders; + private volatile InitContext initContext; --- End diff -- Since BundleClassLoaders is a singleton, this (initContext) should be static too. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...
Github user mattf-horton commented on a diff in the pull request: https://github.com/apache/metron/pull/530#discussion_r133049444 --- Diff: bundles-lib/src/main/java/org/apache/metron/bundles/ExtensionManager.java --- @@ -0,0 +1,540 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.metron.bundles; + + +import com.google.common.collect.ImmutableList; +import com.google.common.collect.ImmutableMap; +import com.google.common.collect.ImmutableSet; +import java.util.ArrayList; +import java.util.Collections; +import java.util.HashMap; +import java.util.HashSet; +import java.util.List; +import java.util.Map; +import java.util.Set; +import org.apache.commons.vfs2.FileObject; +import org.apache.commons.vfs2.FileSystemException; +import org.apache.commons.vfs2.FileSystemManager; +import org.apache.metron.bundles.bundle.Bundle; +import org.apache.metron.bundles.bundle.BundleCoordinates; +import org.apache.metron.bundles.bundle.BundleDetails; +import org.apache.metron.bundles.util.BundleProperties; +import org.apache.metron.bundles.util.DummyFileObject; +import org.apache.metron.bundles.util.FileUtils; +import org.apache.metron.bundles.util.ImmutableCollectionUtils; +import org.apache.metron.bundles.util.StringUtils; +import org.apache.metron.bundles.annotation.behavior.RequiresInstanceClassLoading; + +import org.atteo.classindex.ClassIndex; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.io.IOException; +import java.lang.reflect.Modifier; +import java.net.URI; +import java.net.URISyntaxException; +import java.net.URL; +import java.net.URLClassLoader; +import java.util.concurrent.ConcurrentHashMap; + +/** + * A Singleton class for scanning through the classpath to load all extension components using + * the ClassIndex and running through all classloaders (root, BUNDLEs). + * + * + * @ThreadSafe - is immutable + */ +@SuppressWarnings("rawtypes") +public class ExtensionManager { + + private static volatile ExtensionManager extensionManager; + private volatile InitContext initContext; + + private static final Logger logger = LoggerFactory.getLogger(ExtensionManager.class); + + public static final BundleCoordinates SYSTEM_BUNDLE_COORDINATE = new BundleCoordinates( + BundleCoordinates.DEFAULT_GROUP, "system", BundleCoordinates.DEFAULT_VERSION); + + private static final class InitContext { + +// Maps a service definition (interface) to those classes that implement the interface +private final Map> definitionMap; +private final Map> classNameBundleLookup; +private final Map bundleCoordinateBundleLookup; +private final Map classLoaderBundleLookup; +private final Set requiresInstanceClassLoading; +private final Map instanceClassloaderLookup; + +private InitContext(Map> definitionMap, +Map> classNameBundleLookup, +Map bundleCoordinateBundleLookup, +Map classLoaderBundleLookup, +Set requiresInstanceClassLoading, +Map instanceClassloaderLookup) { + + this.definitionMap = ImmutableCollectionUtils.immutableMapOfSets(definitionMap); + this.classNameBundleLookup = ImmutableCollectionUtils + .immutableMapOfLists(classNameBundleLookup); + this.bundleCoordinateBundleLookup = ImmutableMap.copyOf(bundleCoordinateBundleLookup); + this.classLoaderBundleLookup = ImmutableMap.copyOf(classLoaderBundleLookup); + this.requiresInstanceClassLoading = ImmutableSet.copyOf(requiresInstanceClassLoading); + this.instanceClassloaderLookup = new ConcurrentHashMap<>(instanceClassloaderLookup); +} --- End diff -- Good that these member fields are now immutable. Just a nit: Does instanceClassloaderLookup need to be mutable? One notes that https://google.github.io/guava/releases/19.0/api/docs/co
[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...
Github user mattf-horton commented on a diff in the pull request: https://github.com/apache/metron/pull/530#discussion_r133037865 --- Diff: bundles-lib/src/main/java/org/apache/metron/bundles/BundleClassLoaders.java --- @@ -0,0 +1,376 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.metron.bundles; + +import com.google.common.collect.ImmutableList; +import com.google.common.collect.ImmutableMap; +import java.net.URISyntaxException; +import java.util.*; + +import org.apache.commons.collections.CollectionUtils; +import org.apache.commons.vfs2.FileObject; +import org.apache.commons.vfs2.FileSystemException; +import org.apache.commons.vfs2.FileSystemManager; +import org.apache.metron.bundles.bundle.Bundle; +import org.apache.metron.bundles.bundle.BundleCoordinates; +import org.apache.metron.bundles.bundle.BundleDetails; +import org.apache.metron.bundles.util.BundleProperties; +import org.apache.metron.bundles.util.BundleSelector; +import org.apache.metron.bundles.util.FileUtils; +import org.apache.metron.bundles.util.BundleUtil; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +/** + * A singleton class used to initialize the extension and framework classloaders. + */ +public final class BundleClassLoaders { + + private static volatile BundleClassLoaders bundleClassLoaders; + private volatile InitContext initContext; + private static final Logger logger = LoggerFactory.getLogger(BundleClassLoaders.class); + + /** + * Holds the context from {@code BundleClassLoaders} initialization, + * being the coordinate to bundle mapping. + * + * After initialization these are not changed, and as such they + * are immutable. + * + */ + private final static class InitContext { + +private final List extensionDirs; +private final Map bundles; +private final BundleProperties properties; + +private InitContext( +final List extensionDirs, +final Map bundles, +final BundleProperties properties) { + this.extensionDirs = ImmutableList.copyOf(extensionDirs); + this.bundles = ImmutableMap.copyOf(bundles); + this.properties = properties; +} + } + + private BundleClassLoaders() { + } + + /** + * @return The singleton instance of the BundleClassLoaders + */ + public static BundleClassLoaders getInstance() { +BundleClassLoaders result = bundleClassLoaders; +if (result == null) { + synchronized (BundleClassLoaders.class) { +result = bundleClassLoaders; +if (result == null) { + bundleClassLoaders = new BundleClassLoaders(); + result = bundleClassLoaders; +} + } +} +return result; + } + + /** + * Uninitializes the BundleClassloaders. After calling this init must be called + * before the rest of the methods are called afterwards. + * This is for TESTING ONLY at this time. Reset does not unload or clear any loaded classloaders. + */ + public static void reset() { +synchronized (BundleClassLoaders.class) { + getInstance().unInit(); + bundleClassLoaders = null; +} + } + + private void unInit() { +synchronized (this) { + if(initContext != null) { +initContext = null; + } --- End diff -- Recommend combining reset and unInit methods. This avoids the issues of - having reset static while unInit is per-instance (doesn't matter which since is singleton, but confusing to use both) - using double synchronize calls (ok even with both static or both per-instance, because Java locks are reentrant, but unnecessary and raises concerns in paranoid reviewers like me) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does n
[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...
Github user mattf-horton commented on a diff in the pull request: https://github.com/apache/metron/pull/530#discussion_r133043905 --- Diff: bundles-lib/src/main/java/org/apache/metron/bundles/BundleThreadContextClassLoader.java --- @@ -0,0 +1,213 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.metron.bundles; + +import java.io.IOException; +import java.io.InputStream; +import java.lang.reflect.Constructor; +import java.lang.reflect.InvocationTargetException; +import java.net.URL; +import java.net.URLClassLoader; +import java.util.ArrayList; +import java.util.Enumeration; +import java.util.List; +import java.util.concurrent.atomic.AtomicBoolean; + +import org.apache.metron.bundles.bundle.Bundle; +import org.apache.metron.bundles.util.BundleProperties; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +/** + * THREAD SAFE + */ +public class BundleThreadContextClassLoader extends URLClassLoader { +static final Logger LOG = LoggerFactory.getLogger(BundleThreadContextClassLoader.class); +static final ContextSecurityManager contextSecurityManager = new ContextSecurityManager(); +private final ClassLoader forward = ClassLoader.getSystemClassLoader(); + --- End diff -- I very much like that you made BundleThreadContextClassLoader statically self-initializing, by using ExtensionManager.getInstance().getExtensionClasses() instead of keeping a copy here that had to be initialized. Good job. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...
Github user mattf-horton commented on a diff in the pull request: https://github.com/apache/metron/pull/530#discussion_r133034846 --- Diff: bundles-lib/src/main/java/org/apache/metron/bundles/BundleClassLoaders.java --- @@ -0,0 +1,376 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.metron.bundles; + +import com.google.common.collect.ImmutableList; +import com.google.common.collect.ImmutableMap; +import java.net.URISyntaxException; +import java.util.*; + +import org.apache.commons.collections.CollectionUtils; +import org.apache.commons.vfs2.FileObject; +import org.apache.commons.vfs2.FileSystemException; +import org.apache.commons.vfs2.FileSystemManager; +import org.apache.metron.bundles.bundle.Bundle; +import org.apache.metron.bundles.bundle.BundleCoordinates; +import org.apache.metron.bundles.bundle.BundleDetails; +import org.apache.metron.bundles.util.BundleProperties; +import org.apache.metron.bundles.util.BundleSelector; +import org.apache.metron.bundles.util.FileUtils; +import org.apache.metron.bundles.util.BundleUtil; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +/** + * A singleton class used to initialize the extension and framework classloaders. + */ +public final class BundleClassLoaders { + + private static volatile BundleClassLoaders bundleClassLoaders; + private volatile InitContext initContext; + private static final Logger logger = LoggerFactory.getLogger(BundleClassLoaders.class); + + /** + * Holds the context from {@code BundleClassLoaders} initialization, + * being the coordinate to bundle mapping. + * + * After initialization these are not changed, and as such they + * are immutable. + * + */ + private final static class InitContext { + +private final List extensionDirs; +private final Map bundles; +private final BundleProperties properties; + +private InitContext( +final List extensionDirs, +final Map bundles, +final BundleProperties properties) { + this.extensionDirs = ImmutableList.copyOf(extensionDirs); + this.bundles = ImmutableMap.copyOf(bundles); + this.properties = properties; +} + } + + private BundleClassLoaders() { + } + + /** + * @return The singleton instance of the BundleClassLoaders + */ + public static BundleClassLoaders getInstance() { +BundleClassLoaders result = bundleClassLoaders; +if (result == null) { + synchronized (BundleClassLoaders.class) { +result = bundleClassLoaders; +if (result == null) { + bundleClassLoaders = new BundleClassLoaders(); + result = bundleClassLoaders; +} + } +} +return result; + } --- End diff -- But this can return an uninitialized singleton. It is better to have getInstance() just test for null and throw UnitializedException or return an initialized instance. The creation of the singleton should be in init() -- which must be static so you can invoke before the instance exists. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...
Github user mattf-horton commented on a diff in the pull request: https://github.com/apache/metron/pull/530#discussion_r133046812 --- Diff: bundles-lib/src/main/java/org/apache/metron/bundles/BundleThreadContextClassLoader.java --- @@ -0,0 +1,213 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.metron.bundles; + +import java.io.IOException; +import java.io.InputStream; +import java.lang.reflect.Constructor; +import java.lang.reflect.InvocationTargetException; +import java.net.URL; +import java.net.URLClassLoader; +import java.util.ArrayList; +import java.util.Enumeration; +import java.util.List; +import java.util.concurrent.atomic.AtomicBoolean; + +import org.apache.metron.bundles.bundle.Bundle; +import org.apache.metron.bundles.util.BundleProperties; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +/** + * THREAD SAFE + */ +public class BundleThreadContextClassLoader extends URLClassLoader { +static final Logger LOG = LoggerFactory.getLogger(BundleThreadContextClassLoader.class); +static final ContextSecurityManager contextSecurityManager = new ContextSecurityManager(); +private final ClassLoader forward = ClassLoader.getSystemClassLoader(); + +private BundleThreadContextClassLoader() { +super(new URL[0]); +} + +@Override +public void clearAssertionStatus() { +lookupClassLoader().clearAssertionStatus(); +} + +@Override +public URL getResource(String name) { +return lookupClassLoader().getResource(name); +} + +@Override +public InputStream getResourceAsStream(String name) { +return lookupClassLoader().getResourceAsStream(name); +} + +@Override +public Enumeration getResources(String name) throws IOException { +return lookupClassLoader().getResources(name); +} + +@Override +public Class loadClass(String name) throws ClassNotFoundException { +return lookupClassLoader().loadClass(name); +} + +@Override +public void setClassAssertionStatus(String className, boolean enabled) { +lookupClassLoader().setClassAssertionStatus(className, enabled); +} + +@Override +public void setDefaultAssertionStatus(boolean enabled) { +lookupClassLoader().setDefaultAssertionStatus(enabled); +} + +@Override +public void setPackageAssertionStatus(String packageName, boolean enabled) { +lookupClassLoader().setPackageAssertionStatus(packageName, enabled); +} + +private ClassLoader lookupClassLoader() { +final Class[] classStack = contextSecurityManager.getExecutionStack(); + +for (Class currentClass : classStack) { +final Class bundleClass = findBundleClass(currentClass); +if (bundleClass != null) { +final ClassLoader desiredClassLoader = bundleClass.getClassLoader(); + +// When new Threads are created, the new Thread inherits the ClassLoaderContext of +// the caller. However, the call stack of that new Thread may not trace back to any app-specific +// code. Therefore, the BundleThreadContextClassLoader will be unable to find the appropriate Bundle +// ClassLoader. As a result, we want to set the ContextClassLoader to the Bundle ClassLoader that +// contains the class or resource that we are looking for. +// This locks the current Thread into the appropriate Bundle ClassLoader Context. The framework will change +// the ContextClassLoader back to the BundleThreadContextClassLoader as appropriate via the +// +// TL;DR +// We need to make sure the classloader for the thread is setup correctly to use the bundle
[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...
Github user mattf-horton commented on a diff in the pull request: https://github.com/apache/metron/pull/530#discussion_r133047967 --- Diff: bundles-lib/src/main/java/org/apache/metron/bundles/ExtensionManager.java --- @@ -0,0 +1,540 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.metron.bundles; + + +import com.google.common.collect.ImmutableList; +import com.google.common.collect.ImmutableMap; +import com.google.common.collect.ImmutableSet; +import java.util.ArrayList; +import java.util.Collections; +import java.util.HashMap; +import java.util.HashSet; +import java.util.List; +import java.util.Map; +import java.util.Set; +import org.apache.commons.vfs2.FileObject; +import org.apache.commons.vfs2.FileSystemException; +import org.apache.commons.vfs2.FileSystemManager; +import org.apache.metron.bundles.bundle.Bundle; +import org.apache.metron.bundles.bundle.BundleCoordinates; +import org.apache.metron.bundles.bundle.BundleDetails; +import org.apache.metron.bundles.util.BundleProperties; +import org.apache.metron.bundles.util.DummyFileObject; +import org.apache.metron.bundles.util.FileUtils; +import org.apache.metron.bundles.util.ImmutableCollectionUtils; +import org.apache.metron.bundles.util.StringUtils; +import org.apache.metron.bundles.annotation.behavior.RequiresInstanceClassLoading; + +import org.atteo.classindex.ClassIndex; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.io.IOException; +import java.lang.reflect.Modifier; +import java.net.URI; +import java.net.URISyntaxException; +import java.net.URL; +import java.net.URLClassLoader; +import java.util.concurrent.ConcurrentHashMap; + +/** + * A Singleton class for scanning through the classpath to load all extension components using + * the ClassIndex and running through all classloaders (root, BUNDLEs). + * + * + * @ThreadSafe - is immutable --- End diff -- You've commented out the @ThreadSafe annotation. Should either use it or delete it. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...
Github user mattf-horton commented on a diff in the pull request: https://github.com/apache/metron/pull/530#discussion_r133052945 --- Diff: bundles-lib/src/main/java/org/apache/metron/bundles/ExtensionManager.java --- @@ -0,0 +1,540 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.metron.bundles; + + +import com.google.common.collect.ImmutableList; +import com.google.common.collect.ImmutableMap; +import com.google.common.collect.ImmutableSet; +import java.util.ArrayList; +import java.util.Collections; +import java.util.HashMap; +import java.util.HashSet; +import java.util.List; +import java.util.Map; +import java.util.Set; +import org.apache.commons.vfs2.FileObject; +import org.apache.commons.vfs2.FileSystemException; +import org.apache.commons.vfs2.FileSystemManager; +import org.apache.metron.bundles.bundle.Bundle; +import org.apache.metron.bundles.bundle.BundleCoordinates; +import org.apache.metron.bundles.bundle.BundleDetails; +import org.apache.metron.bundles.util.BundleProperties; +import org.apache.metron.bundles.util.DummyFileObject; +import org.apache.metron.bundles.util.FileUtils; +import org.apache.metron.bundles.util.ImmutableCollectionUtils; +import org.apache.metron.bundles.util.StringUtils; +import org.apache.metron.bundles.annotation.behavior.RequiresInstanceClassLoading; + +import org.atteo.classindex.ClassIndex; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.io.IOException; +import java.lang.reflect.Modifier; +import java.net.URI; +import java.net.URISyntaxException; +import java.net.URL; +import java.net.URLClassLoader; +import java.util.concurrent.ConcurrentHashMap; + +/** + * A Singleton class for scanning through the classpath to load all extension components using + * the ClassIndex and running through all classloaders (root, BUNDLEs). + * + * + * @ThreadSafe - is immutable + */ +@SuppressWarnings("rawtypes") +public class ExtensionManager { + + private static volatile ExtensionManager extensionManager; + private volatile InitContext initContext; + + private static final Logger logger = LoggerFactory.getLogger(ExtensionManager.class); + + public static final BundleCoordinates SYSTEM_BUNDLE_COORDINATE = new BundleCoordinates( + BundleCoordinates.DEFAULT_GROUP, "system", BundleCoordinates.DEFAULT_VERSION); + + private static final class InitContext { + +// Maps a service definition (interface) to those classes that implement the interface +private final Map> definitionMap; +private final Map> classNameBundleLookup; +private final Map bundleCoordinateBundleLookup; +private final Map classLoaderBundleLookup; +private final Set requiresInstanceClassLoading; +private final Map instanceClassloaderLookup; + +private InitContext(Map> definitionMap, +Map> classNameBundleLookup, +Map bundleCoordinateBundleLookup, +Map classLoaderBundleLookup, +Set requiresInstanceClassLoading, +Map instanceClassloaderLookup) { + + this.definitionMap = ImmutableCollectionUtils.immutableMapOfSets(definitionMap); + this.classNameBundleLookup = ImmutableCollectionUtils + .immutableMapOfLists(classNameBundleLookup); + this.bundleCoordinateBundleLookup = ImmutableMap.copyOf(bundleCoordinateBundleLookup); + this.classLoaderBundleLookup = ImmutableMap.copyOf(classLoaderBundleLookup); + this.requiresInstanceClassLoading = ImmutableSet.copyOf(requiresInstanceClassLoading); + this.instanceClassloaderLookup = new ConcurrentHashMap<>(instanceClassloaderLookup); +} + } + + private ExtensionManager(){} + + /** + * @return The singleton instance of the ExtensionManager + */ + public static ExtensionManager getInstance() { +ExtensionManager
[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...
Github user mattf-horton commented on a diff in the pull request: https://github.com/apache/metron/pull/530#discussion_r133049087 --- Diff: bundles-lib/src/main/java/org/apache/metron/bundles/ExtensionManager.java --- @@ -0,0 +1,540 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.metron.bundles; + + +import com.google.common.collect.ImmutableList; +import com.google.common.collect.ImmutableMap; +import com.google.common.collect.ImmutableSet; +import java.util.ArrayList; +import java.util.Collections; +import java.util.HashMap; +import java.util.HashSet; +import java.util.List; +import java.util.Map; +import java.util.Set; +import org.apache.commons.vfs2.FileObject; +import org.apache.commons.vfs2.FileSystemException; +import org.apache.commons.vfs2.FileSystemManager; +import org.apache.metron.bundles.bundle.Bundle; +import org.apache.metron.bundles.bundle.BundleCoordinates; +import org.apache.metron.bundles.bundle.BundleDetails; +import org.apache.metron.bundles.util.BundleProperties; +import org.apache.metron.bundles.util.DummyFileObject; +import org.apache.metron.bundles.util.FileUtils; +import org.apache.metron.bundles.util.ImmutableCollectionUtils; +import org.apache.metron.bundles.util.StringUtils; +import org.apache.metron.bundles.annotation.behavior.RequiresInstanceClassLoading; + +import org.atteo.classindex.ClassIndex; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.io.IOException; +import java.lang.reflect.Modifier; +import java.net.URI; +import java.net.URISyntaxException; +import java.net.URL; +import java.net.URLClassLoader; +import java.util.concurrent.ConcurrentHashMap; + +/** + * A Singleton class for scanning through the classpath to load all extension components using + * the ClassIndex and running through all classloaders (root, BUNDLEs). + * + * + * @ThreadSafe - is immutable + */ +@SuppressWarnings("rawtypes") +public class ExtensionManager { + + private static volatile ExtensionManager extensionManager; + private volatile InitContext initContext; --- End diff -- make initContext static, see below --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...
Github user mattf-horton commented on a diff in the pull request: https://github.com/apache/metron/pull/530#discussion_r133062854 --- Diff: bundles-lib/src/main/java/org/apache/metron/bundles/ExtensionMapping.java --- @@ -0,0 +1,152 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.metron.bundles; + +import org.apache.metron.bundles.bundle.BundleCoordinates; + +import java.util.*; +import java.util.function.BiFunction; + +/** + * The ExtensionMapping represents a mapping of the extensions available to the system. + * It is the product of the BundleMapper. + * + * It is NOT used at runtime for loading extensions, rather it may be used by a system to --- End diff -- Please note which method does load extensions at startup. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...
Github user mattf-horton commented on a diff in the pull request: https://github.com/apache/metron/pull/530#discussion_r133063087 --- Diff: bundles-lib/src/main/java/org/apache/metron/bundles/ExtensionMapping.java --- @@ -0,0 +1,152 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.metron.bundles; + +import org.apache.metron.bundles.bundle.BundleCoordinates; + +import java.util.*; +import java.util.function.BiFunction; + +/** + * The ExtensionMapping represents a mapping of the extensions available to the system. + * It is the product of the BundleMapper. + * + * It is NOT used at runtime for loading extensions, rather it may be used by a system to + * have details about the Extensions that exist in a system + */ +public class ExtensionMapping { + + /* +The extensionNameMap is a map of the following +Extension Type -> Map of Extension Class Types to a set of BundleCoordinates + +For example : +Parser -> MessageParser -> [ bundles with parsers] + +BundleProperties files define with Property names the type and class types, such as: + + bundle.extension.type.parser=org.apache.metron.parsers.interfaces.MessageParser + +This is done to give a namespace to extensions, while supporting future extension types +and classes. This is different from the inspirational Nar system, which defined an explicit set +of supported classes, and a separate map for each. + + */ + private final Map>> extensionNameMap = new HashMap<>(); + + private final BiFunction, Set, Set> merger = (oldValue, newValue) -> { +final Set merged = new HashSet<>(); +merged.addAll(oldValue); +merged.addAll(newValue); +return merged; + }; + + void addExtension(final String extensionName, final BundleCoordinates coordinate, + final String type) { +if (!extensionNameMap.containsKey(extensionName)) { + Map> bundles = new HashMap<>(); + bundles.put(type, new HashSet<>()); + extensionNameMap.put(extensionName, bundles); +} +extensionNameMap.get(extensionName).computeIfAbsent(type, name -> new HashSet<>()) +.add(coordinate); + } + + void addAllExtensions(final String extensionName, final BundleCoordinates coordinate, + final Collection types) { +if (!extensionNameMap.containsKey(extensionName)) { + Map> bundles = new HashMap<>(); + extensionNameMap.put(extensionName, bundles); +} +types.forEach(name -> { + addExtension(extensionName, coordinate, name); +}); + } + + /** + * Returns a Map of the extension class types to a Set of BundleCoordinates for a given + * extension type. + * @param extensionTypeName the extension type name, such as parser, stellar, indexing + * @return Map of extension class name to a Set of BundleCoordinates + */ --- End diff -- good. Thanks for adding all the javadocs. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...
Github user mattf-horton commented on a diff in the pull request: https://github.com/apache/metron/pull/530#discussion_r133049372 --- Diff: bundles-lib/src/main/java/org/apache/metron/bundles/ExtensionManager.java --- @@ -0,0 +1,540 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.metron.bundles; + + +import com.google.common.collect.ImmutableList; +import com.google.common.collect.ImmutableMap; +import com.google.common.collect.ImmutableSet; +import java.util.ArrayList; +import java.util.Collections; +import java.util.HashMap; +import java.util.HashSet; +import java.util.List; +import java.util.Map; +import java.util.Set; +import org.apache.commons.vfs2.FileObject; +import org.apache.commons.vfs2.FileSystemException; +import org.apache.commons.vfs2.FileSystemManager; +import org.apache.metron.bundles.bundle.Bundle; +import org.apache.metron.bundles.bundle.BundleCoordinates; +import org.apache.metron.bundles.bundle.BundleDetails; +import org.apache.metron.bundles.util.BundleProperties; +import org.apache.metron.bundles.util.DummyFileObject; +import org.apache.metron.bundles.util.FileUtils; +import org.apache.metron.bundles.util.ImmutableCollectionUtils; +import org.apache.metron.bundles.util.StringUtils; +import org.apache.metron.bundles.annotation.behavior.RequiresInstanceClassLoading; + +import org.atteo.classindex.ClassIndex; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.io.IOException; +import java.lang.reflect.Modifier; +import java.net.URI; +import java.net.URISyntaxException; +import java.net.URL; +import java.net.URLClassLoader; +import java.util.concurrent.ConcurrentHashMap; + +/** + * A Singleton class for scanning through the classpath to load all extension components using + * the ClassIndex and running through all classloaders (root, BUNDLEs). + * + * + * @ThreadSafe - is immutable + */ +@SuppressWarnings("rawtypes") +public class ExtensionManager { + + private static volatile ExtensionManager extensionManager; + private volatile InitContext initContext; + + private static final Logger logger = LoggerFactory.getLogger(ExtensionManager.class); + + public static final BundleCoordinates SYSTEM_BUNDLE_COORDINATE = new BundleCoordinates( + BundleCoordinates.DEFAULT_GROUP, "system", BundleCoordinates.DEFAULT_VERSION); + + private static final class InitContext { + +// Maps a service definition (interface) to those classes that implement the interface +private final Map> definitionMap; +private final Map> classNameBundleLookup; +private final Map bundleCoordinateBundleLookup; +private final Map classLoaderBundleLookup; +private final Set requiresInstanceClassLoading; +private final Map instanceClassloaderLookup; + +private InitContext(Map> definitionMap, +Map> classNameBundleLookup, +Map bundleCoordinateBundleLookup, +Map classLoaderBundleLookup, +Set requiresInstanceClassLoading, +Map instanceClassloaderLookup) { + + this.definitionMap = ImmutableCollectionUtils.immutableMapOfSets(definitionMap); + this.classNameBundleLookup = ImmutableCollectionUtils + .immutableMapOfLists(classNameBundleLookup); + this.bundleCoordinateBundleLookup = ImmutableMap.copyOf(bundleCoordinateBundleLookup); + this.classLoaderBundleLookup = ImmutableMap.copyOf(classLoaderBundleLookup); + this.requiresInstanceClassLoading = ImmutableSet.copyOf(requiresInstanceClassLoading); + this.instanceClassloaderLookup = new ConcurrentHashMap<>(instanceClassloaderLookup); +} + } + + private ExtensionManager(){} + + /** + * @return The singleton instance of the ExtensionManager + */ + public static ExtensionManager getInstance() { +ExtensionManager
[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...
Github user mattf-horton commented on a diff in the pull request: https://github.com/apache/metron/pull/530#discussion_r133060635 --- Diff: bundles-lib/src/main/java/org/apache/metron/bundles/BundleSystem.java --- @@ -0,0 +1,182 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.metron.bundles; + +import java.lang.invoke.MethodHandles; +import java.net.URI; +import java.util.ArrayList; +import java.util.LinkedList; +import java.util.List; +import org.apache.commons.vfs2.FileObject; +import org.apache.commons.vfs2.FileSystemManager; +import org.apache.metron.bundles.bundle.Bundle; +import org.apache.metron.bundles.util.BundleProperties; +import org.apache.metron.bundles.util.FileSystemManagerFactory; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +/** + * High level interface to the Bundle System. While you may want to use the lower level classes it + * is not required, as BundleSystem provides the base required interface for initializing the system + * and instantiating classes + */ +public class BundleSystem { + + private static final Logger LOG = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass()); + + /** + * Builder for a BundleSystem. only {@link BundleProperties} are required. Beyond that, the + * BundleProperties, if they are the only parameter must have archive extension and bundle + * extension types properties present. + */ + public static class Builder { + +private BundleProperties properties; +private FileSystemManager fileSystemManager; +private List extensionClasses = new LinkedList<>(); +private Bundle systemBundle; + +/** + * The BundleProperties to use. Unless other builder parameters override options + * (withExtensionClasses ), they must have archive extension and bundle extensions types + * specified + * + * @param properties The BundleProperties + * @return Builder + */ +public Builder withBundleProperties(BundleProperties properties) { + this.properties = properties; + return this; +} + +/** + * Provide a {@link FileSystemManager} to overide the default + * + * @param fileSystemManager override + * @return Builder + */ +public Builder withFileSystemManager(FileSystemManager fileSystemManager) { + this.fileSystemManager = fileSystemManager; + return this; +} + +/** + * Provide Extension Classes. If not provided with this override then the classes will be + * configured from the BundleProperties. If provided, the properties file will not be used. + * + * @param extensionClasses override + * @return Builder + */ +public Builder withExtensionClasses(List extensionClasses) { + this.extensionClasses.addAll(extensionClasses); + return this; +} + +/** + * Provide a SystemBundle. If not provided with this override then the default SystemBundle + * will be created. + */ +public Builder withSystemBundle(Bundle systemBundle) { + this.systemBundle = systemBundle; + return this; +} + +/** + * Builds a new BundleSystem. + * + * @return BundleSystem + * @throws NotInitializedException if any errors happen during build + */ +public BundleSystem build() throws NotInitializedException { + if (this.properties == null) { +throw new IllegalArgumentException("BundleProperties are required"); --- End diff -- But you also said the only use of properties file is for the extension classes. This suggests an either/or with the "extensionClasses" field. But if you feel making properties mandatory is future-proofing, that's okay. --- If your project is set up for it, you can reply to this email and have your
[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...
Github user mattf-horton commented on a diff in the pull request: https://github.com/apache/metron/pull/530#discussion_r133035166 --- Diff: bundles-lib/src/main/java/org/apache/metron/bundles/BundleClassLoaders.java --- @@ -0,0 +1,376 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.metron.bundles; + +import com.google.common.collect.ImmutableList; +import com.google.common.collect.ImmutableMap; +import java.net.URISyntaxException; +import java.util.*; + +import org.apache.commons.collections.CollectionUtils; +import org.apache.commons.vfs2.FileObject; +import org.apache.commons.vfs2.FileSystemException; +import org.apache.commons.vfs2.FileSystemManager; +import org.apache.metron.bundles.bundle.Bundle; +import org.apache.metron.bundles.bundle.BundleCoordinates; +import org.apache.metron.bundles.bundle.BundleDetails; +import org.apache.metron.bundles.util.BundleProperties; +import org.apache.metron.bundles.util.BundleSelector; +import org.apache.metron.bundles.util.FileUtils; +import org.apache.metron.bundles.util.BundleUtil; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +/** + * A singleton class used to initialize the extension and framework classloaders. + */ +public final class BundleClassLoaders { + + private static volatile BundleClassLoaders bundleClassLoaders; + private volatile InitContext initContext; + private static final Logger logger = LoggerFactory.getLogger(BundleClassLoaders.class); + + /** + * Holds the context from {@code BundleClassLoaders} initialization, + * being the coordinate to bundle mapping. + * + * After initialization these are not changed, and as such they + * are immutable. + * + */ + private final static class InitContext { + +private final List extensionDirs; +private final Map bundles; +private final BundleProperties properties; + +private InitContext( +final List extensionDirs, +final Map bundles, +final BundleProperties properties) { + this.extensionDirs = ImmutableList.copyOf(extensionDirs); + this.bundles = ImmutableMap.copyOf(bundles); + this.properties = properties; +} + } + + private BundleClassLoaders() { + } + + /** + * @return The singleton instance of the BundleClassLoaders + */ + public static BundleClassLoaders getInstance() { +BundleClassLoaders result = bundleClassLoaders; +if (result == null) { + synchronized (BundleClassLoaders.class) { +result = bundleClassLoaders; +if (result == null) { + bundleClassLoaders = new BundleClassLoaders(); + result = bundleClassLoaders; +} + } +} +return result; + } + + /** + * Uninitializes the BundleClassloaders. After calling this init must be called + * before the rest of the methods are called afterwards. + * This is for TESTING ONLY at this time. Reset does not unload or clear any loaded classloaders. + */ + public static void reset() { +synchronized (BundleClassLoaders.class) { + getInstance().unInit(); + bundleClassLoaders = null; +} + } + + private void unInit() { +synchronized (this) { + if(initContext != null) { +initContext = null; + } +} + } + + /** + * Initializes and loads the BundleClassLoaders. This method must be called before the rest of the + * methods to access the classloaders are called. Multiple calls to this method will have no effect, + * unless a different set of extension directories is passed, which will result in an IllegaStateException + * + * @param fileSystemManager the FileSystemManager + * @param extensionsDirs where to find extension artifacts + * @param props BundleProperties + *
[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...
Github user mattf-horton commented on a diff in the pull request: https://github.com/apache/metron/pull/530#discussion_r133048319 --- Diff: bundles-lib/src/main/java/org/apache/metron/bundles/BundleThreadContextClassLoader.java --- @@ -0,0 +1,213 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.metron.bundles; + +import java.io.IOException; +import java.io.InputStream; +import java.lang.reflect.Constructor; +import java.lang.reflect.InvocationTargetException; +import java.net.URL; +import java.net.URLClassLoader; +import java.util.ArrayList; +import java.util.Enumeration; +import java.util.List; +import java.util.concurrent.atomic.AtomicBoolean; + +import org.apache.metron.bundles.bundle.Bundle; +import org.apache.metron.bundles.util.BundleProperties; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +/** + * THREAD SAFE --- End diff -- Is "THREAD SAFE" a declaration or a design recommendation? In either case, it begs supporting information. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...
Github user mattf-horton commented on a diff in the pull request: https://github.com/apache/metron/pull/530#discussion_r133050317 --- Diff: bundles-lib/src/main/java/org/apache/metron/bundles/ExtensionManager.java --- @@ -0,0 +1,540 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.metron.bundles; + + +import com.google.common.collect.ImmutableList; +import com.google.common.collect.ImmutableMap; +import com.google.common.collect.ImmutableSet; +import java.util.ArrayList; +import java.util.Collections; +import java.util.HashMap; +import java.util.HashSet; +import java.util.List; +import java.util.Map; +import java.util.Set; +import org.apache.commons.vfs2.FileObject; +import org.apache.commons.vfs2.FileSystemException; +import org.apache.commons.vfs2.FileSystemManager; +import org.apache.metron.bundles.bundle.Bundle; +import org.apache.metron.bundles.bundle.BundleCoordinates; +import org.apache.metron.bundles.bundle.BundleDetails; +import org.apache.metron.bundles.util.BundleProperties; +import org.apache.metron.bundles.util.DummyFileObject; +import org.apache.metron.bundles.util.FileUtils; +import org.apache.metron.bundles.util.ImmutableCollectionUtils; +import org.apache.metron.bundles.util.StringUtils; +import org.apache.metron.bundles.annotation.behavior.RequiresInstanceClassLoading; + +import org.atteo.classindex.ClassIndex; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.io.IOException; +import java.lang.reflect.Modifier; +import java.net.URI; +import java.net.URISyntaxException; +import java.net.URL; +import java.net.URLClassLoader; +import java.util.concurrent.ConcurrentHashMap; + +/** + * A Singleton class for scanning through the classpath to load all extension components using + * the ClassIndex and running through all classloaders (root, BUNDLEs). + * + * + * @ThreadSafe - is immutable + */ +@SuppressWarnings("rawtypes") +public class ExtensionManager { + + private static volatile ExtensionManager extensionManager; + private volatile InitContext initContext; + + private static final Logger logger = LoggerFactory.getLogger(ExtensionManager.class); + + public static final BundleCoordinates SYSTEM_BUNDLE_COORDINATE = new BundleCoordinates( + BundleCoordinates.DEFAULT_GROUP, "system", BundleCoordinates.DEFAULT_VERSION); + + private static final class InitContext { + +// Maps a service definition (interface) to those classes that implement the interface +private final Map> definitionMap; +private final Map> classNameBundleLookup; +private final Map bundleCoordinateBundleLookup; +private final Map classLoaderBundleLookup; +private final Set requiresInstanceClassLoading; +private final Map instanceClassloaderLookup; + +private InitContext(Map> definitionMap, +Map> classNameBundleLookup, +Map bundleCoordinateBundleLookup, +Map classLoaderBundleLookup, +Set requiresInstanceClassLoading, +Map instanceClassloaderLookup) { + + this.definitionMap = ImmutableCollectionUtils.immutableMapOfSets(definitionMap); + this.classNameBundleLookup = ImmutableCollectionUtils + .immutableMapOfLists(classNameBundleLookup); + this.bundleCoordinateBundleLookup = ImmutableMap.copyOf(bundleCoordinateBundleLookup); + this.classLoaderBundleLookup = ImmutableMap.copyOf(classLoaderBundleLookup); + this.requiresInstanceClassLoading = ImmutableSet.copyOf(requiresInstanceClassLoading); + this.instanceClassloaderLookup = new ConcurrentHashMap<>(instanceClassloaderLookup); +} + } + + private ExtensionManager(){} + + /** + * @return The singleton instance of the ExtensionManager + */ + public static ExtensionManager getInstance() { +ExtensionManager
[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...
Github user mattf-horton commented on a diff in the pull request: https://github.com/apache/metron/pull/530#discussion_r133056482 --- Diff: bundles-lib/src/main/java/org/apache/metron/bundles/util/ImmutableCollectionUtils.java --- @@ -0,0 +1,65 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.metron.bundles.util; + +import com.google.common.collect.ImmutableList; +import com.google.common.collect.ImmutableMap; +import com.google.common.collect.ImmutableSet; +import java.util.List; +import java.util.Map; +import java.util.Set; + +/** + * Utility class for creating immutable versions of collections, where the inner collection needs to + * be immutable as well. + * + * {@see com.google.common.collect} + */ +public class ImmutableCollectionUtils { --- End diff -- I guess you're trying to assure thread-safeness. Is it worth the extra effort and object spawning? Are the underlying objects likely to be attempted to be modified by other threads, given first-level ("shallow") immutability? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...
Github user mattf-horton commented on a diff in the pull request: https://github.com/apache/metron/pull/530#discussion_r133049820 --- Diff: bundles-lib/src/main/java/org/apache/metron/bundles/ExtensionManager.java --- @@ -0,0 +1,540 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.metron.bundles; + + +import com.google.common.collect.ImmutableList; +import com.google.common.collect.ImmutableMap; +import com.google.common.collect.ImmutableSet; +import java.util.ArrayList; +import java.util.Collections; +import java.util.HashMap; +import java.util.HashSet; +import java.util.List; +import java.util.Map; +import java.util.Set; +import org.apache.commons.vfs2.FileObject; +import org.apache.commons.vfs2.FileSystemException; +import org.apache.commons.vfs2.FileSystemManager; +import org.apache.metron.bundles.bundle.Bundle; +import org.apache.metron.bundles.bundle.BundleCoordinates; +import org.apache.metron.bundles.bundle.BundleDetails; +import org.apache.metron.bundles.util.BundleProperties; +import org.apache.metron.bundles.util.DummyFileObject; +import org.apache.metron.bundles.util.FileUtils; +import org.apache.metron.bundles.util.ImmutableCollectionUtils; +import org.apache.metron.bundles.util.StringUtils; +import org.apache.metron.bundles.annotation.behavior.RequiresInstanceClassLoading; + +import org.atteo.classindex.ClassIndex; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.io.IOException; +import java.lang.reflect.Modifier; +import java.net.URI; +import java.net.URISyntaxException; +import java.net.URL; +import java.net.URLClassLoader; +import java.util.concurrent.ConcurrentHashMap; + +/** + * A Singleton class for scanning through the classpath to load all extension components using + * the ClassIndex and running through all classloaders (root, BUNDLEs). + * + * + * @ThreadSafe - is immutable + */ +@SuppressWarnings("rawtypes") +public class ExtensionManager { + + private static volatile ExtensionManager extensionManager; + private volatile InitContext initContext; + + private static final Logger logger = LoggerFactory.getLogger(ExtensionManager.class); + + public static final BundleCoordinates SYSTEM_BUNDLE_COORDINATE = new BundleCoordinates( + BundleCoordinates.DEFAULT_GROUP, "system", BundleCoordinates.DEFAULT_VERSION); + + private static final class InitContext { + +// Maps a service definition (interface) to those classes that implement the interface +private final Map> definitionMap; +private final Map> classNameBundleLookup; +private final Map bundleCoordinateBundleLookup; +private final Map classLoaderBundleLookup; +private final Set requiresInstanceClassLoading; +private final Map instanceClassloaderLookup; + +private InitContext(Map> definitionMap, +Map> classNameBundleLookup, +Map bundleCoordinateBundleLookup, +Map classLoaderBundleLookup, +Set requiresInstanceClassLoading, +Map instanceClassloaderLookup) { + + this.definitionMap = ImmutableCollectionUtils.immutableMapOfSets(definitionMap); + this.classNameBundleLookup = ImmutableCollectionUtils + .immutableMapOfLists(classNameBundleLookup); + this.bundleCoordinateBundleLookup = ImmutableMap.copyOf(bundleCoordinateBundleLookup); + this.classLoaderBundleLookup = ImmutableMap.copyOf(classLoaderBundleLookup); + this.requiresInstanceClassLoading = ImmutableSet.copyOf(requiresInstanceClassLoading); + this.instanceClassloaderLookup = new ConcurrentHashMap<>(instanceClassloaderLookup); +} + } + + private ExtensionManager(){} + + /** + * @return The singleton instance of the ExtensionManager + */ + public static ExtensionManager getInstance() { +ExtensionManager
[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...
Github user mattf-horton commented on a diff in the pull request: https://github.com/apache/metron/pull/530#discussion_r133047499 --- Diff: bundles-lib/src/main/java/org/apache/metron/bundles/BundleThreadContextClassLoader.java --- @@ -0,0 +1,213 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.metron.bundles; + +import java.io.IOException; +import java.io.InputStream; +import java.lang.reflect.Constructor; +import java.lang.reflect.InvocationTargetException; +import java.net.URL; +import java.net.URLClassLoader; +import java.util.ArrayList; +import java.util.Enumeration; +import java.util.List; +import java.util.concurrent.atomic.AtomicBoolean; + +import org.apache.metron.bundles.bundle.Bundle; +import org.apache.metron.bundles.util.BundleProperties; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +/** + * THREAD SAFE + */ +public class BundleThreadContextClassLoader extends URLClassLoader { +static final Logger LOG = LoggerFactory.getLogger(BundleThreadContextClassLoader.class); +static final ContextSecurityManager contextSecurityManager = new ContextSecurityManager(); +private final ClassLoader forward = ClassLoader.getSystemClassLoader(); + +private BundleThreadContextClassLoader() { +super(new URL[0]); +} + +@Override +public void clearAssertionStatus() { +lookupClassLoader().clearAssertionStatus(); +} + +@Override +public URL getResource(String name) { +return lookupClassLoader().getResource(name); +} + +@Override +public InputStream getResourceAsStream(String name) { +return lookupClassLoader().getResourceAsStream(name); +} + +@Override +public Enumeration getResources(String name) throws IOException { +return lookupClassLoader().getResources(name); +} + +@Override +public Class loadClass(String name) throws ClassNotFoundException { +return lookupClassLoader().loadClass(name); +} + +@Override +public void setClassAssertionStatus(String className, boolean enabled) { +lookupClassLoader().setClassAssertionStatus(className, enabled); +} + +@Override +public void setDefaultAssertionStatus(boolean enabled) { +lookupClassLoader().setDefaultAssertionStatus(enabled); +} + +@Override +public void setPackageAssertionStatus(String packageName, boolean enabled) { +lookupClassLoader().setPackageAssertionStatus(packageName, enabled); +} + +private ClassLoader lookupClassLoader() { +final Class[] classStack = contextSecurityManager.getExecutionStack(); + +for (Class currentClass : classStack) { +final Class bundleClass = findBundleClass(currentClass); +if (bundleClass != null) { +final ClassLoader desiredClassLoader = bundleClass.getClassLoader(); + +// When new Threads are created, the new Thread inherits the ClassLoaderContext of +// the caller. However, the call stack of that new Thread may not trace back to any app-specific +// code. Therefore, the BundleThreadContextClassLoader will be unable to find the appropriate Bundle +// ClassLoader. As a result, we want to set the ContextClassLoader to the Bundle ClassLoader that +// contains the class or resource that we are looking for. +// This locks the current Thread into the appropriate Bundle ClassLoader Context. The framework will change +// the ContextClassLoader back to the BundleThreadContextClassLoader as appropriate via the +// +// TL;DR +// We need to make sure the classloader for the thread is setup correctly to use the bundle
[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/530#discussion_r133050916 --- Diff: bundles-lib/src/main/java/org/apache/metron/bundles/ExtensionMapping.java --- @@ -0,0 +1,104 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.metron.bundles; + +import org.apache.metron.bundles.bundle.BundleCoordinates; + +import java.util.*; +import java.util.function.BiFunction; + +public class ExtensionMapping { + + private final Map>> extensionNameMap = new HashMap<>(); --- End diff -- Which I have now done --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/530#discussion_r133061877 --- Diff: bundles-lib/src/main/java/org/apache/metron/bundles/ExtensionManager.java --- @@ -0,0 +1,436 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.metron.bundles; + + +import org.apache.commons.vfs2.FileObject; +import org.apache.commons.vfs2.FileSystemException; +import org.apache.commons.vfs2.FileSystemManager; +import org.apache.metron.bundles.bundle.Bundle; +import org.apache.metron.bundles.bundle.BundleCoordinates; +import org.apache.metron.bundles.bundle.BundleDetails; +import org.apache.metron.bundles.util.BundleProperties; +import org.apache.metron.bundles.util.FileUtils; +import org.apache.metron.bundles.util.StringUtils; +import org.apache.metron.bundles.annotation.behavior.RequiresInstanceClassLoading; + +import org.atteo.classindex.ClassIndex; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.io.IOException; +import java.lang.reflect.Modifier; +import java.net.URI; +import java.net.URISyntaxException; +import java.net.URL; +import java.net.URLClassLoader; +import java.util.*; +import java.util.concurrent.ConcurrentHashMap; +import java.util.concurrent.atomic.AtomicBoolean; + +/** + * Scans through the classpath to load all extension components using the service provider API and + * running through all classloaders (root, BUNDLEs). + * + * @ThreadSafe - is immutable + */ +@SuppressWarnings("rawtypes") +public class ExtensionManager { + + private static final Logger logger = LoggerFactory.getLogger(ExtensionManager.class); + + public static final BundleCoordinates SYSTEM_BUNDLE_COORDINATE = new BundleCoordinates( + BundleCoordinates.DEFAULT_GROUP, "system", BundleCoordinates.DEFAULT_VERSION); + + // Maps a service definition (interface) to those classes that implement the interface + private static final Map> definitionMap = new HashMap<>(); + + private static final Map> classNameBundleLookup = new HashMap<>(); + private static final Map bundleCoordinateBundleLookup = new HashMap<>(); + private static final Map classLoaderBundleLookup = new HashMap<>(); + + private static final Set requiresInstanceClassLoading = new HashSet<>(); + private static final Map instanceClassloaderLookup = new ConcurrentHashMap<>(); + + private static AtomicBoolean inited = new AtomicBoolean(false); + + // should initialize class definitions + public static void initClassDefinitions(final List classes) { +if (classes != null) { + for (Class clazz : classes) { +definitionMap.put(clazz, new HashSet<>()); + } +} +inited.set(true); + } + + public static void resetClassDefinitions() { +definitionMap.clear(); +inited.set(false); + } + + /** + * Loads all extension class types that can be found on the bootstrap classloader and by creating + * classloaders for all BUNDLES found within the classpath. + * + * @param bundles the bundles to scan through in search of extensions + */ + public static void discoverExtensions(final Bundle systemBundle, final Set bundles) + throws NotInitializedException { +checkInitialized(); +// get the current context class loader +ClassLoader currentContextClassLoader = Thread.currentThread().getContextClassLoader(); + +// load the system bundle first so that any extensions found in JARs directly in lib will be registered as +// being from the system bundle and not from all the other Bundles +loadExtensions(systemBundle); + bundleCoordinateBundleLookup.put(systemBundle.getBundleDetails().getCoordinates(), systemBundle); + +// consider each bundle class loader +for (final Bundle bundle : bundles) { + // Must
[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/530#discussion_r133033031 --- Diff: bundles-lib/src/main/java/org/apache/metron/bundles/ExtensionMapping.java --- @@ -0,0 +1,104 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.metron.bundles; + +import org.apache.metron.bundles.bundle.BundleCoordinates; + +import java.util.*; +import java.util.function.BiFunction; + +public class ExtensionMapping { + + private final Map>> extensionNameMap = new HashMap<>(); --- End diff -- Note - that with the BundleSystem, the properties file IS now used for loading, unlike before - IF the classes are not explicitly set. I will have to write something to explain the ExtensionMappings vs. the ExtensionManager --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/530#discussion_r133030801 --- Diff: bundles-lib/src/main/java/org/apache/metron/bundles/ExtensionMapping.java --- @@ -0,0 +1,104 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.metron.bundles; + +import org.apache.metron.bundles.bundle.BundleCoordinates; + +import java.util.*; +import java.util.function.BiFunction; + +public class ExtensionMapping { + + private final Map>> extensionNameMap = new HashMap<>(); --- End diff -- Added. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/530#discussion_r133029181 --- Diff: bundles-lib/src/main/java/org/apache/metron/bundles/ExtensionManager.java --- @@ -0,0 +1,436 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.metron.bundles; + + +import org.apache.commons.vfs2.FileObject; +import org.apache.commons.vfs2.FileSystemException; +import org.apache.commons.vfs2.FileSystemManager; +import org.apache.metron.bundles.bundle.Bundle; +import org.apache.metron.bundles.bundle.BundleCoordinates; +import org.apache.metron.bundles.bundle.BundleDetails; +import org.apache.metron.bundles.util.BundleProperties; +import org.apache.metron.bundles.util.FileUtils; +import org.apache.metron.bundles.util.StringUtils; +import org.apache.metron.bundles.annotation.behavior.RequiresInstanceClassLoading; + +import org.atteo.classindex.ClassIndex; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.io.IOException; +import java.lang.reflect.Modifier; +import java.net.URI; +import java.net.URISyntaxException; +import java.net.URL; +import java.net.URLClassLoader; +import java.util.*; +import java.util.concurrent.ConcurrentHashMap; +import java.util.concurrent.atomic.AtomicBoolean; + +/** + * Scans through the classpath to load all extension components using the service provider API and + * running through all classloaders (root, BUNDLEs). + * + * @ThreadSafe - is immutable + */ +@SuppressWarnings("rawtypes") +public class ExtensionManager { + + private static final Logger logger = LoggerFactory.getLogger(ExtensionManager.class); + + public static final BundleCoordinates SYSTEM_BUNDLE_COORDINATE = new BundleCoordinates( + BundleCoordinates.DEFAULT_GROUP, "system", BundleCoordinates.DEFAULT_VERSION); + + // Maps a service definition (interface) to those classes that implement the interface + private static final Map> definitionMap = new HashMap<>(); + + private static final Map> classNameBundleLookup = new HashMap<>(); + private static final Map bundleCoordinateBundleLookup = new HashMap<>(); + private static final Map classLoaderBundleLookup = new HashMap<>(); + + private static final Set requiresInstanceClassLoading = new HashSet<>(); + private static final Map instanceClassloaderLookup = new ConcurrentHashMap<>(); + + private static AtomicBoolean inited = new AtomicBoolean(false); + + // should initialize class definitions + public static void initClassDefinitions(final List classes) { +if (classes != null) { + for (Class clazz : classes) { +definitionMap.put(clazz, new HashSet<>()); + } +} +inited.set(true); + } + + public static void resetClassDefinitions() { +definitionMap.clear(); +inited.set(false); + } + + /** + * Loads all extension class types that can be found on the bootstrap classloader and by creating + * classloaders for all BUNDLES found within the classpath. + * + * @param bundles the bundles to scan through in search of extensions + */ + public static void discoverExtensions(final Bundle systemBundle, final Set bundles) + throws NotInitializedException { +checkInitialized(); +// get the current context class loader +ClassLoader currentContextClassLoader = Thread.currentThread().getContextClassLoader(); + +// load the system bundle first so that any extensions found in JARs directly in lib will be registered as +// being from the system bundle and not from all the other Bundles +loadExtensions(systemBundle); + bundleCoordinateBundleLookup.put(systemBundle.getBundleDetails().getCoordinates(), systemBundle); + +// consider each bundle class loader +for (final Bundle bundle : bundles) { + // Must
[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...
Github user mattf-horton commented on a diff in the pull request: https://github.com/apache/metron/pull/530#discussion_r133016408 --- Diff: bundles-lib/src/main/java/org/apache/metron/bundles/ExtensionMapping.java --- @@ -0,0 +1,104 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.metron.bundles; + +import org.apache.metron.bundles.bundle.BundleCoordinates; + +import java.util.*; +import java.util.function.BiFunction; + +public class ExtensionMapping { + + private final Map>> extensionNameMap = new HashMap<>(); --- End diff -- also would be good to document in comments, for future spelunkers. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...
Github user mattf-horton commented on a diff in the pull request: https://github.com/apache/metron/pull/530#discussion_r133016174 --- Diff: bundles-lib/src/main/java/org/apache/metron/bundles/ExtensionManager.java --- @@ -0,0 +1,436 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.metron.bundles; + + +import org.apache.commons.vfs2.FileObject; +import org.apache.commons.vfs2.FileSystemException; +import org.apache.commons.vfs2.FileSystemManager; +import org.apache.metron.bundles.bundle.Bundle; +import org.apache.metron.bundles.bundle.BundleCoordinates; +import org.apache.metron.bundles.bundle.BundleDetails; +import org.apache.metron.bundles.util.BundleProperties; +import org.apache.metron.bundles.util.FileUtils; +import org.apache.metron.bundles.util.StringUtils; +import org.apache.metron.bundles.annotation.behavior.RequiresInstanceClassLoading; + +import org.atteo.classindex.ClassIndex; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.io.IOException; +import java.lang.reflect.Modifier; +import java.net.URI; +import java.net.URISyntaxException; +import java.net.URL; +import java.net.URLClassLoader; +import java.util.*; +import java.util.concurrent.ConcurrentHashMap; +import java.util.concurrent.atomic.AtomicBoolean; + +/** + * Scans through the classpath to load all extension components using the service provider API and + * running through all classloaders (root, BUNDLEs). + * + * @ThreadSafe - is immutable + */ +@SuppressWarnings("rawtypes") +public class ExtensionManager { + + private static final Logger logger = LoggerFactory.getLogger(ExtensionManager.class); + + public static final BundleCoordinates SYSTEM_BUNDLE_COORDINATE = new BundleCoordinates( + BundleCoordinates.DEFAULT_GROUP, "system", BundleCoordinates.DEFAULT_VERSION); + + // Maps a service definition (interface) to those classes that implement the interface + private static final Map> definitionMap = new HashMap<>(); + + private static final Map> classNameBundleLookup = new HashMap<>(); + private static final Map bundleCoordinateBundleLookup = new HashMap<>(); + private static final Map classLoaderBundleLookup = new HashMap<>(); + + private static final Set requiresInstanceClassLoading = new HashSet<>(); + private static final Map instanceClassloaderLookup = new ConcurrentHashMap<>(); + + private static AtomicBoolean inited = new AtomicBoolean(false); + + // should initialize class definitions + public static void initClassDefinitions(final List classes) { +if (classes != null) { + for (Class clazz : classes) { +definitionMap.put(clazz, new HashSet<>()); + } +} +inited.set(true); + } + + public static void resetClassDefinitions() { +definitionMap.clear(); +inited.set(false); + } + + /** + * Loads all extension class types that can be found on the bootstrap classloader and by creating + * classloaders for all BUNDLES found within the classpath. + * + * @param bundles the bundles to scan through in search of extensions + */ + public static void discoverExtensions(final Bundle systemBundle, final Set bundles) + throws NotInitializedException { +checkInitialized(); +// get the current context class loader +ClassLoader currentContextClassLoader = Thread.currentThread().getContextClassLoader(); + +// load the system bundle first so that any extensions found in JARs directly in lib will be registered as +// being from the system bundle and not from all the other Bundles +loadExtensions(systemBundle); + bundleCoordinateBundleLookup.put(systemBundle.getBundleDetails().getCoordinates(), systemBundle); + +// consider each bundle class loader +for (final Bundle bundle : bundles) { + // Must s
[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/530#discussion_r132820993 --- Diff: bundles-lib/src/main/java/org/apache/metron/bundles/ExtensionManager.java --- @@ -0,0 +1,436 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.metron.bundles; + + +import org.apache.commons.vfs2.FileObject; +import org.apache.commons.vfs2.FileSystemException; +import org.apache.commons.vfs2.FileSystemManager; +import org.apache.metron.bundles.bundle.Bundle; +import org.apache.metron.bundles.bundle.BundleCoordinates; +import org.apache.metron.bundles.bundle.BundleDetails; +import org.apache.metron.bundles.util.BundleProperties; +import org.apache.metron.bundles.util.FileUtils; +import org.apache.metron.bundles.util.StringUtils; +import org.apache.metron.bundles.annotation.behavior.RequiresInstanceClassLoading; + +import org.atteo.classindex.ClassIndex; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.io.IOException; +import java.lang.reflect.Modifier; +import java.net.URI; +import java.net.URISyntaxException; +import java.net.URL; +import java.net.URLClassLoader; +import java.util.*; +import java.util.concurrent.ConcurrentHashMap; +import java.util.concurrent.atomic.AtomicBoolean; + +/** + * Scans through the classpath to load all extension components using the service provider API and + * running through all classloaders (root, BUNDLEs). + * + * @ThreadSafe - is immutable --- End diff -- I agree, and I am working on a refactoring. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/530#discussion_r132820984 --- Diff: bundles-lib/src/main/java/org/apache/metron/bundles/ExtensionManager.java --- @@ -0,0 +1,436 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.metron.bundles; + + +import org.apache.commons.vfs2.FileObject; +import org.apache.commons.vfs2.FileSystemException; +import org.apache.commons.vfs2.FileSystemManager; +import org.apache.metron.bundles.bundle.Bundle; +import org.apache.metron.bundles.bundle.BundleCoordinates; +import org.apache.metron.bundles.bundle.BundleDetails; +import org.apache.metron.bundles.util.BundleProperties; +import org.apache.metron.bundles.util.FileUtils; +import org.apache.metron.bundles.util.StringUtils; +import org.apache.metron.bundles.annotation.behavior.RequiresInstanceClassLoading; + +import org.atteo.classindex.ClassIndex; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.io.IOException; +import java.lang.reflect.Modifier; +import java.net.URI; +import java.net.URISyntaxException; +import java.net.URL; +import java.net.URLClassLoader; +import java.util.*; +import java.util.concurrent.ConcurrentHashMap; +import java.util.concurrent.atomic.AtomicBoolean; + +/** + * Scans through the classpath to load all extension components using the service provider API and + * running through all classloaders (root, BUNDLEs). + * + * @ThreadSafe - is immutable + */ +@SuppressWarnings("rawtypes") +public class ExtensionManager { + + private static final Logger logger = LoggerFactory.getLogger(ExtensionManager.class); + + public static final BundleCoordinates SYSTEM_BUNDLE_COORDINATE = new BundleCoordinates( + BundleCoordinates.DEFAULT_GROUP, "system", BundleCoordinates.DEFAULT_VERSION); + + // Maps a service definition (interface) to those classes that implement the interface + private static final Map> definitionMap = new HashMap<>(); + + private static final Map> classNameBundleLookup = new HashMap<>(); + private static final Map bundleCoordinateBundleLookup = new HashMap<>(); + private static final Map classLoaderBundleLookup = new HashMap<>(); + + private static final Set requiresInstanceClassLoading = new HashSet<>(); + private static final Map instanceClassloaderLookup = new ConcurrentHashMap<>(); + + private static AtomicBoolean inited = new AtomicBoolean(false); + + // should initialize class definitions + public static void initClassDefinitions(final List classes) { +if (classes != null) { + for (Class clazz : classes) { +definitionMap.put(clazz, new HashSet<>()); + } +} +inited.set(true); + } + + public static void resetClassDefinitions() { +definitionMap.clear(); +inited.set(false); + } + + /** + * Loads all extension class types that can be found on the bootstrap classloader and by creating + * classloaders for all BUNDLES found within the classpath. + * + * @param bundles the bundles to scan through in search of extensions + */ + public static void discoverExtensions(final Bundle systemBundle, final Set bundles) + throws NotInitializedException { +checkInitialized(); +// get the current context class loader +ClassLoader currentContextClassLoader = Thread.currentThread().getContextClassLoader(); + +// load the system bundle first so that any extensions found in JARs directly in lib will be registered as +// being from the system bundle and not from all the other Bundles +loadExtensions(systemBundle); + bundleCoordinateBundleLookup.put(systemBundle.getBundleDetails().getCoordinates(), systemBundle); + +// consider each bundle class loader +for (final Bundle bundle : bundles) { + // Must
[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/530#discussion_r132820881 --- Diff: bundles-lib/src/main/java/org/apache/metron/bundles/VFSBundleClassLoader.java --- @@ -0,0 +1,515 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.metron.bundles; + +import java.io.IOException; +import java.net.URL; +import java.security.CodeSource; +import java.security.Permission; +import java.security.PermissionCollection; +import java.security.Permissions; +import java.security.SecureClassLoader; +import java.security.cert.Certificate; +import java.util.ArrayList; +import java.util.Collections; +import java.util.Enumeration; +import java.util.Iterator; +import java.util.List; +import java.util.jar.Attributes; +import java.util.jar.Attributes.Name; +import org.apache.commons.vfs2.FileObject; +import org.apache.commons.vfs2.FileSystemException; +import org.apache.commons.vfs2.FileSystemManager; +import org.apache.commons.vfs2.NameScope; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + + +/** + * + * A ClassLoader for loading BUNDLES (plugin archives). BUNDLEs are designed to + * allow isolating bundles of code (comprising one-or-more + * plugin classes and their + * dependencies) from other such bundles; this allows for dependencies and + * processors that require conflicting, incompatible versions of the same + * dependency to run in a single instance of a given process. + * + * + * BundleClassLoader follows the delegation model described in + * {@link ClassLoader#findClass(java.lang.String) ClassLoader.findClass(...)}; + * classes are first loaded from the parent ClassLoader, and only if + * they cannot be found there does the BundleClassLoader provide a + * definition. Specifically, this means that resources are loaded from the application's + * conf + * and lib directories first, and if they cannot be found there, are + * loaded from the BUNDLE. + * + * + * The packaging of a BUNDLE is such that it is a ZIP file with the following + * directory structure: + * + * + * +META-INF/ + * +-- bundled-dependencies/ + * +--+ * +-- MANIFEST.MF + * + * + * + * + * The MANIFEST.MF file contains the same information as a typical JAR file but + * also includes two additional bundle properties: {@code Bundle-Id} and + * {@code Bundle-Dependency-Id}. + * + * + * + * The {@code Bundle-Id} provides a unique identifier for this BUNDLE. + * + * + * + * The {@code Bundle-Dependency-Id} is optional. If provided, it indicates that + * this BUNDLE should inherit all of the dependencies of the BUNDLE with the provided + * ID. Often times, the BUNDLE that is depended upon is referred to as the Parent. + * This is because its ClassLoader will be the parent ClassLoader of the + * dependent BUNDLE. + * + * + * + * If a BUNDLE is built using the Bundles Maven Plugin, the {@code Bundle-Id} property + * will be set to the artifactId of the BUNDLE. The {@code Bundle-Dependency-Id} will + * be set to the artifactId of the BUNDLE that is depended upon. For example, if + * BUNDLE A is defined as such: + * + * + * ... + * bundle-a + *bundle + * ... + *+ * + * + * + * + * + * + * Then the MANIFEST.MF file that is created for Bundle A will have the following + * properties set: + * + * {@code {Foo}-Id: bundle-a} + * {@code {Foo}-Dependency-Id: bundle-z} + * + * Where is configurable by BundleProperty META_ID_PREFIX [ default Bundle ] + *+ * + *group + *bundle-z + *bundle + *
[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/530#discussion_r132808269 --- Diff: bundles-lib/src/main/java/org/apache/metron/bundles/ExtensionManager.java --- @@ -0,0 +1,436 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.metron.bundles; + + +import org.apache.commons.vfs2.FileObject; +import org.apache.commons.vfs2.FileSystemException; +import org.apache.commons.vfs2.FileSystemManager; +import org.apache.metron.bundles.bundle.Bundle; +import org.apache.metron.bundles.bundle.BundleCoordinates; +import org.apache.metron.bundles.bundle.BundleDetails; +import org.apache.metron.bundles.util.BundleProperties; +import org.apache.metron.bundles.util.FileUtils; +import org.apache.metron.bundles.util.StringUtils; +import org.apache.metron.bundles.annotation.behavior.RequiresInstanceClassLoading; + +import org.atteo.classindex.ClassIndex; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.io.IOException; +import java.lang.reflect.Modifier; +import java.net.URI; +import java.net.URISyntaxException; +import java.net.URL; +import java.net.URLClassLoader; +import java.util.*; +import java.util.concurrent.ConcurrentHashMap; +import java.util.concurrent.atomic.AtomicBoolean; + +/** + * Scans through the classpath to load all extension components using the service provider API and + * running through all classloaders (root, BUNDLEs). + * + * @ThreadSafe - is immutable + */ +@SuppressWarnings("rawtypes") +public class ExtensionManager { + + private static final Logger logger = LoggerFactory.getLogger(ExtensionManager.class); + + public static final BundleCoordinates SYSTEM_BUNDLE_COORDINATE = new BundleCoordinates( + BundleCoordinates.DEFAULT_GROUP, "system", BundleCoordinates.DEFAULT_VERSION); + + // Maps a service definition (interface) to those classes that implement the interface + private static final Map> definitionMap = new HashMap<>(); + + private static final Map> classNameBundleLookup = new HashMap<>(); + private static final Map bundleCoordinateBundleLookup = new HashMap<>(); + private static final Map classLoaderBundleLookup = new HashMap<>(); + + private static final Set requiresInstanceClassLoading = new HashSet<>(); + private static final Map instanceClassloaderLookup = new ConcurrentHashMap<>(); + + private static AtomicBoolean inited = new AtomicBoolean(false); + + // should initialize class definitions + public static void initClassDefinitions(final List classes) { +if (classes != null) { + for (Class clazz : classes) { +definitionMap.put(clazz, new HashSet<>()); + } +} +inited.set(true); + } + + public static void resetClassDefinitions() { +definitionMap.clear(); +inited.set(false); + } + + /** + * Loads all extension class types that can be found on the bootstrap classloader and by creating + * classloaders for all BUNDLES found within the classpath. + * + * @param bundles the bundles to scan through in search of extensions + */ + public static void discoverExtensions(final Bundle systemBundle, final Set bundles) + throws NotInitializedException { +checkInitialized(); +// get the current context class loader +ClassLoader currentContextClassLoader = Thread.currentThread().getContextClassLoader(); + +// load the system bundle first so that any extensions found in JARs directly in lib will be registered as +// being from the system bundle and not from all the other Bundles +loadExtensions(systemBundle); + bundleCoordinateBundleLookup.put(systemBundle.getBundleDetails().getCoordinates(), systemBundle); + +// consider each bundle class loader +for (final Bundle bundle : bundles) { + // Must
[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/530#discussion_r132808241 --- Diff: bundles-lib/src/main/java/org/apache/metron/bundles/ExtensionManager.java --- @@ -0,0 +1,436 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.metron.bundles; + + +import org.apache.commons.vfs2.FileObject; +import org.apache.commons.vfs2.FileSystemException; +import org.apache.commons.vfs2.FileSystemManager; +import org.apache.metron.bundles.bundle.Bundle; +import org.apache.metron.bundles.bundle.BundleCoordinates; +import org.apache.metron.bundles.bundle.BundleDetails; +import org.apache.metron.bundles.util.BundleProperties; +import org.apache.metron.bundles.util.FileUtils; +import org.apache.metron.bundles.util.StringUtils; +import org.apache.metron.bundles.annotation.behavior.RequiresInstanceClassLoading; + +import org.atteo.classindex.ClassIndex; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.io.IOException; +import java.lang.reflect.Modifier; +import java.net.URI; +import java.net.URISyntaxException; +import java.net.URL; +import java.net.URLClassLoader; +import java.util.*; +import java.util.concurrent.ConcurrentHashMap; +import java.util.concurrent.atomic.AtomicBoolean; + +/** + * Scans through the classpath to load all extension components using the service provider API and + * running through all classloaders (root, BUNDLEs). + * + * @ThreadSafe - is immutable + */ +@SuppressWarnings("rawtypes") +public class ExtensionManager { + + private static final Logger logger = LoggerFactory.getLogger(ExtensionManager.class); + + public static final BundleCoordinates SYSTEM_BUNDLE_COORDINATE = new BundleCoordinates( + BundleCoordinates.DEFAULT_GROUP, "system", BundleCoordinates.DEFAULT_VERSION); + + // Maps a service definition (interface) to those classes that implement the interface + private static final Map> definitionMap = new HashMap<>(); + + private static final Map> classNameBundleLookup = new HashMap<>(); + private static final Map bundleCoordinateBundleLookup = new HashMap<>(); + private static final Map classLoaderBundleLookup = new HashMap<>(); + + private static final Set requiresInstanceClassLoading = new HashSet<>(); + private static final Map instanceClassloaderLookup = new ConcurrentHashMap<>(); + + private static AtomicBoolean inited = new AtomicBoolean(false); + + // should initialize class definitions + public static void initClassDefinitions(final List classes) { +if (classes != null) { + for (Class clazz : classes) { +definitionMap.put(clazz, new HashSet<>()); + } +} +inited.set(true); + } + + public static void resetClassDefinitions() { +definitionMap.clear(); +inited.set(false); + } + + /** + * Loads all extension class types that can be found on the bootstrap classloader and by creating + * classloaders for all BUNDLES found within the classpath. + * + * @param bundles the bundles to scan through in search of extensions + */ + public static void discoverExtensions(final Bundle systemBundle, final Set bundles) + throws NotInitializedException { +checkInitialized(); +// get the current context class loader +ClassLoader currentContextClassLoader = Thread.currentThread().getContextClassLoader(); + +// load the system bundle first so that any extensions found in JARs directly in lib will be registered as +// being from the system bundle and not from all the other Bundles +loadExtensions(systemBundle); + bundleCoordinateBundleLookup.put(systemBundle.getBundleDetails().getCoordinates(), systemBundle); + +// consider each bundle class loader +for (final Bundle bundle : bundles) { + // Must
[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/530#discussion_r132808147 --- Diff: bundles-lib/src/main/java/org/apache/metron/bundles/ExtensionManager.java --- @@ -0,0 +1,436 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.metron.bundles; + + +import org.apache.commons.vfs2.FileObject; +import org.apache.commons.vfs2.FileSystemException; +import org.apache.commons.vfs2.FileSystemManager; +import org.apache.metron.bundles.bundle.Bundle; +import org.apache.metron.bundles.bundle.BundleCoordinates; +import org.apache.metron.bundles.bundle.BundleDetails; +import org.apache.metron.bundles.util.BundleProperties; +import org.apache.metron.bundles.util.FileUtils; +import org.apache.metron.bundles.util.StringUtils; +import org.apache.metron.bundles.annotation.behavior.RequiresInstanceClassLoading; + +import org.atteo.classindex.ClassIndex; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.io.IOException; +import java.lang.reflect.Modifier; +import java.net.URI; +import java.net.URISyntaxException; +import java.net.URL; +import java.net.URLClassLoader; +import java.util.*; +import java.util.concurrent.ConcurrentHashMap; +import java.util.concurrent.atomic.AtomicBoolean; + +/** + * Scans through the classpath to load all extension components using the service provider API and + * running through all classloaders (root, BUNDLEs). + * + * @ThreadSafe - is immutable + */ +@SuppressWarnings("rawtypes") +public class ExtensionManager { + + private static final Logger logger = LoggerFactory.getLogger(ExtensionManager.class); + + public static final BundleCoordinates SYSTEM_BUNDLE_COORDINATE = new BundleCoordinates( + BundleCoordinates.DEFAULT_GROUP, "system", BundleCoordinates.DEFAULT_VERSION); + + // Maps a service definition (interface) to those classes that implement the interface + private static final Map> definitionMap = new HashMap<>(); + + private static final Map> classNameBundleLookup = new HashMap<>(); + private static final Map bundleCoordinateBundleLookup = new HashMap<>(); + private static final Map classLoaderBundleLookup = new HashMap<>(); + + private static final Set requiresInstanceClassLoading = new HashSet<>(); + private static final Map instanceClassloaderLookup = new ConcurrentHashMap<>(); + + private static AtomicBoolean inited = new AtomicBoolean(false); + + // should initialize class definitions + public static void initClassDefinitions(final List classes) { +if (classes != null) { + for (Class clazz : classes) { +definitionMap.put(clazz, new HashSet<>()); + } +} +inited.set(true); + } + + public static void resetClassDefinitions() { +definitionMap.clear(); +inited.set(false); + } + + /** + * Loads all extension class types that can be found on the bootstrap classloader and by creating + * classloaders for all BUNDLES found within the classpath. + * + * @param bundles the bundles to scan through in search of extensions + */ + public static void discoverExtensions(final Bundle systemBundle, final Set bundles) + throws NotInitializedException { +checkInitialized(); +// get the current context class loader +ClassLoader currentContextClassLoader = Thread.currentThread().getContextClassLoader(); + +// load the system bundle first so that any extensions found in JARs directly in lib will be registered as +// being from the system bundle and not from all the other Bundles +loadExtensions(systemBundle); + bundleCoordinateBundleLookup.put(systemBundle.getBundleDetails().getCoordinates(), systemBundle); + +// consider each bundle class loader +for (final Bundle bundle : bundles) { + // Must
[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/530#discussion_r132808084 --- Diff: bundles-lib/src/main/java/org/apache/metron/bundles/ExtensionMapping.java --- @@ -0,0 +1,104 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.metron.bundles; + +import org.apache.metron.bundles.bundle.BundleCoordinates; + +import java.util.*; +import java.util.function.BiFunction; + +public class ExtensionMapping { + + private final Map>> extensionNameMap = new HashMap<>(); --- End diff -- So - the goal with the Extension* stuff was to remove the necessity to statically declare the classes of extensions used. They are passed in to the ExtensionManager and the BundleThreadContextClassLoader. The ExtensionMapping is different, it ends up getting the classes from discover in the nifi properties, where the classes are defined, and typed. So yes, the type does subdivide the namespace. Type -> Classname -> coordinates Parser -> MessageParser -> Parser Extension Coordinates --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/530#discussion_r132807655 --- Diff: bundles-lib/src/main/java/org/apache/metron/bundles/BundleThreadContextClassLoader.java --- @@ -0,0 +1,221 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.metron.bundles; + +import java.io.IOException; +import java.io.InputStream; +import java.lang.reflect.Constructor; +import java.lang.reflect.InvocationTargetException; +import java.net.URL; +import java.net.URLClassLoader; +import java.util.ArrayList; +import java.util.Enumeration; +import java.util.List; +import java.util.concurrent.atomic.AtomicBoolean; + +import org.apache.metron.bundles.bundle.Bundle; +import org.apache.metron.bundles.util.BundleProperties; + +/** + * THREAD SAFE + */ +public class BundleThreadContextClassLoader extends URLClassLoader { + +static final ContextSecurityManager contextSecurityManager = new ContextSecurityManager(); +private final ClassLoader forward = ClassLoader.getSystemClassLoader(); +private static final List> bundleSpecificClasses = new ArrayList<>(); +private static AtomicBoolean inited = new AtomicBoolean(false); +// should initialize class definitions +public static void initClasses(final List classes){ --- End diff -- I don't like them either. I thought about DI and other things, but I was avoiding major changes of that type. I'm more willing now to part ways. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...
Github user mattf-horton commented on a diff in the pull request: https://github.com/apache/metron/pull/530#discussion_r132801000 --- Diff: bundles-lib/src/main/java/org/apache/metron/bundles/BundleThreadContextClassLoader.java --- @@ -0,0 +1,221 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.metron.bundles; + +import java.io.IOException; +import java.io.InputStream; +import java.lang.reflect.Constructor; +import java.lang.reflect.InvocationTargetException; +import java.net.URL; +import java.net.URLClassLoader; +import java.util.ArrayList; +import java.util.Enumeration; +import java.util.List; +import java.util.concurrent.atomic.AtomicBoolean; + +import org.apache.metron.bundles.bundle.Bundle; +import org.apache.metron.bundles.util.BundleProperties; + +/** + * THREAD SAFE + */ +public class BundleThreadContextClassLoader extends URLClassLoader { + +static final ContextSecurityManager contextSecurityManager = new ContextSecurityManager(); +private final ClassLoader forward = ClassLoader.getSystemClassLoader(); +private static final List> bundleSpecificClasses = new ArrayList<>(); +private static AtomicBoolean inited = new AtomicBoolean(false); +// should initialize class definitions +public static void initClasses(final List classes){ +bundleSpecificClasses.clear(); +if(classes != null){ +for( Class clazz : classes){ +bundleSpecificClasses.add(clazz); +} +} +inited.set(true); +} + +public static void resetClasses(){ +bundleSpecificClasses.clear(); +inited.set(false); +} + +private BundleThreadContextClassLoader() { +super(new URL[0]); +} + +@Override +public void clearAssertionStatus() { +lookupClassLoader().clearAssertionStatus(); +} + +@Override +public URL getResource(String name) { +return lookupClassLoader().getResource(name); +} + +@Override +public InputStream getResourceAsStream(String name) { +return lookupClassLoader().getResourceAsStream(name); +} + +@Override +public Enumeration getResources(String name) throws IOException { +return lookupClassLoader().getResources(name); +} + +@Override +public Class loadClass(String name) throws ClassNotFoundException { +return lookupClassLoader().loadClass(name); +} + +@Override +public void setClassAssertionStatus(String className, boolean enabled) { +lookupClassLoader().setClassAssertionStatus(className, enabled); +} + +@Override +public void setDefaultAssertionStatus(boolean enabled) { +lookupClassLoader().setDefaultAssertionStatus(enabled); +} + +@Override +public void setPackageAssertionStatus(String packageName, boolean enabled) { +lookupClassLoader().setPackageAssertionStatus(packageName, enabled); +} + +private ClassLoader lookupClassLoader() { +final Class[] classStack = contextSecurityManager.getExecutionStack(); + +for (Class currentClass : classStack) { +final Class bundleClass = findBundleClass(currentClass); +if (bundleClass != null) { +final ClassLoader desiredClassLoader = bundleClass.getClassLoader(); + +// When new Threads are created, the new Thread inherits the ClassLoaderContext of +// the caller. However, the call stack of that new Thread may not trace back to any app-specific +// code. Therefore, the BundleThreadContextClassLoader will be unable to find the appropriate Bundle +// ClassLoader. As a result, we want to set the ContextClassLoader to the Bundel ClassLoader th
[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...
Github user mattf-horton commented on a diff in the pull request: https://github.com/apache/metron/pull/530#discussion_r132797108 --- Diff: bundles-lib/src/main/java/org/apache/metron/bundles/ExtensionManager.java --- @@ -0,0 +1,436 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.metron.bundles; + + +import org.apache.commons.vfs2.FileObject; +import org.apache.commons.vfs2.FileSystemException; +import org.apache.commons.vfs2.FileSystemManager; +import org.apache.metron.bundles.bundle.Bundle; +import org.apache.metron.bundles.bundle.BundleCoordinates; +import org.apache.metron.bundles.bundle.BundleDetails; +import org.apache.metron.bundles.util.BundleProperties; +import org.apache.metron.bundles.util.FileUtils; +import org.apache.metron.bundles.util.StringUtils; +import org.apache.metron.bundles.annotation.behavior.RequiresInstanceClassLoading; + +import org.atteo.classindex.ClassIndex; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.io.IOException; +import java.lang.reflect.Modifier; +import java.net.URI; +import java.net.URISyntaxException; +import java.net.URL; +import java.net.URLClassLoader; +import java.util.*; +import java.util.concurrent.ConcurrentHashMap; +import java.util.concurrent.atomic.AtomicBoolean; + +/** + * Scans through the classpath to load all extension components using the service provider API and + * running through all classloaders (root, BUNDLEs). + * + * @ThreadSafe - is immutable + */ +@SuppressWarnings("rawtypes") +public class ExtensionManager { + + private static final Logger logger = LoggerFactory.getLogger(ExtensionManager.class); + + public static final BundleCoordinates SYSTEM_BUNDLE_COORDINATE = new BundleCoordinates( + BundleCoordinates.DEFAULT_GROUP, "system", BundleCoordinates.DEFAULT_VERSION); + + // Maps a service definition (interface) to those classes that implement the interface + private static final Map> definitionMap = new HashMap<>(); + + private static final Map> classNameBundleLookup = new HashMap<>(); + private static final Map bundleCoordinateBundleLookup = new HashMap<>(); + private static final Map classLoaderBundleLookup = new HashMap<>(); + + private static final Set requiresInstanceClassLoading = new HashSet<>(); + private static final Map instanceClassloaderLookup = new ConcurrentHashMap<>(); + + private static AtomicBoolean inited = new AtomicBoolean(false); + + // should initialize class definitions + public static void initClassDefinitions(final List classes) { +if (classes != null) { + for (Class clazz : classes) { +definitionMap.put(clazz, new HashSet<>()); + } +} +inited.set(true); + } + + public static void resetClassDefinitions() { +definitionMap.clear(); +inited.set(false); + } + + /** + * Loads all extension class types that can be found on the bootstrap classloader and by creating + * classloaders for all BUNDLES found within the classpath. + * + * @param bundles the bundles to scan through in search of extensions + */ + public static void discoverExtensions(final Bundle systemBundle, final Set bundles) + throws NotInitializedException { +checkInitialized(); +// get the current context class loader +ClassLoader currentContextClassLoader = Thread.currentThread().getContextClassLoader(); + +// load the system bundle first so that any extensions found in JARs directly in lib will be registered as +// being from the system bundle and not from all the other Bundles +loadExtensions(systemBundle); + bundleCoordinateBundleLookup.put(systemBundle.getBundleDetails().getCoordinates(), systemBundle); + +// consider each bundle class loader +for (final Bundle bundle : bundles) { + // Must s
[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...
Github user mattf-horton commented on a diff in the pull request: https://github.com/apache/metron/pull/530#discussion_r132799099 --- Diff: bundles-lib/src/main/java/org/apache/metron/bundles/ExtensionMapping.java --- @@ -0,0 +1,104 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.metron.bundles; + +import org.apache.metron.bundles.bundle.BundleCoordinates; + +import java.util.*; +import java.util.function.BiFunction; + +public class ExtensionMapping { + + private final Map>> extensionNameMap = new HashMap<>(); --- End diff -- Nifi had separate namespaces for the different categories of extension. This collapses it into one namespace. That's probably okay, but worth noting. Or does the later use of "types" subdivide in a different way? I didn't fully follow. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...
Github user mattf-horton commented on a diff in the pull request: https://github.com/apache/metron/pull/530#discussion_r132800554 --- Diff: bundles-lib/src/main/java/org/apache/metron/bundles/BundleThreadContextClassLoader.java --- @@ -0,0 +1,221 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.metron.bundles; + +import java.io.IOException; +import java.io.InputStream; +import java.lang.reflect.Constructor; +import java.lang.reflect.InvocationTargetException; +import java.net.URL; +import java.net.URLClassLoader; +import java.util.ArrayList; +import java.util.Enumeration; +import java.util.List; +import java.util.concurrent.atomic.AtomicBoolean; + +import org.apache.metron.bundles.bundle.Bundle; +import org.apache.metron.bundles.util.BundleProperties; + +/** + * THREAD SAFE + */ +public class BundleThreadContextClassLoader extends URLClassLoader { + +static final ContextSecurityManager contextSecurityManager = new ContextSecurityManager(); +private final ClassLoader forward = ClassLoader.getSystemClassLoader(); +private static final List> bundleSpecificClasses = new ArrayList<>(); +private static AtomicBoolean inited = new AtomicBoolean(false); +// should initialize class definitions +public static void initClasses(final List classes){ --- End diff -- The comparable code in Nifi used a static initializer to init the classes list. This uses a public method, and an AtomicBoolean flag to say it's been done. Why doesn't the initClasses() method need to be synchronized? (BTW, I really dislike these all-static or mostly-static classes, which seem to be trying to be singletons without engaging in the discipline of singleton-ness, especially wrt thread safeness. But I understand they originated in the nifi code, so I'm not after you about that :-) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...
Github user mattf-horton commented on a diff in the pull request: https://github.com/apache/metron/pull/530#discussion_r132797485 --- Diff: bundles-lib/src/main/java/org/apache/metron/bundles/ExtensionManager.java --- @@ -0,0 +1,436 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.metron.bundles; + + +import org.apache.commons.vfs2.FileObject; +import org.apache.commons.vfs2.FileSystemException; +import org.apache.commons.vfs2.FileSystemManager; +import org.apache.metron.bundles.bundle.Bundle; +import org.apache.metron.bundles.bundle.BundleCoordinates; +import org.apache.metron.bundles.bundle.BundleDetails; +import org.apache.metron.bundles.util.BundleProperties; +import org.apache.metron.bundles.util.FileUtils; +import org.apache.metron.bundles.util.StringUtils; +import org.apache.metron.bundles.annotation.behavior.RequiresInstanceClassLoading; + +import org.atteo.classindex.ClassIndex; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.io.IOException; +import java.lang.reflect.Modifier; +import java.net.URI; +import java.net.URISyntaxException; +import java.net.URL; +import java.net.URLClassLoader; +import java.util.*; +import java.util.concurrent.ConcurrentHashMap; +import java.util.concurrent.atomic.AtomicBoolean; + +/** + * Scans through the classpath to load all extension components using the service provider API and + * running through all classloaders (root, BUNDLEs). + * + * @ThreadSafe - is immutable + */ +@SuppressWarnings("rawtypes") +public class ExtensionManager { + + private static final Logger logger = LoggerFactory.getLogger(ExtensionManager.class); + + public static final BundleCoordinates SYSTEM_BUNDLE_COORDINATE = new BundleCoordinates( + BundleCoordinates.DEFAULT_GROUP, "system", BundleCoordinates.DEFAULT_VERSION); + + // Maps a service definition (interface) to those classes that implement the interface + private static final Map> definitionMap = new HashMap<>(); + + private static final Map> classNameBundleLookup = new HashMap<>(); + private static final Map bundleCoordinateBundleLookup = new HashMap<>(); + private static final Map classLoaderBundleLookup = new HashMap<>(); + + private static final Set requiresInstanceClassLoading = new HashSet<>(); + private static final Map instanceClassloaderLookup = new ConcurrentHashMap<>(); + + private static AtomicBoolean inited = new AtomicBoolean(false); + + // should initialize class definitions + public static void initClassDefinitions(final List classes) { +if (classes != null) { + for (Class clazz : classes) { +definitionMap.put(clazz, new HashSet<>()); + } +} +inited.set(true); + } + + public static void resetClassDefinitions() { +definitionMap.clear(); +inited.set(false); + } + + /** + * Loads all extension class types that can be found on the bootstrap classloader and by creating + * classloaders for all BUNDLES found within the classpath. + * + * @param bundles the bundles to scan through in search of extensions + */ + public static void discoverExtensions(final Bundle systemBundle, final Set bundles) + throws NotInitializedException { +checkInitialized(); +// get the current context class loader +ClassLoader currentContextClassLoader = Thread.currentThread().getContextClassLoader(); + +// load the system bundle first so that any extensions found in JARs directly in lib will be registered as +// being from the system bundle and not from all the other Bundles +loadExtensions(systemBundle); + bundleCoordinateBundleLookup.put(systemBundle.getBundleDetails().getCoordinates(), systemBundle); + +// consider each bundle class loader +for (final Bundle bundle : bundles) { + // Must s
[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...
Github user mattf-horton commented on a diff in the pull request: https://github.com/apache/metron/pull/530#discussion_r132801955 --- Diff: bundles-lib/src/main/java/org/apache/metron/bundles/VFSBundleClassLoader.java --- @@ -0,0 +1,515 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.metron.bundles; + +import java.io.IOException; +import java.net.URL; +import java.security.CodeSource; +import java.security.Permission; +import java.security.PermissionCollection; +import java.security.Permissions; +import java.security.SecureClassLoader; +import java.security.cert.Certificate; +import java.util.ArrayList; +import java.util.Collections; +import java.util.Enumeration; +import java.util.Iterator; +import java.util.List; +import java.util.jar.Attributes; +import java.util.jar.Attributes.Name; +import org.apache.commons.vfs2.FileObject; +import org.apache.commons.vfs2.FileSystemException; +import org.apache.commons.vfs2.FileSystemManager; +import org.apache.commons.vfs2.NameScope; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + + +/** + * + * A ClassLoader for loading BUNDLES (plugin archives). BUNDLEs are designed to + * allow isolating bundles of code (comprising one-or-more + * plugin classes and their + * dependencies) from other such bundles; this allows for dependencies and + * processors that require conflicting, incompatible versions of the same + * dependency to run in a single instance of a given process. + * + * + * BundleClassLoader follows the delegation model described in + * {@link ClassLoader#findClass(java.lang.String) ClassLoader.findClass(...)}; + * classes are first loaded from the parent ClassLoader, and only if + * they cannot be found there does the BundleClassLoader provide a + * definition. Specifically, this means that resources are loaded from the application's + * conf + * and lib directories first, and if they cannot be found there, are + * loaded from the BUNDLE. + * + * + * The packaging of a BUNDLE is such that it is a ZIP file with the following + * directory structure: + * + * + * +META-INF/ + * +-- bundled-dependencies/ + * +--+ * +-- MANIFEST.MF + * + * + * + * + * The MANIFEST.MF file contains the same information as a typical JAR file but + * also includes two additional bundle properties: {@code Bundle-Id} and + * {@code Bundle-Dependency-Id}. + * + * + * + * The {@code Bundle-Id} provides a unique identifier for this BUNDLE. + * + * + * + * The {@code Bundle-Dependency-Id} is optional. If provided, it indicates that + * this BUNDLE should inherit all of the dependencies of the BUNDLE with the provided + * ID. Often times, the BUNDLE that is depended upon is referred to as the Parent. + * This is because its ClassLoader will be the parent ClassLoader of the + * dependent BUNDLE. + * + * + * + * If a BUNDLE is built using the Bundles Maven Plugin, the {@code Bundle-Id} property + * will be set to the artifactId of the BUNDLE. The {@code Bundle-Dependency-Id} will + * be set to the artifactId of the BUNDLE that is depended upon. For example, if + * BUNDLE A is defined as such: + * + * + * ... + * bundle-a + *bundle + * ... + *+ * + * + * + * + * + * + * Then the MANIFEST.MF file that is created for Bundle A will have the following + * properties set: + * + * {@code {Foo}-Id: bundle-a} + * {@code {Foo}-Dependency-Id: bundle-z} + * + * Where is configurable by BundleProperty META_ID_PREFIX [ default Bundle ] + *+ * + *group + *bundle-z + *bundle + *
[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...
Github user mattf-horton commented on a diff in the pull request: https://github.com/apache/metron/pull/530#discussion_r132798637 --- Diff: bundles-lib/src/main/java/org/apache/metron/bundles/ExtensionManager.java --- @@ -0,0 +1,436 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.metron.bundles; + + +import org.apache.commons.vfs2.FileObject; +import org.apache.commons.vfs2.FileSystemException; +import org.apache.commons.vfs2.FileSystemManager; +import org.apache.metron.bundles.bundle.Bundle; +import org.apache.metron.bundles.bundle.BundleCoordinates; +import org.apache.metron.bundles.bundle.BundleDetails; +import org.apache.metron.bundles.util.BundleProperties; +import org.apache.metron.bundles.util.FileUtils; +import org.apache.metron.bundles.util.StringUtils; +import org.apache.metron.bundles.annotation.behavior.RequiresInstanceClassLoading; + +import org.atteo.classindex.ClassIndex; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.io.IOException; +import java.lang.reflect.Modifier; +import java.net.URI; +import java.net.URISyntaxException; +import java.net.URL; +import java.net.URLClassLoader; +import java.util.*; +import java.util.concurrent.ConcurrentHashMap; +import java.util.concurrent.atomic.AtomicBoolean; + +/** + * Scans through the classpath to load all extension components using the service provider API and + * running through all classloaders (root, BUNDLEs). + * + * @ThreadSafe - is immutable --- End diff -- I'm worried about ExtensionManager. It is effectively a fully static class, hence a de-facto singleton. But the implementor seems under the impression that by making all top-level static fields "final", that somehow makes it threadsafe. Which is blatantly untrue, since those fields contain data structures that are not immutable, and in most cases not concurrent. Also, it probably currently works because it is only called once at startup time. But I'm concerned what happens if it gets called from multiple topologies running in the same VM. And it certainly won't work for dynamic addition of bundles during runtime. Let me emphasize again: This class is NOT threadsafe. @ottobackwards , if you believe it needs to be threadsafe, then please do a reset and discuss with me how to fix this. If it's probably okay for current usages, then it is sufficient to mention it for improvement in a jira. Do you think METRON-1100 is an okay place for such mention, or should we open another jira? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...
Github user mattf-horton commented on a diff in the pull request: https://github.com/apache/metron/pull/530#discussion_r132795983 --- Diff: bundles-lib/src/main/java/org/apache/metron/bundles/ExtensionManager.java --- @@ -0,0 +1,436 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.metron.bundles; + + +import org.apache.commons.vfs2.FileObject; +import org.apache.commons.vfs2.FileSystemException; +import org.apache.commons.vfs2.FileSystemManager; +import org.apache.metron.bundles.bundle.Bundle; +import org.apache.metron.bundles.bundle.BundleCoordinates; +import org.apache.metron.bundles.bundle.BundleDetails; +import org.apache.metron.bundles.util.BundleProperties; +import org.apache.metron.bundles.util.FileUtils; +import org.apache.metron.bundles.util.StringUtils; +import org.apache.metron.bundles.annotation.behavior.RequiresInstanceClassLoading; + +import org.atteo.classindex.ClassIndex; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.io.IOException; +import java.lang.reflect.Modifier; +import java.net.URI; +import java.net.URISyntaxException; +import java.net.URL; +import java.net.URLClassLoader; +import java.util.*; +import java.util.concurrent.ConcurrentHashMap; +import java.util.concurrent.atomic.AtomicBoolean; + +/** + * Scans through the classpath to load all extension components using the service provider API and + * running through all classloaders (root, BUNDLEs). + * + * @ThreadSafe - is immutable + */ +@SuppressWarnings("rawtypes") +public class ExtensionManager { + + private static final Logger logger = LoggerFactory.getLogger(ExtensionManager.class); + + public static final BundleCoordinates SYSTEM_BUNDLE_COORDINATE = new BundleCoordinates( + BundleCoordinates.DEFAULT_GROUP, "system", BundleCoordinates.DEFAULT_VERSION); + + // Maps a service definition (interface) to those classes that implement the interface + private static final Map> definitionMap = new HashMap<>(); + + private static final Map> classNameBundleLookup = new HashMap<>(); + private static final Map bundleCoordinateBundleLookup = new HashMap<>(); + private static final Map classLoaderBundleLookup = new HashMap<>(); + + private static final Set requiresInstanceClassLoading = new HashSet<>(); + private static final Map instanceClassloaderLookup = new ConcurrentHashMap<>(); + + private static AtomicBoolean inited = new AtomicBoolean(false); + + // should initialize class definitions + public static void initClassDefinitions(final List classes) { +if (classes != null) { + for (Class clazz : classes) { +definitionMap.put(clazz, new HashSet<>()); + } +} +inited.set(true); + } + + public static void resetClassDefinitions() { +definitionMap.clear(); +inited.set(false); + } + + /** + * Loads all extension class types that can be found on the bootstrap classloader and by creating + * classloaders for all BUNDLES found within the classpath. + * + * @param bundles the bundles to scan through in search of extensions + */ + public static void discoverExtensions(final Bundle systemBundle, final Set bundles) + throws NotInitializedException { +checkInitialized(); +// get the current context class loader +ClassLoader currentContextClassLoader = Thread.currentThread().getContextClassLoader(); + +// load the system bundle first so that any extensions found in JARs directly in lib will be registered as +// being from the system bundle and not from all the other Bundles +loadExtensions(systemBundle); + bundleCoordinateBundleLookup.put(systemBundle.getBundleDetails().getCoordinates(), systemBundle); + +// consider each bundle class loader +for (final Bundle bundle : bundles) { + // Must s
[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...
Github user mattf-horton commented on a diff in the pull request: https://github.com/apache/metron/pull/530#discussion_r132801851 --- Diff: bundles-lib/src/main/java/org/apache/metron/bundles/VFSBundleClassLoader.java --- @@ -0,0 +1,515 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.metron.bundles; + +import java.io.IOException; +import java.net.URL; +import java.security.CodeSource; +import java.security.Permission; +import java.security.PermissionCollection; +import java.security.Permissions; +import java.security.SecureClassLoader; +import java.security.cert.Certificate; +import java.util.ArrayList; +import java.util.Collections; +import java.util.Enumeration; +import java.util.Iterator; +import java.util.List; +import java.util.jar.Attributes; +import java.util.jar.Attributes.Name; +import org.apache.commons.vfs2.FileObject; +import org.apache.commons.vfs2.FileSystemException; +import org.apache.commons.vfs2.FileSystemManager; +import org.apache.commons.vfs2.NameScope; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + + +/** + * + * A ClassLoader for loading BUNDLES (plugin archives). BUNDLEs are designed to + * allow isolating bundles of code (comprising one-or-more + * plugin classes and their + * dependencies) from other such bundles; this allows for dependencies and + * processors that require conflicting, incompatible versions of the same + * dependency to run in a single instance of a given process. + * + * + * BundleClassLoader follows the delegation model described in + * {@link ClassLoader#findClass(java.lang.String) ClassLoader.findClass(...)}; + * classes are first loaded from the parent ClassLoader, and only if + * they cannot be found there does the BundleClassLoader provide a + * definition. Specifically, this means that resources are loaded from the application's + * conf + * and lib directories first, and if they cannot be found there, are + * loaded from the BUNDLE. + * + * + * The packaging of a BUNDLE is such that it is a ZIP file with the following + * directory structure: + * + * + * +META-INF/ + * +-- bundled-dependencies/ + * +--+ * +-- MANIFEST.MF + * + * + * + * + * The MANIFEST.MF file contains the same information as a typical JAR file but + * also includes two additional bundle properties: {@code Bundle-Id} and + * {@code Bundle-Dependency-Id}. + * + * + * + * The {@code Bundle-Id} provides a unique identifier for this BUNDLE. + * + * + * + * The {@code Bundle-Dependency-Id} is optional. If provided, it indicates that + * this BUNDLE should inherit all of the dependencies of the BUNDLE with the provided + * ID. Often times, the BUNDLE that is depended upon is referred to as the Parent. + * This is because its ClassLoader will be the parent ClassLoader of the + * dependent BUNDLE. + * + * + * + * If a BUNDLE is built using the Bundles Maven Plugin, the {@code Bundle-Id} property + * will be set to the artifactId of the BUNDLE. The {@code Bundle-Dependency-Id} will + * be set to the artifactId of the BUNDLE that is depended upon. For example, if + * BUNDLE A is defined as such: + * + * + * ... + * bundle-a + *bundle + * ... + *+ * + * + * + * + * + * + * Then the MANIFEST.MF file that is created for Bundle A will have the following + * properties set: + * + * {@code {Foo}-Id: bundle-a} + * {@code {Foo}-Dependency-Id: bundle-z} + * + * Where is configurable by BundleProperty META_ID_PREFIX [ default Bundle ] + *+ * + *group + *bundle-z + *bundle + *
[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...
Github user mattf-horton commented on a diff in the pull request: https://github.com/apache/metron/pull/530#discussion_r132796537 --- Diff: bundles-lib/src/main/java/org/apache/metron/bundles/ExtensionManager.java --- @@ -0,0 +1,436 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.metron.bundles; + + +import org.apache.commons.vfs2.FileObject; +import org.apache.commons.vfs2.FileSystemException; +import org.apache.commons.vfs2.FileSystemManager; +import org.apache.metron.bundles.bundle.Bundle; +import org.apache.metron.bundles.bundle.BundleCoordinates; +import org.apache.metron.bundles.bundle.BundleDetails; +import org.apache.metron.bundles.util.BundleProperties; +import org.apache.metron.bundles.util.FileUtils; +import org.apache.metron.bundles.util.StringUtils; +import org.apache.metron.bundles.annotation.behavior.RequiresInstanceClassLoading; + +import org.atteo.classindex.ClassIndex; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.io.IOException; +import java.lang.reflect.Modifier; +import java.net.URI; +import java.net.URISyntaxException; +import java.net.URL; +import java.net.URLClassLoader; +import java.util.*; +import java.util.concurrent.ConcurrentHashMap; +import java.util.concurrent.atomic.AtomicBoolean; + +/** + * Scans through the classpath to load all extension components using the service provider API and + * running through all classloaders (root, BUNDLEs). + * + * @ThreadSafe - is immutable + */ +@SuppressWarnings("rawtypes") +public class ExtensionManager { + + private static final Logger logger = LoggerFactory.getLogger(ExtensionManager.class); + + public static final BundleCoordinates SYSTEM_BUNDLE_COORDINATE = new BundleCoordinates( + BundleCoordinates.DEFAULT_GROUP, "system", BundleCoordinates.DEFAULT_VERSION); + + // Maps a service definition (interface) to those classes that implement the interface + private static final Map> definitionMap = new HashMap<>(); + + private static final Map> classNameBundleLookup = new HashMap<>(); + private static final Map bundleCoordinateBundleLookup = new HashMap<>(); + private static final Map classLoaderBundleLookup = new HashMap<>(); + + private static final Set requiresInstanceClassLoading = new HashSet<>(); + private static final Map instanceClassloaderLookup = new ConcurrentHashMap<>(); + + private static AtomicBoolean inited = new AtomicBoolean(false); + + // should initialize class definitions + public static void initClassDefinitions(final List classes) { +if (classes != null) { + for (Class clazz : classes) { +definitionMap.put(clazz, new HashSet<>()); + } +} +inited.set(true); + } + + public static void resetClassDefinitions() { +definitionMap.clear(); +inited.set(false); + } + + /** + * Loads all extension class types that can be found on the bootstrap classloader and by creating + * classloaders for all BUNDLES found within the classpath. + * + * @param bundles the bundles to scan through in search of extensions + */ + public static void discoverExtensions(final Bundle systemBundle, final Set bundles) + throws NotInitializedException { +checkInitialized(); +// get the current context class loader +ClassLoader currentContextClassLoader = Thread.currentThread().getContextClassLoader(); + +// load the system bundle first so that any extensions found in JARs directly in lib will be registered as +// being from the system bundle and not from all the other Bundles +loadExtensions(systemBundle); + bundleCoordinateBundleLookup.put(systemBundle.getBundleDetails().getCoordinates(), systemBundle); + +// consider each bundle class loader +for (final Bundle bundle : bundles) { + // Must s
[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/530#discussion_r132284538 --- Diff: bundles-lib/src/main/java/org/apache/metron/bundles/bundle/BundleDetails.java --- @@ -0,0 +1,191 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.metron.bundles.bundle; + +import org.apache.commons.vfs2.FileObject; + +import java.text.ParseException; +import java.text.SimpleDateFormat; +import java.util.Date; + +/** + * Metadata about a bundle. + * the withCoordinates and withBundleFile properties are required + */ +public class BundleDetails { + +private final FileObject bundleFile; + +private final BundleCoordinates coordinates; +private final BundleCoordinates dependencyCoordinates; + +private final String buildTag; +private final String buildRevision; +private final String buildBranch; +private final String buildTimestamp; +private final String buildJdk; +private final String builtBy; + +private BundleDetails(final Builder builder) { +this.bundleFile = builder.bundleFile; +this.coordinates = builder.coordinates; +this.dependencyCoordinates = builder.dependencyCoordinates; + +this.buildTag = builder.buildTag; +this.buildRevision = builder.buildRevision; +this.buildBranch = builder.buildBranch; +this.buildTimestamp = builder.buildTimestamp; +this.buildJdk = builder.buildJdk; +this.builtBy = builder.builtBy; + +if (this.coordinates == null) { +if (this.bundleFile == null) { +throw new IllegalStateException("Coordinate cannot be null"); +} else { +throw new IllegalStateException("Coordinate cannot be null for " + this.bundleFile.getName()); +} +} + +if (this.bundleFile == null) { +throw new IllegalStateException("bundleFile cannot be null for " + this.coordinates +.getId()); +} +} + +public FileObject getBundleFile() { +return bundleFile; +} + +public BundleCoordinates getCoordinates() { +return coordinates; +} + +public BundleCoordinates getDependencyCoordinates() { +return dependencyCoordinates; +} + +public String getBuildTag() { +return buildTag; +} + +public String getBuildRevision() { +return buildRevision; +} + +public String getBuildBranch() { +return buildBranch; +} + +public String getBuildTimestamp() { +return buildTimestamp; +} + +public String getBuildJdk() { +return buildJdk; +} + +public String getBuiltBy() { +return builtBy; +} + +@Override +public String toString() { +return coordinates.toString(); +} + +public Date getBuildTimestampDate() { +if (buildTimestamp != null && !buildTimestamp.isEmpty()) { +try { +SimpleDateFormat buildTimestampFormat = new SimpleDateFormat("-MM-dd'T'HH:mm:ss'Z'"); +Date buildTimestampDate = buildTimestampFormat.parse(buildTimestamp); +return buildTimestampDate; +} catch (ParseException parseEx) { +return null; +} +} else { +return null; +} +} + +/** + * Builder for BundleDetails. --- End diff -- done. I also did the validation you had mentioned. When I got in there it made more sense than not. Sorry I didn't see it before --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not
[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/530#discussion_r132284429 --- Diff: bundles-lib/src/main/java/org/apache/metron/bundles/bundle/BundleCoordinates.java --- @@ -0,0 +1,93 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.metron.bundles.bundle; + +/** + * The coordinates of a bundle (group, artifact, version). + */ +public class BundleCoordinates { + +public static final String DEFAULT_GROUP = "default"; +public static final String DEFAULT_VERSION = "unversioned"; + +public static final BundleCoordinates UNKNOWN_COORDINATE = new BundleCoordinates(DEFAULT_GROUP, "unknown", DEFAULT_VERSION); + +private final String group; +private final String id; +private final String version; +private final String coordinate; + +public BundleCoordinates(final String group, final String id, final String version) { +this.group = isBlank(group) ? DEFAULT_GROUP : group; +this.id = id; +this.version = isBlank(version) ? DEFAULT_VERSION : version; + +if (isBlank(id)) { +throw new IllegalStateException("Id is required for BundleCoordinates"); +} + +if(this.group.contains(":") || this.id.contains(":") || this.version.contains(":")) { +throw new IllegalStateException(String.format("Invalid coordinates: cannot contain : character group[%s] id[%s] version[%s]",this.group,this.id,this.version)); +} +this.coordinate = this.group + ":" + this.id + ":" + this.version; +} + +private boolean isBlank(String str) { +return str == null || str.trim().length() == 0; +} + +public String getGroup() { +return group; +} + +public String getId() { +return id; +} + +public String getVersion() { +return version; +} + +public final String getCoordinate() { --- End diff -- done --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...
Github user mattf-horton commented on a diff in the pull request: https://github.com/apache/metron/pull/530#discussion_r132280796 --- Diff: bundles-lib/src/main/java/org/apache/metron/bundles/bundle/BundleCoordinates.java --- @@ -0,0 +1,93 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.metron.bundles.bundle; + +/** + * The coordinates of a bundle (group, artifact, version). + */ +public class BundleCoordinates { + +public static final String DEFAULT_GROUP = "default"; +public static final String DEFAULT_VERSION = "unversioned"; + +public static final BundleCoordinates UNKNOWN_COORDINATE = new BundleCoordinates(DEFAULT_GROUP, "unknown", DEFAULT_VERSION); + +private final String group; +private final String id; +private final String version; +private final String coordinate; + +public BundleCoordinates(final String group, final String id, final String version) { +this.group = isBlank(group) ? DEFAULT_GROUP : group; +this.id = id; +this.version = isBlank(version) ? DEFAULT_VERSION : version; + +if (isBlank(id)) { +throw new IllegalStateException("Id is required for BundleCoordinates"); +} + +if(this.group.contains(":") || this.id.contains(":") || this.version.contains(":")) { +throw new IllegalStateException(String.format("Invalid coordinates: cannot contain : character group[%s] id[%s] version[%s]",this.group,this.id,this.version)); +} +this.coordinate = this.group + ":" + this.id + ":" + this.version; +} + +private boolean isBlank(String str) { +return str == null || str.trim().length() == 0; +} + +public String getGroup() { +return group; +} + +public String getId() { +return id; +} + +public String getVersion() { +return version; +} + +public final String getCoordinate() { --- End diff -- Please rename this to getCoordinates() too, since it is public. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...
Github user mattf-horton commented on a diff in the pull request: https://github.com/apache/metron/pull/530#discussion_r132279911 --- Diff: bundles-lib/src/main/java/org/apache/metron/bundles/bundle/BundleDetails.java --- @@ -0,0 +1,191 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.metron.bundles.bundle; + +import org.apache.commons.vfs2.FileObject; + +import java.text.ParseException; +import java.text.SimpleDateFormat; +import java.util.Date; + +/** + * Metadata about a bundle. + * the withCoordinates and withBundleFile properties are required + */ +public class BundleDetails { + +private final FileObject bundleFile; + +private final BundleCoordinates coordinates; +private final BundleCoordinates dependencyCoordinates; + +private final String buildTag; +private final String buildRevision; +private final String buildBranch; +private final String buildTimestamp; +private final String buildJdk; +private final String builtBy; + +private BundleDetails(final Builder builder) { +this.bundleFile = builder.bundleFile; +this.coordinates = builder.coordinates; +this.dependencyCoordinates = builder.dependencyCoordinates; + +this.buildTag = builder.buildTag; +this.buildRevision = builder.buildRevision; +this.buildBranch = builder.buildBranch; +this.buildTimestamp = builder.buildTimestamp; +this.buildJdk = builder.buildJdk; +this.builtBy = builder.builtBy; + +if (this.coordinates == null) { +if (this.bundleFile == null) { +throw new IllegalStateException("Coordinate cannot be null"); +} else { +throw new IllegalStateException("Coordinate cannot be null for " + this.bundleFile.getName()); +} +} + +if (this.bundleFile == null) { +throw new IllegalStateException("bundleFile cannot be null for " + this.coordinates +.getId()); +} +} + +public FileObject getBundleFile() { +return bundleFile; +} + +public BundleCoordinates getCoordinates() { +return coordinates; +} + +public BundleCoordinates getDependencyCoordinates() { +return dependencyCoordinates; +} + +public String getBuildTag() { +return buildTag; +} + +public String getBuildRevision() { +return buildRevision; +} + +public String getBuildBranch() { +return buildBranch; +} + +public String getBuildTimestamp() { +return buildTimestamp; +} + +public String getBuildJdk() { +return buildJdk; +} + +public String getBuiltBy() { +return builtBy; +} + +@Override +public String toString() { +return coordinates.toString(); +} + +public Date getBuildTimestampDate() { +if (buildTimestamp != null && !buildTimestamp.isEmpty()) { +try { +SimpleDateFormat buildTimestampFormat = new SimpleDateFormat("-MM-dd'T'HH:mm:ss'Z'"); +Date buildTimestampDate = buildTimestampFormat.parse(buildTimestamp); +return buildTimestampDate; +} catch (ParseException parseEx) { +return null; +} +} else { +return null; +} +} + +/** + * Builder for BundleDetails. --- End diff -- Please also javadoc the required properties on the Builder, since that the public i/f that actually gets used. > + * the withCoordinates and withBundleFile properties are required --- If your project is set up for it, you can reply to this email and have your reply appea
[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...
Github user mattf-horton commented on a diff in the pull request: https://github.com/apache/metron/pull/530#discussion_r132263102 --- Diff: bundles-lib/src/main/java/org/apache/metron/bundles/BundleClassLoaders.java --- @@ -0,0 +1,353 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.metron.bundles; + +import java.net.URISyntaxException; +import java.util.*; + +import org.apache.commons.collections.CollectionUtils; +import org.apache.commons.vfs2.FileObject; +import org.apache.commons.vfs2.FileSystemException; +import org.apache.commons.vfs2.FileSystemManager; +import org.apache.metron.bundles.bundle.Bundle; +import org.apache.metron.bundles.bundle.BundleCoordinate; +import org.apache.metron.bundles.bundle.BundleDetails; +import org.apache.metron.bundles.util.BundleProperties; +import org.apache.metron.bundles.util.BundleSelector; +import org.apache.metron.bundles.util.FileUtils; +import org.apache.metron.bundles.util.BundleUtil; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +/** + * A singleton class used to initialize the extension and framework classloaders. + */ +public final class BundleClassLoaders { + + private static volatile BundleClassLoaders bundleClassLoaders; + private volatile InitContext initContext; + private static final Logger logger = LoggerFactory.getLogger(BundleClassLoaders.class); + + private final static class InitContext { + +private final List extensionDirs; +private final Map bundles; + +private InitContext( +final List extensionDirs, +final Map bundles) { + this.extensionDirs = extensionDirs; + this.bundles = bundles; +} + } + + private BundleClassLoaders() { + } + + /** + * @return The singleton instance of the BundleClassLoaders + */ + public static BundleClassLoaders getInstance() { +BundleClassLoaders result = bundleClassLoaders; +if (result == null) { + synchronized (BundleClassLoaders.class) { +result = bundleClassLoaders; +if (result == null) { + bundleClassLoaders = result = new BundleClassLoaders(); +} + } +} +return result; + } + + public static void reset() { +getInstance().unInit(); + } + + private void unInit() { +synchronized (this) { + initContext = null; +} + } + + /** + * Initializes and loads the BundleClassLoaders. This method must be called before the rest of the + * methods to access the classloaders are called and it can be safely called any number of times + * provided the same framework and extension working dirs are used. + * + * @param fileSystemManager the FileSystemManager + * @param extensionsDirs where to find extension artifacts + * @param props BundleProperties + * @throws FileSystemException if any issue occurs while exploding bundle working directories. + * @throws java.lang.ClassNotFoundException if unable to load class definition + * @throws IllegalStateException already initialized with a given pair of directories cannot + * reinitialize or use a different pair of directories. + */ + public void init(final FileSystemManager fileSystemManager, final List extensionsDirs, + BundleProperties props) + throws FileSystemException, ClassNotFoundException, URISyntaxException { +if (extensionsDirs == null || fileSystemManager == null) { + throw new NullPointerException("cannot have empty arguments"); +} +InitContext ic = initContext; +if (ic == null) { + synchronized (this) { +ic = initContext; +if (ic == null) { + initContext = ic = load(fileSystemManager, extensionsDirs, props); +} + } +} else { + boolean matching = CollectionUtils +
[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...
Github user mattf-horton commented on a diff in the pull request: https://github.com/apache/metron/pull/530#discussion_r132261444 --- Diff: bundles-lib/src/main/java/org/apache/metron/bundles/BundleClassLoaders.java --- @@ -0,0 +1,353 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.metron.bundles; + +import java.net.URISyntaxException; +import java.util.*; + +import org.apache.commons.collections.CollectionUtils; +import org.apache.commons.vfs2.FileObject; +import org.apache.commons.vfs2.FileSystemException; +import org.apache.commons.vfs2.FileSystemManager; +import org.apache.metron.bundles.bundle.Bundle; +import org.apache.metron.bundles.bundle.BundleCoordinate; +import org.apache.metron.bundles.bundle.BundleDetails; +import org.apache.metron.bundles.util.BundleProperties; +import org.apache.metron.bundles.util.BundleSelector; +import org.apache.metron.bundles.util.FileUtils; +import org.apache.metron.bundles.util.BundleUtil; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +/** + * A singleton class used to initialize the extension and framework classloaders. + */ +public final class BundleClassLoaders { + + private static volatile BundleClassLoaders bundleClassLoaders; + private volatile InitContext initContext; + private static final Logger logger = LoggerFactory.getLogger(BundleClassLoaders.class); + + private final static class InitContext { + +private final List extensionDirs; +private final Map bundles; + +private InitContext( +final List extensionDirs, +final Map bundles) { + this.extensionDirs = extensionDirs; + this.bundles = bundles; +} + } + + private BundleClassLoaders() { + } + + /** + * @return The singleton instance of the BundleClassLoaders + */ + public static BundleClassLoaders getInstance() { +BundleClassLoaders result = bundleClassLoaders; +if (result == null) { + synchronized (BundleClassLoaders.class) { +result = bundleClassLoaders; +if (result == null) { + bundleClassLoaders = result = new BundleClassLoaders(); +} + } +} +return result; + } + + public static void reset() { +getInstance().unInit(); + } + + private void unInit() { +synchronized (this) { + initContext = null; +} + } + + /** + * Initializes and loads the BundleClassLoaders. This method must be called before the rest of the + * methods to access the classloaders are called and it can be safely called any number of times + * provided the same framework and extension working dirs are used. --- End diff -- But now I'm also concerned, if your design point was single extensions only: If I add multiple extensions, will they all be loaded and available? It looked to me like they would, but maybe not if you were specifically thinking otherwise? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...
Github user mattf-horton commented on a diff in the pull request: https://github.com/apache/metron/pull/530#discussion_r132260802 --- Diff: bundles-lib/src/main/java/org/apache/metron/bundles/BundleClassLoaders.java --- @@ -0,0 +1,353 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.metron.bundles; + +import java.net.URISyntaxException; +import java.util.*; + +import org.apache.commons.collections.CollectionUtils; +import org.apache.commons.vfs2.FileObject; +import org.apache.commons.vfs2.FileSystemException; +import org.apache.commons.vfs2.FileSystemManager; +import org.apache.metron.bundles.bundle.Bundle; +import org.apache.metron.bundles.bundle.BundleCoordinate; +import org.apache.metron.bundles.bundle.BundleDetails; +import org.apache.metron.bundles.util.BundleProperties; +import org.apache.metron.bundles.util.BundleSelector; +import org.apache.metron.bundles.util.FileUtils; +import org.apache.metron.bundles.util.BundleUtil; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +/** + * A singleton class used to initialize the extension and framework classloaders. + */ +public final class BundleClassLoaders { + + private static volatile BundleClassLoaders bundleClassLoaders; + private volatile InitContext initContext; + private static final Logger logger = LoggerFactory.getLogger(BundleClassLoaders.class); + + private final static class InitContext { + +private final List extensionDirs; +private final Map bundles; + +private InitContext( +final List extensionDirs, +final Map bundles) { + this.extensionDirs = extensionDirs; + this.bundles = bundles; +} + } + + private BundleClassLoaders() { + } + + /** + * @return The singleton instance of the BundleClassLoaders + */ + public static BundleClassLoaders getInstance() { +BundleClassLoaders result = bundleClassLoaders; +if (result == null) { + synchronized (BundleClassLoaders.class) { +result = bundleClassLoaders; +if (result == null) { + bundleClassLoaders = result = new BundleClassLoaders(); +} + } +} +return result; + } + + public static void reset() { +getInstance().unInit(); + } + + private void unInit() { +synchronized (this) { + initContext = null; +} + } --- End diff -- If you wish to add it, the annotation comes from > import com.google.common.annotations.VisibleForTesting; --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...
Github user mattf-horton commented on a diff in the pull request: https://github.com/apache/metron/pull/530#discussion_r132258789 --- Diff: bundles-lib/src/main/java/org/apache/metron/bundles/bundle/BundleDetails.java --- @@ -0,0 +1,189 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.metron.bundles.bundle; + +import org.apache.commons.vfs2.FileObject; + +import java.text.ParseException; +import java.text.SimpleDateFormat; +import java.util.Date; + +/** + * Metadata about a bundle. + */ +public class BundleDetails { + +private final FileObject bundleFile; + +private final BundleCoordinate coordinate; +private final BundleCoordinate dependencyCoordinate; + +private final String buildTag; +private final String buildRevision; +private final String buildBranch; +private final String buildTimestamp; +private final String buildJdk; +private final String builtBy; + +private BundleDetails(final Builder builder) { +this.bundleFile = builder.bundleFile; +this.coordinate = builder.coordinate; +this.dependencyCoordinate = builder.dependencyCoordinate; + +this.buildTag = builder.buildTag; +this.buildRevision = builder.buildRevision; +this.buildBranch = builder.buildBranch; +this.buildTimestamp = builder.buildTimestamp; +this.buildJdk = builder.buildJdk; +this.builtBy = builder.builtBy; + +if (this.coordinate == null) { +if (this.bundleFile == null) { +throw new IllegalStateException("Coordinate cannot be null"); +} else { +throw new IllegalStateException("Coordinate cannot be null for " + this.bundleFile.getName()); +} +} + +if (this.bundleFile == null) { +throw new IllegalStateException("Working directory cannot be null for " + this.coordinate.getId()); +} +} + +public FileObject getBundleFile() { +return bundleFile; +} + +public BundleCoordinate getCoordinate() { +return coordinate; +} + +public BundleCoordinate getDependencyCoordinate() { +return dependencyCoordinate; +} + +public String getBuildTag() { +return buildTag; +} + +public String getBuildRevision() { +return buildRevision; +} + +public String getBuildBranch() { +return buildBranch; +} + +public String getBuildTimestamp() { +return buildTimestamp; +} + +public String getBuildJdk() { +return buildJdk; +} + +public String getBuiltBy() { +return builtBy; +} + +@Override +public String toString() { +return coordinate.toString(); +} + +public Date getBuildTimestampDate() { +if (buildTimestamp != null && !buildTimestamp.isEmpty()) { +try { +SimpleDateFormat buildTimestampFormat = new SimpleDateFormat("-MM-dd'T'HH:mm:ss'Z'"); +Date buildTimestampDate = buildTimestampFormat.parse(buildTimestamp); +return buildTimestampDate; +} catch (ParseException parseEx) { +return null; +} +} else { +return null; +} +} + +/** + * Builder for BundleDetails. + */ +public static class Builder { + +private FileObject bundleFile; + +private BundleCoordinate coordinate; +private BundleCoordinate dependencyCoordinate; + +private String buildTag; +private String buildRevision; +private String buildBranch; +private String buildTimestamp; +private
[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...
Github user mattf-horton commented on a diff in the pull request: https://github.com/apache/metron/pull/530#discussion_r132255328 --- Diff: bundles-lib/src/main/java/org/apache/metron/bundles/bundle/Bundle.java --- @@ -0,0 +1,48 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.metron.bundles.bundle; + +/** + * Represents a bundle that contains one or more extensions. + */ +public class Bundle { + +private final BundleDetails bundleDetails; + +private final ClassLoader classLoader; + +public Bundle(final BundleDetails bundleDetails, final ClassLoader classLoader) { +this.bundleDetails = bundleDetails; +this.classLoader = classLoader; + +if (this.bundleDetails == null) { +throw new IllegalStateException("BundleDetails cannot be null"); +} + +if (this.classLoader == null) { +throw new IllegalStateException("ClassLoader cannot be null"); --- End diff -- @ottobackwards , regarding > That and it is a runtime exception? Both IllegalStateException and IllegalArgumentException are RuntimeExceptions, hence unchecked. My comment "It is also a RuntimeException" was intended to say "It is easy to substitute IllegalArgumentException here, because it doesn't change the exception-checking behavior." --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...
Github user mattf-horton commented on a diff in the pull request: https://github.com/apache/metron/pull/530#discussion_r132254112 --- Diff: bundles-lib/src/main/java/org/apache/metron/bundles/util/FileSystemManagerFactory.java --- @@ -0,0 +1,99 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.metron.bundles.util; + +import org.apache.accumulo.start.classloader.vfs.UniqueFileReplicator; +import org.apache.commons.vfs2.*; +import org.apache.commons.vfs2.cache.SoftRefFilesCache; +import org.apache.commons.vfs2.impl.DefaultFileSystemManager; +import org.apache.commons.vfs2.impl.FileContentInfoFilenameFactory; +import org.apache.commons.vfs2.provider.hdfs.HdfsFileProvider; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.io.File; + +public class FileSystemManagerFactory { + private static final Logger LOG = LoggerFactory.getLogger(FileSystemManagerFactory.class); + + /** + * Create a FileSystem manager suitable for our purposes. + * This manager supports files of the following types: + * * res - resource files + * * jar + * * tar + * * bz2 + * * tgz + * * zip + * * HDFS + * * FTP + * * HTTP/S + * * file + * @return + * @throws FileSystemException + */ + public static FileSystemManager createFileSystemManager() throws FileSystemException { +return createFileSystemManager(null); + } + public static FileSystemManager createFileSystemManager(String jarExtensionToRegister) throws FileSystemException { +DefaultFileSystemManager vfs = new DefaultFileSystemManager(); + +if(jarExtensionToRegister != null || !StringUtils.isBlank(jarExtensionToRegister) ) { + vfs.addExtensionMap(jarExtensionToRegister, "jar"); + vfs.addProvider(jarExtensionToRegister, new org.apache.commons.vfs2.provider.jar.JarFileProvider()); +} + +vfs.addProvider("res", new org.apache.commons.vfs2.provider.res.ResourceFileProvider()); +vfs.addProvider("zip", new org.apache.commons.vfs2.provider.zip.ZipFileProvider()); +vfs.addProvider("gz", new org.apache.commons.vfs2.provider.gzip.GzipFileProvider()); +vfs.addProvider("ram", new org.apache.commons.vfs2.provider.ram.RamFileProvider()); +vfs.addProvider("file", new org.apache.commons.vfs2.provider.local.DefaultLocalFileProvider()); +vfs.addProvider("jar", new org.apache.commons.vfs2.provider.jar.JarFileProvider()); +vfs.addProvider("http", new org.apache.commons.vfs2.provider.http.HttpFileProvider()); +vfs.addProvider("https", new org.apache.commons.vfs2.provider.https.HttpsFileProvider()); +vfs.addProvider("ftp", new org.apache.commons.vfs2.provider.ftp.FtpFileProvider()); +vfs.addProvider("ftps", new org.apache.commons.vfs2.provider.ftps.FtpsFileProvider()); +vfs.addProvider("war", new org.apache.commons.vfs2.provider.jar.JarFileProvider()); +vfs.addProvider("par", new org.apache.commons.vfs2.provider.jar.JarFileProvider()); +vfs.addProvider("ear", new org.apache.commons.vfs2.provider.jar.JarFileProvider()); +vfs.addProvider("sar", new org.apache.commons.vfs2.provider.jar.JarFileProvider()); +vfs.addProvider("ejb3", new org.apache.commons.vfs2.provider.jar.JarFileProvider()); +vfs.addProvider("tmp", new org.apache.commons.vfs2.provider.temp.TemporaryFileProvider()); +vfs.addProvider("tar", new org.apache.commons.vfs2.provider.tar.TarFileProvider()); +vfs.addProvider("tbz2", new org.apache.commons.vfs2.provider.tar.TarFileProvider()); +vfs.addProvider("tgz", new org.apache.commons.vfs2.provider.tar.TarFileProvider()); +vfs.addProvider("bz2", new org.apache.commons.vfs2.provider.bzip2.Bzip2FileProvider()); +vfs.addProvider("hdfs", new HdfsFileProvider()); +vfs.addExtensionMap("jar", "jar"); +vfs.addExtensionMap("zip", "zip"); +vfs.addExtensionMap("gz", "gz"); +vf
[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...
Github user bbende commented on a diff in the pull request: https://github.com/apache/metron/pull/530#discussion_r132012486 --- Diff: bundles-lib/src/main/java/org/apache/metron/bundles/bundle/Bundle.java --- @@ -0,0 +1,48 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.metron.bundles.bundle; + +/** + * Represents a bundle that contains one or more extensions. + */ +public class Bundle { + +private final BundleDetails bundleDetails; + +private final ClassLoader classLoader; + +public Bundle(final BundleDetails bundleDetails, final ClassLoader classLoader) { +this.bundleDetails = bundleDetails; +this.classLoader = classLoader; + +if (this.bundleDetails == null) { +throw new IllegalStateException("BundleDetails cannot be null"); +} + +if (this.classLoader == null) { +throw new IllegalStateException("ClassLoader cannot be null"); --- End diff -- No worries, sounds good. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/530#discussion_r131999633 --- Diff: bundles-lib/src/main/java/org/apache/metron/bundles/bundle/Bundle.java --- @@ -0,0 +1,48 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.metron.bundles.bundle; + +/** + * Represents a bundle that contains one or more extensions. + */ +public class Bundle { + +private final BundleDetails bundleDetails; + +private final ClassLoader classLoader; + +public Bundle(final BundleDetails bundleDetails, final ClassLoader classLoader) { +this.bundleDetails = bundleDetails; +this.classLoader = classLoader; + +if (this.bundleDetails == null) { +throw new IllegalStateException("BundleDetails cannot be null"); +} + +if (this.classLoader == null) { +throw new IllegalStateException("ClassLoader cannot be null"); --- End diff -- @bbende sorry to pull you into the middle of this opus. I was planning on talking to you about this after it passed review and settled a bit. I *think* there a couple of things in here that may be worth a thought for nifi, and my plan is to bring that up later. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/530#discussion_r131997042 --- Diff: bundles-lib/src/main/java/org/apache/metron/bundles/bundle/Bundle.java --- @@ -0,0 +1,48 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.metron.bundles.bundle; + +/** + * Represents a bundle that contains one or more extensions. + */ +public class Bundle { + +private final BundleDetails bundleDetails; + +private final ClassLoader classLoader; + +public Bundle(final BundleDetails bundleDetails, final ClassLoader classLoader) { +this.bundleDetails = bundleDetails; +this.classLoader = classLoader; + +if (this.bundleDetails == null) { +throw new IllegalStateException("BundleDetails cannot be null"); +} + +if (this.classLoader == null) { +throw new IllegalStateException("ClassLoader cannot be null"); --- End diff -- That and it is a runtime exception? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/530#discussion_r131995115 --- Diff: bundles-lib/src/main/java/org/apache/metron/bundles/BundleClassLoaders.java --- @@ -0,0 +1,353 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.metron.bundles; + +import java.net.URISyntaxException; +import java.util.*; + +import org.apache.commons.collections.CollectionUtils; +import org.apache.commons.vfs2.FileObject; +import org.apache.commons.vfs2.FileSystemException; +import org.apache.commons.vfs2.FileSystemManager; +import org.apache.metron.bundles.bundle.Bundle; +import org.apache.metron.bundles.bundle.BundleCoordinate; +import org.apache.metron.bundles.bundle.BundleDetails; +import org.apache.metron.bundles.util.BundleProperties; +import org.apache.metron.bundles.util.BundleSelector; +import org.apache.metron.bundles.util.FileUtils; +import org.apache.metron.bundles.util.BundleUtil; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +/** + * A singleton class used to initialize the extension and framework classloaders. + */ +public final class BundleClassLoaders { + + private static volatile BundleClassLoaders bundleClassLoaders; + private volatile InitContext initContext; + private static final Logger logger = LoggerFactory.getLogger(BundleClassLoaders.class); + + private final static class InitContext { + +private final List extensionDirs; +private final Map bundles; + +private InitContext( +final List extensionDirs, +final Map bundles) { + this.extensionDirs = extensionDirs; + this.bundles = bundles; +} + } + + private BundleClassLoaders() { + } + + /** + * @return The singleton instance of the BundleClassLoaders + */ + public static BundleClassLoaders getInstance() { +BundleClassLoaders result = bundleClassLoaders; +if (result == null) { + synchronized (BundleClassLoaders.class) { +result = bundleClassLoaders; +if (result == null) { + bundleClassLoaders = result = new BundleClassLoaders(); +} + } +} +return result; + } + + public static void reset() { +getInstance().unInit(); + } + + private void unInit() { +synchronized (this) { + initContext = null; +} + } + + /** + * Initializes and loads the BundleClassLoaders. This method must be called before the rest of the + * methods to access the classloaders are called and it can be safely called any number of times + * provided the same framework and extension working dirs are used. + * + * @param fileSystemManager the FileSystemManager + * @param extensionsDirs where to find extension artifacts + * @param props BundleProperties + * @throws FileSystemException if any issue occurs while exploding bundle working directories. + * @throws java.lang.ClassNotFoundException if unable to load class definition + * @throws IllegalStateException already initialized with a given pair of directories cannot + * reinitialize or use a different pair of directories. + */ + public void init(final FileSystemManager fileSystemManager, final List extensionsDirs, + BundleProperties props) + throws FileSystemException, ClassNotFoundException, URISyntaxException { +if (extensionsDirs == null || fileSystemManager == null) { + throw new NullPointerException("cannot have empty arguments"); +} +InitContext ic = initContext; +if (ic == null) { + synchronized (this) { +ic = initContext; +if (ic == null) { + initContext = ic = load(fileSystemManager, extensionsDirs, props); +} + } +} else { + boolean matching = CollectionUtils +
[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/530#discussion_r131994814 --- Diff: bundles-lib/src/main/java/org/apache/metron/bundles/BundleClassLoaders.java --- @@ -0,0 +1,353 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.metron.bundles; + +import java.net.URISyntaxException; +import java.util.*; + +import org.apache.commons.collections.CollectionUtils; +import org.apache.commons.vfs2.FileObject; +import org.apache.commons.vfs2.FileSystemException; +import org.apache.commons.vfs2.FileSystemManager; +import org.apache.metron.bundles.bundle.Bundle; +import org.apache.metron.bundles.bundle.BundleCoordinate; +import org.apache.metron.bundles.bundle.BundleDetails; +import org.apache.metron.bundles.util.BundleProperties; +import org.apache.metron.bundles.util.BundleSelector; +import org.apache.metron.bundles.util.FileUtils; +import org.apache.metron.bundles.util.BundleUtil; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +/** + * A singleton class used to initialize the extension and framework classloaders. + */ +public final class BundleClassLoaders { + + private static volatile BundleClassLoaders bundleClassLoaders; + private volatile InitContext initContext; + private static final Logger logger = LoggerFactory.getLogger(BundleClassLoaders.class); + + private final static class InitContext { + +private final List extensionDirs; +private final Map bundles; + +private InitContext( +final List extensionDirs, +final Map bundles) { + this.extensionDirs = extensionDirs; + this.bundles = bundles; +} + } + + private BundleClassLoaders() { + } + + /** + * @return The singleton instance of the BundleClassLoaders + */ + public static BundleClassLoaders getInstance() { +BundleClassLoaders result = bundleClassLoaders; +if (result == null) { + synchronized (BundleClassLoaders.class) { +result = bundleClassLoaders; +if (result == null) { + bundleClassLoaders = result = new BundleClassLoaders(); +} + } +} +return result; + } + + public static void reset() { +getInstance().unInit(); + } + + private void unInit() { +synchronized (this) { + initContext = null; +} + } + + /** + * Initializes and loads the BundleClassLoaders. This method must be called before the rest of the + * methods to access the classloaders are called and it can be safely called any number of times + * provided the same framework and extension working dirs are used. + * + * @param fileSystemManager the FileSystemManager + * @param extensionsDirs where to find extension artifacts + * @param props BundleProperties + * @throws FileSystemException if any issue occurs while exploding bundle working directories. + * @throws java.lang.ClassNotFoundException if unable to load class definition + * @throws IllegalStateException already initialized with a given pair of directories cannot + * reinitialize or use a different pair of directories. + */ + public void init(final FileSystemManager fileSystemManager, final List extensionsDirs, + BundleProperties props) + throws FileSystemException, ClassNotFoundException, URISyntaxException { +if (extensionsDirs == null || fileSystemManager == null) { + throw new NullPointerException("cannot have empty arguments"); +} +InitContext ic = initContext; +if (ic == null) { + synchronized (this) { +ic = initContext; +if (ic == null) { + initContext = ic = load(fileSystemManager, extensionsDirs, props); +} + } +} else { + boolean matching = CollectionUtils +
[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/530#discussion_r131991571 --- Diff: bundles-lib/src/main/java/org/apache/metron/bundles/BundleClassLoaders.java --- @@ -0,0 +1,353 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.metron.bundles; + +import java.net.URISyntaxException; +import java.util.*; + +import org.apache.commons.collections.CollectionUtils; +import org.apache.commons.vfs2.FileObject; +import org.apache.commons.vfs2.FileSystemException; +import org.apache.commons.vfs2.FileSystemManager; +import org.apache.metron.bundles.bundle.Bundle; +import org.apache.metron.bundles.bundle.BundleCoordinate; +import org.apache.metron.bundles.bundle.BundleDetails; +import org.apache.metron.bundles.util.BundleProperties; +import org.apache.metron.bundles.util.BundleSelector; +import org.apache.metron.bundles.util.FileUtils; +import org.apache.metron.bundles.util.BundleUtil; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +/** + * A singleton class used to initialize the extension and framework classloaders. + */ +public final class BundleClassLoaders { + + private static volatile BundleClassLoaders bundleClassLoaders; + private volatile InitContext initContext; + private static final Logger logger = LoggerFactory.getLogger(BundleClassLoaders.class); + + private final static class InitContext { + +private final List extensionDirs; +private final Map bundles; + +private InitContext( +final List extensionDirs, +final Map bundles) { + this.extensionDirs = extensionDirs; + this.bundles = bundles; +} + } + + private BundleClassLoaders() { + } + + /** + * @return The singleton instance of the BundleClassLoaders + */ + public static BundleClassLoaders getInstance() { +BundleClassLoaders result = bundleClassLoaders; +if (result == null) { + synchronized (BundleClassLoaders.class) { +result = bundleClassLoaders; +if (result == null) { + bundleClassLoaders = result = new BundleClassLoaders(); +} + } +} +return result; + } + + public static void reset() { +getInstance().unInit(); + } + + private void unInit() { +synchronized (this) { + initContext = null; +} + } --- End diff -- No, in fact we count on the system taking care of that. @visible for testing seems more correct. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/530#discussion_r131991006 --- Diff: bundles-lib/src/main/java/org/apache/metron/bundles/BundleClassLoaders.java --- @@ -0,0 +1,353 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.metron.bundles; + +import java.net.URISyntaxException; +import java.util.*; + +import org.apache.commons.collections.CollectionUtils; +import org.apache.commons.vfs2.FileObject; +import org.apache.commons.vfs2.FileSystemException; +import org.apache.commons.vfs2.FileSystemManager; +import org.apache.metron.bundles.bundle.Bundle; +import org.apache.metron.bundles.bundle.BundleCoordinate; +import org.apache.metron.bundles.bundle.BundleDetails; +import org.apache.metron.bundles.util.BundleProperties; +import org.apache.metron.bundles.util.BundleSelector; +import org.apache.metron.bundles.util.FileUtils; +import org.apache.metron.bundles.util.BundleUtil; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +/** + * A singleton class used to initialize the extension and framework classloaders. + */ +public final class BundleClassLoaders { + + private static volatile BundleClassLoaders bundleClassLoaders; + private volatile InitContext initContext; + private static final Logger logger = LoggerFactory.getLogger(BundleClassLoaders.class); + + private final static class InitContext { + +private final List extensionDirs; +private final Map bundles; + +private InitContext( +final List extensionDirs, +final Map bundles) { + this.extensionDirs = extensionDirs; + this.bundles = bundles; +} + } + + private BundleClassLoaders() { + } + + /** + * @return The singleton instance of the BundleClassLoaders + */ + public static BundleClassLoaders getInstance() { +BundleClassLoaders result = bundleClassLoaders; +if (result == null) { + synchronized (BundleClassLoaders.class) { +result = bundleClassLoaders; +if (result == null) { + bundleClassLoaders = result = new BundleClassLoaders(); +} + } +} +return result; + } + + public static void reset() { +getInstance().unInit(); + } + + private void unInit() { +synchronized (this) { + initContext = null; +} + } + + /** + * Initializes and loads the BundleClassLoaders. This method must be called before the rest of the + * methods to access the classloaders are called and it can be safely called any number of times + * provided the same framework and extension working dirs are used. --- End diff -- No, it is I who doesn't understand. I was not thinking about the supervisor correctly. So, adding extensions without restart needs more thought. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...
Github user bbende commented on a diff in the pull request: https://github.com/apache/metron/pull/530#discussion_r131986154 --- Diff: bundles-lib/src/main/java/org/apache/metron/bundles/bundle/Bundle.java --- @@ -0,0 +1,48 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.metron.bundles.bundle; + +/** + * Represents a bundle that contains one or more extensions. + */ +public class Bundle { + +private final BundleDetails bundleDetails; + +private final ClassLoader classLoader; + +public Bundle(final BundleDetails bundleDetails, final ClassLoader classLoader) { +this.bundleDetails = bundleDetails; +this.classLoader = classLoader; + +if (this.bundleDetails == null) { +throw new IllegalStateException("BundleDetails cannot be null"); +} + +if (this.classLoader == null) { +throw new IllegalStateException("ClassLoader cannot be null"); --- End diff -- The overall goal was just to ensure that if any code later on encounters a Bundle, it can be sure that the details and class loader are present. There isn't a realistic case in NiFi where you should encounter these exceptions, it was just being defensive. I think I got in the habit of using IllegalStateException here since at this point you are checking the member variables of the object and saying that something in the state of the object is not valid, as opposed to checking the arguments before assigning them which would maybe better to use IllegalArgumentException. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---