[ANNOUNCE] Apache Storm 1.1.3 Released
The Apache Storm community is pleased to announce the release of Apache Storm version 1.1.3. Storm is a distributed, fault-tolerant, and high-performance realtime computation system that provides strong guarantees on the processing of data. You can read more about Storm on the project website: http://storm.apache.org Downloads of source and binary distributions are listed in our download section: http://storm.apache.org/downloads.html You can read more about this release in the following blog post: http://storm.apache.org/2018/02/19/storm121-released.html Distribution artifacts are available in Maven Central at the following coordinates: groupId: org.apache.storm artifactId: storm-core version: 1.1.3 The full list of changes is available here[1]. Please let us know [2] if you encounter any problems. Regards, The Apache Storm Team [1]: http://www.us.apache.org/dist/storm/apache-storm-1.1.3/RELEASE_NOTES.html [2]: https://issues.apache.org/jira/browse/STORM
[ANNOUNCE] Apache Storm 1.2.2 Released
The Apache Storm community is pleased to announce the release of Apache Storm version 1.2.2. Storm is a distributed, fault-tolerant, and high-performance realtime computation system that provides strong guarantees on the processing of data. You can read more about Storm on the project website: http://storm.apache.org Downloads of source and binary distributions are listed in our download section: http://storm.apache.org/downloads.html You can read more about this release in the following blog post: http://storm.apache.org/2018/02/19/storm121-released.html Distribution artifacts are available in Maven Central at the following coordinates: groupId: org.apache.storm artifactId: storm-core version: 1.2.2 The full list of changes is available here[1]. Please let us know [2] if you encounter any problems. Regards, The Apache Storm Team [1]: http://www.us.apache.org/dist/storm/apache-storm-1.2.2/RELEASE_NOTES.html [2]: https://issues.apache.org/jira/browse/STORM
[GitHub] storm issue #2702: STORM-3092: Refactor code to correct launch initializatio...
Github user danny0405 commented on the issue: https://github.com/apache/storm/pull/2702 +1 thx for your contribution. ---
[GitHub] storm pull request #2698: STORM-2882: shade storm-client dependencies
Github user srdo commented on a diff in the pull request:
https://github.com/apache/storm/pull/2698#discussion_r192818795
--- Diff:
storm-client/src/jvm/org/apache/storm/container/cgroup/SystemOperation.java ---
@@ -54,8 +55,8 @@ public static String exec(String cmd) throws IOException {
Process process = new ProcessBuilder(new String[]{ "/bin/bash",
"-c", cmd }).start();
try {
process.waitFor();
-String output = IOUtils.toString(process.getInputStream());
-String errorOutput =
IOUtils.toString(process.getErrorStream());
+String output = IOUtils.toString(process.getInputStream(),
Charsets.UTF_8);
--- End diff --
Won't this break if the system's standard charset isn't UTF-8?
---
[GitHub] storm pull request #2698: STORM-2882: shade storm-client dependencies
Github user srdo commented on a diff in the pull request: https://github.com/apache/storm/pull/2698#discussion_r192816102 --- Diff: shaded-deps/pom.xml --- @@ -0,0 +1,272 @@ + + + +http://maven.apache.org/POM/4.0.0; + xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance; + xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd;> +4.0.0 + + +storm +org.apache.storm +2.0.0-SNAPSHOT +.. + + +org.apache.storm +shaded-deps +jar +Shaded Deps for Storm Client +Shaded version of dependencies used only for internal storm code. + + + --- End diff -- I think jaxb-api is missing from here. It was in the dependency list for storm-client. ---
[GitHub] storm pull request #2698: STORM-2882: shade storm-client dependencies
Github user srdo commented on a diff in the pull request:
https://github.com/apache/storm/pull/2698#discussion_r192822576
--- Diff:
storm-client/test/jvm/org/apache/storm/security/auth/ClientAuthUtilsTest.java
---
@@ -172,59 +178,47 @@ public void makeDigestPayloadTest() throws
NoSuchAlgorithmException {
public void invalidConfigResultsInIOException() throws
RuntimeException {
HashMap conf = new HashMap<>();
conf.put("java.security.auth.login.config", "__FAKE_FILE__");
-Assert.assertNotNull(AuthUtils.GetConfiguration(conf));
+Assert.assertNotNull(ClientAuthUtils.getConfiguration(conf));
}
@Test
public void validConfigResultsInNotNullConfigurationTest() throws
IOException {
File file1 = folder.newFile("mockfile.txt");
HashMap conf = new HashMap<>();
conf.put("java.security.auth.login.config",
file1.getAbsolutePath());
-Assert.assertNotNull(AuthUtils.GetConfiguration(conf));
-}
-
-@Test
-public void uiHttpCredentialsPluginTest() {
-Map conf = new HashMap<>();
-conf.put(
-Config.UI_HTTP_CREDS_PLUGIN,
-"org.apache.storm.security.auth.AuthUtilsTestMock");
-conf.put(
-Config.DRPC_HTTP_CREDS_PLUGIN,
-"org.apache.storm.security.auth.AuthUtilsTestMock");
-conf.put(
-Config.STORM_PRINCIPAL_TO_LOCAL_PLUGIN,
-"org.apache.storm.security.auth.AuthUtilsTestMock");
-conf.put(
-Config.STORM_GROUP_MAPPING_SERVICE_PROVIDER_PLUGIN,
-"org.apache.storm.security.auth.AuthUtilsTestMock");
-
-Assert.assertTrue(
-AuthUtils.GetUiHttpCredentialsPlugin(conf).getClass() ==
AuthUtilsTestMock.class);
-Assert.assertTrue(
-AuthUtils.GetDrpcHttpCredentialsPlugin(conf).getClass() ==
AuthUtilsTestMock.class);
-Assert.assertTrue(
-AuthUtils.GetPrincipalToLocalPlugin(conf).getClass() ==
AuthUtilsTestMock.class);
-Assert.assertTrue(
-
AuthUtils.GetGroupMappingServiceProviderPlugin(conf).getClass() ==
AuthUtilsTestMock.class);
+Assert.assertNotNull(ClientAuthUtils.getConfiguration(conf));
}
@Test(expected = RuntimeException.class)
public void updateSubjectWithNullThrowsTest() {
-AuthUtils.updateSubject(null, null, null);
+ClientAuthUtils.updateSubject(null, null, null);
}
@Test(expected = RuntimeException.class)
public void updateSubjectWithNullAutosThrowsTest() {
-AuthUtils.updateSubject(new Subject(), null, null);
+ClientAuthUtils.updateSubject(new Subject(), null, null);
}
@Test
public void updateSubjectWithNullAutosTest() {
AuthUtilsTestMock mock = Mockito.mock(AuthUtilsTestMock.class);
Collection autos = Arrays.asList(new
IAutoCredentials[]{ mock });
Subject s = new Subject();
-AuthUtils.updateSubject(s, autos, null);
+ClientAuthUtils.updateSubject(s, autos, null);
Mockito.verify(mock, Mockito.times(1)).updateSubject(s, null);
}
+
+@Test
+public void uiHttpCredentialsPluginTest() {
--- End diff --
Is this test correctly named?
---
[GitHub] storm pull request #2698: STORM-2882: shade storm-client dependencies
Github user srdo commented on a diff in the pull request: https://github.com/apache/storm/pull/2698#discussion_r192822955 --- Diff: storm-core/pom.xml --- @@ -193,10 +193,6 @@ curator-framework compile - --- End diff -- Why is only Zookeeper being removed? ---
[GitHub] storm pull request #2698: STORM-2882: shade storm-client dependencies
Github user srdo commented on a diff in the pull request: https://github.com/apache/storm/pull/2698#discussion_r192823787 --- Diff: storm-server/src/main/java/org/apache/storm/testing/InProcessZookeeper.java --- @@ -12,15 +12,16 @@ package org.apache.storm.testing; -import java.util.List; +import org.apache.storm.shade.org.apache.zookeeper.server.NIOServerCnxnFactory; import org.apache.storm.zookeeper.Zookeeper; -import org.apache.zookeeper.server.NIOServerCnxnFactory; /** * A local Zookeeper instance available for testing. + * ```java --- End diff -- I don't think this will get formatted correctly unless markdown syntax is allowed in Javadoc? ---
[GitHub] storm pull request #2698: STORM-2882: shade storm-client dependencies
Github user srdo commented on a diff in the pull request: https://github.com/apache/storm/pull/2698#discussion_r192814731 --- Diff: shaded-deps/pom.xml --- @@ -0,0 +1,272 @@ + + + +http://maven.apache.org/POM/4.0.0; + xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance; + xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd;> +4.0.0 + + +storm +org.apache.storm +2.0.0-SNAPSHOT +.. + + +org.apache.storm +shaded-deps +jar +Shaded Deps for Storm Client +Shaded version of dependencies used only for internal storm code. + + + +com.google.guava +guava + + +org.apache.curator +curator-framework + + +org.apache.curator +curator-client + + + +org.apache.curator +curator-recipes + + +org.apache.zookeeper +zookeeper + + + +jline +jline + + + +org.apache.yetus +audience-annotations + + + + +commons-io +commons-io +compile --- End diff -- Nit: I think compile scope is the default ---
[GitHub] storm pull request #2698: STORM-2882: shade storm-client dependencies
Github user srdo commented on a diff in the pull request: https://github.com/apache/storm/pull/2698#discussion_r192814411 --- Diff: shaded-deps/pom.xml --- @@ -0,0 +1,272 @@ + + + +http://maven.apache.org/POM/4.0.0; + xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance; + xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd;> +4.0.0 + + +storm +org.apache.storm +2.0.0-SNAPSHOT +.. + + +org.apache.storm +shaded-deps +jar +Shaded Deps for Storm Client +Shaded version of dependencies used only for internal storm code. + + + +com.google.guava +guava + + +org.apache.curator +curator-framework + + +org.apache.curator +curator-client + +
[GitHub] storm pull request #2685: STORM-1294: Port netty-unit-test to Java
Github user asfgit closed the pull request at: https://github.com/apache/storm/pull/2685 ---
[GitHub] storm pull request #2700: [STORM-3093] Cache the storm id to executors mappi...
Github user kishorvpatil commented on a diff in the pull request:
https://github.com/apache/storm/pull/2700#discussion_r192811084
--- Diff:
storm-server/src/main/java/org/apache/storm/daemon/nimbus/Nimbus.java ---
@@ -1333,6 +1336,21 @@ private AssignmentDistributionService
getAssignmentsDistributer() {
return heartbeatsCache;
}
+public AtomicReference>>>
getIdToExecutors() {
+return idToExecutors;
+}
+
+private Set> getOrUpdateExecutors(String topoId,
StormBase base, Map topoConf,
+StormTopology topology)
+throws IOException, AuthorizationException,
InvalidTopologyException, KeyNotFoundException {
+Set> executors = idToExecutors.get().get(topoId);
+if (null == executors) {
+executors = new HashSet<>(computeExecutors(topoId, base,
topoConf, topology));
+idToExecutors.getAndUpdate(new Assoc<>(topoId, executors));
--- End diff --
Do we intend to create new HashMap every time? If not, would
`idToExecutors.get().put(topoId, executors);` should suffice?
---
[GitHub] storm pull request #2700: [STORM-3093] Cache the storm id to executors mappi...
Github user asfgit closed the pull request at: https://github.com/apache/storm/pull/2700 ---
[GitHub] storm issue #2700: [STORM-3093] Cache the storm id to executors mapping on m...
Github user revans2 commented on the issue: https://github.com/apache/storm/pull/2700 The travis failure looks unrelated. ---
[GitHub] storm issue #2691: STORM-3061: Update version of hbase
Github user revans2 commented on the issue: https://github.com/apache/storm/pull/2691 I am planning right now to get STORM-2882 in first, and then I will come back and do as much manual testing as possible for the different components, and update thing accordingly. ---
[GitHub] storm issue #2698: STORM-2882: shade storm-client dependencies
Github user revans2 commented on the issue: https://github.com/apache/storm/pull/2698 @HeartSaVioR I have addressed the other review comments, but I could not find a place in the documentation for instructions on how to do a release. I agree that we want to publish the shaded-deps too, or clients will not be able to find their dependencies. Just not sure where that is documented, or what scripts there are that do it, if any. ---
[GitHub] storm pull request #2702: STORM-3092: Refactor code to correct launch initia...
GitHub user zd-project opened a pull request: https://github.com/apache/storm/pull/2702 STORM-3092: Refactor code to correct launch initialization of Supervisor Fixed the issue STORM-3092 You can merge this pull request into a Git repository by running: $ git pull https://github.com/zd-project/storm master Alternatively you can review and apply these changes as the patch at: https://github.com/apache/storm/pull/2702.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2702 commit 44e272735688113ff0b49de6aa1cab2fa9e9fdfc Author: Zhengdai Date: 2018-06-04T15:44:48Z STORM-3092: Refactor code to correct launch initialization of Supervisor ---
[GitHub] storm issue #2700: [STORM-3093] Cache the storm id to executors mapping on m...
Github user Ethanlm commented on the issue: https://github.com/apache/storm/pull/2700 @danny0405 No it's fine. Thanks for the contribution. ---
[GitHub] storm issue #2700: [STORM-3093] Cache the storm id to executors mapping on m...
Github user danny0405 commented on the issue: https://github.com/apache/storm/pull/2700 @Ethanlm Sorry i have not did the benchmark test, but we can see the promotion through the storm-core module building time, it's about 10 seconds faster on my laptop. I can add in the test cases if if it needed. ---
[GitHub] storm issue #2701: STORM-3091 don't allow workers to create heartbeat direct...
Github user danny0405 commented on the issue: https://github.com/apache/storm/pull/2701 + 1, thx for your contribution. ---
[GitHub] storm issue #2700: [STORM-3093] Cache the storm id to executors mapping on m...
Github user Ethanlm commented on the issue: https://github.com/apache/storm/pull/2700 Looks good. Do you have any performance test results to share so that we can have an idea how much performance gain this patch gets? It's fine if you don't have it ---
[GitHub] storm pull request #2701: STORM-3091 don't allow workers to create heartbeat...
Github user revans2 commented on a diff in the pull request:
https://github.com/apache/storm/pull/2701#discussion_r192757264
--- Diff: storm-client/src/jvm/org/apache/storm/utils/VersionedStore.java
---
@@ -25,9 +25,17 @@
private String _root;
-public VersionedStore(String path) throws IOException {
+/**
+ * Creates a store at the given path.
+ *
+ * @path The path for the store
--- End diff --
Should be
`@param path The path for the store`
---
[GitHub] storm pull request #2701: STORM-3091 don't allow workers to create heartbeat...
GitHub user agresch opened a pull request: https://github.com/apache/storm/pull/2701 STORM-3091 don't allow workers to create heartbeat directory You can merge this pull request into a Git repository by running: $ git pull https://github.com/agresch/storm agresch_storm-3091 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/storm/pull/2701.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2701 commit dbae31d8507aea5992d2e669da1de43d66442075 Author: Aaron Gresch Date: 2018-06-04T14:16:10Z STORM-3091 don't allow workers to create heartbeat directory ---
