[GitHub] storm issue #459: [STORM-2999] Add generics to static type of method paramet...
Github user knusbaum commented on the issue: https://github.com/apache/storm/pull/459 +1 ---
[GitHub] storm issue #459: avoiding rawtypes in IBold.prepare in order to allow usage...
Github user knusbaum commented on the issue: https://github.com/apache/storm/pull/459 @krichter722 I would either rename that Jira and update the description a bit, or open a new one. Once you've done that, add the jira number into the PR title - look at other Storm PRs for examples. After that and the checkstyle stuff, I don't see anything else that should prevent this from going in. :+1: ---
[GitHub] storm pull request #459: avoiding rawtypes in IBold.prepare in order to allo...
Github user knusbaum commented on a diff in the pull request: https://github.com/apache/storm/pull/459#discussion_r175268334 --- Diff: external/storm-kafka/pom.xml --- @@ -57,7 +57,7 @@ maven-checkstyle-plugin -557 +561 --- End diff -- What we did with checkstyle, for better or worse, was agree on a set of rules and then apply those rules. We didn't go through and fix every checkstyle violation when we did that. However, we agreed that any pull requests should not increase the number of alerts, and preferably decrease them. It would be better if you could reformat any new 80+character lines to please the checkstyle monster. ---
[GitHub] storm issue #459: avoiding rawtypes in IBold.prepare in order to allow usage...
Github user knusbaum commented on the issue: https://github.com/apache/storm/pull/459 @krichter722 Yes, these maps are vestiges of the old clojure code. It would probably be preferable to do as you suggest, but I'm not sure how much work there is to do there. IIRC, the config objects are pretty well spread and used throughout the code, and there's some other support code, tests, validation, etc. That may have to be significantly changed. I think it's definitely worth consideration, though. Maybe a dev list email would be a good place to get opinions. ---
[GitHub] storm issue #459: avoiding rawtypes in IBold.prepare in order to allow usage...
Github user knusbaum commented on the issue: https://github.com/apache/storm/pull/459 Yes, I'll retracting my -1. Sorry for letting this one languish. ---
[GitHub] storm issue #2347: STORM-2760: Add Blobstore Migration Scripts
Github user knusbaum commented on the issue: https://github.com/apache/storm/pull/2347 integration test failure looks unrelated. ---
[GitHub] storm pull request #2041: STORM-2452: Storm Metric classes are not thread sa...
Github user knusbaum closed the pull request at: https://github.com/apache/storm/pull/2041 ---
[GitHub] storm pull request #1817: STORM-2237: Nimbus reports bad supervisor heartbea...
Github user knusbaum closed the pull request at: https://github.com/apache/storm/pull/1817 ---
[GitHub] storm pull request #2347: STORM-2760: Add Blobstore Migration Scripts
Github user knusbaum commented on a diff in the pull request:
https://github.com/apache/storm/pull/2347#discussion_r141420947
--- Diff:
external/storm-blobstore-migration/src/main/java/org/apache/storm/blobstore/ListHDFS.java
---
@@ -0,0 +1,52 @@
+package org.apache.storm.blobstore;
+
+import java.util.Map;
+
+import javax.security.auth.Subject;
+
+import org.apache.storm.Config;
+import org.apache.storm.blobstore.ClientBlobStore;
+import org.apache.storm.hdfs.blobstore.HdfsBlobStore;
+import org.apache.storm.hdfs.blobstore.HdfsClientBlobStore;
+import org.apache.storm.utils.Utils;
+
+public class ListHDFS {
+
+public static void main(String[] args) throws Exception {
--- End diff --
[STORM-2763](https://issues.apache.org/jira/browse/STORM-2763)
---
[GitHub] storm pull request #2347: STORM-2760: Add Blobstore Migration Scripts
GitHub user knusbaum opened a pull request: https://github.com/apache/storm/pull/2347 STORM-2760: Add Blobstore Migration Scripts Add code and helper scripts for migrating active Storm clusters from a locally-backed BlobStore to an HDFS-backed BlobStore. You can merge this pull request into a Git repository by running: $ git pull https://github.com/knusbaum/incubator-storm blobstore-migrator Alternatively you can review and apply these changes as the patch at: https://github.com/apache/storm/pull/2347.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2347 commit 4af7da0ff48bd347aa49509744c0e6a8e749341f Author: Kyle Nusbaum <knusb...@yahoo-inc.com> Date: 2017-09-26T20:17:32Z Adding storm blobstore migrator. ---
[GitHub] storm pull request #2338: [STORM-2744] add in restart timeout for backpressu...
Github user knusbaum commented on a diff in the pull request:
https://github.com/apache/storm/pull/2338#discussion_r140345783
--- Diff:
storm-client/src/jvm/org/apache/storm/cluster/StormClusterStateImpl.java ---
@@ -423,32 +424,35 @@ public void supervisorHeartbeat(String supervisorId,
SupervisorInfo info) {
}
/**
- * if znode exists and to be not on?, delete; if exists and on?, do
nothing; if not exists and to be on?, create; if not exists and not on?, do
nothing;
+ * if znode exists and timestamp is 0?, delete; if exists and
timestamp is larger than 0?, do nothing;
--- End diff --
The question mark on the `on` in the old comment appears to be left-over
from the clojure translation where the on argument was spelled `on?`.
It should go away in any new comment.
---
[GitHub] storm issue #2330: STORM-2742: Logviewer leaking file descriptors
Github user knusbaum commented on the issue:
https://github.com/apache/storm/pull/2330
I just added a try {...} finally {...} block. Not sure why the diff is so
ugly.
---
[GitHub] storm pull request #2330: STORM-2742: Logviewer leaking file descriptors
GitHub user knusbaum opened a pull request: https://github.com/apache/storm/pull/2330 STORM-2742: Logviewer leaking file descriptors You can merge this pull request into a Git repository by running: $ git pull https://github.com/knusbaum/incubator-storm STORM-2742 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/storm/pull/2330.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2330 commit 1e81a907fd3722601e3d1e765baf4a9360b95e99 Author: Kyle Nusbaum <knusb...@yahoo-inc.com> Date: 2017-09-18T20:06:07Z Fix FD leak ---
[GitHub] storm issue #2289: STORM-2702: storm-loadgen
Github user knusbaum commented on the issue: https://github.com/apache/storm/pull/2289 +1 ---
[GitHub] storm pull request #2289: STORM-2702: storm-loadgen
Github user knusbaum commented on a diff in the pull request:
https://github.com/apache/storm/pull/2289#discussion_r135317773
--- Diff:
examples/storm-loadgen/src/main/java/org/apache/storm/loadgen/InputStream.java
---
@@ -0,0 +1,263 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.storm.loadgen;
+
+import java.io.ByteArrayInputStream;
+import java.io.ObjectInputStream;
+import java.io.Serializable;
+import java.util.HashMap;
+import java.util.Map;
+import org.apache.storm.generated.GlobalStreamId;
+import org.apache.storm.generated.Grouping;
+import org.apache.storm.grouping.PartialKeyGrouping;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A set of measurements about a stream so we can statistically reproduce
it.
+ */
+public class InputStream implements Serializable {
+private static final Logger LOG =
LoggerFactory.getLogger(InputStream.class);
+public final String fromComponent;
+public final String toComponent;
+public final String id;
+public final NormalDistStats execTime;
+public final NormalDistStats processTime;
+public final GroupingType groupingType;
+//Cached GlobalStreamId
+private GlobalStreamId gsid = null;
+
+/**
+ * Create an output stream from a config.
+ * @param conf the config to read from.
+ * @return the read OutputStream.
+ */
+public static InputStream fromConf(Map<String, Object> conf) {
+String component = (String) conf.get("from");
+String toComp = (String) conf.get("to");
+NormalDistStats execTime = NormalDistStats.fromConf((Map<String,
Object>) conf.get("execTime"));
+NormalDistStats processTime =
NormalDistStats.fromConf((Map<String, Object>) conf.get("processTime"));
+Map<String, Object> grouping = (Map<String, Object>)
conf.get("grouping");
+GroupingType groupingType = GroupingType.fromConf((String)
grouping.get("type"));
+String streamId = (String) grouping.getOrDefault("streamId",
"default");
+return new InputStream(component, toComp, streamId, execTime,
processTime, groupingType);
+}
+
+/**
+ * Convert this to a conf.
+ * @return the conf.
+ */
+public Map<String, Object> toConf() {
+Map<String, Object> ret = new HashMap<>();
+ret.put("from", fromComponent);
+ret.put("to", toComponent);
+ret.put("execTime", execTime.toConf());
+ret.put("processTime", processTime.toConf());
+
+Map<String, Object> grouping = new HashMap<>();
+grouping.put("streamId", id);
+grouping.put("type", groupingType.toConf());
+ret.put("grouping", grouping);
+
+return ret;
+}
+
+public static class Builder {
+private String fromComponent;
+private String toComponent;
+private String id;
+private NormalDistStats execTime;
+private NormalDistStats processTime;
+private GroupingType groupingType = GroupingType.SHUFFLE;
+
+public String getFromComponent() {
--- End diff --
Ahh, sorry. I missed their use.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm issue #2289: STORM-2702: storm-loadgen
Github user knusbaum commented on the issue: https://github.com/apache/storm/pull/2289 Partial review done. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request #2289: STORM-2702: storm-loadgen
Github user knusbaum commented on a diff in the pull request:
https://github.com/apache/storm/pull/2289#discussion_r135084038
--- Diff:
examples/storm-loadgen/src/main/java/org/apache/storm/loadgen/CaptureLoad.java
---
@@ -0,0 +1,468 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.storm.loadgen;
+
+import java.io.File;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+import org.apache.commons.cli.CommandLine;
+import org.apache.commons.cli.CommandLineParser;
+import org.apache.commons.cli.DefaultParser;
+import org.apache.commons.cli.HelpFormatter;
+import org.apache.commons.cli.Option;
+import org.apache.commons.cli.Options;
+import org.apache.commons.cli.ParseException;
+import org.apache.storm.Config;
+import org.apache.storm.generated.Bolt;
+import org.apache.storm.generated.BoltStats;
+import org.apache.storm.generated.ClusterSummary;
+import org.apache.storm.generated.ComponentCommon;
+import org.apache.storm.generated.ExecutorSummary;
+import org.apache.storm.generated.GlobalStreamId;
+import org.apache.storm.generated.Grouping;
+import org.apache.storm.generated.Nimbus;
+import org.apache.storm.generated.SpoutSpec;
+import org.apache.storm.generated.StormTopology;
+import org.apache.storm.generated.StreamInfo;
+import org.apache.storm.generated.TopologyInfo;
+import org.apache.storm.generated.TopologyPageInfo;
+import org.apache.storm.generated.TopologySummary;
+import org.apache.storm.generated.WorkerSummary;
+import org.apache.storm.utils.NimbusClient;
+import org.apache.storm.utils.ObjectReader;
+import org.json.simple.JSONObject;
+import org.json.simple.JSONValue;
+import org.json.simple.parser.JSONParser;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Capture running topologies for load gen later on.
+ */
+public class CaptureLoad {
+private static final Logger LOG =
LoggerFactory.getLogger(CaptureLoad.class);
+public static final String DEFAULT_OUT_DIR = "./loadgen/";
+
+private static List extractBoltValues(List
summaries,
+ GlobalStreamId id,
+ Function<BoltStats,
Map<String, Map<GlobalStreamId, Double>>> func) {
+
+List ret = new ArrayList<>();
+if (summaries != null) {
+for (ExecutorSummary summ : summaries) {
+if (summ != null && summ.is_set_stats()) {
+Map<String, Map<GlobalStreamId, Double>> data =
func.apply(summ.get_stats().get_specific().get_bolt());
+if (data != null) {
+List subvalues = data.values().stream()
+.map((subMap) -> subMap.get(id))
+.filter((value) -> value != null)
+.mapToDouble((value) -> value.doubleValue())
+.boxed().collect(Collectors.toList());
+ret.addAll(subvalues);
+}
+}
+}
+}
+return ret;
+}
+
+static TopologyLoadConf captureTopology(Nimbus.Iface client,
TopologySummary topologySummary) throws Exception {
+String topologyName = topologySummary.get_name();
+LOG.info("Capturing {}...", topologyName);
+String topologyId = topologySummary.get_id();
+TopologyInfo info = client.getTopologyInfo(topologyId);
+TopologyPageInfo tpinfo = client.getTopologyPageInfo(topologyId,
":all-time", false);
+@SuppressWarnings("checkstyle:VariableDeclarationUsageDistance&qu
[GitHub] storm pull request #2289: STORM-2702: storm-loadgen
Github user knusbaum commented on a diff in the pull request: https://github.com/apache/storm/pull/2289#discussion_r135073224 --- Diff: examples/storm-loadgen/README.md --- @@ -0,0 +1,195 @@ +# Storm Load Generation Tools + +A set of tools to place an artificial load on a storm cluster to compare against a different storm cluster. This is particularly helpful when making changes to the data path in storm to see what if any impact the changes had. This is also useful for end users that want to compare different hardware setups to see what the trade-offs are, although actually running your real topologies is going to be more accurate. + +## Methodology +The idea behind all of these tools is to measure the trade-offs between latency, throughput, and cost when processing data using Apache Storm. + +When processing data you typically will know a few things. First you will know about how much data you are going to be processing. This will typically be a range of values that change throughput the day. You also will have an idea of how quickly you need the data processed by. Often this is measured in terms of the latency it takes to process data at the some percentile or set of percentiles. This is because of most use cases the value of the data declines over time, and being able to react to the data quickly is more valuable. You probably also have a budget for how much you are willing to spend to be able to process this data. There are always trade-offs in how quickly you can process some data and how efficiently you can processes that data both in terms of resource usage (cost) and latency. These tools are designed to help you explore that space. --- End diff -- Couple nits: 1. ``` Often this is measured in terms of the latency it takes to process data at the some percentile or set of percentiles. ``` `the some` 2. ```This is because of most use cases ...``` *in* most use cases? *for* most use cases? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request #2289: STORM-2702: storm-loadgen
Github user knusbaum commented on a diff in the pull request: https://github.com/apache/storm/pull/2289#discussion_r135074782 --- Diff: examples/storm-loadgen/README.md --- @@ -0,0 +1,195 @@ +# Storm Load Generation Tools + +A set of tools to place an artificial load on a storm cluster to compare against a different storm cluster. This is particularly helpful when making changes to the data path in storm to see what if any impact the changes had. This is also useful for end users that want to compare different hardware setups to see what the trade-offs are, although actually running your real topologies is going to be more accurate. + +## Methodology +The idea behind all of these tools is to measure the trade-offs between latency, throughput, and cost when processing data using Apache Storm. + +When processing data you typically will know a few things. First you will know about how much data you are going to be processing. This will typically be a range of values that change throughput the day. You also will have an idea of how quickly you need the data processed by. Often this is measured in terms of the latency it takes to process data at the some percentile or set of percentiles. This is because of most use cases the value of the data declines over time, and being able to react to the data quickly is more valuable. You probably also have a budget for how much you are willing to spend to be able to process this data. There are always trade-offs in how quickly you can process some data and how efficiently you can processes that data both in terms of resource usage (cost) and latency. These tools are designed to help you explore that space. + +A note on how latency is measured. Storm typically measures latency from when a message is emitted by a spout until the point it is fully acked or failed (in many versions of storm it actually does this in the acker instead of the spout so it is trying to be a measure of how long it takes for the actual processing, removing as much of the acker overhead as possible). For these tools we do it differently. We simulate a throughput and measure the start time of the tuple from when it would have been emitted if the topology could keep up with the load. In the normal case this should not be an issue, but if the topology cannot keep up with the throughput you will see the latency grow very high compared to the latency reported by storm. + +## Tools +### CaptureLoad + +`CaptureLoad` will look at the topologies on a running cluster and store the structure of and metrics about each of theses topologies storing them in a format that can be used later to reproduce a similar load on the cluster. + + Usage +``` +storm jar storm-loadgen.jar org.apache.storm.loadgen.CaptureLoad [options] [topologyName]* +``` +|Option| Description| +|-|-| +|-a,--anonymize | Strip out any possibly identifiable information| +| -h,--help | Print a help message | +| -o,--output-dir | Where to write (defaults to ./loadgen/)| + + Limitations +This is still a work in progress. It does not currently capture CPU or memory usage of a topology. Resource requests (used by RAS when scheduling) within the topology are also not captured yet, nor is the user that actually ran the topology. + +### GenLoad + +`GenLoad` will take the files produced by `CaptureLoad` and replay them in a simulated way on a cluster. It also offers lots of ways to capture metrics about those simulated topologies to be able to compare different software versions of different hardware setups. You can also make adjustments to the topology before submitting it to change the size or throughput of the topology. + +### Usage +``` +storm jar storm-loadgen.jar org.apache.storm.loadgen.GenLoad [options] [capture_file]* +``` + +|Option| Description| +|-|-| +| --debug | Print debug information about the adjusted topology before submitting it. | +|-h,--help | Print a help message | +| --local-or-shuffle | Replace shuffle grouping with local or shuffle grouping. | +| --parallel MULTIPLIER(:TOPO:COMP)?> | How much to scale the topology up or down in parallelism. The new parallelism will round up to the next whole number. If a topology + component is supplied only that component will be scaled. If topo or component is blank or a `'*'` all topologies or components matched the other part will be scaled. Only 1 scaling rule, the most specific, will be applied to a component. Providing a topology name is considered more specific than not providing one. (defaults to 1.0 no scaling) | +| -r,--report-interval INTERVAL_SECS> | How long in between reported metrics. Will be rounded up to the next 10 sec boundary. default 30 | +| --reporter TYPE:FILE?OPTIONS>
[GitHub] storm pull request #2289: STORM-2702: storm-loadgen
Github user knusbaum commented on a diff in the pull request:
https://github.com/apache/storm/pull/2289#discussion_r135078488
--- Diff:
examples/storm-loadgen/src/main/java/org/apache/storm/loadgen/CaptureLoad.java
---
@@ -0,0 +1,468 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.storm.loadgen;
+
+import java.io.File;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+import org.apache.commons.cli.CommandLine;
+import org.apache.commons.cli.CommandLineParser;
+import org.apache.commons.cli.DefaultParser;
+import org.apache.commons.cli.HelpFormatter;
+import org.apache.commons.cli.Option;
+import org.apache.commons.cli.Options;
+import org.apache.commons.cli.ParseException;
+import org.apache.storm.Config;
+import org.apache.storm.generated.Bolt;
+import org.apache.storm.generated.BoltStats;
+import org.apache.storm.generated.ClusterSummary;
+import org.apache.storm.generated.ComponentCommon;
+import org.apache.storm.generated.ExecutorSummary;
+import org.apache.storm.generated.GlobalStreamId;
+import org.apache.storm.generated.Grouping;
+import org.apache.storm.generated.Nimbus;
+import org.apache.storm.generated.SpoutSpec;
+import org.apache.storm.generated.StormTopology;
+import org.apache.storm.generated.StreamInfo;
+import org.apache.storm.generated.TopologyInfo;
+import org.apache.storm.generated.TopologyPageInfo;
+import org.apache.storm.generated.TopologySummary;
+import org.apache.storm.generated.WorkerSummary;
+import org.apache.storm.utils.NimbusClient;
+import org.apache.storm.utils.ObjectReader;
+import org.json.simple.JSONObject;
+import org.json.simple.JSONValue;
+import org.json.simple.parser.JSONParser;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Capture running topologies for load gen later on.
+ */
+public class CaptureLoad {
+private static final Logger LOG =
LoggerFactory.getLogger(CaptureLoad.class);
+public static final String DEFAULT_OUT_DIR = "./loadgen/";
+
+private static List extractBoltValues(List
summaries,
+ GlobalStreamId id,
+ Function<BoltStats,
Map<String, Map<GlobalStreamId, Double>>> func) {
+
+List ret = new ArrayList<>();
+if (summaries != null) {
+for (ExecutorSummary summ : summaries) {
+if (summ != null && summ.is_set_stats()) {
+Map<String, Map<GlobalStreamId, Double>> data =
func.apply(summ.get_stats().get_specific().get_bolt());
+if (data != null) {
+List subvalues = data.values().stream()
+.map((subMap) -> subMap.get(id))
+.filter((value) -> value != null)
+.mapToDouble((value) -> value.doubleValue())
+.boxed().collect(Collectors.toList());
--- End diff --
I think the `mapToDouble` and the `boxed` are unnecessary.
They are inverse operations, first taking a `Stream` from
`filter(...)` and making a `DoubleStream`, then reboxing and turning the
`DoubleStream` into a `Stream`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request #2289: STORM-2702: storm-loadgen
Github user knusbaum commented on a diff in the pull request: https://github.com/apache/storm/pull/2289#discussion_r135074180 --- Diff: examples/storm-loadgen/README.md --- @@ -0,0 +1,195 @@ +# Storm Load Generation Tools + +A set of tools to place an artificial load on a storm cluster to compare against a different storm cluster. This is particularly helpful when making changes to the data path in storm to see what if any impact the changes had. This is also useful for end users that want to compare different hardware setups to see what the trade-offs are, although actually running your real topologies is going to be more accurate. + +## Methodology +The idea behind all of these tools is to measure the trade-offs between latency, throughput, and cost when processing data using Apache Storm. + +When processing data you typically will know a few things. First you will know about how much data you are going to be processing. This will typically be a range of values that change throughput the day. You also will have an idea of how quickly you need the data processed by. Often this is measured in terms of the latency it takes to process data at the some percentile or set of percentiles. This is because of most use cases the value of the data declines over time, and being able to react to the data quickly is more valuable. You probably also have a budget for how much you are willing to spend to be able to process this data. There are always trade-offs in how quickly you can process some data and how efficiently you can processes that data both in terms of resource usage (cost) and latency. These tools are designed to help you explore that space. + +A note on how latency is measured. Storm typically measures latency from when a message is emitted by a spout until the point it is fully acked or failed (in many versions of storm it actually does this in the acker instead of the spout so it is trying to be a measure of how long it takes for the actual processing, removing as much of the acker overhead as possible). For these tools we do it differently. We simulate a throughput and measure the start time of the tuple from when it would have been emitted if the topology could keep up with the load. In the normal case this should not be an issue, but if the topology cannot keep up with the throughput you will see the latency grow very high compared to the latency reported by storm. + +## Tools +### CaptureLoad + +`CaptureLoad` will look at the topologies on a running cluster and store the structure of and metrics about each of theses topologies storing them in a format that can be used later to reproduce a similar load on the cluster. --- End diff -- theses --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request #2289: STORM-2702: storm-loadgen
Github user knusbaum commented on a diff in the pull request:
https://github.com/apache/storm/pull/2289#discussion_r135085622
--- Diff:
examples/storm-loadgen/src/main/java/org/apache/storm/loadgen/EstimateThroughput.java
---
@@ -0,0 +1,108 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.storm.loadgen;
+
+import java.util.ArrayList;
+import java.util.List;
+import org.apache.commons.cli.CommandLine;
+import org.apache.commons.cli.CommandLineParser;
+import org.apache.commons.cli.DefaultParser;
+import org.apache.commons.cli.HelpFormatter;
+import org.apache.commons.cli.Option;
+import org.apache.commons.cli.Options;
+import org.apache.commons.cli.ParseException;
+import org.apache.storm.Config;
+import org.apache.storm.generated.ClusterSummary;
+import org.apache.storm.generated.Nimbus;
+import org.apache.storm.generated.TopologySummary;
+import org.apache.storm.loadgen.CaptureLoad;
+import org.apache.storm.utils.NimbusClient;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Estimate the throughput of all topologies.
+ */
+public class EstimateThroughput {
+private static final Logger LOG =
LoggerFactory.getLogger(EstimateThroughput.class);
+
+/**
+ * Main entry point for estimate throughput command.
+ * @param args the command line arguments.
+ * @throws Exception on any error.
+ */
+public static void main(String[] args) throws Exception {
+Options options = new Options();
+options.addOption(Option.builder("h")
+.longOpt("help")
+.desc("Print a help message")
+.build());
+CommandLineParser parser = new DefaultParser();
+CommandLine cmd = null;
+ParseException pe = null;
+try {
+cmd = parser.parse(options, args);
+} catch (ParseException e) {
+pe = e;
+}
+if (pe != null || cmd.hasOption('h')) {
+if (pe != null) {
+System.err.println("ERROR " + pe.getMessage());
--- End diff --
(same)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request #2289: STORM-2702: storm-loadgen
Github user knusbaum commented on a diff in the pull request:
https://github.com/apache/storm/pull/2289#discussion_r135086409
--- Diff:
examples/storm-loadgen/src/main/java/org/apache/storm/loadgen/InputStream.java
---
@@ -0,0 +1,263 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.storm.loadgen;
+
+import java.io.ByteArrayInputStream;
+import java.io.ObjectInputStream;
+import java.io.Serializable;
+import java.util.HashMap;
+import java.util.Map;
+import org.apache.storm.generated.GlobalStreamId;
+import org.apache.storm.generated.Grouping;
+import org.apache.storm.grouping.PartialKeyGrouping;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A set of measurements about a stream so we can statistically reproduce
it.
+ */
+public class InputStream implements Serializable {
+private static final Logger LOG =
LoggerFactory.getLogger(InputStream.class);
+public final String fromComponent;
+public final String toComponent;
+public final String id;
+public final NormalDistStats execTime;
+public final NormalDistStats processTime;
+public final GroupingType groupingType;
+//Cached GlobalStreamId
+private GlobalStreamId gsid = null;
+
+/**
+ * Create an output stream from a config.
+ * @param conf the config to read from.
+ * @return the read OutputStream.
+ */
+public static InputStream fromConf(Map<String, Object> conf) {
+String component = (String) conf.get("from");
+String toComp = (String) conf.get("to");
+NormalDistStats execTime = NormalDistStats.fromConf((Map<String,
Object>) conf.get("execTime"));
+NormalDistStats processTime =
NormalDistStats.fromConf((Map<String, Object>) conf.get("processTime"));
+Map<String, Object> grouping = (Map<String, Object>)
conf.get("grouping");
+GroupingType groupingType = GroupingType.fromConf((String)
grouping.get("type"));
+String streamId = (String) grouping.getOrDefault("streamId",
"default");
+return new InputStream(component, toComp, streamId, execTime,
processTime, groupingType);
+}
+
+/**
+ * Convert this to a conf.
+ * @return the conf.
+ */
+public Map<String, Object> toConf() {
+Map<String, Object> ret = new HashMap<>();
+ret.put("from", fromComponent);
+ret.put("to", toComponent);
+ret.put("execTime", execTime.toConf());
+ret.put("processTime", processTime.toConf());
+
+Map<String, Object> grouping = new HashMap<>();
+grouping.put("streamId", id);
+grouping.put("type", groupingType.toConf());
+ret.put("grouping", grouping);
+
+return ret;
+}
+
+public static class Builder {
+private String fromComponent;
+private String toComponent;
+private String id;
+private NormalDistStats execTime;
+private NormalDistStats processTime;
+private GroupingType groupingType = GroupingType.SHUFFLE;
+
+public String getFromComponent() {
--- End diff --
It doesn't really bother me, but do builders usually have getters?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm issue #2202: STORM-2623: Add in a whitelist for scheduler strategies
Github user knusbaum commented on the issue: https://github.com/apache/storm/pull/2202 @HeartSaVioR Sounds good. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm issue #2202: STORM-2623: Add in a whitelist for scheduler strategies
Github user knusbaum commented on the issue: https://github.com/apache/storm/pull/2202 @HeartSaVioR @kishorvpatil Please have a look. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm issue #2230: [STORM-2644] Show message when result not found on deep s...
Github user knusbaum commented on the issue: https://github.com/apache/storm/pull/2230 I would prefer *one* message if nothing is found. When there are hundreds of workers, this would make users scroll through many "Result not found" messages before getting to results that *are* returned. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm issue #2220: STORM-2638: Add in maximum object size for Pacemaker Thri...
Github user knusbaum commented on the issue: https://github.com/apache/storm/pull/2220 @vesense Done. I think I've cleared the checkstyle warnings on all the lines I changed. Let me know if I've missed something. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm issue #2220: STORM-2638: Add in maximum object size for Pacemaker Thri...
Github user knusbaum commented on the issue: https://github.com/apache/storm/pull/2220 @vesense Agreed. How do I run the linter? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request #2220: STORM-2638: Add in maximum object size for Pacemak...
GitHub user knusbaum opened a pull request: https://github.com/apache/storm/pull/2220 STORM-2638: Add in maximum object size for Pacemaker Thrift objects You can merge this pull request into a Git repository by running: $ git pull https://github.com/knusbaum/incubator-storm STORM-2638 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/storm/pull/2220.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2220 commit e6465b5dc5c7e31e0b92a18de578a13d05a5abf2 Author: Kyle Nusbaum <knusb...@yahoo-inc.com> Date: 2017-07-17T19:40:15Z Adding in maximum/minimum object size --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request #2202: STORM-2623: Add in a whitelist for scheduler strat...
GitHub user knusbaum opened a pull request: https://github.com/apache/storm/pull/2202 STORM-2623: Add in a whitelist for scheduler strategies You can merge this pull request into a Git repository by running: $ git pull https://github.com/knusbaum/incubator-storm STORM-2623 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/storm/pull/2202.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2202 commit eabfa7df94c22a6e1842a6d3bf3fdc9f345850dc Author: Kyle Nusbaum <knusb...@yahoo-inc.com> Date: 2017-07-10T21:30:22Z Adding whitelist for scheduler strategies --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm issue #2187: [STORM-2093] Fix permissions in multi-tenant, secure mode
Github user knusbaum commented on the issue: https://github.com/apache/storm/pull/2187 +1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request #2187: [STORM-2093] Fix permissions in multi-tenant, secu...
Github user knusbaum commented on a diff in the pull request: https://github.com/apache/storm/pull/2187#discussion_r125767551 --- Diff: storm-core/src/clj/org/apache/storm/daemon/logviewer.clj --- @@ -502,6 +503,20 @@ (-> (resp/response "Page not found") (resp/status 404) +(defnk set-log-file-permissions [fname root-dir] + (let [file (.getCanonicalFile (File. root-dir fname)) + run-as-user (*STORM-CONF* SUPERVISOR-RUN-WORKER-AS-USER) + parent (.getParent (File. root-dir fname)) + md-file (if (nil? parent) nil (get-metadata-file-for-wroker-logdir parent)) + topo-owner (if (nil? md-file) nil (get-topo-owner-from-metadata-file (.getCanonicalPath md-file)))] + (if (and run-as-user --- End diff -- Nit: since there is no 'else' branch, replacing `if` with `when` will allow you to remove the `do`. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request #2187: [STORM-2093] Fix permissions in multi-tenant, secu...
Github user knusbaum commented on a diff in the pull request: https://github.com/apache/storm/pull/2187#discussion_r125766911 --- Diff: storm-core/src/clj/org/apache/storm/daemon/logviewer.clj --- @@ -502,6 +503,20 @@ (-> (resp/response "Page not found") (resp/status 404) +(defnk set-log-file-permissions [fname root-dir] + (let [file (.getCanonicalFile (File. root-dir fname)) --- End diff -- Indentation on this function is off. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm issue #2180: [STORM-2602] storm.zookeeper.topology.auth.payload doesn'...
Github user knusbaum commented on the issue: https://github.com/apache/storm/pull/2180 +1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm issue #2183: STORM-2609: Simple command line DRPC Client
Github user knusbaum commented on the issue: https://github.com/apache/storm/pull/2183 +1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm issue #2050: STORM-2448: Add in Storm and JDK versions when submitting...
Github user knusbaum commented on the issue: https://github.com/apache/storm/pull/2050 +1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm issue #2051: STORM-2448: Add in Storm and JDK versions when submitting...
Github user knusbaum commented on the issue: https://github.com/apache/storm/pull/2051 +1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm issue #2049: STORM-2448: Add in Storm and JDK versions when submitting...
Github user knusbaum commented on the issue: https://github.com/apache/storm/pull/2049 +1 LGTM. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm issue #2129: [STORM-2206] - initial checkin of visualization using vis...
Github user knusbaum commented on the issue: https://github.com/apache/storm/pull/2129 Thanks a lot for this. It looks really good in general. One thing I noticed that could use some improvement is the bolt-info dropdown. The info can run off the window with no option to scroll. It seems like the scroll bar doesn't show up until the info is well out of view, and when it does show up, I can't scroll enough to bring everything back in view. Two more minor things: 1. When clicking on a spout, the info window still says "Bolt Info" 2. It's not exactly clear what the things labeled "Up:" are. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm issue #2130: STORM-2526: Revert changes mistakenly made to generated f...
Github user knusbaum commented on the issue: https://github.com/apache/storm/pull/2130 +1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm issue #2078: Add support for configuring memory load and cpu load in F...
Github user knusbaum commented on the issue: https://github.com/apache/storm/pull/2078 This looks good to me, but I'm pretty sure we've frozen features on 1.0.x-branch. It would be better to put it on 1.x-branch, and needs to go into master for 2.x as well. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm issue #2096: [STORM-2498] Fix Download Full File link in 1.x branch
Github user knusbaum commented on the issue: https://github.com/apache/storm/pull/2096 +1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm issue #2095: STORM-1642: Rethrow exception on serialization error and ...
Github user knusbaum commented on the issue: https://github.com/apache/storm/pull/2095 +1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm issue #2060: STORM-2468: Remove clojure from storm-client
Github user knusbaum commented on the issue: https://github.com/apache/storm/pull/2060 Missed the previous request. This all looks good to me. +1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request #2041: STORM-2452: Storm Metric classes are not thread sa...
GitHub user knusbaum opened a pull request: https://github.com/apache/storm/pull/2041 STORM-2452: Storm Metric classes are not thread safe Synchronizing metrics methods You can merge this pull request into a Git repository by running: $ git pull https://github.com/knusbaum/incubator-storm STORM-2452 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/storm/pull/2041.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2041 commit f02845f1314e2e7b21f1d5daf3918f2b00c96467 Author: Kyle Nusbaum <knusb...@yahoo-inc.com> Date: 2017-04-03T15:47:50Z Synchronizing metrics methods --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm issue #2013: STORM-2420: fix build
Github user knusbaum commented on the issue: https://github.com/apache/storm/pull/2013 +1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request #1973: STORM-2385: pacemaker_state_factory.clj does not c...
GitHub user knusbaum opened a pull request: https://github.com/apache/storm/pull/1973 STORM-2385: pacemaker_state_factory.clj does not compile on branch-1.0.x Code cherry-picked onto this branch in PR #1953 broke this. 1.0.x-branch does not have Pacemaker-HA. You can merge this pull request into a Git repository by running: $ git pull https://github.com/knusbaum/incubator-storm 1.0.x-branch Alternatively you can review and apply these changes as the patch at: https://github.com/apache/storm/pull/1973.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1973 commit 9b179bfd435525b29bf2eafc0fd22dbe2072fb9f Author: Kyle Nusbaum <knusb...@yahoo-inc.com> Date: 2017-02-27T19:51:10Z Revert bad change to pacemaker client. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request #1954: STORM-2372: Pacemaker client doesn't clean up hear...
GitHub user knusbaum reopened a pull request: https://github.com/apache/storm/pull/1954 STORM-2372: Pacemaker client doesn't clean up heartbeats properly. (2.x) See #1952 You can merge this pull request into a Git repository by running: $ git pull https://github.com/knusbaum/incubator-storm master Alternatively you can review and apply these changes as the patch at: https://github.com/apache/storm/pull/1954.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1954 commit a64f135d2859d178e4645787e99638a84e1b192f Author: Kyle Nusbaum <knusb...@yahoo-inc.com> Date: 2017-02-21T23:14:09Z Fixing pacemaker delete_pulse commit 6839fec10b94a56f6d9614547f4cabafcd9b355a Author: Kyle Nusbaum <knusb...@yahoo-inc.com> Date: 2017-02-22T18:26:31Z Fixing minor issue. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request #1954: STORM-2372: Pacemaker client doesn't clean up hear...
Github user knusbaum closed the pull request at: https://github.com/apache/storm/pull/1954 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm issue #1954: STORM-2372: Pacemaker client doesn't clean up heartbeats ...
Github user knusbaum commented on the issue: https://github.com/apache/storm/pull/1954 Testing done. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm issue #1954: STORM-2372: Pacemaker client doesn't clean up heartbeats ...
Github user knusbaum commented on the issue: https://github.com/apache/storm/pull/1954 Need to test this one 1.* branches are tested. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request #1954: STORM-2372: Pacemaker client doesn't clean up hear...
GitHub user knusbaum opened a pull request: https://github.com/apache/storm/pull/1954 STORM-2372: Pacemaker client doesn't clean up heartbeats properly. (2.x) See #1952 You can merge this pull request into a Git repository by running: $ git pull https://github.com/knusbaum/incubator-storm master Alternatively you can review and apply these changes as the patch at: https://github.com/apache/storm/pull/1954.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1954 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request #1953: STORM-2372: Pacemaker client doesn't clean up hear...
GitHub user knusbaum opened a pull request: https://github.com/apache/storm/pull/1953 STORM-2372: Pacemaker client doesn't clean up heartbeats properly. (1.0.x) See #1952 You can merge this pull request into a Git repository by running: $ git pull https://github.com/knusbaum/incubator-storm 1.0.x-branch Alternatively you can review and apply these changes as the patch at: https://github.com/apache/storm/pull/1953.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1953 commit a6b9668c98b46a7acfb9d4f39c6e90342b2f41f7 Author: Kyle Nusbaum <knusb...@yahoo-inc.com> Date: 2017-02-21T20:18:31Z Fixing pacemaker delete-path bug. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request #1952: STORM-2372: Pacemaker client doesn't clean up hear...
GitHub user knusbaum opened a pull request: https://github.com/apache/storm/pull/1952 STORM-2372: Pacemaker client doesn't clean up heartbeats properly. You can merge this pull request into a Git repository by running: $ git pull https://github.com/knusbaum/incubator-storm 1.x-branch Alternatively you can review and apply these changes as the patch at: https://github.com/apache/storm/pull/1952.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1952 commit ebed1c8b01397b09f4083e66f574a25f9b7c585d Author: Kyle Nusbaum <knusb...@yahoo-inc.com> Date: 2017-02-21T20:18:31Z Fixing pacemaker delete-path bug. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request #1759: STORM-2185: Storm Supervisor doesn't delete direct...
Github user knusbaum closed the pull request at: https://github.com/apache/storm/pull/1759 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm issue #1817: STORM-2237: Nimbus reports bad supervisor heartbeat - unk...
Github user knusbaum commented on the issue: https://github.com/apache/storm/pull/1817 I'll have a version up for 2.x as well. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request #1817: STORM-2237:Nimbus reports bad supervisor heartbeat...
GitHub user knusbaum opened a pull request: https://github.com/apache/storm/pull/1817 STORM-2237:Nimbus reports bad supervisor heartbeat - unknown version or thrift exception You can merge this pull request into a Git repository by running: $ git pull https://github.com/knusbaum/incubator-storm STORM-2237 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/storm/pull/1817.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1817 commit 6a2241d9795e0582bb5b1fb9bb469665247f3a72 Author: Kyle Nusbaum <knusb...@yahoo-inc.com> Date: 2016-12-07T23:41:40Z Fixing bad supervisor bug and adding list of bad supervisors to the UI. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request #1812: Updating Trident RAS Documentation.
GitHub user knusbaum opened a pull request: https://github.com/apache/storm/pull/1812 Updating Trident RAS Documentation. Adding documentation for defaults, clarifying other parts. You can merge this pull request into a Git repository by running: $ git pull https://github.com/knusbaum/incubator-storm Trident_RAS_Doc Alternatively you can review and apply these changes as the patch at: https://github.com/apache/storm/pull/1812.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1812 commit 32b42758b4dd8ba54e4201f2918405cc520ff7da Author: Kyle Nusbaum <knusb...@yahoo-inc.com> Date: 2016-12-02T19:46:21Z Updating Trident RAS Documentation. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request #1804: STORM-2222: Repeated NPEs thrown in nimbus if reba...
GitHub user knusbaum opened a pull request: https://github.com/apache/storm/pull/1804 STORM-: Repeated NPEs thrown in nimbus if rebalance fails You can merge this pull request into a Git repository by running: $ git pull https://github.com/knusbaum/incubator-storm 1.0.x-branch Alternatively you can review and apply these changes as the patch at: https://github.com/apache/storm/pull/1804.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1804 commit c9b80da918510e16ce239a4e8bfbb0b8c22f9358 Author: Kyle Nusbaum <knusb...@yahoo-inc.com> Date: 2016-11-28T19:09:17Z STORM- Repeated NPEs thrown in nimbus if rebalance fails --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm issue #1766: [STORM-2192] Add a new IAutoCredentials plugin to support...
Github user knusbaum commented on the issue: https://github.com/apache/storm/pull/1766 +1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request #1769: STORM-2195: Clean up some of worker-launcher code
GitHub user knusbaum opened a pull request: https://github.com/apache/storm/pull/1769 STORM-2195: Clean up some of worker-launcher code You can merge this pull request into a Git repository by running: $ git pull https://github.com/knusbaum/incubator-storm worker-launcher-comm Alternatively you can review and apply these changes as the patch at: https://github.com/apache/storm/pull/1769.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1769 commit eaa415a10a2d51aa87651c1adf7f68f2f0ac75de Author: Kyle Nusbaum <knusb...@yahoo-inc.com> Date: 2016-11-02T20:44:48Z Cleaning up some of worker-launcher --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm issue #1759: STORM-2185: Storm Supervisor doesn't delete directories p...
Github user knusbaum commented on the issue: https://github.com/apache/storm/pull/1759 @HeartSaVioR Are you using harshach's script to merge these? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request #1753: STORM-2180: KafkaSpout ack() should handle re-bala...
Github user knusbaum closed the pull request at: https://github.com/apache/storm/pull/1753 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm issue #1753: STORM-2180: KafkaSpout ack() should handle re-balance - 1...
Github user knusbaum commented on the issue: https://github.com/apache/storm/pull/1753 Closing for #1696 per request. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request #1752: STORM-2180: KafkaSpout ack() should handle re-bala...
Github user knusbaum closed the pull request at: https://github.com/apache/storm/pull/1752 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm issue #1752: STORM-2180: KafkaSpout ack() should handle re-balance
Github user knusbaum commented on the issue: https://github.com/apache/storm/pull/1752 Fine by me. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm issue #1735: STORM-203 Adding paths to default java library path
Github user knusbaum commented on the issue: https://github.com/apache/storm/pull/1735 +1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request #1758: STORM-2185: Storm Supervisor doesn't delete direct...
GitHub user knusbaum opened a pull request: https://github.com/apache/storm/pull/1758 STORM-2185: Storm Supervisor doesn't delete directories properly sometimes You can merge this pull request into a Git repository by running: $ git pull https://github.com/knusbaum/incubator-storm STORM-2185 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/storm/pull/1758.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1758 commit 3ac3b8329121c38de774e57f694882eac9d1f043 Author: Kyle Nusbaum <kylejnusb...@gmail.com> Date: 2016-10-31T21:05:44Z Fixing KafkaSpout acking. commit 2d3d7b109094fe966abd83f12e1ed388075b3be5 Author: Kyle Nusbaum <kylejnusb...@gmail.com> Date: 2016-10-31T21:11:18Z Fixing spacing. commit 52da50044f0fba65e99af6a4fd2c5af55d54d455 Author: Kyle Nusbaum <knusb...@yahoo-inc.com> Date: 2016-11-02T20:29:53Z Fix. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request #1759: STORM-2185: Storm Supervisor doesn't delete direct...
GitHub user knusbaum opened a pull request: https://github.com/apache/storm/pull/1759 STORM-2185: Storm Supervisor doesn't delete directories properly sometimes - 1.x branch You can merge this pull request into a Git repository by running: $ git pull https://github.com/knusbaum/incubator-storm STORM-2185-1.x-branch Alternatively you can review and apply these changes as the patch at: https://github.com/apache/storm/pull/1759.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1759 commit 0035a2cc91095dc440f0ca44df3a61d85db9494c Author: Kyle Nusbaum <kylejnusb...@gmail.com> Date: 2016-10-31T21:05:44Z Fixing KafkaSpout acking. commit b4dc3410c3dca405560de8b5a2649aecb4c2934d Author: Kyle Nusbaum <kylejnusb...@gmail.com> Date: 2016-10-31T21:11:18Z Fixing spacing. commit 5d6b6e4ee6ac55f7905082d01343740ca43594ce Author: Kyle Nusbaum <knusb...@yahoo-inc.com> Date: 2016-11-02T20:29:53Z Fix. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request #1753: STORM-2180: KafkaSpout ack() should handle re-bala...
GitHub user knusbaum opened a pull request: https://github.com/apache/storm/pull/1753 STORM-2180: KafkaSpout ack() should handle re-balance - 1.x branch You can merge this pull request into a Git repository by running: $ git pull https://github.com/knusbaum/incubator-storm STORM-2180-1.x-branch Alternatively you can review and apply these changes as the patch at: https://github.com/apache/storm/pull/1753.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1753 commit 0035a2cc91095dc440f0ca44df3a61d85db9494c Author: Kyle Nusbaum <kylejnusb...@gmail.com> Date: 2016-10-31T21:05:44Z Fixing KafkaSpout acking. commit b4dc3410c3dca405560de8b5a2649aecb4c2934d Author: Kyle Nusbaum <kylejnusb...@gmail.com> Date: 2016-10-31T21:11:18Z Fixing spacing. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request #1752: STORM-2180: KafkaSpout ack() should handle re-bala...
GitHub user knusbaum opened a pull request: https://github.com/apache/storm/pull/1752 STORM-2180: KafkaSpout ack() should handle re-balance You can merge this pull request into a Git repository by running: $ git pull https://github.com/knusbaum/incubator-storm STORM-2180 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/storm/pull/1752.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1752 commit 3ac3b8329121c38de774e57f694882eac9d1f043 Author: Kyle Nusbaum <kylejnusb...@gmail.com> Date: 2016-10-31T21:05:44Z Fixing KafkaSpout acking. commit 2d3d7b109094fe966abd83f12e1ed388075b3be5 Author: Kyle Nusbaum <kylejnusb...@gmail.com> Date: 2016-10-31T21:11:18Z Fixing spacing. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm issue #1603: STORM-2012: Upgrade Kafka to 0.8.2.2
Github user knusbaum commented on the issue: https://github.com/apache/storm/pull/1603 Builds fine locally. +1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm issue #1728: STORM-2142 ReportErrorAndDie runs suicide function only w...
Github user knusbaum commented on the issue: https://github.com/apache/storm/pull/1728 (Nice catch, BTW) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm issue #1728: STORM-2142 ReportErrorAndDie runs suicide function only w...
Github user knusbaum commented on the issue: https://github.com/apache/storm/pull/1728 +1 Looks like mistranslation. This restores the behavior of the old Clojure code. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm issue #1735: STORM-203 Adding paths to default java library path
Github user knusbaum commented on the issue: https://github.com/apache/storm/pull/1735 /usr/lib64 is FHS standard. /usr/lib/x86_64-linux-gnu is Debian-specific. I think we should avoid setting distro-specific defaults. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm issue #1720: STORM-1546: Adding Read and Write Aggregations for Pacema...
Github user knusbaum commented on the issue: https://github.com/apache/storm/pull/1720 Also updated the docs. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm issue #1720: STORM-1546: Adding Read and Write Aggregations for Pacema...
Github user knusbaum commented on the issue: https://github.com/apache/storm/pull/1720 @HeartSaVioR I've synced the java files between the branches so there are almost no differences. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request #1724: STORM-2131: Add blob command to worker-launcher, m...
Github user knusbaum commented on a diff in the pull request:
https://github.com/apache/storm/pull/1724#discussion_r82672822
--- Diff: storm-core/src/native/worker-launcher/impl/worker-launcher.c ---
@@ -414,33 +414,30 @@ static int copy_file(int input, const char*
in_filename,
return 0;
}
-int setup_stormdist(FTSENT* entry, uid_t euser) {
+static int setup_permissions(FTSENT* entry, uid_t euser, int user_write) {
if (lchown(entry->fts_path, euser, launcher_gid) != 0) {
-fprintf(ERRORFILE, "Failure to exec app initialization process - %s\n",
- strerror(errno));
+fprintf(ERRORFILE, "Failure to exec app initialization process - %s,
fts_path=%s\n",
+strerror(errno), entry->fts_path);
return -1;
}
mode_t mode = entry->fts_statp->st_mode;
- mode_t new_mode = (mode & (S_IRWXU)) | S_IRGRP | S_IWGRP;
- if ((mode & S_IXUSR) == S_IXUSR) {
-new_mode = new_mode | S_IXGRP;
+ mode_t new_mode = (mode & (S_IRUSR | S_IXUSR)) | S_IRGRP | S_IWGRP;
+ if (user_write) {
+new_mode = new_mode | S_IWUSR;
--- End diff --
ð
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request #1724: STORM-2131: Add blob command to worker-launcher, m...
Github user knusbaum commented on a diff in the pull request:
https://github.com/apache/storm/pull/1724#discussion_r82672834
--- Diff: storm-core/src/native/worker-launcher/impl/worker-launcher.c ---
@@ -414,33 +414,30 @@ static int copy_file(int input, const char*
in_filename,
return 0;
}
-int setup_stormdist(FTSENT* entry, uid_t euser) {
+static int setup_permissions(FTSENT* entry, uid_t euser, int user_write) {
if (lchown(entry->fts_path, euser, launcher_gid) != 0) {
-fprintf(ERRORFILE, "Failure to exec app initialization process - %s\n",
- strerror(errno));
+fprintf(ERRORFILE, "Failure to exec app initialization process - %s,
fts_path=%s\n",
+strerror(errno), entry->fts_path);
return -1;
}
mode_t mode = entry->fts_statp->st_mode;
- mode_t new_mode = (mode & (S_IRWXU)) | S_IRGRP | S_IWGRP;
- if ((mode & S_IXUSR) == S_IXUSR) {
-new_mode = new_mode | S_IXGRP;
+ mode_t new_mode = (mode & (S_IRUSR | S_IXUSR)) | S_IRGRP | S_IWGRP;
+ if (user_write) {
+new_mode = new_mode | S_IWUSR;
}
if ((mode & S_IFDIR) == S_IFDIR) {
-new_mode = new_mode | S_ISGID;
+new_mode = new_mode | S_IXGRP | S_ISGID;
}
if (chmod(entry->fts_path, new_mode) != 0) {
-fprintf(ERRORFILE, "Failure to exec app initialization process - %s\n",
- strerror(errno));
+fprintf(ERRORFILE, "Failure to exec app initialization process - %s,
fts_path=%s\n",
+strerror(errno), entry->fts_path);
return -1;
}
return 0;
}
-int setup_stormdist_dir(const char* local_dir) {
- //This is the same as
- //> chmod g+rwX -R $local_dir
- //> chown -no-dereference -R $user:$supervisor-group $local_dir
--- End diff --
ð
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request #1724: STORM-2131: Add blob command to worker-launcher, m...
Github user knusbaum commented on a diff in the pull request:
https://github.com/apache/storm/pull/1724#discussion_r82634841
--- Diff:
storm-core/src/jvm/org/apache/storm/daemon/supervisor/SupervisorUtils.java ---
@@ -104,6 +104,16 @@ public static void setupStormCodeDir(Map<String,
Object> conf, Map<String, Objec
}
}
+public static void setupWorkerArtifactsDir(Map<String, Object> conf,
Map<String, Object> stormConf, String dir) throws IOException {
+if
(Utils.getBoolean(conf.get(Config.SUPERVISOR_RUN_WORKER_AS_USER), false)) {
+String logPrefix = "setup conf for " + dir;
--- End diff --
No idea. I copied this over from setupStormCodeDir so they would look
similar in the logs.
Can change both to actually make sense.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request #1724: STORM-2131: Add blob command to worker-launcher, m...
Github user knusbaum commented on a diff in the pull request:
https://github.com/apache/storm/pull/1724#discussion_r82634237
--- Diff: storm-core/src/native/worker-launcher/impl/worker-launcher.c ---
@@ -485,14 +482,14 @@ int setup_stormdist_dir(const char* local_dir) {
case FTS_DP:// A directory being visited in post-order
case FTS_DOT: // A dot directory
+ case FTS_SL:// A symbolic link
+ case FTS_SLNONE:// A broken symbolic link
--- End diff --
This is ported from a change @tgravescs made. We can ask him why.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm issue #1724: STORM-2131: Add blob command to worker-launcher, make sto...
Github user knusbaum commented on the issue: https://github.com/apache/storm/pull/1724 @kishorvpatil @revans2 @jerrypeng --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request #1724: STORM-2131: Add blob command to worker-launcher, m...
GitHub user knusbaum opened a pull request: https://github.com/apache/storm/pull/1724 STORM-2131: Add blob command to worker-launcher, make stormdist directory not writeable by topo owner Re-opening this, as I reverted the last one to fix a typo (#1723) You can merge this pull request into a Git repository by running: $ git pull https://github.com/knusbaum/incubator-storm STORM-2131 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/storm/pull/1724.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1724 commit 49514b6e9265d0510ce61ba998a53000e0b2e57e Author: Kyle Nusbaum <kylejnusb...@gmail.com> Date: 2016-10-03T19:26:24Z Updating worker launcher for blobs. commit 5825e14a11c3d91c10036971e7757c8e4f82283d Author: Kyle Nusbaum <kylejnusb...@gmail.com> Date: 2016-09-26T20:42:18Z Fixing permissions for stormdist directory. commit f930b9a7e5c6d86c5782a44f5154c81bff82d9d8 Author: Kyle Nusbaum <kylejnusb...@gmail.com> Date: 2016-10-03T20:08:33Z Fixing workers artifacts permissions. commit 1ce0f639c53a8fd1d22a74ea78cfbdb15a81b90b Author: Kyle Nusbaum <kylejnusb...@gmail.com> Date: 2016-10-05T22:19:19Z fixing typo. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm issue #1468: STORM-1885. python script for squashing and merging prs.
Github user knusbaum commented on the issue: https://github.com/apache/storm/pull/1468 -1 I am generally opposed to this. Most PRs only have a small number of commits and aren't a problem. For PRs with a large number of commits, it's simple enough to ask the contributor to squash their own commits. I'm not sure I see the benefit in adding another script, which we will have to maintain, in order to do something we should rarely be doing. Also, I worry that having this script will lead to a sharp rise in totally-squashed PR merges, even when there's not really any benefit (and in fact, loss of authorship info) since some people are likely just going to use the script whenever they're doing a merge. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm issue #1396: [storm-elasticsearch]Upgrade elasticsearch version from 1...
Github user knusbaum commented on the issue: https://github.com/apache/storm/pull/1396 I'm going to close this. We're not going to drop 1.x support and there's another PR up adding support for 2.x without dropping 1.x. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm issue #1337: STORM-1475: Add storm-elasticsearch2 module
Github user knusbaum commented on the issue: https://github.com/apache/storm/pull/1337 #1676 is a no-go. We need guava in storm-core. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm issue #1154: fix storm-starter README.markdown links
Github user knusbaum commented on the issue: https://github.com/apache/storm/pull/1154 This has been fixed elsewhere. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm issue #1290: [YSTORM-1661] Added acl validation config 1.x Branch PR
Github user knusbaum commented on the issue: https://github.com/apache/storm/pull/1290 This was cherry picked. Closing. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm issue #1538: Sending the Data to Kafka as a batch, instead of sending ...
Github user knusbaum commented on the issue: https://github.com/apache/storm/pull/1538 I'm going to close this PR. It hasn't been touched in a few months, it is still missing a Jira, and we have decided to halt any non-bug work on 10.* branches. In addition, it looks to me like `batch.size` is available for Kafka at least as far back as 0.8.0, so utilizing that sounds like the correct way to address this issue. However, I only spent a brief time looking at the docs, so If you think this is still a valuable change for the 1.x and master branches, please file a Jira and open PRs to those branches. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm issue #1316: [STORM-1642] Catch Exception when deserialization failed....
Github user knusbaum commented on the issue: https://github.com/apache/storm/pull/1316 If you want to clean up the log messages, this still looks good. We'd want this instead to go into master and 1.x, though. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm issue #1242: Storm 1643 - Performance Fix: Optimize clojure lookups re...
Github user knusbaum commented on the issue: https://github.com/apache/storm/pull/1242 This hasn't been updated for ~7 months, and I'm -1 on the concept anyhow, so I'm going to close it. If you'd like to discuss, feel free to reopen. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm issue #1067: STORM-1515: Reset LocalState if corrupted after a hard re...
Github user knusbaum commented on the issue: https://github.com/apache/storm/pull/1067 @torbiak Would you like to do work on this, or should it be closed? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm issue #1018: Improve mention of aggregate() in the partitionAggregate(...
Github user knusbaum commented on the issue: https://github.com/apache/storm/pull/1018 +1 I'm fine with this change. If you want to upmerge, I'll get it in. Otherwise I plan to close this in a few weeks or so. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm issue #647: (STORM-956) When the execute() or nextTuple() hang on exte...
Github user knusbaum commented on the issue: https://github.com/apache/storm/pull/647 Closing this. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm issue #1710: STORM-1546: Adding Read and Write Aggregations for Pacema...
Github user knusbaum commented on the issue: https://github.com/apache/storm/pull/1710 Documentation updated. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request #1723: STORM-2131: Add blob command to worker-launcher, m...
GitHub user knusbaum opened a pull request: https://github.com/apache/storm/pull/1723 STORM-2131: Add blob command to worker-launcher, make stormdist directory not writeable by topo owner You can merge this pull request into a Git repository by running: $ git pull https://github.com/knusbaum/incubator-storm STORM-2131 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/storm/pull/1723.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1723 commit 49514b6e9265d0510ce61ba998a53000e0b2e57e Author: Kyle Nusbaum <kylejnusb...@gmail.com> Date: 2016-10-03T19:26:24Z Updating worker launcher for blobs. commit 5825e14a11c3d91c10036971e7757c8e4f82283d Author: Kyle Nusbaum <kylejnusb...@gmail.com> Date: 2016-09-26T20:42:18Z Fixing permissions for stormdist directory. commit f930b9a7e5c6d86c5782a44f5154c81bff82d9d8 Author: Kyle Nusbaum <kylejnusb...@gmail.com> Date: 2016-10-03T20:08:33Z Fixing workers artifacts permissions. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm issue #1710: STORM-1546: Adding Read and Write Aggregations for Pacema...
Github user knusbaum commented on the issue: https://github.com/apache/storm/pull/1710 @d2r @revans2 Any additional comments? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request #1720: STORM-1546: Adding Read and Write Aggregations for...
GitHub user knusbaum opened a pull request: https://github.com/apache/storm/pull/1720 STORM-1546: Adding Read and Write Aggregations for Pacemaker to make it HA compatible (1.x-branch) 1.x-branch version of Pacemaker-HA You can merge this pull request into a Git repository by running: $ git pull https://github.com/knusbaum/incubator-storm Pacemaker-HA-1.x-branch2 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/storm/pull/1720.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1720 commit 7b34582463857eba8bc08ff8abab338f4e83fbd6 Author: Kyle Nusbaum <kylejnusb...@gmail.com> Date: 2016-09-29T18:58:54Z Moving files into old position for merge. commit dd7ff1e0a0cecfe581c7441f941e3b9bbcc90c9d Author: Sanket <schintap@untilservice-lm> Date: 2016-03-04T22:41:02Z Porting Pacemaker-HA to community. commit a6171bd6770676d4d0bdb9c9d05a4785df35b671 Author: Kyle Nusbaum <kylejnusb...@gmail.com> Date: 2016-09-29T20:43:09Z Ported successfully. commit 20180cca1de9b25dc391722a9088e10107a31443 Author: Kyle Nusbaum <kylejnusb...@gmail.com> Date: 2016-09-29T20:45:01Z Minor cleanup. commit 9c5860498a71c04fda1e201283a63ec2c6352062 Author: Kyle Nusbaum <kylejnusb...@gmail.com> Date: 2016-09-29T22:06:16Z Making HA backwards-compatible --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request #1710: STORM-1546: Adding Read and Write Aggregations for...
Github user knusbaum commented on a diff in the pull request: https://github.com/apache/storm/pull/1710#discussion_r81219539 --- Diff: storm-core/src/jvm/org/apache/storm/Config.java --- @@ -926,10 +926,10 @@ public static final String UI_HTTPS_NEED_CLIENT_AUTH = "ui.https.need.client.auth"; /** - * The host that Pacemaker is running on. + * The list of servers that Pacemaker is running on. */ -@isString -public static final String PACEMAKER_HOST = "pacemaker.host"; +@isStringList +public static final String PACEMAKER_SERVERS = "pacemaker.servers"; --- End diff -- @HeartSaVioR Yep, the 1.x-branch PR will leave pacemaker.host in, but have it deprecated. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request #1710: STORM-1546: Adding Read and Write Aggregations for...
Github user knusbaum commented on a diff in the pull request:
https://github.com/apache/storm/pull/1710#discussion_r80798282
--- Diff:
storm-core/src/jvm/org/apache/storm/cluster/PaceMakerStateStorage.java ---
@@ -159,10 +162,6 @@ public void set_worker_hb(String path, byte[] data,
List acls) {
ret = details;
}
}
-if(ret == null) {
-throw new HBExecutionException("Failed to get a
response.");
-}
-LOG.debug("Successful get_worker_hb");
return ret;
--- End diff --
Yes, I wasn't quite sure how we wanted to handle this, NULL responses are
valid, and right now the code treats mismatched responses as if they were NULL.
We can distinguish between them if we want the behavior to differ whether the
pacemakers returned NULL details or returned an invalid message. Throwing on
null is not what we wanted, though, since it will cause Nimbus to crash when
nimbus tries to read a topology's heartbeats before it has started sending them.
Maybe we want to add some logic so that if ALL pacemakers return an invalid
response, then we throw?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
