date:20170724

Github user satishd commented on a diff in the pull request:

https://github.com/apache/storm/pull/2241#discussion_r129208678
  
--- Diff: conf/defaults.yaml ---
@@ -304,6 +303,7 @@ storm.cgroup.resources:
 storm.cgroup.hierarchy.name: "storm"
 storm.supervisor.cgroup.rootdir: "storm"
 storm.cgroup.cgexec.cmd: "/bin/cgexec"
+storm.cgroup.cgexec.cmd: "/bin/cgexec"
--- End diff --

may be an accidental copy, needs to be removed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request #2241: STORM-2306 : Messaging subsystem redesign.

Github user satishd commented on a diff in the pull request:

https://github.com/apache/storm/pull/2241#discussion_r129217706
  
--- Diff: 
storm-client/src/jvm/org/apache/storm/executor/spout/SpoutExecutor.java ---
@@ -105,19 +107,21 @@ public void expire(Long key, TupleInfo tupleInfo) {
 }
 });
 
-this.spoutThrottlingMetrics.registerAll(topoConf, 
idToTask.values().iterator().next().getUserContext());
+this.spoutThrottlingMetrics.registerAll(topoConf, 
idToTask.get(taskIds.get(0)).getUserContext());
 this.outputCollectors = new ArrayList<>();
-for (Map.Entry entry : idToTask.entrySet()) {
-Task taskData = entry.getValue();
+for (int i=0; i

[GitHub] storm pull request #2241: STORM-2306 : Messaging subsystem redesign.

Github user satishd commented on a diff in the pull request:

https://github.com/apache/storm/pull/2241#discussion_r129212155
  
--- Diff: 
storm-client/src/jvm/org/apache/storm/daemon/worker/WorkerState.java ---
@@ -488,84 +476,55 @@ public void registerCallbacks() {
 LOG.info("Registering IConnectionCallbacks for {}:{}", 
assignmentId, port);
 receiver.registerRecv(new 
DeserializingConnectionCallback(topologyConf,
 getWorkerTopologyContext(),
-this::transferLocal));
+this::transferLocalBatch));
 }
 
-public void transferLocal(List tupleBatch) {
-Map> grouped = new HashMap<>();
-for (AddressedTuple tuple : tupleBatch) {
-Integer executor = taskToShortExecutor.get(tuple.dest);
-if (null == executor) {
-LOG.warn("Received invalid messages for unknown tasks. 
Dropping... ");
-continue;
-}
-List current = grouped.get(executor);
-if (null == current) {
-current = new ArrayList<>();
-grouped.put(executor, current);
-}
-current.add(tuple);
-}
 
-for (Map.Entry> entry : 
grouped.entrySet()) {
-DisruptorQueue queue = 
shortExecutorReceiveQueueMap.get(entry.getKey());
-if (null != queue) {
-queue.publish(entry.getValue());
-} else {
-LOG.warn("Received invalid messages for unknown tasks. 
Dropping... ");
+private void transferLocalBatch(List tupleBatch) {
+try {
+for (int i = 0; i < tupleBatch.size(); i++) {
--- End diff --

Does `foreach` have significant perf issue?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request #2241: STORM-2306 : Messaging subsystem redesign.

Github user satishd commented on a diff in the pull request:

https://github.com/apache/storm/pull/2241#discussion_r129209017
  
--- Diff: storm-client/src/jvm/org/apache/storm/task/IOutputCollector.java 
---
@@ -30,4 +31,5 @@
 void ack(Tuple input);
 void fail(Tuple input);
 void resetTimeout(Tuple input);
+void flush();
--- End diff --

 May want to add some javadoc about the same. It seems we are ready to 
break the APIs with new set of changes in this redesign. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request #2241: STORM-2306 : Messaging subsystem redesign.

Github user satishd commented on a diff in the pull request:

https://github.com/apache/storm/pull/2241#discussion_r129208440
  
--- Diff: conf/defaults.yaml ---
@@ -146,7 +149,7 @@ supervisor.run.worker.as.user: false
 #how long supervisor will wait to ensure that a worker process is started
 supervisor.worker.start.timeout.secs: 120
 #how long between heartbeats until supervisor considers that worker dead 
and tries to restart it
-supervisor.worker.timeout.secs: 30
+supervisor.worker.timeout.secs: 3
--- End diff --

Is this really a deliberate change?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request #2241: STORM-2306 : Messaging subsystem redesign.

Github user satishd commented on a diff in the pull request:

https://github.com/apache/storm/pull/2241#discussion_r129215798
  
--- Diff: storm-client/src/jvm/org/apache/storm/daemon/Task.java ---
@@ -246,4 +291,26 @@ private void addTaskHooks() {
 }
 }
 
+private static HashMap> getGroupersPerStream(Map> streamComponentToGrouper) {
--- End diff --

nit: This can return `Map` 
instead of implementations.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request #2241: STORM-2306 : Messaging subsystem redesign.

Github user satishd commented on a diff in the pull request:

https://github.com/apache/storm/pull/2241#discussion_r129208570
  
--- Diff: conf/defaults.yaml ---
@@ -253,11 +247,16 @@ topology.trident.batch.emit.interval.millis: 500
 topology.testing.always.try.serialize: false
 topology.classpath: null
 topology.environment: null
-topology.bolts.outgoing.overflow.buffer.enable: false
-topology.disruptor.wait.timeout.millis: 1000
-topology.disruptor.batch.size: 100
-topology.disruptor.batch.timeout.millis: 1
-topology.disable.loadaware.messaging: false
+topology.bolts.outgoing.overflow.buffer.enable: false # TODO: Roshan : 
Whats this ?
+topology.disruptor.wait.timeout.millis: 1000  # TODO: Roshan: not used, 
but we may/not want this behavior
+topology.transfer.buffer.size: 5
+topology.transfer.batch.size: 10
+topology.executor.receive.buffer.size: 5
+topology.producer.batch.size: 1000  # TODO: Roshan:  rename
+topology.flush.tuple.freq.millis: 5000
--- End diff --

nit: Better to add a comment describing about this property.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request #2241: STORM-2306 : Messaging subsystem redesign.

Github user satishd commented on a diff in the pull request:

https://github.com/apache/storm/pull/2241#discussion_r129222974
  
--- Diff: storm-client/src/jvm/org/apache/storm/utils/TransferDrainer.java 
---
@@ -21,101 +21,100 @@
 import java.util.HashMap;
 import java.util.Iterator;
 import java.util.Map;
+import java.util.Map.Entry;
 
 import org.apache.storm.generated.NodeInfo;
 import org.apache.storm.messaging.IConnection;
 import org.apache.storm.messaging.TaskMessage;
-import com.google.common.collect.Maps;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
 public class TransferDrainer {
 
-  private Map>> bundles = new 
HashMap();
+  private Map> bundles = new HashMap();
+
   private static final Logger LOG = 
LoggerFactory.getLogger(TransferDrainer.class);
-  
-  public void add(HashMap> 
taskTupleSetMap) {
-for (Map.Entry> entry : 
taskTupleSetMap.entrySet()) {
-  addListRefToMap(this.bundles, entry.getKey(), entry.getValue());
+
+  // Cache the msgs grouped by destination node
+  public void add(TaskMessage taskMsg) {
+int destId = taskMsg.task();
+ArrayList msgs = bundles.get(destId);
--- End diff --

we can have `bundles.computeIfAbsent(destId, integer -> new 
ArrayList<>());` removing null check, creating list and put that into bundles 
map.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request #2241: STORM-2306 : Messaging subsystem redesign.

Github user satishd commented on a diff in the pull request:

https://github.com/apache/storm/pull/2241#discussion_r129219489
  
--- Diff: 
storm-client/src/jvm/org/apache/storm/grouping/ShuffleGrouping.java ---
@@ -26,38 +26,25 @@
 import java.util.ArrayList;
 import java.util.Arrays;
 import java.util.Collections;
-import java.util.concurrent.atomic.AtomicInteger;
+
 
 public class ShuffleGrouping implements CustomStreamGrouping, Serializable 
{
-private Random random;
 private ArrayList> choices;
-private AtomicInteger current;
+private int current = 0 ;
--- End diff --

You changed this to non threadsafe. Is this instance not shared by multiple 
components?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request #2241: STORM-2306 : Messaging subsystem redesign.

Github user satishd commented on a diff in the pull request:

https://github.com/apache/storm/pull/2241#discussion_r129216374
  
--- Diff: storm-client/src/jvm/org/apache/storm/daemon/Task.java ---
@@ -122,28 +130,33 @@ public Task(Executor executor, Integer taskId) throws 
IOException {
 return new ArrayList<>(0);
 }
 
+
 public List getOutgoingTasks(String stream, List 
values) {
 if (debug) {
 LOG.info("Emitting Tuple: taskId={} componentId={} stream={} 
values={}", taskId, componentId, stream, values);
 }
 
-List outTasks = new ArrayList<>();
-if (!streamComponentToGrouper.containsKey(stream)) {
-throw new IllegalArgumentException("Unknown stream ID: " + 
stream);
-}
-if (null != streamComponentToGrouper.get(stream)) {
-// null value for __system
-for (LoadAwareCustomStreamGrouping grouper : 
streamComponentToGrouper.get(stream).values()) {
+ArrayList outTasks = new ArrayList<>();
+
+// TODO: PERF: expensive hashtable lookup in critical path
--- End diff --

Is this an expensive hit? This map may not contain many keys(no of streams 
defined for this task). 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request #2241: STORM-2306 : Messaging subsystem redesign.

Github user satishd commented on a diff in the pull request:

https://github.com/apache/storm/pull/2241#discussion_r129217748
  
--- Diff: 
storm-client/src/jvm/org/apache/storm/executor/spout/SpoutExecutor.java ---
@@ -105,19 +107,21 @@ public void expire(Long key, TupleInfo tupleInfo) {
 }
 });
 
-this.spoutThrottlingMetrics.registerAll(topoConf, 
idToTask.values().iterator().next().getUserContext());
+this.spoutThrottlingMetrics.registerAll(topoConf, 
idToTask.get(taskIds.get(0)).getUserContext());
 this.outputCollectors = new ArrayList<>();
-for (Map.Entry entry : idToTask.entrySet()) {
-Task taskData = entry.getValue();
+for (int i=0; i

[GitHub] storm pull request #2241: STORM-2306 : Messaging subsystem redesign.

Github user satishd commented on a diff in the pull request:

https://github.com/apache/storm/pull/2241#discussion_r129214432
  
--- Diff: storm-client/src/jvm/org/apache/storm/daemon/Acker.java ---
@@ -66,6 +67,7 @@ public void prepare(Map topoConf, 
TopologyContext context, Outpu
 
 @Override
 public void execute(Tuple input) {
+long start = System.currentTimeMillis();
--- End diff --

nit: `start` is never used.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request #2241: STORM-2306 : Messaging subsystem redesign.

Github user satishd commented on a diff in the pull request:

https://github.com/apache/storm/pull/2241#discussion_r129218102
  
--- Diff: 
storm-client/src/jvm/org/apache/storm/executor/ExecutorShutdown.java ---
@@ -34,17 +34,19 @@
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
+import java.util.ArrayList;
 import java.util.List;
 import java.util.Map;
 
 public class ExecutorShutdown implements Shutdownable, IRunningExecutor {
 
 private static final Logger LOG = 
LoggerFactory.getLogger(ExecutorShutdown.class);
+
 private final Executor executor;
 private final List threads;
-private final Map taskDatas;
+private final ArrayList taskDatas;
 
-public ExecutorShutdown(Executor executor, List 
threads, Map taskDatas) {
+public ExecutorShutdown(Executor executor, List 
threads, ArrayList taskDatas) {
--- End diff --

nit: taskDatas as List


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request #2241: STORM-2306 : Messaging subsystem redesign.

Github user satishd commented on a diff in the pull request:

https://github.com/apache/storm/pull/2241#discussion_r129218772
  
--- Diff: 
storm-client/src/jvm/org/apache/storm/executor/ExecutorTransfer.java ---
@@ -17,72 +17,92 @@
  */
 package org.apache.storm.executor;
 
-import com.google.common.annotations.VisibleForTesting;
-import com.lmax.disruptor.EventHandler;
 import org.apache.storm.Config;
 import org.apache.storm.daemon.worker.WorkerState;
 import org.apache.storm.serialization.KryoTupleSerializer;
 import org.apache.storm.tuple.AddressedTuple;
 import org.apache.storm.tuple.Tuple;
-import org.apache.storm.utils.DisruptorQueue;
-import org.apache.storm.utils.MutableObject;
+import org.apache.storm.utils.JCQueue;
 import org.apache.storm.utils.ObjectReader;
+import org.apache.storm.utils.Utils;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
 import java.util.ArrayList;
+import java.util.Collections;
 import java.util.Map;
-import java.util.concurrent.Callable;
 
-public class ExecutorTransfer implements EventHandler, Callable {
+public class ExecutorTransfer  {
 private static final Logger LOG = 
LoggerFactory.getLogger(ExecutorTransfer.class);
 
 private final WorkerState workerData;
-private final DisruptorQueue batchTransferQueue;
-private final Map topoConf;
 private final KryoTupleSerializer serializer;
-private final MutableObject cachedEmit;
 private final boolean isDebug;
+private final int producerBatchSz;
+private int remotesBatchSz = 0;
+private final ArrayList localReceiveQueues; // [taksid]=queue
+private final ArrayList outboundQueues; // [taksid]=queue, 
some entries can be null
 
-public ExecutorTransfer(WorkerState workerData, DisruptorQueue 
batchTransferQueue, Map topoConf) {
+
+public ExecutorTransfer(WorkerState workerData, Map 
topoConf) {
 this.workerData = workerData;
-this.batchTransferQueue = batchTransferQueue;
-this.topoConf = topoConf;
 this.serializer = new KryoTupleSerializer(topoConf, 
workerData.getWorkerTopologyContext());
-this.cachedEmit = new MutableObject(new ArrayList<>());
 this.isDebug = 
ObjectReader.getBoolean(topoConf.get(Config.TOPOLOGY_DEBUG), false);
+this.producerBatchSz = 
ObjectReader.getInt(topoConf.get(Config.TOPOLOGY_PRODUCER_BATCH_SIZE));
+this.localReceiveQueues = 
Utils.convertToArray(workerData.getShortExecutorReceiveQueueMap());
+this.outboundQueues = new 
ArrayList(Collections.nCopies(localReceiveQueues.size(), null) );
 }
 
-public void transfer(int task, Tuple tuple) {
-AddressedTuple val = new AddressedTuple(task, tuple);
+
+public void transfer(int task, Tuple tuple) throws 
InterruptedException {
+AddressedTuple addressedTuple = new AddressedTuple(task, tuple);
 if (isDebug) {
-LOG.info("TRANSFERRING tuple {}", val);
+LOG.info("TRANSFERRING tuple {}", addressedTuple);
 }
-batchTransferQueue.publish(val);
-}
 
-@VisibleForTesting
-public DisruptorQueue getBatchTransferQueue() {
-return this.batchTransferQueue;
+boolean isLocal = transferLocal(addressedTuple);
+if (!isLocal) {
+transferRemote(addressedTuple);
+++remotesBatchSz;
+if(remotesBatchSz >=producerBatchSz) {
+flushRemotes();
+remotesBatchSz =0;
+}
+}
 }
 
-@Override
-public Object call() throws Exception {
-batchTransferQueue.consumeBatchWhenAvailable(this);
-return 0L;
+private void transferRemote(AddressedTuple tuple) throws 
InterruptedException {
+workerData.transferRemote(tuple);
 }
 
-public String getName() {
-return batchTransferQueue.getName();
+// flushes local and remote messages
+public void flush() throws InterruptedException {
+flushLocal();
+flushRemotes();
 }
 
-@Override
-public void onEvent(Object event, long sequence, boolean endOfBatch) 
throws Exception {
-ArrayList cachedEvents = (ArrayList) cachedEmit.getObject();
-cachedEvents.add(event);
-if (endOfBatch) {
-workerData.transfer(serializer, cachedEvents);
-cachedEmit.setObject(new ArrayList<>());
+private void flushLocal() throws InterruptedException {
+for (int i = 0; i < outboundQueues.size(); i++) {
+JCQueue q = outboundQueues.get(i);
+if(q!=null)
+q

[GitHub] storm pull request #2241: STORM-2306 : Messaging subsystem redesign.

Github user satishd commented on a diff in the pull request:

https://github.com/apache/storm/pull/2241#discussion_r129214771
  
--- Diff: storm-client/src/jvm/org/apache/storm/daemon/GrouperFactory.java 
---
@@ -137,7 +137,7 @@ public void prepare(WorkerTopologyContext context, 
GlobalStreamId stream, List targetTasks;
+private ArrayList > targetTasks;
--- End diff --

nit: No need to change from List to ArrayList.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request #2241: STORM-2306 : Messaging subsystem redesign.

Github user satishd commented on a diff in the pull request:

https://github.com/apache/storm/pull/2241#discussion_r129220412
  
--- Diff: storm-client/src/jvm/org/apache/storm/tuple/TupleImpl.java ---
@@ -24,50 +24,56 @@
 import org.apache.storm.task.GeneralTopologyContext;
 
 public class TupleImpl implements Tuple {
-private final List values;
-private final int taskId;
-private final String streamId;
-private final GeneralTopologyContext context;
-private final MessageId id;
+private List values;
+private int taskId;
+private String streamId;
+private GeneralTopologyContext context;
+private MessageId id;
+private final String srcComponent;
 private Long _processSampleStartTime;
 private Long _executeSampleStartTime;
 private long _outAckVal = 0;
-
+
 public TupleImpl(Tuple t) {
 this.values = t.getValues();
 this.taskId = t.getSourceTask();
 this.streamId = t.getSourceStreamId();
 this.id = t.getMessageId();
 this.context = t.getContext();
-if (t instanceof TupleImpl) {
+this.srcComponent = t.getSourceComponent();
+try {
 TupleImpl ti = (TupleImpl) t;
 this._processSampleStartTime = ti._processSampleStartTime;
 this._executeSampleStartTime = ti._executeSampleStartTime;
 this._outAckVal = ti._outAckVal;
+} catch (ClassCastException e) {
+// ignore ... if t is not a TupleImpl type .. faster than 
checking and then casting
 }
 }
 
-public TupleImpl(GeneralTopologyContext context, List values, 
int taskId, String streamId, MessageId id) {
+public TupleImpl(GeneralTopologyContext context, List values, 
String srcComponent, int taskId, String streamId, MessageId id) {
--- End diff --

Why do we need to have this to be passed in constructor as that can be 
derived like below from the existing arguments? This constructor change is 
spread across all usages.
`srcComponent = context.getComponentId(taskId)`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request #2241: STORM-2306 : Messaging subsystem redesign.

Github user satishd commented on a diff in the pull request:

https://github.com/apache/storm/pull/2241#discussion_r129223991
  
--- Diff: storm-client/src/jvm/org/apache/storm/utils/TransferDrainer.java 
---
@@ -21,101 +21,100 @@
 import java.util.HashMap;
 import java.util.Iterator;
 import java.util.Map;
+import java.util.Map.Entry;
 
 import org.apache.storm.generated.NodeInfo;
 import org.apache.storm.messaging.IConnection;
 import org.apache.storm.messaging.TaskMessage;
-import com.google.common.collect.Maps;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
 public class TransferDrainer {
 
-  private Map>> bundles = new 
HashMap();
+  private Map> bundles = new HashMap();
+
   private static final Logger LOG = 
LoggerFactory.getLogger(TransferDrainer.class);
-  
-  public void add(HashMap> 
taskTupleSetMap) {
-for (Map.Entry> entry : 
taskTupleSetMap.entrySet()) {
-  addListRefToMap(this.bundles, entry.getKey(), entry.getValue());
+
+  // Cache the msgs grouped by destination node
+  public void add(TaskMessage taskMsg) {
--- End diff --

It seems `add` and `send` are always invoked in the same thread from 
(JCQueue.Consumer's `accept` and `flush`) and there is no contention. Is that 
right?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: problem with the size of the result after submitting

2017-07-24 Thread sam mohel

in supervisor.log got
2017-07-25 07:35:20.137 b.s.d.supervisor [INFO] Worker Process
42c31e1a-d6d9-45e7-99ce-ef2cb6293810 exited with code: 20
2017-07-25 07:35:21.988 b.s.d.supervisor [INFO] Worker Process
42c31e1a-d6d9-45e7-99ce-ef2cb6293810 has died!
2017-07-25 07:35:21.994 b.s.d.supervisor [INFO] Shutting down and clearing
state for id 42c31e1a-d6d9-45e7-99ce-ef2cb6293810. Current supervisor time:
1500960921. State: :timed-out, Heartbeat: {:time-secs 1500960919, :storm-id
"top-1-1500960906", :executors [[8 8] [47 47] [12 12] [54 54] [2 2] [32 32]
[41 41] [7 7] [51 51] [22 22] [42 42] [3 3] [56 56] [33 33] [24 24] [1 1]
[39 39] [53 53] [40 40] [35 35] [43 43] [18 18] [6 6] [28 28] [38 38] [20
20] [9 9] [57 57] [48 48] [30 30] [23 23] [50 50] [11 11] [31 31] [-1 -1]
[44 44] [16 16] [55 55] [13 13] [19 19] [52 52] [21 21] [5 5] [27 27] [29
29] [26 26] [45 45] [10 10] [37 37] [46 46] [14 14] [4 4] [36 36] [15 15]
[34 34] [25 25] [49 49] [17 17]], :port 6703}
2017-07-25 07:35:21.995 b.s.d.supervisor [INFO] Shutting down
94209043-2d37-47a0-b8b9-c37edfae8ccd:42c31e1a-d6d9-45e7-99ce-ef2cb6293810
2017-07-25 07:35:21.996 b.s.config [INFO] GET worker-user
42c31e1a-d6d9-45e7-99ce-ef2cb6293810
2017-07-25 07:35:22.005 b.s.util [INFO] Error when trying to kill 4984.
Process is probably already dead.
2017-07-25 07:35:22.005 b.s.d.supervisor [INFO] Sleep 1 seconds for
execution of cleanup threads on worker.
2017-07-25 07:35:23.007 b.s.util [INFO] Error when trying to kill 4984.
Process is probably already dead.
2017-07-25 07:35:23.009 b.s.config [INFO] REMOVE worker-user
42c31e1a-d6d9-45e7-99ce-ef2cb6293810
2017-07-25 07:35:23.009 b.s.d.supervisor [INFO] Shut down
94209043-2d37-47a0-b8b9-c37edfae8ccd:42c31e1a-d6d9-45e7-99ce-ef2cb6293810

On Tue, Jul 25, 2017 at 1:08 AM, sam mohel  wrote:

> i submitted a topology using apache-storm -0.10.2 with localhost 127.0.0.1
> and got results with size 50 KB only. Each time i submit the topology ,i
> got same size. It suppose to get more than it . I checked the log files and
> here is the worker.log
>
> [INFO] Activating spout spout0:(57)
> 2017-07-24 23:27:55.167 b.s.d.executor [INFO] Prepared bolt b-6:(10)
> 2017-07-24 23:27:55.141 b.s.u.StormBoundedExponentialBackoffRetry [INFO]
> The baseSleepTimeMs [2000] the maxSleepTimeMs [6] the maxRetries [5]
> 2017-07-24 23:27:55.390 b.s.u.StormBoundedExponentialBackoffRetry [INFO]
> The baseSleepTimeMs [2000] the maxSleepTimeMs [6] the maxRetries [5]
> 2017-07-24 23:28:00.342 b.s.d.worker [INFO] Shutting down worker
> top-1-1500931668 10dd0bf8-daf1-496e-8ea7-142810464250 6703
> 2017-07-24 23:28:00.342 b.s.d.worker [INFO] Shutting down receive thread
> 2017-07-24 23:28:00.378 b.s.m.n.Client [INFO] creating Netty Client,
> connecting to lenovo:6703, bufferSize: 5242880
> 2017-07-24 23:28:00.383 b.s.m.loader [INFO] Shutting down
> receiving-thread: [top-1-1500931668, 6703]
> 2017-07-24 23:28:00.383 b.s.m.n.Client [INFO] closing Netty Client
> Netty-Client-lenovo/192.168.x.x:6703 
> 2017-07-24 23:28:00.383 b.s.m.n.Client [INFO] waiting up to 60 ms to
> send 0 pending messages to Netty-Client-lenovo/192.168.x.x:6703
> 
> 2017-07-24 23:28:00.383 b.s.m.loader [INFO] Waiting for
> receiving-thread:[top-1-1500931668, 6703] to die
> 2017-07-24 23:28:00.384 b.s.m.loader [INFO] Shutdown receiving-thread:
> [top-1-1500931668, 6703]
> 2017-07-24 23:28:00.385 b.s.d.worker [INFO] Shut down receive thread
> 2017-07-24 23:28:00.385 b.s.d.worker [INFO] Terminating messaging context
> 2017-07-24 23:28:00.385 b.s.d.worker [INFO] Shutting down executors
> 2017-07-24 23:28:00.386 b.s.d.executor [INFO] Shutting down executor
> b-6:[8 8]
> 2017-07-24 23:28:00.387 b.s.util [INFO] Async loop interrupted!
> 2017-07-24 23:28:00.387 b.s.util [INFO] Async loop interrupted!
> 2017-07-24 23:28:00.388 b.s.d.executor [INFO] Shut down executor b-6:[8 8]
> 2017-07-24 23:28:00.388 b.s.d.executor [INFO] Shutting down executor
> b-9:[47 47]
> 2017-07-24 23:28:00.389 b.s.util [INFO] Async loop interrupted!
> 2017-07-24 23:28:00.389 b.s.util [INFO] Async loop interrupted!
>
>
> i hope i can find help , Thanks
>

[GitHub] storm pull request #2218: STORM-2614: Enhance stateful windowing to persist ...

2017-07-24 Thread ben-manes

Github user ben-manes commented on a diff in the pull request:

https://github.com/apache/storm/pull/2218#discussion_r129211231
  
--- Diff: 
storm-client/src/jvm/org/apache/storm/topology/SimpleWindowPartitionCache.java 
---
@@ -0,0 +1,182 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.storm.topology;
+
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.Map;
+import java.util.concurrent.ConcurrentMap;
+import java.util.concurrent.ConcurrentSkipListMap;
+import java.util.concurrent.locks.ReentrantLock;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A simple implementation that evicts the largest un-pinned entry from 
the cache. This works well
+ * for caching window partitions since the access pattern is mostly 
sequential scans.
+ */
+public class SimpleWindowPartitionCache implements 
WindowPartitionCache {
+private static final Logger LOG = 
LoggerFactory.getLogger(SimpleWindowPartitionCache.class);
+
+private final ConcurrentSkipListMap map = new 
ConcurrentSkipListMap<>();
+private final Map pinned = new HashMap<>();
+private final long maximumSize;
+private final RemovalListener removalListener;
+private final CacheLoader cacheLoader;
+private final ReentrantLock lock = new ReentrantLock(true);
+private int size;
+
+@Override
+public V get(K key) {
+return getOrLoad(key, false);
+}
+
+@Override
+public V getPinned(K key) {
+return getOrLoad(key, true);
+}
+
+@Override
+public boolean unpin(K key) {
+LOG.debug("unpin '{}'", key);
+boolean res = false;
+try {
+lock.lock();
+if (pinned.compute(key, (k, v) -> v == null ? 0 : v - 1) <= 0) 
{
+pinned.remove(key);
+res = true;
+}
+} finally {
+lock.unlock();
+}
+LOG.debug("pinned '{}'", pinned);
+return res;
+}
+
+@Override
+public ConcurrentMap asMap() {
+return map;
+}
+
+@Override
+public void invalidate(K key) {
+if (isPinned(key)) {
+LOG.debug("Entry '{}' is pinned, skipping invalidation", key);
+} else {
+LOG.debug("Invalidating entry '{}'", key);
+V val = map.remove(key);
+try {
+lock.lock();
+--size;
--- End diff --

if the key isn't in the cache (e.g. concurrent invalidations) then this 
will decrease incorrectly. I think you need to check `val` and skip the locking 
code if not present.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm issue #2218: STORM-2614: Enhance stateful windowing to persist the win...

2017-07-24 Thread arunmahadevan

Github user arunmahadevan commented on the issue:

https://github.com/apache/storm/pull/2218
  
@ben-manes , thanks for the insights. Right now the custom logic is kind of 
straightforward. May be I will attempt to provide an alternate implementation 
on top of Caffeine which could be useful in future.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request #2241: STORM-2306 : Messaging subsystem redesign.

2017-07-24 Thread roshannaik

GitHub user roshannaik opened a pull request:

https://github.com/apache/storm/pull/2241

STORM-2306 : Messaging subsystem redesign.

Having spent a lot of time on this, I am happy to share some good news and
some even better news with you.

Before venturing further, I must add, to limit the scope of this PR, no
attempt was made to improve ACK-ing mode perf. Although there are some big
latency gains seen in ACK mode, these are a side effect of the new messaging
design and work remains to be done to improve ACK mode.

Please see the design docs posted on the STORM-2306 jira for details on
what is being done

So, first the good news .. a quick competitive evaluation:

# 1) Competitive Perf evaluation :

Here are some quick comparison of Storm numbers taken on my laptop against
numbers for similar/identical topologies published by Heron, Flink and Apex.
Shall provide just rolled up summary here and leave the detailed analysis for
later.

Storm numbers here were run on my MacBook Pro (2015) with 16GB ram and a
single 4 core Intel i7 chip.

### A) Compared To Heron and Flink:
--
Heron recently published this blog about the big perf improvements (~4-6x)
they achieved.

https://blog.twitter.com/engineering/en_us/topics/open-source/2017/optimizing-twitter-heron.html
They ran it on dual 12-core Intel Xeon chips (didn't say how many machines).

They use a simplified word count topology that I have emulated for
comparison purposes and included it as part of this PR
(SimplifiedWordCountTopo).

Flink also publishes numbers for a similar setup here
https://flink.apache.org/features.html#streaming

Below are per core throughput numbers.

**[:HERON:]**
Acking Disabled: per core = **~475 k/sec**.
Acking Enabled: per core = **~150 k/sec**. Latency = **30ms**

**[:FLINK:]**
Per core: **~1 mill/sec**

**[:STORM:]**
Acking Disabled: per core = **2 mill/sec.**(1 spout & 1 counter bolt)
Acking Enabled: per core = **0.6 mill/sec**, Latency = **0.73 ms** (+1
acker)

**Takeaways:**
- Storm's with-ACK throughput is better than Heron's no-ACK throughput.
- Without ACKs, Storm is far ahead of Heron and also better than Flink.
- Storm's Latency (in microseconds) is also significantly better than both
(although this metric is better to compared with multiple machines in the run).
AFAIKT, Flink is generally not known for having good latency (as per Flink's
own comparison with Storm -
https://data-artisans.com/blog/high-throughput-low-latency-and-exactly-once-stream-processing-with-apache-flink).

### B) Compared to Apex:
---
Apex appears to be the best performer among the opensource lot.. by a
reasonably good margin. Some numbers they published in their early days (when
it was called DataTorrent) were misleading/dubious IMO, but the newer numbers
appear credible.

Here we look at how fast inter spout/bolt communication can be achieved
using an ultra minimalist topology.
A ConstSpout emits a short string to a DevNull bolt that discards the
tuples it receives. This topo has been in storm-perf for sometime now.

Apex provides numbers for a identical setup ... what they call "container
local" performance here:
https://www.datatorrent.com/blog/blog-apex-performance-benchmark/

Other than the fact that Storm numbers were on my laptop, these numbers are
a good apples to apples comparison.

**[:APEX:]**
Container local Throughput : **~4.2 mill/sec**

**[:STORM:]**
Worker local throughput : **8.1 mill/sec**

# 2) Core messaging Performance

Now for the better news. The redesigned messaging system is actually much
faster and able to move messages between threads at an astounding rate :

- **120 mill/sec** (batchSz=1000, 2 producers writing to 1 consumer).
- **67 mill/sec** (batchSz=1000, 1 producers writing to 1 consumer).

I have included JCQueuePerfTest.java in this PR to help get quick
measurements from within the IDE.

That naturally begs the question .. why is Storm pushing only 8.1 mill/sec
between a ConstSpout and DevNullBolt ? The short answer is ... there are big
bottlenecks in other parts of the code. In this PR I have tackled some such
bottlenecks but many still remain. We are faster than the competition, but
still have room to be much much faster. If anyone is interested in pursuing
these to push Storm's perf to the next level, I am happy to point them in the
right direction.

Again, please refer to the design docs in the JIRA for details on the new
design and the rationale behind them.

You can merge this pull request into a Git repository by running:

$ git pull https://

[GitHub] storm issue #2240: [STORM-2657] Update SECURITY.MD

2017-07-24 Thread liu-zhaokun

Github user liu-zhaokun commented on the issue:

https://github.com/apache/storm/pull/2240
  
@revans2 
Hello,I have updated SECURITY.MD,review it,please.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request #2240: [STORM-2657] Update SECURITY.MD

2017-07-24 Thread liu-zhaokun

GitHub user liu-zhaokun opened a pull request:

https://github.com/apache/storm/pull/2240

[STORM-2657] Update SECURITY.MD


[https://issues.apache.org/jira/browse/STORM-2657](https://issues.apache.org/jira/browse/STORM-2657)

There are several PRs about storm acl merged into master,so I update 
SECURITY.md.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/liu-zhaokun/storm master07250849

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/storm/pull/2240.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2240


commit b604ea0abcef950105d4bec9315b6eb211b98c95
Author: liuzhaokun 
Date:   2017-07-25T02:38:56Z

[STORM-2657] Update SECURITY.MD




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm issue #2224: STORM-2640: Deprecate KafkaConsumer.subscribe API option,...

2017-07-24 Thread hmcl

Github user hmcl commented on the issue:

https://github.com/apache/storm/pull/2224
  
+1


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

problem with the size of the result after submitting

2017-07-24 Thread sam mohel

i submitted a topology using apache-storm -0.10.2 with localhost 127.0.0.1
and got results with size 50 KB only. Each time i submit the topology ,i
got same size. It suppose to get more than it . I checked the log files and
here is the worker.log

[INFO] Activating spout spout0:(57)
2017-07-24 23:27:55.167 b.s.d.executor [INFO] Prepared bolt b-6:(10)
2017-07-24 23:27:55.141 b.s.u.StormBoundedExponentialBackoffRetry [INFO]
The baseSleepTimeMs [2000] the maxSleepTimeMs [6] the maxRetries [5]
2017-07-24 23:27:55.390 b.s.u.StormBoundedExponentialBackoffRetry [INFO]
The baseSleepTimeMs [2000] the maxSleepTimeMs [6] the maxRetries [5]
2017-07-24 23:28:00.342 b.s.d.worker [INFO] Shutting down worker
top-1-1500931668 10dd0bf8-daf1-496e-8ea7-142810464250 6703
2017-07-24 23:28:00.342 b.s.d.worker [INFO] Shutting down receive thread
2017-07-24 23:28:00.378 b.s.m.n.Client [INFO] creating Netty Client,
connecting to lenovo:6703, bufferSize: 5242880
2017-07-24 23:28:00.383 b.s.m.loader [INFO] Shutting down receiving-thread:
[top-1-1500931668, 6703]
2017-07-24 23:28:00.383 b.s.m.n.Client [INFO] closing Netty Client
Netty-Client-lenovo/192.168.x.x:6703 
2017-07-24 23:28:00.383 b.s.m.n.Client [INFO] waiting up to 60 ms to
send 0 pending messages to Netty-Client-lenovo/192.168.x.x:6703

2017-07-24 23:28:00.383 b.s.m.loader [INFO] Waiting for
receiving-thread:[top-1-1500931668, 6703] to die
2017-07-24 23:28:00.384 b.s.m.loader [INFO] Shutdown receiving-thread:
[top-1-1500931668, 6703]
2017-07-24 23:28:00.385 b.s.d.worker [INFO] Shut down receive thread
2017-07-24 23:28:00.385 b.s.d.worker [INFO] Terminating messaging context
2017-07-24 23:28:00.385 b.s.d.worker [INFO] Shutting down executors
2017-07-24 23:28:00.386 b.s.d.executor [INFO] Shutting down executor b-6:[8
8]
2017-07-24 23:28:00.387 b.s.util [INFO] Async loop interrupted!
2017-07-24 23:28:00.387 b.s.util [INFO] Async loop interrupted!
2017-07-24 23:28:00.388 b.s.d.executor [INFO] Shut down executor b-6:[8 8]
2017-07-24 23:28:00.388 b.s.d.executor [INFO] Shutting down executor
b-9:[47 47]
2017-07-24 23:28:00.389 b.s.util [INFO] Async loop interrupted!
2017-07-24 23:28:00.389 b.s.util [INFO] Async loop interrupted!


i hope i can find help , Thanks

[GitHub] storm issue #2239: [STORM-2655] fix: log users cannot view worker.log on Sto...

Github user Ethanlm commented on the issue:

https://github.com/apache/storm/pull/2239
  
Tested it manually.

![image](https://user-images.githubusercontent.com/14900612/28543136-63cd9718-7084-11e7-9ba5-a3509cc8b13b.png)



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request #2239: [STORM-2655] fix: log users cannot view worker.log...

GitHub user Ethanlm opened a pull request:

https://github.com/apache/storm/pull/2239

[STORM-2655] fix: log users cannot view worker.log on Storm UI on secâ¦

See: https://issues.apache.org/jira/browse/STORM-2655

solution: add princialToLocal inside of isAuthorizedLogUser function.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/Ethanlm/storm STORM-2655

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/storm/pull/2239.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2239


commit 7cf019b874ca937e134ae7a77de0f361965783dc
Author: Ethan Li 
Date:   2017-07-24T20:16:22Z

[STORM-2655] fix: log users cannot view worker.log on Storm UI on secure 
cluster




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm issue #2218: STORM-2614: Enhance stateful windowing to persist the win...

2017-07-24 Thread ben-manes

Github user ben-manes commented on the issue:

https://github.com/apache/storm/pull/2218
  
fyi, you can emulate pinning by setting the entry's weight to zero. Then 
the entry will not be evicted, but is eligible for expiration, etc. Of course 
zero weight entries means the cache allows more entries overall, whereas 
pinning might still restrict overall capacity. But its not clear how to a cache 
should handle evictions when not enough space can be freed from unpinned 
entries.

The other approach is less elegant. A `CacheWriter` can intercept an 
eviction and a `CacheLoader` intercepts gets to fetch a missing entry. Since 
recursive computations are disallowed, the writer could re-put the entry 
asynchronously. This races with a `get`, so the loader could fetch from the map 
prior to some other data store. In some ways not too dissimilar to your own 
pinned cache.

In general pinning breaks the caching model of recomputable transient state 
and the eviction policy will try to avoid pollution by removing low-value 
entries early. Generally it can lead to O(n) evictions unless maintained in a 
dedicated LRU queue so that potential victims are not evaluated. Since pinning 
vs capacity is confusing, right now Caffeine favors zero weights since the 
use-cases are unclear.

Hope that helps. I agree your own logic may be more straightforward and 
preferable.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm issue #2238: [STORM-2654] preserve the formatting of the exception sta...

Github user Ethanlm commented on the issue:

https://github.com/apache/storm/pull/2238
  
Changes:

![image](https://user-images.githubusercontent.com/14900612/28540603-908a9b42-707b-11e7-8fd3-9c8d1ad5d4b3.png)



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request #2238: [STORM-2654] preserve the formatting of the except...

GitHub user Ethanlm opened a pull request:

https://github.com/apache/storm/pull/2238

[STORM-2654] preserve the formatting of the exception stacktrace on UI

See https://issues.apache.org/jira/browse/STORM-2654


![image](https://user-images.githubusercontent.com/14900612/28540592-87a8155e-707b-11e7-85c1-9dc658c26b21.png)


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/Ethanlm/storm STORM-2654

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/storm/pull/2238.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2238


commit db47bf51ad52cfbdb59d503d7fee28e1a38deb1e
Author: Ethan Li 
Date:   2017-07-24T19:21:48Z

[STORM-2654] preserve the formatting of the exception stacktrace on UI




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[VOTE] Release Apache Storm 1.0.4 (rc1)

2017-07-24 Thread P. Taylor Goetz

This is a call to vote on releasing Apache Storm 1.0.4 (rc1)

Full list of changes in this release:

https://git-wip-us.apache.org/repos/asf?p=storm.git;a=blob_plain;f=CHANGELOG.md;h=50878fab679973a1230466920006dc0746ffddd5;hb=eac433b0beb3798c4723deb39b3c4fad446378f4

The tag/commit to be voted upon is v1.0.4:

https://git-wip-us.apache.org/repos/asf?p=storm.git;a=blob_plain;f=CHANGELOG.md;hb=a5e1c154b5b2ae74fd78bf10d4c130afb1ad4513

The source archive being voted upon can be found here:

https://dist.apache.org/repos/dist/dev/storm/apache-storm-1.0.4-rc1/apache-storm-1.0.4-src.tar.gz

Other release files, signatures and digests can be found here:

https://dist.apache.org/repos/dist/dev/storm/apache-storm-1.0.4-rc1/

The release artifacts are signed with the following key:

https://git-wip-us.apache.org/repos/asf?p=storm.git;a=blob_plain;f=KEYS;hb=22b832708295fa2c15c4f3c70ac0d2bc6fded4bd

The Nexus staging repository for this release is:

https://repository.apache.org/content/repositories/orgapachestorm-1048

Please vote on releasing this package as Apache Storm 1.0.4.

When voting, please list the actions taken to verify the release.

This vote will be open for at least 72 hours.

[ ] +1 Release this package as Apache Storm 1.0.4
[ ]  0 No opinion
[ ] -1 Do not release this package because...

Thanks to everyone who contributed to this release.

-Taylor

[GitHub] storm pull request #2207: [STORM-2626] Provided a template for drpc-auth-acl...

2017-07-24 Thread asfgit

Github user asfgit closed the pull request at:

https://github.com/apache/storm/pull/2207


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm issue #2207: [STORM-2626] Provided a template for drpc-auth-acl.yaml

2017-07-24 Thread liu-zhaokun

Github user liu-zhaokun commented on the issue:

https://github.com/apache/storm/pull/2207
  
@revans2
You are right.I created several PRs which has been mergerd that is related 
to ACL. I will update SECURITY.md in a new PR with these changes  later.Could 
you help me merge this PR firstly?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm issue #2204: STORM-1280 port backtype.storm.daemon.logviewer to java

Github user Ethanlm commented on the issue:

https://github.com/apache/storm/pull/2204
  
@HeartSaVioR  Yes. Sure. Should I wait until this PR being merged in?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm issue #2204: STORM-1280 port backtype.storm.daemon.logviewer to java

Github user revans2 commented on the issue:

https://github.com/apache/storm/pull/2204
  
@Ethanlm since you found this could you file a follow on JIRA to use the 
PrincipalToLocal plugin in the logviewer when authorizing users?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm issue #2204: STORM-1280 port backtype.storm.daemon.logviewer to java

Github user Ethanlm commented on the issue:

https://github.com/apache/storm/pull/2204
  
@HeartSaVioR @revans2  I think I figured it out. 
I added "et...@derekd.ystorm.net" to logs.users and then it works. I traced 
down to the 
[isAuthorizedLogUser](https://github.com/apache/storm/pull/2204/files#diff-aad25f432acc108d98b3126bbd914873R74)
 function and found that it doesn't translate "principal" to "user" so that 
"et...@derekd.ystorm.net" doesn't match with logs.user "ethan" in the 
configuration. @revans2 and I think the code is identical to the old code. So 
this should an existing bug.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm issue #2207: [STORM-2626] Provided a template for drpc-auth-acl.yaml

Github user revans2 commented on the issue:

https://github.com/apache/storm/pull/2207
  
I do agree with @Ethanlm that updating SECURITY.md would probably be more 
helpful, but if you want this to be a first step then I am happy to merge it in.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm issue #2237: [STORM-2653] Pacemaker code improvement

Github user revans2 commented on the issue:

https://github.com/apache/storm/pull/2237
  
@Ethanlm Great, now we need to wait for the full 24 hours to be up in case 
others want to respond to the pull request.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm issue #2237: PacemakerClient code improvement

Github user Ethanlm commented on the issue:

https://github.com/apache/storm/pull/2237
  
@revans2  Thanks. Create a JIRA: https://github.com/apache/storm/pull/2237


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm issue #2204: STORM-1280 port backtype.storm.daemon.logviewer to java

Github user revans2 commented on the issue:

https://github.com/apache/storm/pull/2204
  
The change looks good to me and I am +1.  Will wait for @Ethanlm to work 
through his issues first though.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm issue #2204: STORM-1280 port backtype.storm.daemon.logviewer to java

Github user Ethanlm commented on the issue:

https://github.com/apache/storm/pull/2204
  
Still not authorized.. But I guess it's my problem.. Will try to figure it 
out. Thanks.


![image](https://user-images.githubusercontent.com/14900612/28523809-8d57133e-7043-11e7-85cc-7cd72c94a198.png)

![image](https://user-images.githubusercontent.com/14900612/28523855-b8bdc266-7043-11e7-828b-b51e1de2bef8.png)



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request #2228: STORM-2646 NimbusClient Class cast exception when ...

2017-07-24 Thread ecararus

Github user ecararus closed the pull request at:

https://github.com/apache/storm/pull/2228


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm issue #2218: STORM-2614: Enhance stateful windowing to persist the win...