there another way that you can recommend?
I'm currently using logback but will happily switch over to log4j2 if that
makes it easier.
I'm using storm 1.2.1 & have tried adding the appenders/loggers to both the
local cluster.xml & worker.xml but nothing.
Thanks in advance,
Kevin.
So, how the messages are distributed depends on the type of Stream Grouping
you select. Probably the most common choice is ShuffleGrouping which does
send an equal number of tuples to each bolt instance.
The various types of stream groupings are described here (scroll down to
the Stream Groupings
The most common reason for spout failures but no bolt failures is tuple
timeouts. Try increasing MESSAGE_TIMEOUT_SECONDS. If you are seeing very
low execute latencies for your bolts but very high complete latency, you
can also try decreasing SPOUT_MAX_PENDING.
Hopefully this helps.
On Fri, Dec 9,
/defaults.yaml#L261
Based on these, I believe the LoadAwareShuffleGrouping is the default. Can
anyone more familiar with the code confirm this?
On Tue, Nov 22, 2016 at 11:43 AM, Kevin Peek wrote:
> Hello. I have a question about the load aware shuffle grouping.
>
> If I create a grouping like be
Hello. I have a question about the load aware shuffle grouping.
If I create a grouping like below, will I end up with a
LoadAwareShuffleGrouping by default?
topologyBuilder.setBolt("boltId", new
SomeBolt()).shuffleGrouping("otherBoltId");
I would expect this to result in a traditional ShuffleGro
I played around a little bit with Stephen's test and it seems that the
Collection.shuffle() call here is causing the problem (at least the problem
Stephen is talking about).
https://github.com/apache/storm/blob/1.0.x-branch/storm-core/src/jvm/org/apache/storm/grouping/ShuffleGrouping.java#L58
I cr
rtion of worker hooks
> functions properly, but not shutdown. Hopefully we will be able to fix both
> these issues in an upcoming release.
>
> -Taylor
>
>
> On Oct 21, 2016, at 9:58 AM, Kevin Peek wrote:
>
> I am running into problems with WorkerHooks on a local cluster. Ev
I am running into problems with WorkerHooks on a local cluster. Even using
only a BaseWorkerHook, I get an Exception. When I run the following code,
an EOFException is thrown - it seems the Worker is trying to deserialize an
empty byte[] for one of the WorkerHooks. Comment out the line adding the
h
Hello,I have this set and have storm running fine, deployed topologies, etc. Just wondering if there is any description of the directories that get created under storm.local.dir.Thanks,Kevin
Thanks for the clarification. Just odd to see a jump from 0.1x to a 1.x release... On 09/27/16, Junfeng Chen wrote: If you are newbie to Storm, I recommend you to try 1.0.* , as it introduces so many new features. Regard,Junfeng ChenOn Sun, Sep 25, 2016 at 3:04 AM, Kevin <kevinde...@verizon.
I think you have to drill down into the topology to see that On 09/26/16, S G wrote: Hi,I am unable to see the YAML file (with properties reduced to their actual values) on the storm UI.Does anyone know how to do that?Would it be a nice addition to the UI ?Thx,SG
Hello,I'm just getting started with Storm and a bit confused about what I see for releases. There is a 0.10.x version and a 1.0x version - which one do I want? What's the difference? I've downloaded both and I think one has the command line code and the other is the source with examples?Thanks i
see the progress of Storm SQL, you can have a
> look at epic issues below. Contributions would be more appreciated!
>
> - Storm SQL Phase II <https://issues.apache.org/jira/browse/STORM-1433>
> - Storm SQL Phase III <https://issues.apache.org/jira/browse/STORM-2075>
> (Adding
&
gt; with other companies under the control of BAE Systems Applied Intelligence
> Limited, details of which can be found at http://www.baesystems.com/
> Businesses/index.htm.
>
--
Kevin Conaway
http://www.linkedin.com/pub/kevin-conaway/7/107/580/
https://github.com/kevinconaway
Wed, Jul 27, 2016 at 1:32 PM, Kevin Peek wrote:
>
>> Thanks for the reply.
>>
>> In either of these cases, shouldn't storm stop letting the spout emit
>> tuples once max_spout_pending is reached? In that case, the tuples already
>> in the topology (or droppe
ects a batch and at some
> point ack's them all. In that case thes situation when batch bigger than
> max_spout_pending and some tuples fails.
>
> 2) Bolt doesn't ack tuple at all. Make sure Bolt acks or fails tuples
> without any exclusions.
>
> On Wed, Jul 27, 2
topology will show about
50K spout failures and zero bolt failures.
Am I misunderstanding something that would allow more tuples to time out,
or is there another source of spout failures?
Thanks in advance,
Kevin Peek
Thanks very much for all the options. I think I can probably achieve what
I'm looking for.
I appreciate the help very much.
Cheers,
Kevin
On 8 July 2016 at 06:33, Abhishek Agarwal wrote:
> I agree with Nathan. It would be great if you can narrow down your
> problem. You should t
ricted to just replace resource files,
> using distributed cache on 1.0.0 makes it possible.
>
> Hope it helps.
>
> Thanks,
> Jungtaek Lim (HeartSaVioR)
>
> 2016년 7월 8일 (금) 오전 6:30, Kevin Stembridge 님이
> 작성:
>
>> Hi all,
>>
>> I'm just looking at S
Hi all,
I'm just looking at Storm to see if it would be suitable for my use case.
What I need to be able to do is allow users to edit topologies on the fly.
Is it possible to do this?
>From what I can see, a topology is basically baked into a jar file and
deployed.
Thanks in advance.
Kevin
]cost: 1623692
> [2016-05-31 09:14:01]cost: 2524674
> [2016-05-31 09:14:02]cost: 3383506
> [2016-05-31 09:14:03]cost: 3898478
> [2016-05-31 09:14:04]cost: 2120949
> [2016-05-31 09:14:05]cost: 3756272
> [2016-05-31 09:14:06]cost: 2877997
> [2016-05-31 09:14:07]cost: 3432532
> [2016-05-31 09:14:08]cost: 3638306
> [2016-05-31 09:14:09]cost: 2958907
> [2016-05-31 09:14:10]cost: 2742666
> [2016-05-31 09:14:11]cost: 3024576
> [2016-05-31 09:14:12]cost: 2822562
> [2016-05-31 09:14:13]cost: 2623060
> [2016-05-31 09:14:14]cost: 4045938
>
> Obviously, there is a 2ms latency approximately. It seems not good for me.
> How can I reduce the latency?
>
--
Kevin Conaway
http://www.linkedin.com/pub/kevin-conaway/7/107/580/
https://github.com/kevinconaway
read.java:745)
Looking in the store code, I see that they are sleeping here:
(while (not @(:storm-active-atom executor-data))
(Thread/sleep 100))
At what point does the topology become "active"?
On Mon, May 2, 2016 at 11:38 AM, Kevin Conaway
wrote:
> Some more interesting info.
>
e?
On Mon, May 2, 2016 at 10:48 PM, Jungtaek Lim wrote:
> Kevin,
>
> For specific task, you can register your own metrics which resides per
> task.
> But metrics doc on Storm is not kind enough to let users follow, so I
> addressed this and submitted pull request.
> http
evidenced
by lack of log messages indicating so)
On the one supervisor which _did_ receive the timeout message, the 2
workers came up just fine.
On Mon, May 2, 2016 at 11:21 AM, Kevin Conaway
wrote:
> Unfortunately we're not capturing disk i/o in our metrics, I can look in
> to doing that fo
e
>
> I'm assuming you're saying that multiple workers had state :timed-out at
> once? Was that on the same host? If the state is :disallowed, that is
> perfectly normal when the reassignment happens, as I described earlier.
>
> - Erik
>
> On Sunday, May 1, 2016, K
is hard to determine.
>
> - Erik
>
> On Sunday, May 1, 2016, Kevin Conaway > wrote:
>
>> Thanks Erik. We're using Storm 0.10 so Pacemaker doesn't come in to play
>> here.
>>
>> 3. Worker process died due to exception (this is almost always wha
gt; <https://issues.apache.org/jira/browse/STORM-1699>
>> - Provide topology metrics in detail (metrics per each stream)
>> <https://issues.apache.org/jira/browse/STORM-1719>
>> - (WIP) Introduce Cluster Metrics Consumer
>>
>> As I don't maintain l
t config values you have on the worker
> and nimbus hosts.)
>
> - Erik
>
>
> On Saturday, April 30, 2016, Kevin Conaway
> wrote:
>
>> We are using Storm 0.10 and we noticed that Nimbus decided to restart our
>> topology. From researching past threads it seems li
oughts on what could have happened here and how to debug further?
--
Kevin Conaway
http://www.linkedin.com/pub/kevin-conaway/7/107/580/
https://github.com/kevinconaway
ods on the same
> thread."
>
> So if your topology spends non-ignorable time on nextTuple(), that could
> be affected to tuple latency.
>
> Hope this helps.
>
> Thanks,
> Jungtaek Lim (HeartSaVioR)
>
> 2016년 4월 20일 (수) 오전 10:21, Kevin Conaway 님이 작성:
>
>&
Yes it appears that the skipped-max-spout metric is only in 1.0.0 as part
of the automatic back pressure fixes (
https://issues.apache.org/jira/browse/STORM-886)
On Tue, Apr 19, 2016 at 9:21 PM, Kevin Conaway
wrote:
> We are already sending our metrics to graphite but I don't see
>
m to see
> what time is spent where for each executor thread as well as the netty and
> lmax layers.
>
> --John
>
> Sent from my iPhone
>
> On Apr 19, 2016, at 8:41 PM, Kevin Conaway
> wrote:
>
> In Storm 0.10, is there a way to monitor the _maxSpoutPending_ valu
or each is very low, like 0.01% of
capacity (< 10 entries at any given time) so I'm not sure that items are
sitting in those particular disruptor queues
On Tue, Apr 19, 2016 at 8:54 PM, Jungtaek Lim wrote:
> Hi Kevin,
>
> You can attach metrics consumer to log additional informatio
r topology has one spout and one bolt. The average bolt execute latency
is ~20ms but the overall spout latency is ~60ms. I'd like to know where
those other 40ms are being spent.
Thank you
--
Kevin Conaway
http://www.linkedin.com/pub/kevin-conaway/7/107/580/
https://github.com/kevinconaway
gt; 21,000 tuples it should be receiving.
>
>
> On Friday, April 15, 2016 12:56 PM, Kevin Conaway <
> kevin.a.cona...@gmail.com
> > wrote:
>
>
> How are you verifying that the tuples are failing? If you're looking at
> storm UI for the exact counts you may be m
guess, that would
appear to be whatever process is responsible for collecting metrics via
*IMetric.getValueAndReset()*
Does that provide any further insight in to what happened? I will keep
digging on my end.
Thanks,
Kevin
On Fri, Apr 15, 2016 at 2:17 PM, Kevin Conaway
wrote:
> I took thread
> You might want to check the thread dump and verify if some bolt is stuck
> somewhere
>
> Excuse typos
> On Apr 15, 2016 11:08 PM, "Kevin Conaway"
> wrote:
>
>> Was the bolt really "stuck" though given that the failure was at the
>> spout leve
metric
consumer until we killed and restarted the topology.
On Fri, Apr 15, 2016 at 1:31 PM, Abhishek Agarwal
wrote:
> Kevin,
> That would explain it. A stuck bolt will stall the whole topology.
> MetricConsumer runs as a bolt so it will be blocked as well
>
> Excuse typos
>
, which I thought
> would help, but still have not seen any improvement. Or is there something
> else that might be causing this problem?
>
> Thanks
>
--
Kevin Conaway
http://www.linkedin.com/pub/kevin-conaway/7/107/580/
https://github.com/kevinconaway
(like heap size, gc time) still have
been sent?
On Thu, Apr 14, 2016 at 10:09 PM, Kevin Conaway
wrote:
> Thank you for taking the time to respond.
>
> In my bolt I am registering 3 custom metrics (each a ReducedMetric to
> track the latency of individual operations in the bolt). The m
trics consumer wasn't
"keeping up". In storm UI, the processing latency was very low for that
pseudo-bolt, as was the capacity. Storm UI just showed that no tuples were
being delivered to the bolt.
Thanks!
On Thu, Apr 14, 2016 at 9:00 PM, Jungtaek Lim wrote:
> Kevin,
>
> D
of the workers
were restarted during this time.
Any thoughts on what could be causing this and how to diagnose further?
Thank you
--
Kevin Conaway
http://www.linkedin.com/pub/kevin-conaway/7/107/580/
https://github.com/kevinconaway
Hi,
I have used the the non-transactional spout and bolt for some tasks.
And it works great.
But when I tried to use the trident topology, I got confused.
What I want to do is to count unique user id in 5 minutes, and save the
number to db.
If I use the non-transactional spout and bolt, I use tic
1T01:40:39.305+ s.k.ZkCoordinator [INFO] Task [4/7] Finished
>>>> refreshing
>>>> 2015-07-01T01:41:39.311+ s.k.ZkCoordinator [INFO] Task [4/7]
>>>> Refreshing partition manager connections
>>>> 2015-07-01T01:41:39.454+ s.k.DynamicBrokersReader
Thank you for quick response! It has nothing different using IBatchSpout
and the data is keeping changing while refreshing the page. For more people
to join us, let's discuss in English. Chinese also is ok for me :).
之前是用的kafka-spout，发现了这个问题，所以测了一下一个空的spout会怎么样，结果就是这样。Why!
Best Regards,
Kev
Unsubscribe
发自我的 iPhone
> 在 2015年2月6日，03:49，Paul Fortin 写道：
>
> Unsubscribe
46 matches
Mail list logo
