hakraborty <
rudraneel.chakrabo...@gmail.com> wrote:
> You can assign executors of the same topology in a worker process. For an
> example if a topology has 10 executors , you can assign all of them in a
> single worker slot
>
>
> On Wednesday, 9 March 2016, Nick R. Katsipoulakis
&g
Hello all,
I was going through the backtype.storm.scheduler package and I came across
class Cluster. What is the difference between the following two methods:
Cluster.getAssignableSlots(SupervisorDetails supervisor) and
Cluster.getAvailableSlots(SupervisorDetails supervisor)
Also, if an execut
haras/StormTopologyTest>. (I only included
>> > files that I used, not whole project)
>> >
>> > Basically, I have a few questions if you don't mint to answer them
>> > 1) How to use HDFS to read and write?
>> > 2) Is my "scikit-learn" implementation correct?
>> > 3) How to create a Storm project? (Currently working in
>> "storm-starter")
>> >
>> > These questions may sound a bit silly, but I really can't find a
>> > proper solution.
>> >
>> > Thank you for your attention to this matter.
>> > Sincerely, Zharas.
>> >
>> >
>> >
>> >
>> > --
>> > Best regards,
>> > Zharas
>>
>>
>
>
> --
> Best regards,
> Zharas
>
--
Nick R. Katsipoulakis,
Department of Computer Science
University of Pittsburgh
gt;>>> Hi Everyone,
>>>>>>
>>>>>> I have a large fan-out that I've posted questions about before with
>>>>>> the following new, updated info:
>>>>>>
>>>>>> 1. Incoming tu
true>
>>> >> <http://docs.oracle.com/javase/6/docs/api/java/lang/Object.html?is-external=true>
>>> > tuple) method
>>> 4. I have throughput where I need it to be if I just receive tuples in
>>> Bolt B, ack, and drop. If I do actual process
.BlockingWaitStrategy--irrespective if I drop the
>> tuples or process in Bolt B
>>
>> I am wondering if the acking of the anchor tuples is what's resulting in
>> so much time spent in the LMAX messaging layer. What do y'all think? Any
>> ideas appreciated as always.
>>
>> Thanks! :)
>>
>> --John
>>
>
>
>
> --
> Nick R. Katsipoulakis,
> Department of Computer Science
> University of Pittsburgh
>
--
Nick R. Katsipoulakis,
Department of Computer Science
University of Pittsburgh
degrades a bunch.
> 5. I profiled the Bolt B worker yesterday and see that over 90% is spent
> in com.lmax.disruptor.BlockingWaitStrategy--irrespective if I drop the
> tuples or process in Bolt B
>
> I am wondering if the acking of the anchor tuples is what's resulting in
> so much time spent in the
work on the smallest number of emails to
> do it faster.
>
> But I still want to be able to have an XML that represent 10 000+ emails
> at the end.
>
>
>
> I can’t think of topology to address this.
>
> Can someone give me some pointers to the best way to handle this ?
developers
team.
Thanks
--
Nick R. Katsipoulakis,
Department of Computer Science
University of Pittsburgh
has any idea what goes wrong and I get this radical/buffling
behavior?
Thank you.
P.S.: I have attached my YAML file for further information and again I
apologize for the long post.
--
Nick R. Katsipoulakis,
Department of Computer Science
University of Pittsburgh
storm.yaml
Description: Binary data
;> On Tue, Nov 17, 2015 at 6:36 AM, Dmitry Sergeev
>> wrote:
>>
>>> First, I started Nimbus with
>>> ${storm.home}/bin/storm nimbus
>>> It starts successfully
>>> When I'm trying to start supervisor with this command:
>>> ${storm.home}/
is problem with security configuration.
> Storm 0.9.4 works fine.
> Can you help me?
>
> Thanks
>
--
Nick R. Katsipoulakis,
Department of Computer Science
University of Pittsburgh
r part of your log.
Nick
On Sat, Nov 7, 2015 at 8:17 PM, researcher cs
wrote:
> i'm just asking why you guess that there are another process using port
> 6703 ?!!!
> and my supervisor using one worker only
>
> On Sat, Nov 7, 2015 at 5:13 PM, Nick R. Katsipoulakis <
&
ess using port 6703 ?! i'm now
> running storm and think that this process for it ! am i right or wrong ?
>
> 3299 2766 0 02:55 pts/7 00:00:00 grep --color=auto 6703
>
> On Sat, Nov 7, 2015 at 5:06 PM, Nick R. Katsipoulakis <
> nick.kat...@gmail.com> wrote:
>
>> T
color=auto 6703
>>
>> and this command netstat -netulp | grep 2181 and got this
>> (Not all processes could be identified, non-owned process info will not
>> be shown, you would have to be root to see it all.) tcp6 0 0 :::2181 :::*
>> LISTEN 1000 14805 2301/java
>&g
,
Nick
On Sat, Nov 7, 2015 at 7:34 PM, researcher cs
wrote:
> sorry i'm new to storm
> is this command that check !?
> ps -ef|grep 67xx
>
> On Sat, Nov 7, 2015 at 4:17 PM, Nick R. Katsipoulakis <
> nick.kat...@gmail.com> wrote:
>
>> Hello,
>>
>>
ava:423)
>> [clojure-1.5.1.jar:na] at
>> backtype.storm.daemon.worker$fn__6959$mk_worker__7015.doInvoke(worker.clj:391)
>> [storm-core-0.9.5.jar:0.9.5] at clojure.lang.RestFn.invoke(RestFn.java:512)
>> [clojure-1.5.1.jar:na] at
>> backtype.storm.daemon.worker$_main.invoke(worker.clj:502)
>> [storm-core-0.9.5.jar:0.9.5] at
>> clojure.lang.AFn.applyToHelper(AFn.java:172) [clojure-1.5.1.jar:na] at
>> clojure.lang.AFn.applyTo(AFn.java:151) [clojure-1.5.1.jar:na] at
>> backtype.storm.daemon.worker.main(Unknown Source)
>> [storm-core-0.9.5.jar:0.9.5]*
>>
>> *storm version 0.9.5 *
>>
>> *zookeeper -3.4.6 *
>>
>
>
--
Nick R. Katsipoulakis,
Department of Computer Science
University of Pittsburgh
e on the same machine.
Thank you,
Nick
On Thu, Nov 5, 2015 at 10:06 AM, Nick R. Katsipoulakis <
nick.kat...@gmail.com> wrote:
> Hello all,
>
> Thank you very much for your replies. I actually uses Matthias' suggestion
> and it worked.
>
> Thanks,
> Nick
>
> On
ch component. Therefore Spout 1 task IDs are 0 to (n-1), Bolt 1
> > task IDs are 0 to (m-1), etc where n = number of Spout 1 tasks and m =
> > number of Bolt 1 tasks.
> >
> > If you want a cyclic graph in your topology I believe you have to use a
> > non-default strea
Hello,
I have a question regarding direct streaming and sending a tuple from a
downstream node to its upstream node. To be more precise, let us assume we
have the following topology:
Spout-1 --(direct-grouping)--> Bolt-1 --(direct-grouping)--> Bolt-2
Can Bolt-2 call emitDirect() and send a tuple
ker per node, my latency was bad Probably my
> use case was different.
>
> On Mon, Oct 5, 2015 at 10:29 AM, Nick R. Katsipoulakis <
> nick.kat...@gmail.com> wrote:
>
>> Hello guys,
>>
>> This is a really interesting discussion. I am also trying to fine
Hello guys,
This is a really interesting discussion. I am also trying to fine-tune the
performance of my cluster and especially my end-to-end-latency which ranges
from 200-1200 msec for a topology with 2 spouts (each one with 2k tuples
per second input rate) and 3 bolts. My cluster consists of 3 z
Hello Kashyap,
I have come up with something similar before. However, I have not yet
figured out how I can avoid these kind of Exceptions.
Nick
On Thu, Sep 24, 2015 at 5:19 AM, Kashyap Mhaisekar
wrote:
> Hi,
> To fix Netty reconnect issues, I upgraded to 0.9.5 of storm and it worked
> well in
e Storm calls all the spout methods on the
> same thread." -- to indicate it would be a good idea to fire up a separate
> thread in our custom spouts in what sounds like a similar way to you. So
> far we haven't had any issues doing this.
>
>
> On Fri, Sep 18, 2015 a
Hello all,
I have spouts that read input from files and send the data inside my
topology. In order to achieve higher input rates, I do some buffering of
data, by having them read by a thread, spawned after the spout is initiated
(in the open() function). The data are stored in an ArrayBlockingQueu
the worker context switches my task.
Thanks,
Nick
On Tue, Sep 15, 2015 at 10:20 AM, Nick R. Katsipoulakis <
nick.kat...@gmail.com> wrote:
> Hello again,
>
> Thank you for the link and the info. I am going to look into this in more
> detail.
>
> Cheers,
> Nick
>
>
to a
> specific node, you need to provide a custom scheduler.
>
> See here for an example:
>
> https://xumingming.sinaapp.com/885/twitter-storm-how-to-develop-a-pluggable-scheduler/
>
>
> -Matthias
>
> On 09/15/2015 03:31 PM, Nick R. Katsipoulakis wrote:
> > Hey
/hub/cs/dbis/aeolus/spouts/FixedStreamRateDriverSpout.java
>
> Feel free to use and/or modify both.
>
> -Matthias
>
>
> On 09/10/2015 10:18 PM, Nick R. Katsipoulakis wrote:
> > Hello,
> >
> > I am currently running some experiments and in order to send data t
Hello,
I am currently running some experiments and in order to send data to my
spouts, I do the following:
I spawn external processes which read the data from files (on disk) and
they send them through TCP sockets to Spouts. I do the former because (a) I
want to control the input rate of the spou
hold of emits/acks/fails that needs to be met before
> > the numbers show up on the UI.
> >
> > Often I will see 0 on the UI until, for example, the number of emits
> > reaches 20. And very often the numbers will increment by 20s too.--
> > Derek
&
s/acks/fails that needs to be met before the numbers
> show up on the UI.
>
> Often I will see 0 on the UI until, for example, the number of emits
> reaches 20. And very often the numbers will increment by 20s too.--
> Derek
>
>
> ________
>
g used. You
> need to do something like:
>
> Object id= ;
> _collector.emit(id,tuple);
>
> Regards,
> Javier
> On Sep 8, 2015 3:19 PM, "Nick R. Katsipoulakis"
> wrote:
>
>> Hello all,
>>
>> I am running a topology for bench marking my clust
Hello all,
I am running a topology for bench marking my cluster. In it, I anchor
tuples and I acknowledge them for exactly-once processing and in order to
see the complete latency metric on the Storm UI. However, the "Complete
Latency" and the "Acked" metric values for my spouts remain 0 and I gue
he bolt can be obtained in Storm UI with no extra
> code.
>
> I wanted to have also the network cost when distributing a computation.
Hence, the latencies on the UI are not suitable for me. On the other hand,
capacity is a useful metric. Do you know how I can get the capacity from
the Bol
h 12 worker slots), each of the 11
> executors is running on a single worker of a single supervisor (host).
>
> I am not idea why you observe a different behavior...
>
> -Matthias
>
> On 09/03/2015 12:20 AM, Nick R. Katsipoulakis wrote:
> > When I say co-locate, what I ha
gt; previous stable version of 0.8.2.
>
> I did not verify this thoroughly on my cluster though.
>
>
>
>
>
> *From:* Nick R. Katsipoulakis [mailto:nick.kat...@gmail.com]
> *Sent:* Thursday, September 03, 2015 9:08 AM
>
> *To:* user@storm.apache.org
> *Subject:* Re: Net
p=10) connected via shuffle grouping and have 12
>>>> supervisor available (each with 12 worker slots), each of the 11
>>>> executors is running on a single worker of a single supervisor (host).
>>>>
>>>> I am not idea why you observe a different beh
a shuffle grouping and have 12
>>> supervisor available (each with 12 worker slots), each of the 11
>>> executors is running on a single worker of a single supervisor (host).
>>>
>>> I am not idea why you observe a different behavior...
>>>
>>> -
, it is evidence that one or more of the workers are not
> starting up, which results in connections either not occuring or
> reconnecting occuring when supervisors kill workers that don't start up
> properly. I recommend checking the supervisor and nimbus logs to see if
> th
Hello Kashyap,
I have been having the same issue for some time now on my AWS cluster. To
be honest, I do not know how to resolve it.
Regards,
Nick
2015-09-03 0:07 GMT-04:00 Kashyap Mhaisekar :
> Hi,
> Has anyone experienced Netty reconnects repeatedly? My workers seem to be
> eternally in recon
oit this, you should use
> shuffle-or-local grouping instead of shuffle. The disadvantage of
> shuffle-or-local might be missing load-balancing. Shuffle always ensures
> good load balancing.
>
>
> -Matthias
>
>
>
> On 09/02/2015 10:31 PM, Nick R. Katsipoulakis wrote:
>
> multiple? Storm isolates topologies for fault-tolerance reasons. Thus, a
> single worker cannot process executors from different topologies. If you
> run out of workers, a topology might not start up completely.
>
> -Matthias
>
>
>
> On 09/02/2015 09:54 PM, Nick R. Kat
Hardware, ie, number of cores per node?
> - How many node/supervisor are available?
> - Configured number of workers for the topology?
> - What is the number of task for each spout/bolt?
> - What is the number of executors for each spout/bolt?
>
> -Matthias
>
> On 09/02/
Hello all,
I am working on a project in which I submit a topology to my Storm cluster,
but for some reason, some of my tasks do not start executing.
I can see that the above is happening because every bolt I have needs to
connect to an external server and do a registration to a service. However,
Hello all,
Is there a way to set the number of workers when submitting a topology. For
instance, when I type in:
$>storm jar path-to.jar com.MyTopology
If yes, how can I do the above?
Thanks,
Nikos
Hello,
I had the same issue when I tried to collocate all my executors to one
worker. I believe it has to do with resource contention among the different
threads (at least the spouts that are not starting). I would advise you to
make sure that you do not over-provision the system's resources and e
Hello Dimitri,
I also get similar errors on my AWS cluster and I have not figured out how
to resolve them. However, I do not think that they necessarily affect your
topologies.
Cheers,
Nikos
2015-07-27 12:22 GMT-04:00 Dimitris Sarlis :
> Hi all,
>
> I'm trying to run a Storm topology and I'm g
Hello,
Well the error you get means either that you have not properly defined the
Main class, or that the jar has not been built properly (and the jar does
not contain the built source code).
Cheers,
Nikos
2015-07-27 9:39 GMT-04:00 Vamsikrishna Vinjam <
vamsikrishna.vin...@inndata.in>:
> storm
ledged
> then there are settings that have to do with timeouts and maximum number of
> tuples "in flight".
> Set these "wrong" and you may see the effects you have.
> Or perhaps you forget to ack all tuples?
>
> Niels Basjes
>
> On Sun, 19 Jul 2015 22:46
Hello Manish,
Do you make sure that you are ACK-ing tuples after processing?
Nikos
2015-07-21 10:57 GMT-04:00 Manish Nema :
> Hi,
> I am running a topology in the 6 node cluster on RHEL 5 and Oracle JDK
> 1.7. Schematic for topology as attached, there are 6 workers for spout and
> bolts. Topol
performance difference between regular and direct
> streams (or streams with default name vs use specified name).
>
> (Internally, the first three methods just call the last method and add
> boolean flag "false" and/or stream name "default".)
>
> -Matthi
Hello,
I was wondering whether there is a performance gain if in the
declareOutputFields() of my spouts/bolts I use declareStream() instead of
declare()? In other words, if I give a hint to Storm that I am using direct
streams instead of letting the runtime figure it out by itself, will I see
bett
Hello,
No, I am not. Also, I am using direct-grouping for sending tuples between
the spout and the bolts.
Nikos
2015-07-19 14:40 GMT-04:00 Niels Basjes :
> Do you use Trident or the more low level API?
>
> Niels
>
> On Sun, Jul 19, 2015 at 7:40 PM, Nick R. Katsipoulakis
> wr
Hello all,
I have a topology in which a Spout (A) emits tuples to a Bolt (B) and in
turn, B emits tuples to a Bolt (C).
In order to perform some measurements in my topology I have Spout A send
some two types of tuples: normal data tuples and latency-measure tuples.
After sending a user-defined n
sk's id.
>>
>> Hope this helps.
>>
>> Kindly yours,
>>
>> Andrew Grammenos
>>
>> -- PGP PKey --
>> <https://www.dropbox.com/s/2kcxe59zsi9nrdt/pgpsig.txt>
>> https://www.dropbox.com/s/ei2nqsen641daei/pgpsig.txt
>>
>> On Thu,
t grouping depends on your code (i.e.
> you can create hotspots), fields grouping depends on your key distribution,
> etc.
>
> On Thu, Jul 16, 2015 at 3:50 PM, Nick R. Katsipoulakis <
> nick.kat...@gmail.com> wrote:
>
>> Hello all,
>>
>> I have two questions:
&
Hello all,
I have two questions:
1) How do you exactly measure latency? I am doing the same thing and I have
a problem getting the exact milliseconds of latency (mainly because of
clock drifting).
2) (to Nathan) Is there a difference in speeds among different groupings?
For instance, is shuffle f
and I computed that the latencies between spout
> emit and bolt consume and bolt emit to next bolt consume varies from 2 to
> 25ms. Have still not figured out what the problem is.
>
> The thread dumps indicate lmax disruptor but I guess that is expected.
>
> Thanks
> Kashyap
>
Hello all,
I have been working in Storm on low-latency applications. Specifically, I
currently work on an application in which I join 4 different streams
(relational join for those familiar with Relational Algebra). Before I jump
into the discussion, I would like to present you the technical detai
s are replayed again.
>
>
>
> http://storm.apache.org/documentation/Fault-tolerance.html
>
> http://storm.apache.org/documentation/FAQ.html
>
>
>
>
>
> *From:* Nick R. Katsipoulakis [mailto:nick.kat...@gmail.com]
> *Sent:* Thursday, June 25, 2015 12:18 PM
> *To:* u
rocesses, since we were testing
> on a shared machine. Is it your case?
>
> Regards,
> JG
>
> On Sun, Jun 28, 2015 at 11:46 AM, Nick R. Katsipoulakis <
> nick.kat...@gmail.com> wrote:
>
>> Javier thank you for your response.
>>
>> So, do you suggest
it's still working. I think we saw something like this some
> time during our PoC development, and it was fixed by allocating more memory
> to our workers - not enough memory was causing the workers to incur in
> heavy GC cycles.
>
> Regards,
> Javier
>
> On F
Hello,
I have been running a sample topology and I can see on the nimbus.log
messages like the following:
2015-06-26T19:46:35.556+ b.s.d.nimbus [INFO] Executor
tpch-q5-top-1-1435347835:[5 5] not alive
2015-06-26T19:46:35.557+ b.s.d.nimbus [INFO] Executor
tpch-q5-top-1-1435347835:[13 13] n
(or
design decision point) in Storm currently, or can I change something in my
configuration to resolve it.
Thank you very much for your time.
Regards,
Nick
-- Forwarded message --
From: Nick R. Katsipoulakis
Date: 2015-06-25 15:57 GMT-04:00
Subject: Problems with Storm EC2 setup
problem is that your
> host is somehow resolved as "Netty-Client-*", which is not pingable. You
> may modify /etc/hosts to map the hostnames to IP addresses appropriately if
> it is allowed.
>
> —
> Sincerely,
> Fan Jiang
>
>
> On Thu, Jun 25, 2015 at 3:5
", which is not pingable. You
> may modify /etc/hosts to map the hostnames to IP addresses appropriately if
> it is allowed.
>
> —
> Sincerely,
> Fan Jiang
>
>
> On Thu, Jun 25, 2015 at 3:57 PM, Nick R. Katsipoulakis <
> nick.kat...@gmail.com> wrote:
>
>
Hello all,
I apologize for the long message, but I have no idea what is going wrong in
my setup and I tried to give a lot of info about my cluster. I have the
following EC2 setup:
1) 3x m4.xlarge nodes for a 3-node ZooKeeper ensemble and a nimbus
2) 4x m4.xlarge nodes for my Supervisors.
All of
Hello,
I have the problem that at some point in a running topology, one of the
tasks running gets restarted by Storm. Under which circumstances can the
previous happen? Can it happen because of Netty (input rate of tuples is
higher than the process rate)?
I do not understand why the previous is h
run on that supervisor,
> whereas the latter is set per topology.
>
> On Thu, Jun 25, 2015 at 11:19 AM, Nick R. Katsipoulakis <
> nick.kat...@gmail.com> wrote:
>
>> I see. I will try to debug and see what's going on. Also, what is the
>> difference between worker.chil
er hash, it will give supervisor node and port assignments but
> requires some decoding). Then take a look at the worker log. Maybe your
> initialization is taking too long?
>
> On Thu, Jun 25, 2015 at 11:06 AM, Nick R. Katsipoulakis <
> nick.kat...@gmail.com> wrote:
>
>>
b50025 closed
2015-06-24T19:08:39.748+ o.a.s.z.ClientCnxn [INFO] EventThread shut down
But no indication on why the above is happening.
Thanks,
Nick
2015-06-25 10:52 GMT-04:00 Nathan Leung :
> Any problems in supervisor or nimbus logs?
>
> On Thu, Jun 25, 2015 at 10:49 AM, Nick R.
I am using m4.xlarge instances, each one with 4 workers per supervisor.
Yes, they are listed.
Nick
2015-06-25 10:47 GMT-04:00 Nathan Leung :
> How big are your EC2 instances? Are your supervisors listed in the storm
> UI?
>
> On Thu, Jun 25, 2015 at 10:43 AM, Nick R. Katsipoulakis
t; In general worker options need to be set in the supervisor config files.
>
> On Thu, Jun 25, 2015 at 10:07 AM, Nick R. Katsipoulakis <
> nick.kat...@gmail.com> wrote:
>
>> Hello sy.pan
>>
>> Thank you for the link. I will try the suggestions.
>>
>
2015年6月25日，02:14，Nick R. Katsipoulakis 写道：
>
> Hello all,
>
> I am working on an EC2 Storm cluster, and I want the workers in the
> supervisor machines to use 4GBs of memory, so I add the following line in
> the machine that hosts the nimbus:
>
> worker.childopts-Xmx4096m -
Hello all,
I am working on an EC2 Storm cluster, and I want the workers in the
supervisor machines to use 4GBs of memory, so I add the following line in
the machine that hosts the nimbus:
worker.childopts-Xmx4096m -XX:+UseConcMarkSweepGC -XX:+UseParNewGC
-XX:+UseConcMarkSweepGC -XX:NewSize=128m
-
Hello all,
I have recently updated my Storm cluster to use an ensemble of zookeeper
servers (3 is the exact number). I am running my VMs on EC2, and in order
to avoid updating the configuration files I used Elastic IPs for each of
the Nimbus and ZooKeeper servers (every time an instance restarts,
e the cause, please, feel free to
comment.
Thank you again,
Nick
2015-05-22 12:21 GMT-04:00 Nick R. Katsipoulakis :
> Hello Dima,
>
> I checked the security groups and they are properly set. I will try the
> storm.local.hostname to see if that can resolve the issue.
>
> Thank yo
s for any limitations.
>
> Also you can setting up "storm.local.hostname" in storm.yml on worker
> machine for using only public IPs.
>
> Best regards,
> Dmytro Dragan
> On May 22, 2015 18:38, "Nick R. Katsipoulakis"
> wrote:
>
>> Hello Jeff,
>>
&
>
>
> On Fri, May 22, 2015 at 10:04 AM, Nick R. Katsipoulakis <
> nick.kat...@gmail.com> wrote:
>
>> 52.7.165.232/52.7.165.232:2181. Will not attempt to authenticate using
>> SASL (unknown error)
>> 2015-05-21T20:45:11.030+ o.a.z.ClientCnxn [INFO] Socket co
Hello all,
I have recently ported my 5-node cluster on Amazon EC2. In detail, I have
one node running Nimbus and ZooKeeper, and 4 nodes running supervisors. All
of my instances are t2.micro type (1Ghz vCPU and 1GB RAM) and I am using
Storm 0.9.4, ZooKeeper 3.4.6, Open JDK v1.7, and all images run
> machines I guess one method is to send a message with a timestamp, get a
> response with the same time-stamp and get the difference.
>
> Thanks,
> Supun..
>
> On Thu, Apr 23, 2015 at 10:45 AM, Nick R. Katsipoulakis <
> nick.kat...@gmail.com> wrote:
>
>>
Hello Jeremy,
I have the same fears on using nano-time.
Cheers,
Nick
2015-04-23 10:29 GMT-04:00 Jeremy Heiler :
> On Wed, Apr 22, 2015 at 3:18 PM, Sushant Kumar
> wrote:
> >
> > look at
> http://docs.oracle.com/javase/1.5.0/docs/api/java/lang/System.html#nanoTime%28%29
> .
>
> I'd be wry of us
indow stats are not really very helpful
> for measuring the performance over time).
> https://github.com/ganglia/gmond_python_modules/tree/master/storm_topology
>
> That may be useful to understand how to get the process, execute latency
> etc. for Storm spout/bolt etc.
>
> On
Hello all,
I have been trying to calculate, as precisely as possible, how much time a
tuple spends waiting in the input queue of a Bolt until it is provided in
the BaseRichBolt.execute() function.
The way I try to get this wait time (latency) is by adding a timestamp
attribute in the tuple, the t
--
Nikolaos Romanos Katsipoulakis,
University of Pittsburgh, PhD candidate
Hello all,
I have been trying to calculate, as precisely as possible, how much time a
tuple spends waiting in the input queue of a Bolt until it is provided in
the BaseRichBolt.execute() function.
The way I try to get this wait time (latency) is by adding a timestamp
attribute in the tuple, the t
Actually Grant, I just figured out that is not difficult to setup NTP on
those machines. I will go with that for now. Thank you very much
Nikos
2015-03-19 10:26 GMT-04:00 Nick R. Katsipoulakis :
> Hello Grant and thanks for your reply. I am aware of NTP, but I would not
> like to go throu
the sender by
> reply email and delete all copies of this message.
>
> Please click here
> <http://www.cisco.com/web/about/doing_business/legal/cri/index.html> for
> Company Registration Information.
>
>
>
>
> From: "Nick R. Katsipoulakis"
> Repl
Hello all,
I came across the following issue in my cluster and I would like to share
it with you, in case you have any proposals/solutions:
I am running my Storm (0.9.2) application in a 5 node cluster. Each bolt is
assigned randomly to one of the available workers and during execution,
each work
t this tutorial [2].
>
> [1]
> https://github.com/nathanmarz/storm/blob/master/storm-core/src/jvm/backtype/storm/metric/LoggingMetricsConsumer.java
> [2] https://www.endgame.com/blog/storm-metrics-how-to.html
>
> 2015-03-13 13:58 GMT+01:00 Nick R. Katsipoulakis :
>
>> Hello a
Hello all,
In Storm UI I can see the following metrics for each bolt/spout in an
active topology: Capacity and Latency. Is there a way to get those metrics
from the bolt's code and store them in a custom log file.
Thank you,
Nikos
--
Nikolaos Romanos Katsipoulakis,
University of Pittsburgh, Ph
eption has anything to do with the version of Storm I am
using (0.9.2)?
Thanks,
Nikos
2015-03-04 13:06 GMT-05:00 Nick R. Katsipoulakis :
> Hello all,
>
> I attempt to submit a topology with one spout and two bolts. During
> initialization I see that the two bolts start execution no
Hello all,
I attempt to submit a topology with one spout and two bolts. During
initialization I see that the two bolts start execution normally, but my
spout sends the following error to the Storm UI and does not start
execution:
java.lang.RuntimeException: Cannot get field 'spout' because union
Hello,
I have had my cluster running for a while and my worker nodes' log files
are getting really big in size. What is the best way to clear up some
space? Should I just erase them and move them back them up somewhere else?
Thanks,
Nikos
--
Nikolaos Romanos Katsipoulakis,
University of Pittsbur
Hello,
I am trying to setup the logviewer service in each node in a storm cluster.
Therefore, I have the following configurations in the storm.yaml files:
nimbus-node:
logviewer.port: 8000
logviewer.childopts: "-Xmx128m"
logviewer.appender.name: "A1"
supervisor-node:
logviewer.port: 8000
logvi
:09 GMT-05:00 Nathan Leung :
> with 4 workers, by default, you would also have 4 acker tasks (1 /
> worker), which is why you might see 8 tasks.
>
> On Mon, Feb 23, 2015 at 1:56 PM, Nick R. Katsipoulakis <
> nick.kat...@gmail.com> wrote:
>
>> Hello,
>>
&g
Hello,
I am trying to control my cluster's parallelism and I want to achieve the
following: 1 worker per component, and 1 executioner per worker and 1 task
per executioner. In order to achieve the former, I submit my components to
the topology as follows:
Config conf = new Config();
TopologyBuil
Hello all,
I recently decided to try the storm-deploy project for deploying my
topologies. Therefore, I create an EC2 account and start an instance. I
follow the configuration steps, I configure the ~/.pallet/config.clj
accordingly and I leave the conf/clusters.yaml file with its default
values. H
e the fat jar included the dependency, which is provided by nimbus
>> (and workers) when deploying in the cluster.
>>
>> ...
>>
>> org.apache.storm
>> storm-core
>> ${storm.version}
>> provid
Hello,
I am currently trying to automate some experiments on Storm and to do so, I
have created a Java process which in turn launches three Java sub-processes
using the ProcessBuilder framework. The three processes are mainly
executions of jar files (2 of them simple java applications, 1 of them
S
1 - 100 of 102 matches
Mail list logo
