Dear all,
I want to specify some worker.childopts for my topology inner it's code,
and I use this way:
conf.put(Config.WORKER_CHILDOPTS, WORKER_OPTS);
but I found it doesn't work.
I don't use storm.yaml file to set worker.childopts, because the memory
requirement of my topologies are widely differ
Hi Chen,
You should remove the while() loop from nextTuple(),initialise the socket
opening code in your Spout's prepare() method(which gets called only once)
and in your nextTuple() you should have something like this:
String line;
if(line = socket.readLine()) != null) {
_collector.emit(new Va
That's not true, it won't make a difference performance-wise whether you
emit one tuple or many in nextTuple. The batching happens automatically
after that.
On Mon, Feb 17, 2014 at 3:51 PM, Danijel Schiavuzzi wrote:
> Also, Trident is batch-based, so in nextTuple() you should really emit
> batch
Also, Trident is batch-based, so in nextTuple() you should really emit
batches of tuples, not single tuples, for maximum performance.
On Tue, Feb 18, 2014 at 12:47 AM, Danijel Schiavuzzi wrote:
> Exactly, you must never block in nextTuple().
>
> Read your socket asynchronously (in a separate thre
Exactly, you must never block in nextTuple().
Read your socket asynchronously (in a separate thread) and fill the data
into a queue. Then in nextTuple() emit data from that queue.
On Tue, Feb 18, 2014 at 12:05 AM, Chen Wang wrote:
> If I remove the while loop in my spout, then everything works f
If I remove the while loop in my spout, then everything works fine.(well
except data loss in the spout..)
On Mon, Feb 17, 2014 at 2:46 PM, Chen Wang wrote:
> Its just a BaseRichSpout.
>
>
> On Mon, Feb 17, 2014 at 1:06 PM, Danijel Schiavuzzi wrote:
>
>> What interface does your spout implement?
Its just a BaseRichSpout.
On Mon, Feb 17, 2014 at 1:06 PM, Danijel Schiavuzzi wrote:
> What interface does your spout implement? Is it IBatchSpout?
>
>
> On Mon, Feb 17, 2014 at 8:49 PM, Chen Wang wrote:
>
>> Hi,
>> I might be missing something apparent. In my spout, next tuple method, I
>> have
This SO answer seems to answer your question:
http://stackoverflow.com/a/17454586/234901
On Mon, Feb 17, 2014 at 7:30 PM, Abhishek Bhattacharjee <
abhishek.bhattacharje...@gmail.com> wrote:
> If you see practically number of workers should always be less than or
> equal to the number of tasks. T
Danijel,
Thanks, this was tremendously helpful in getting these metrics out.
It would be great if this project were listed on:
https://github.com/nathanmarz/storm/wiki/Metrics. /cc Nathan
Cheers,
Mike
On Sun, Feb 16, 2014 at 5:57 PM, Danijel Schiavuzzi
wrote:
> Hi,
>
> You can have a look a
What interface does your spout implement? Is it IBatchSpout?
On Mon, Feb 17, 2014 at 8:49 PM, Chen Wang wrote:
> Hi,
> I might be missing something apparent. In my spout, next tuple method, I
> have an infinite loop to read from our socket server:
>
> @Override
>
> public void nextTuple() {
>
>
Hi,
I might be missing something apparent. In my spout, next tuple method, I
have an infinite loop to read from our socket server:
@Override
public void nextTuple() {
try {
String line;
while ((line = socket.readLine()) != null) {
System.out.println("emitting new entries");
_collector
If you see practically number of workers should always be less than or
equal to the number of tasks. This is not only valid for Comp sci but for
any field. If we have more no. of workers than tasks available then we are
just wasting our resources. That is we are not using our resources
optimally.
S
Hi Svend,
Thank you for your reply.
Unfortunately, the problem doesn't seem to be in the handling of dates at
the Java and database levels. I've confirmed this by eliminating all
java.sql.Date and java.util.Date types from my topology entirely, instead
rappresenting the timestamps as integers (mi
I've been looking at the parallelism of a storm topology. In what situations is
it useful to have more than one task running on the same executor? An executor
is a thread, so if there's several tasks on that executor only one task can run
at a time. So why would you want more than one task on th
Hi Aniket,
Thanks for the help. I finally managed to solve the problem. As suggested I did
three things :-
1) Restarted the entire topology
2) Specified the port of the redis server
3) Did a clean maven compile to generate the jar. Your pom.xml helped me
here.
Thanks for saving
Hello,
I am sending my traffic logs on my storm cluster having topology with one
supervisor
running. On my topology i have 5 bolts . when i send traffic at 3 milli
second delay , every bolts works fine , but as soon as i reduce the delay
to 2 milli second , bolt-1 stops and consequently bolt-2 st
Hello,
I am sending my traffic logs on my storm cluster having topology with one
supervisor
running. On my topology i have 5 bolts . when i send traffic at 3 milli
second delay , every bolts works fine , but as soon as i reduce the delay
to 2 milli second , bolt-1 stops and consequently bolt-2 st
I'm trying to implement a bolt that needs to be notified of tuple failures (it
puts incoming tuples on a queue, and gets notified separately when to emit the
tuples on the queue. If a tuple fails in another bolt, I need to take it off
the queue). However, I've come across multiple issues in impl
Hi Abhinav,
I guess problem is on the worker node where your code gets deployed.
Try installing redis on the machine on which you are deploying your code.
In the mean time you can checkout this repo and compare it with your code.
I recommend Maven.
https://github.com/aniketalhat/StatefulBolts
On
Hi,
I am quite new to both Redis and STORM. I have a bolt running writing data to a
redis map which works perfectly fine when running locally. When I deploy the
code to the cluster, I get a class not found exception specifically telling me
that redis client cannot be found ( Caused by:
java.lan
Hi Mike
I've just pushed the metrics consumer I wrote for Librato to
https://github.com/DigitalPebble/storm-crawler. As you'll see in the code,
the token and username have to be passed via configuration. The only quirk
is that you can specify that a datapoint is sent as a counter by putting
'_coun
Hello ,
Plz explain the impact of callinng these two fucntions on any Output
collector , means how it lets trident to cstch failed exception and retry
Thanks in advance
I think you should write your own implementation for fetching data from
kafka using kafka-api and not use the kafka-storm. It is easier and better
that way. You can refer the above repo for reference.
On Mon, Feb 17, 2014 at 12:06 PM, Aniket Alhat wrote:
>
> Hi Saurabh,
> Sorry I could not ident
23 matches
Mail list logo
