Yes, it's been fixed in 'master' for some time now.
Danijel
On Tuesday, September 16, 2014, M.Tarkeshwar Rao
wrote:
> Hi Danijel,
>
> Is the issue resolved in any version of the storm?
>
> Regards
> Tarkeshwar
>
> On Thu, Jul 17, 2014 at 6:57 PM, Danije
elp much.
>>>>
>>>> Thanks,
>>>>
>>>> Aaron Zimmerman
>>>>
>>>
>>>
>>> --
>>> Kindly yours,
>>>
>>> Andrew Grammenos
>>>
>>> -- PGP PKey --
>>> <
t; On Thu, Jul 17, 2014 at 1:37 AM, Danijel Schiavuzzi <
> dani...@schiavuzzi.com> wrote:
>
>> I see no issues with your cluster configuration.
>>
>> You should definitely share the (simplified if possible) topology
>> code and the steps to reproduce the blockage,
t.getString(i));
> }
> return sb.toString();
> }
> }
>
> However, the HashSet is stored in memory, when the data grows to a very
> large level, I think it will cause a OOM.
> So is there a scalable solution?
>
> 2014-07-14
>
.scheduler.meta:
> name: "supervisor1"
>
>
> worker.childopts: "-Xmx2048m"
>
> mm.hdfs.ipaddress: "10.61.244.7"
> mm.hdfs.port: 9000
> topology.batch.size: 2
> topology.batch.timeout: 1
> topology.workers: 2
> topology.d
expect my topology to have a throughput of around 80,000/s and I've
>>> been seeing a complete latency of around 300ms, so given this formula, I'd
>>> want 2*80000*.3 = 48,000 Max Spout Pending.
>>>
>>> This seems absurdly high to me..
>>>
>>> --
>>> Raphael Hsieh
>>>
>>>
>>>
>>>
>>
>>
>>
>> --
>> Carlos Rodríguez
>> Developer at ENEO Tecnología
>> http://redborder.net/
>> http://lnkd.in/bgfCVF9
>>
>
>
>
> --
> Raphael Hsieh
>
>
>
>
--
Danijel Schiavuzzi
E: dani...@schiavuzzi.com
W: www.schiavuzzi.com
T: +385989035562
Skype: danijels7
> Telles Mota Vidal Nobrega
> M.sc. Candidate at UFCG
> B.sc. in Computer Science at UFCG
> Software Engineer at OpenStack Project - HP/LSD-UFCG
>
--
Danijel Schiavuzzi
E: dani...@schiavuzzi.com
W: www.schiavuzzi.com
T: +385989035562
Skype: danijels7
2014-07-10 23:20 GMT+08:00 Can Gencer :
>>>
>>>> Hi David,
>>>>
>>>> Another option would be to create two seperate persistentAggregates
>>>> from the groupedStream as Sam mentioned, and then use topology.join() to
>>>> join them on
ot;tridentTestOpaqueSpout", new
> OpaqueTridentKafkaSpout(kafkaConfig))
> .name("tridentTestOpaqueSpout")
> .each(new Fields("str"), new PrinterBolt(),
> new Fields("print"));
>
> Config config
config.setNumWorkers(1);
> config.setMaxSpoutPending(3);
>
> StormSubmitter.submitTopologyWithProgressBar(args[0], config,
topology.build());
> }
> }
>
>
> // note
>
> I also resolved issue with reusing spout name across topologies, as per
recommendati
r each emitted tuple?
>
--
Danijel Schiavuzzi
E: dani...@schiavuzzi.com
W: www.schiavuzzi.com
T: +385989035562
Skype: danijels7
he current offset
> is more then 10 behind by default?
>
> This seems like a bad default value, the spout literally skipped over
> months of data without any warning.
>
> Are the core contributors open to accepting a pull request that would set
> the default to Long.MAX_VALUE?
>
> Thanks,
>
> Curtis Allen
>
>
--
Danijel Schiavuzzi
E: dani...@schiavuzzi.com
W: www.schiavuzzi.com
T: +385989035562
Skype: danijels7
);
>
>
> Config config = new Config();
> config.put(Config.TOPOLOGY_TRIDENT_BATCH_EMIT_INTERVAL_MILLIS,
> 2000);
> config.setNumWorkers(1);
> config.setMaxSpoutPending(3);
>
> StormSubmitter.submitTopologyWithProgressBar(args[0], confi
new Sum(),
> new Fields("total_some_int_1", "total_
> some_int_2"))
>
> but got
>
> Exception in thread "main" java.lang.IllegalArgumentException: Combiner
> aggs only take a single field as input. Go
Also, you should paste all your worker logs (worker-*.log files).
On Tuesday, July 8, 2014, Danijel Schiavuzzi wrote:
> I'd double check the Kafka producer to make sure those messages are really
> getting into the right Kafka topic. Also,
> try enabling Config.setDebug(true) and
ing first I tried with 10, than removed it (left
> default value)
>
>
> On Tue, Jul 8, 2014 at 6:31 PM, Danijel Schiavuzzi > wrote:
>
>> Are you sure you are producing new messages into the same Kafka
>> topic? What number did you set maxSpoutPending to?
>>
>&g
t; left from load testing on one machine)
>
> Still same result: no matter what, new messages are not taken from kafka
> after topology is redeployed.
>
>
> On Tue, Jul 8, 2014 at 6:15 PM, Danijel Schiavuzzi > wrote:
>
>> Try lowering setMaxSpoutPending(10) to a
l datastore such as dynamoDb ?
>
> --
> Raphael Hsieh
>
>
>
>
--
Danijel Schiavuzzi
E: dani...@schiavuzzi.com
W: www.schiavuzzi.com
T: +385989035562
Skype: danijels7
e topology deployment and short run
> http://pastie.org/private/4xsk6pijvmulwrcg7zgca
>
> those are VMs that host this storm cluster
> 10.0.0.241 zookeeper1
> 10.0.0.101 supervisor1
> 10.0.0.21 kafka1
> 10.0.0.251 nimbus1
>
> Thanks,
> Milos
>
>
>
>
>
>
>
>
--
Danijel Schiavuzzi
E: dani...@schiavuzzi.com
W: www.schiavuzzi.com
T: +385989035562
Skype: danijels7
killed worker spawn on new slot(new communication port) then it
> working fine. Same behavior as in storm 9.0.1.
>
> Please update me if you get any new development.
>
> Regards
> Tarkeshwar
>
>
> On Thu, Jul 3, 2014 at 7:06 PM, Danijel Schiavuzzi > wrote:
>
>> Hi Bo
ans wrote:
> I have not seen this before, if you could file a JIRA on this that would
> be great.
>
> - Bobby
>
> From: Danijel Schiavuzzi
> Reply-To: "user@storm.incubator.apache.org" <
> user@storm.incubator.apache.org>
> Date: Wednesday, June 4, 20
> On Tue, Jul 1, 2014 at 2:53 PM, Danijel Schiavuzzi > wrote:
>
>> Is the topology a Trident transactional topology maybe?
>> On Jul 1, 2014 9:02 AM, "M.Tarkeshwar Rao"
>> wrote:
>>
>>> Hi All,
>>>
>>> We are using apache storm
Is the topology a Trident transactional topology maybe?
On Jul 1, 2014 9:02 AM, "M.Tarkeshwar Rao" wrote:
> Hi All,
>
> We are using apache storm 9.01.
> Our topology have two worker processes. If one process of the worker
> killed our topology getting hanged.
>
> Any inputs?
> Runs fine with zer
t is the bottleneck!!
>
> BUT!!! I try to increase the parallelism hint of parseDataBolt from 20 to
> 50, and it almost has no effect of throughput. What's the problems? If I
> need to process more tuples in the future, what's the solution ?
>
>
> Best regards,
> Jam
At your current throughput rate, the choice of Java or Kryo serialization
doesn't matter much. The bottleneck seems to be somewhere else.
;>
>>> I can use a persistentAggregate to calculate the running total for each
>>> word. However I can't use another persistentAggregate directly chained to
>>> that as that would only add the new value instead of decrementing the old
>>> value for the group fi
is about 3KB. What types of message do
>> you fetch from kafka?
>>
>> Another import question is what kafka-storm do you use? I see so many
>> different versions of them and make me confused. Can you share storm config
>> in your topology and kafkaSpout's config to me?
Try to run the topology without the MySQL bolt to find out if that's the
bottleneck. Do you update the database in batches? That's an essential
optimization you should implement.
With a two node Storm cluster I can fetch 450 000 messages/s from Kafka,
and that's with a Trident transactional topol
I had a similar problem, with Netty my Trident transactional topology was
getting stuck after several occurences of Kafka spout restarting due to
Kafka SocketTimeouts (I can reproduce this bug by blocking access to Kafka
from the Supervisor machines with iptables, only a few tries are needed to
rep
2014 at 6:22 PM, Raphael Hsieh wrote:
> Is the storm.trident.util.LRUMap distributed among all the hosts in the
> storm cluster ?
>
> if not, is there any way to combine this with a memcache ?
>
> --
> Raphael Hsieh
>
>
>
>
--
Danijel Schiavuzzi
E
of a tie vote from the PPMC, votes from the community
> will be used to break the tie.
>
> This vote will be open until Monday, June 16 11:59 PM UTC.
>
> - Taylor
>
--
Danijel Schiavuzzi
E: dani...@schiavuzzi.com
W: www.schiavuzzi.com
T: +385989035562
Skype: danijels7
k), but this can be scaled
>> with more kafka partitions.
>> However $spoutcoord-spout0 is always a unique thread in trident, whatever
>> spout we provide, and I clearly observed that $spoutcoord-spout0 goes above
>> 1 after some minutes (and latency is above 10 sec or something).
>> Is there a way to improve this ? Or is this an unavoidable consequence of
>> trident's transactional logic that can't be addressed ?
>>
>>
>
--
Danijel Schiavuzzi
E: dani...@schiavuzzi.com
W: www.schiavuzzi.com
T: +385989035562
Skype: danijels7
.9.1-incubating
> kafka: 0.8.0.1
> storm-kafka-0.8-plus 0.3
> trident-redis <https://github.com/kstyrc/trident-redis>: 1.1-SNAPSHOT
>
> Thanks!
>
> Jie
>
--
Danijel Schiavuzzi
E: dani...@schiavuzzi.com
W: www.schiavuzzi.com
T: +385989035562
Skype: danijels7
s on this? Anyone had the same problem? It does seem a serious
issue as it affect the reliability and fault tolerance of the Storm cluster.
In the meantime, I'll try to prepare a reproducible test case for this.
Thanks,
Danijel
On Mon, Mar 31, 2014 at 4:39 PM, Danijel Schiavuzzi
wrote:
illing data.
>>> Is there any documentation on the start offsets that I could take a look
>>> at ?
>>> Or even documentation on kafka.api.OffsetRequest.LatestTime() ?
>>>
>>>
>>> On Wed, May 28, 2014 at 1:01 PM, Raphael Hsieh wrote:
>>>
|
> KiB/r 0 | | KiB/w 4 | MBr/s 0.00 | MBw/s
> 0.75 | avq 1.00 | avio 4.88 ms
>
> Regars,
>
> Andres
>
> El 28/05/2014, a las 04:01, Srinath C
> >
> escribió:
>
> Apart from the autopurge options, also set the number of transac
it left off. I would
> rather manually process that old data, and have my storm system start
> processing the live data.
>
> Thanks
> --
> Raphael Hsieh
>
>
>
>
--
Danijel Schiavuzzi
E: dani...@schiavuzzi.com
W: www.schiavuzzi.com
T: +385989035562
Skype: danijels7
starting from?
> >> From my understanding Storm will use zookeeper in order to tell its
> place in the Kafka stream. Where can I find metrics on this ?
> >> How can I see how large the stream is? What how much data is sitting in
> the stream and what the most recent/oldest positio
utor.clj:398)
> ~[na:na]
> at
> backtype.storm.disruptor$clojure_handler$reify__1894.onEvent(disruptor.clj:58)
> ~[na:na]
> at backtype.storm.utils.DisruptorQueue.consumeBatchToCursor
>
> If someone has some pointers on that issue, they are more than welcome.
>
> Here is the code :
>
> https://github.com/lerouxrgd/kafka-storm-starter/tree/plugin_trient-memcached
>
>
>
>
--
Danijel Schiavuzzi
E: dani...@schiavuzzi.com
W: www.schiavuzzi.com
T: +385989035562
Skype: danijels7
you know how I could fix it? Is there =
> any way to write less frequently in zookeeper?
>
>
> Regards,
>
> Andres
>
--
Danijel Schiavuzzi
E: dani...@schiavuzzi.com
W: www.schiavuzzi.com
T: +385989035562
Skype: danijels7
ng, send it to whatever
> datastore is specified.
>
> Is this the case ? Or will the Persistent Aggregate use the external
> datastore in order to do the aggregations ?
>
> --
> Raphael Hsieh
>
>
>
--
Danijel Schiavuzzi
E: dani...@schiavuzzi.com
W: www.schiavuzzi.com
T: +385989035562
Skype: danijels7
is mailing list's archive (the new one
at Apache and the old one at Google Groups, storm-user and storm-dev) --
lots of useful info is buried here.
Best regards,
Danijel Schiavuzzi
www.schiavuzzi.com
On Sunday, May 25, 2014, Przemek Grzędzielski
wrote:
> Hello Storm community,
>
>
Hi,
(1) do persistent aggregates -- as their name says, they aggregate and
persist those aggregates into an external datastore.
While (2) do aggregations on streams only, without persisting the resulting
aggregates.
Regards,
Danijel Schiavuzzi
www.schiavuzzi.com
On Tue, May 13, 2014 at 9:40
m/u/48250946/4.png
>
> Topology Configuration: http://pastebin.com/uLsVa5Hn
>
> I hope these are useful,
> Thanks for your time!
>
>
> 2014-04-25 14:59 GMT+02:00 Danijel Schiavuzzi :
>
> Could you provide the Storm UI stats for the "spout3" component?
>>
eper ? ).
>
> I have been using storm for quite a while and managed to make my
> production topology quite stable but still have some minor craches on the
> worker nodes at some times.
>
> Checking the Storm UI is a solution but does not offer immediate
> notifications by itself.
&
the throughput is high for this
> configuration (4k msg/sec), or maybe that I have a low number of kafka
> partitions.
>
> I would appreciate any information about what is causing this and any tip
> about kafka & storm performance :)
>
> Thanks!
>
--
Danijel Schiavuzzi
E: dani...@schiavuzzi.com
W: www.schiavuzzi.com
T: +385989035562
Skype: danijels7
ut()) to persist the updated aggregations back to the underlying data
store.
Take a look at the Storm sources under the package "storm.trident.*". A
good starting point for understanding Trident would be the Java class
"storm.trident.state.map.TransactionalMap" (or OpaqueMap or
That's because the metrics consumer you implemented runs on the cluster as
an actual Storm bolt itself, consuming the metrics from the metrics stream.
If you enable the "Show System Stats" option in Storm UI, you'll see your
metrics bolt running on the cluster.
=========
>>
>> Gvain
>>
>> Email: jh.li...@gmail.com
>>
>
>
--
Danijel Schiavuzzi
E: dani...@schiavuzzi.com
W: www.schiavuzzi.com
T: +385989035562
Skype: danijels7
d situation occurs. This would cause the batch
> to be retried, but it's deterministically failing, so the batch will be
> retried forever.
>
>
> On Mon, Apr 7, 2014 at 10:37 AM, Danijel Schiavuzzi <
> dani...@schiavuzzi.com> wrote:
>
>> Hi Jason,
>>
&g
ch.
>
>
>
> On Mon, Mar 31, 2014 at 7:39 AM, Danijel Schiavuzzi <
> dani...@schiavuzzi.com> wrote:
>
> To (partially) answer my own question -- I still have no idea on the cause
> of the stuck topology, but re-submitting the topology helps -- after
> re-submittin
To (partially) answer my own question -- I still have no idea on the cause
of the stuck topology, but re-submitting the topology helps -- after
re-submitting my topology is now running normally.
On Wed, Mar 26, 2014 at 6:04 PM, Danijel Schiavuzzi
wrote:
> Also, I did have multiple cases of
r
why it hasn't been successfully acked by Trident.
Any suggestions on the area (topology component) to focus my research on?
Thanks,
On Wed, Mar 26, 2014 at 5:32 PM, Danijel Schiavuzzi
wrote:
> Hello,
>
> I'm having problems with my transactional Trident topology. It has been
stuck topology? Any other useful info I might
provide?
Thanks,
--
Danijel Schiavuzzi
E: dani...@schiavuzzi.com
W: www.schiavuzzi.com
T: +385989035562
Skype: danijel.schiavuzzi
stuck topology? Any other useful info I might
provide?
Thanks,
--
Danijel Schiavuzzi
E: dani...@schiavuzzi.com
W: www.schiavuzzi.com
T: +385989035562
Skype: danijel.schiavuzzi
In Trident, you can use stateQuery() to query a TridentState. See the Word
Count demo topology for an example of usage.
Best regards,
Danijel Schiavuzzi
www.schiavuzzi.com
On Thu, Mar 13, 2014 at 6:22 PM, Dirk Weissenborn <
dirk.weissenb...@gmail.com> wrote:
> Hey guys,
>
> w
Hi Binita,
You seem to be using Kafka 0.8, in which case you must use the Kafka spout
from this project: https://github.com/wurstmeister/storm-kafka-0.8-plus/
The current storm-kafka KafkaSpout doesn't support the new Kafka version
0.8.
Best regards,
Danijel Schiavuzzi
www.schiavuzzi.com
nce started on QA in QA UI
> (port 8084). The QA UI (port 8084) is still showing supervisor of DEV
> box.
>
> Should I assume we can't use same zookeeper environment for DEV and QA?
>
> Regards,
> Arun
>
--
Danijel Schiavuzzi
E: dani...@schiavuzzi.com
W: www.schiavuzzi.com
T: +385989035562
Skype: danijels7
new Fields());
>
> I was thinking I'd put the flushing logic into ProcessResultFunction. But
> then I would have
> to be able to signal my IBackingMap implementation to clear its state
> since I'm dealing with time windows. And I cannot figure out a proper way
> to do this. :o
>
> - Matti
>
--
Danijel Schiavuzzi
E: dani...@schiavuzzi.com
W: www.schiavuzzi.com
T: +385989035562
Skype: danijels7
anyone know how emitBatch works? How can I avoid blocking in
> emitBatch?
>
> Thanks.
>
> Yunmeng
>
--
Danijel Schiavuzzi
E: dani...@schiavuzzi.com
W: www.schiavuzzi.com
T: +385989035562
Skype: danijels7
ble?
>
> If that cannot be accomplished with the current architecture, are there
> any other facilities in storm to handle a problem like this?
>
> Thanks very much!
>
--
Danijel Schiavuzzi
do you confirm it's the case for this topology?
>
>
> Cheers,
>
> S
>
>
>
>
>
>
> On Mon, Feb 17, 2014 at 6:45 PM, Danijel Schiavuzzi <
> dani...@schiavuzzi.com> wrote:
>
>> Hi Svend,
>>
>> Thank you for your reply.
>>
>
Also, Trident is batch-based, so in nextTuple() you should really emit
batches of tuples, not single tuples, for maximum performance.
On Tue, Feb 18, 2014 at 12:47 AM, Danijel Schiavuzzi wrote:
> Exactly, you must never block in nextTuple().
>
> Read your socket asynchronously (in a
rks fine.(well
> except data loss in the spout..)
>
>
> On Mon, Feb 17, 2014 at 2:46 PM, Chen Wang wrote:
>
>> Its just a BaseRichSpout.
>>
>>
>> On Mon, Feb 17, 2014 at 1:06 PM, Danijel Schiavuzzi
>> wrote:
>>
>>> What interface does your
> When I run it in local mode, it seems that the trident will not catch any
> tuple emitted from my spout. The topology will just spin on the spout
> emitting tuples, while the rest of the topology never get executed. How
> would I fix that?
>
>
> Using normal bolt/topology will be able to receive the tuple.
>
> Thanks,
>
> Chen
>
--
Danijel Schiavuzzi
st added field is different in Java and in MSSQL.
>
>
> I hope this helps!
>
>
> Svend
> http://svendvanderveken.wordpress.com/
>
>
>
>
>
>
>
>
>
>
>
> On Fri, Feb 14, 2014 at 11:48 AM, Danijel Schiavuzzi <
> dani...@schiavuzzi.com>
has some examples it would really save me some time.
>
>
> Thanks!
>
> Mike
>
> --
>
> Mike Heffner
>
> >
> Librato, Inc.
>
>
--
Danijel Schiavuzzi
E: dani...@schiavuzzi.com
W: www.schiavuzzi.com
T: +385989035562
Skype: danijels7
>>>> How should I go about it ?
>>>> Any help is appreciated.
>>>>
>>>> Thanks,
>>>>
>>>> *Abhishek Bhattacharjee*
>>>> *Pune Institute of Computer Technology*
>>>>
>>>
>>
>>
>> --
>> *Abhishek Bhattacharjee*
>> *Pune Institute of Computer Technology*
>>
>
>
>
> --
> *Abhishek Bhattacharjee*
> *Pune Institute of Computer Technology*
>
--
Danijel Schiavuzzi
E: dani...@schiavuzzi.com
W: www.schiavuzzi.com
T: +385989035562
Skype: danijels7
transferred values, and the Ack-ed values . Is that expected?
>
> To answer a simple question of "How many words are being processed by the
> TridentWordCount topology in a second or Minute", What should i do? Please
> advice.
>
> Thanks for your responses, in Advance.
>
> Regards,
> Prem
>
--
Danijel Schiavuzzi
further, I'd welcome any info or hint
>> on avoiding these deadlocks. Should I improve my TridentState
>> implementation somewhere? I see that the State is passed the partitionIndex
>> and numPartition parameters on initialization, which I currently don't use
>> -- and which I suppose are the key to implementing paralellism right?
>>
>> Also, could you recommend an example TridentState implementation which
>> got the paralellism right?
>>
>> Any help is appreciated,
>>
>> --
>> Danijel Schiavuzzi
>>
>
>
--
Danijel Schiavuzzi
E: dani...@schiavuzzi.com
W: www.schiavuzzi.com
T: +385989035562
Skype: danijels7
lization, which I currently don't use --
and which I suppose are the key to implementing paralellism right?
Also, could you recommend an example TridentState implementation which got
the paralellism right?
Any help is appreciated,
--
Danijel Schiavuzzi
> Hi all,
>
>
> I wrote a blog post about error handling in Storm Trident topologies, here
> is the link in case any of you is interested:
>
>
> http://svendvanderveken.wordpress.com/2014/02/05/error-handling-in-storm-trident-topologies/
>
>
> Any comment welcome :D
Kafka version 0.7, and storm-kafka-0.8-plus
in case you use Kafka 0.8.
Best regards,
Danijel Schiavuzzi
www.schiavuzzi.com
On Wed, Feb 12, 2014 at 8:42 AM, Chitra Raveendran <
chitra.raveend...@flutura.com> wrote:
> Hi
>
> I have a topology in production which uses the default
st verify if the value is optimal for
the topology? Also, what approach to setting the "Trident emit batch
interval millis" value would you recommend?
Thanks,
--
Danijel Schiavuzzi
E: dani...@schiavuzzi.com
W: www.schiavuzzi.com
T: +385989035562
Skype: danijels7
change the JDBC database driver used, which should be very
simple to modify accordingly.
Best regards,
Danijel Schiavuzzi
www.schiavuzzi.com
On Thursday, January 16, 2014, Huang, Roger wrote:
> Anyone know of a Trident state implementation for Oracle RDBMS?
>
> I did a quick search on G
whenever a
new bolt instance is started on the cluster.
If you need a way to access relational databases from the Trident API, take
a look at storm-mysql or trident-mssql projects on Github.
Best regards,
Danijel Schiavuzzi
www.schiavuzzi.com
On Wed, Jan 15, 2014 at 5:38 PM, Huang, Roger wr
y
feasible? Any feedback is appreciated.
Thanks,
--
Danijel Schiavuzzi
E: dani...@schiavuzzi.com
W: www.schiavuzzi.com
T: +385989035562
Skype: danijel.schiavuzzi
77 matches
Mail list logo
