Workman
Sent: 15-08-2014 07:53
To: user@storm.incubator.apache.org
Subject: Re: Kafka + Storm
I suppose not directly. It depends on the lifetime of your Kafka queues and on
your latency requirements. You need to make sure you have enough "doctors" or
in storm language workers, in
> load on Storm cluster.
> --
> From: Justin Workman
> Sent: 15-08-2014 07:53
> To: user@storm.incubator.apache.org
> Subject: Re: Kafka + Storm
>
> I suppose not directly. It depends on the lifetime of your Kafka queues
> and on your latency requirements. You need t
Workman
Sent: 15-08-2014 07:53
To: user@storm.incubator.apache.org
Subject: Re: Kafka + Storm
I suppose not directly. It depends on the lifetime of your Kafka queues
and on your latency requirements. You need to make sure you have enough
"doctors" or in storm language workers, in your sto
g"
Subject: Re: Kafka + Storm
I suppose not directly. It depends on the lifetime of your Kafka queues and on
your latency requirements. You need to make sure you have enough "doctors" or
in storm language workers, in your storm cluster to process your messages
within your SLA.
s?
Adaryl "Bob" Wakefield, MBA
Principal
Mass Street Analytics
913.938.6685
www.linkedin.com/in/bobwakefieldmba
Twitter: @BobLovesData
*From:* Justin Workman
*Sent:* Thursday, August 14, 2014 7:47 PM
*To:* user@storm.incubator.apache.org
*Subject:* Re: Kafka + Storm
If you are familiar
Also, since Kafka acts as a buffer, storm is not directly affected by the speed
of your data sources/producers.
-Original Message-
From: "Justin Workman"
Sent: 15-08-2014 07:12
To: "user@storm.incubator.apache.org"
Subject: Re: Kafka + Storm
Good analogy!
Sen
Wakefield, MBA
Principal
Mass Street Analytics
913.938.6685
www.linkedin.com/in/bobwakefieldmba
Twitter: @BobLovesData
*From:* Justin Workman
*Sent:* Thursday, August 14, 2014 7:47 PM
*To:* user@storm.incubator.apache.org
*Subject:* Re: Kafka + Storm
If you are familiar with Weblogic or ActiveM
14, 2014 7:47 PM
To: user@storm.incubator.apache.org
Subject: Re: Kafka + Storm
If you are familiar with Weblogic or ActiveMQ, it is similar. Let's see if I
can explain, I am definitely not a subject matter expert on this.
Within Kafka you can create "queues", ie a webclicks que
on’t have a hard software engineering
background. I know that Kafka is “a message queuing” system, but I don’t
really know what that means.
(I can’t believe you wrote all that from your iPhone)
B.
*From:* Justin Workman
*Sent:* Thursday, August 14, 2014 7:22 PM
*To:* user@storm.incubator.ap
)
B.
From: Justin Workman
Sent: Thursday, August 14, 2014 7:22 PM
To: user@storm.incubator.apache.org
Subject: Re: Kafka + Storm
Personally, we looked at several options, including writing our own storm
source. There are limited storm sources with community support out there. For
us, it
Personally, we looked at several options, including writing our own storm
source. There are limited storm sources with community support out there.
For us, it boiled down to the following;
1) community support and what appeared to be a standard method. Storm has
now included the kafka source as a
Can someone tell me why people put Kafka in front of Storm? Can’t Storm ingest
messages without having Kafka in the middle?
B.
Hello,
We are reading tuples from kafka using storm and putting into HBase.
but due to some performance issue we want to replace *HBase* with *Parquet
file format*.
Is it possible to write tuples read from kafka to the *HDFS (Parquet file
format with partitions)* using storm ?
Thanks in advance.
ith Kafka + Storm +
Trident + OpaqueTridentKafkaSpout
Mentioned below are our setup details :
Storm Topology :
Broker broker = Broker.fromString("localhost:9092")
GlobalPartitionInformation info = new GlobalPartitionInformation()
if(args[4]){
int partitionCount = args[4].toInteger()
We are seeing some performance issues with Kafka + Storm + Trident +
OpaqueTridentKafkaSpout
Mentioned below are our setup details :
Storm Topology :
Broker broker = Broker.fromString("localhost:9092")
GlobalPartitionInformation info = new GlobalPartitionInformation()
FWIW, setting the number of ackers to number of workers gave us a an order
of magnitude gains in latency on our small ec2 test cluster. Our next step
is to do simple microbatching/trident and see how that impacts latency
On Sat, Jun 14, 2014 at 8:28 AM, Haralds Ulmanis
wrote:
> And what about c
der trying the build setup (dependencies as well as
> excludes) in kafka-storm-starter until Storm 0.9.2 -- which includes
> storm-kafka 0.9.2 -- is released.
>
> https://github.com/miguno/kafka-storm-starter/blob/develop/build.sbt
>
> Here I am using a re-packaged version of wurstme
You may consider trying the build setup (dependencies as well as
excludes) in kafka-storm-starter until Storm 0.9.2 -- which includes
storm-kafka 0.9.2 -- is released.
https://github.com/miguno/kafka-storm-starter/blob/develop/build.sbt
Here I am using a re-packaged version of wurstmeister&#
And what about cpu/network/disk utilization ? And load factors per bolt
from storm UI ?
On 14 June 2014 15:53, Shaikh Riyaz wrote:
> Hi,
>
> Daily we are downloaded 28 Million of messages and Monthly it goes up to
> 800+ million.
>
> We want to process this amount of data through our kafka and
Hi,
Daily we are downloaded 28 Million of messages and Monthly it goes up to
800+ million.
We want to process this amount of data through our kafka and storm cluster
and would like to store in HBase cluster.
We are targeting to process one month of data in one day. Is it possible?
We have setup
I am still stuck on this issue. Can someone please help in upgrading the
storm cluster 0.9.0.1 to use curator-framework 1.3.3 and zookeeper 3.4.5
Regards
Anis
On Wed, Jun 11, 2014 at 11:38 PM, Anis Nasir wrote:
> Regarding, 0.9.2-incubating-SNAPSHOT, it comes with the interface
> ILocalCluster
Regarding, 0.9.2-incubating-SNAPSHOT, it comes with the interface
ILocalCluster. I can not find any example using it.
Regards
Anis
On Wed, Jun 11, 2014 at 11:14 PM, Anis Nasir wrote:
> Dear Jonas,
>
> I am getting the same exception running this build
>
> https://github.com/wurstmeister/storm-
Dear Jonas,
I am getting the same exception running this build
https://github.com/wurstmeister/storm-kafka-0.8-plus-test
Regards
Anis
On Wed, Jun 11, 2014 at 11:07 PM, János Háber
wrote:
> Anis: no, you using snapshot version of storm-kafka plugin plus the old
> storm kafka plugin with sto
Anis: no, you using snapshot version of storm-kafka plugin plus the old
storm kafka plugin with storm 0.9.1...
I think storm-kafka 0.9.2-incubating-SNAPSHOT only compatible with
storm-core 0.9.2-incubating-SNAPSHOT
In the 0.9.2-incubating-SNAPSHOT old curator framework (1.0.1) will be
changed to m
Dear Michael,
I am using storm0.9.0.1 with kafka 0.8.1.1 and below mentioned are the
dependencies. I tried with both storm-kafka 0.9.2-incubating-SNAPSHOT
and storm-kafka-0.8-plus
0.4.0. The zookeeper version is 3.3.6.
org.apache.storm
storm-core
0.9.1-incubating
Can you share your build depencies, notably the versions of Storm and
storm-kafka and their respective artifactId and groupId? That is, the
snippets relevant to storm-core and storm-kafka from your pom.xml,
build.gradle, build.sbt, or similar build configuration file. Also,
please add from which
member right, I launched some mvn target from storm-dist/binary directory.
>
>
>
>
> On 11 June 2014 21:18, János Háber wrote:
>
>> Yeah... this is one of the biggest problem of about kafka <-> storm
>> integration (second is the kafka-storm module scala depende
storm incubation project and build everything by
myself, with slightly modified dependencies. I can be wrong, but If I
remember right, I launched some mvn target from storm-dist/binary directory.
On 11 June 2014 21:18, János Háber wrote:
> Yeah... this is one of the biggest problem of about kafka &
Yeah... this is one of the biggest problem of about kafka <-> storm
integration (second is the kafka-storm module scala dependency version, but
I explain this in another topic), and this is why I plan move to Spark...
ok it's only near real time, but more stable than Storm (and native
Looks like a curator version mismatch...
--
Derek
On 6/11/14, 14:52, Anis Nasir wrote:
Dear all,
I am getting below mentioned exception while running storm-kafka. Does
anyone have seen this one before?
7800 [Thread-27-words] ERROR backtype.storm.util - Async loop died!
java.lang.NoSuchMetho
Dear all,
I am getting below mentioned exception while running storm-kafka. Does
anyone have seen this one before?
7800 [Thread-27-words] ERROR backtype.storm.util - Async loop died!
java.lang.NoSuchMethodError: com.netflix.curator.framework.
api.CreateBuilder.creatingParentsIfNeeded()Lcom/
net
where it's coming from, a solution is to just exclude jline from
the kafka/storm-kafka dependency in your pom.xml. ex:
org.apache.kafka
kafka_2.9.2
0.8.1
jline
jline
You can also check your dependency tree and look for other references to
other scala ver
I am using kafka with storm. I am using maven to build my topology and I am
using scala 2.9.2 same as I am using kafka_2.9.2_0.8.1.
Topology build perfectly using maven. But hwn I submit the topology to
storm I get the following Exception:
java.lang.NoSuchMethodError: scala.Predef$.int2Integer(I)
Have you tried increasing the tuple timeout? The default of 30 seconds may
not suit you.
On Thursday, June 5, 2014, Romain Leroux wrote:
> Has anyone ever faced similar or related issue ?
>
>
> 2014-06-03 19:33 GMT+09:00 Romain Leroux >:
>
>> I have a simple trident transactional topology that
Has anyone ever faced similar or related issue ?
2014-06-03 19:33 GMT+09:00 Romain Leroux :
> I have a simple trident transactional topology that does something like
> the following:
>
> kafka transactional spout (~3000 rec/sec, 6 partitions thus paraHint=6)
> -->
> aggregate with reducerAggrega
I have a simple trident transactional topology that does something like the
following:
kafka transactional spout (~3000 rec/sec, 6 partitions thus paraHint=6) -->
aggregate with reducerAggregator (paraHint=20) -->
transactional state (I tried MemoryMapState, MemcachedMapState and
CassandraMapState
uilding a Trident topology, not the plain Storm KafkaSpout one.
>
>
> On Tuesday, May 27, 2014, Romain Leroux wrote:
>
>> Hi,
>>
>> First of all thanks to @miguno for his amazing work on
>> kafka-storm-starter.
>>
>> I am trying to add a memcached state
I believe you should be using a Trident Kafka spout variant if you're
building a Trident topology, not the plain Storm KafkaSpout one.
On Tuesday, May 27, 2014, Romain Leroux wrote:
> Hi,
>
> First of all thanks to @miguno for his amazing work on kafka-storm-starter.
>
>
Hi,
First of all thanks to @miguno for his amazing work on kafka-storm-starter.
I am trying to add a memcached state to it based on :
https://github.com/nathanmarz/trident-memcached
More particularly I'd like to test the full stack:
Kafka->Storm->TransactionalState(Memcached) with Tr
Hi everyone,
to sweeten the upcoming long weekend I have released code examples that
show how to integrate Kafka 0.8+ with Storm 0.9+, while using Apache
Avro as the data serialization format.
https://github.com/miguno/kafka-storm-starter
Since the integration of the latest Kafka and Storm
Hi all,
What are the best practices, for building a multi-tenant app in the context
of Kafka and storm?
For example: creating topic for each tenant and consume multi-topics spout
(using wildcard).
Thanks in advance,
Amikam.
] -
From: Kiran Nagasubramanian
Reply-To: "user@storm.incubator.apache.org"
Date: Saturday, April 19, 2014 at 10:37 PM
To: "user@storm.incubator.apache.org"
Subject: Re: kafka/storm integration help needed
This may be of some help to you
https://github.com/wurstmei
This may be of some help to you
https://github.com/wurstmeister/storm-kafka-0.8-plus
Kiran
On Sat, Apr 19, 2014 at 8:11 PM, Scott C. Cote wrote:
> Is storm-kafka integration still supported?
>
> Am trying to setup a dev environment that encompasses storm + kafka
>
> To that goal, I have:
>
>
m: "Scott C. Cote"
Date: Saturday, April 19, 2014 at 10:11 PM
To: "user@storm.incubator.apache.org"
Subject: kafka/storm integration help needed
Is storm-kafka integration still supported?
Am trying to setup a dev environment that encompasses storm + kafka
To that goal, I
Is storm-kafka integration still supported?
Am trying to setup a dev environment that encompasses storm + kafka
To that goal, I have:
1. kafka 2.9.2-0.8.1 installed on my machine and independently running (all
local host).
2. Successfully configured via log4j a simple java app to write into kaf
Hi,
I'm trying to work Kafka+Storm in distributed mode.
However, it doesn't receive any tuple from Spout when I tried by multiple
supervisors though it worked when I tried by single supervisor as same code.
In addition, I couldn't find any error logs in Nimbus log, Supervisor logs
Hi Mark,
I couldn't view the gists. Well, I am also working on the same
kafka-storm integration. It seems kafkaspout (
https://github.com/nathanmarz/storm-contrib/tree/master/storm-kafka) has
some issues. Instead of using kafkaspout which is readily available, I
included kafka consumer co
Hi All,
I have followed the storm setup guides and have storm working talking to
zookeeper. I also have a fully working local kafka setup with messages
being sent to it. Ideally now I need to link the too but seem to be stuck
finding a simple clojure example.
I have managed to get the storm-start
48 matches
Mail list logo
