Use case

2014-09-04 Thread Aris Alexis
Hello, I am building a big web application that I want to be massively scalable (I am using cassandra and titan as a general db). I want to implement the following: real time web chat that is persisted so that user a in the future can recall his chat with user b,c,d much like facebook. mail like

Re: Use case

2014-09-04 Thread cac...@gmail.com
I believe there are architectures for the chat system that can use Kafka in a sensible fashion to achieve certain of the difficult aspects. However doing partition per user would not be advisable, nor I imagine would relying on Kafka's storage for checking for past or expired messages. (I've done t

Re: Use case

2014-09-04 Thread Sharninder
Since you want all chats and mail history persisted all the time, I personally wouldn't recommend kafka for your requirement. Kafka is more suitable as a streaming system where events expire after a certain time. Look at something more general purpose like hbase for persisting data indefinitely. S

Re: Use case

2014-09-04 Thread Philip O'Toole
Agreed. I can't see this being a good use for Kafka. Philip - http://www.philipotoole.com On Thursday, September 4, 2014 9:57 PM, Sharninder wrote: Since you want all chats and mail history persisted all the time, I personally wouldn't recommend k

Re: Use case

2014-09-05 Thread Aris Alexis
Thanks for the reply. If I use it only for activity streams like twitter: I would want a topic for each #tag and a topic for each user and maybe foreach city. Would that be too many topics or it doesn't matter since most of them will be deleted in a specified interval. Best Regards, Aris Giachn

Re: Use case

2014-09-05 Thread Aris Alexis
Thanks for the reply. If I use it only for activity streams like twitter: I would want a topic for each #tag and a topic for each user and maybe foreach city. Would that be too many topics or it doesn't matter since most of them will be deleted in a specified interval. Best Regards, Aris Giachn

Re: Use case

2014-09-05 Thread Sharninder
I'm not really sure about your exact use-case but I don't think having a topic per user is very efficient. Deleting topics in kafka, at the moment, isn't really straightforward. You should rethink your date pipeline a bit. Also, just because kafka has the ability to store message

Re: Use case

2014-09-05 Thread Philip O'Toole
4:21 AM, Sharninder wrote: I'm not really sure about your exact use-case but I don't think having a topic per user is very efficient. Deleting topics in kafka, at the moment, isn't really straightforward. You should rethink your date pipeline a bit. Also, just because kafka ha

Re: Use case

2014-09-05 Thread Gwen Shapira
ilipotoole.com > > > On Friday, September 5, 2014 4:21 AM, Sharninder wrote: > > > > I'm not really sure about your exact use-case but I don't think having a > topic per user is very efficient. Deleting topics in kafka, at the moment, > isn't really straightfo

Re: Use case

2014-09-05 Thread Jonathan Weeks
use-case outlined might be more tenable. Best Regards, -Jonathan On Sep 5, 2014, at 4:20 AM, Sharninder wrote: > I'm not really sure about your exact use-case but I don't think having a > topic per user is very efficient. Deleting topics in kafka, at the moment, > isn'

Re: Use case

2014-09-05 Thread Christian Csar
rsion of kafka. > > There are also secondary impacts on topic count — e.g. useful operational > tools such as: http://quantifind.com/KafkaOffsetMonitor/ start to become > problematic in terms of UX with a massive number of topics. > > Once topic deletion is a supported feature, the

Re: Use case

2014-09-05 Thread Aris Alexis
ve partitions extend downtime in the case of a > failure; this means fewer topics (e.g. hundreds or thousands) is a best > practice in the published version of kafka. > > > > There are also secondary impacts on topic count — e.g. useful > operational tools such as: http://quantifind.

Kafka use case

2018-09-26 Thread Shibi Ns
I have 2 systems 1. System I - A web based interface based on Oracle DB and No REST API support 2. System II - Supports rest API's which also has web based interface . When a record created or updated in either of the system I want propagate the data to other system . Ca

Kafka use case query

2015-07-10 Thread lrnop
Hello, We have a number of different scenarios within our company that we are considering Kafka for. There is one case in particular that has caused debate. The relevant characteristics are: - Very high throughput - 1000's of messages/second. - Very bursty traffic with unpredictable su

A specific use case

2016-08-01 Thread Hamza HACHANI
Good morning, I'm working on a specific use case. In fact i'm receiving messages from an operator network and trying to do statistics on their number per minute,perhour,per day ... I would like to create a broker that receives the messages and generates a message every minute. These

High Level Consumer use case

2015-07-22 Thread Sreenivasulu NALLAPATI
Hello, I would like to introduce myself first. I am Sreeni working as a technical lead with Amadeus. We planned to use Kafka for our billing messages. Here is my use case: We wanted to run a batch job every 15min with High Level Consumer by changing auto.commit.enable=false. For now, we have a

How to implement use case

2017-04-27 Thread Vladimir Lalovic
y are good choice for this use case but I would like to confirm that with someone from Kafka team or to get some recommendations ... Thanks, Vladimir

Re: A specific use case

2016-08-03 Thread Guozhang Wang
Hello Hamza, By saying "broker" I think you are actually referring to a Kafka Streams instance? Guozhang On Mon, Aug 1, 2016 at 1:01 AM, Hamza HACHANI wrote: > Good morning, > > I'm working on a specific use case. In fact i'm receiving messages from an > o

Re : A specific use case

2016-08-03 Thread Hamza HACHANI
Hi, Yes in fact . And ï found à solution. It was in editing the method punctuate in kafka stream processor. - Message de réponse - De : "Guozhang Wang" Pour : "users@kafka.apache.org" Objet : A specific use case Date : mer., août 3, 2016 23:38 Hello Hamza, By say

Apache Kafka Use Case at WalmartLabs

2014-03-07 Thread Bhavesh Mistry
We are planning to use Apache Kafka to replace Apache Fume for mostly as log transport layer. Please see the attached image which is similar use case ( and deployment architecture ) at Linkedin (according to http://sites.computer.org/debull/A12june/pipeline.pdf ). I have following questions

Re: High Level Consumer use case

2015-07-22 Thread Stevo Slavić
I would like to introduce myself first. I am Sreeni working as a technical > lead with Amadeus. We planned to use Kafka for our billing messages. > > > > Here is my use case: > > We wanted to run a batch job every 15min with High Level Consumer by > changing *auto.commit.en

Flume use case for Kafka & HDFS

2015-09-19 Thread Hemanth Abbina
I'm new to Flume and thinking to use Flume in the below scenario. Our system receives events as HTTP POST, and we need to store them in Kafka(for processing) as well as HDFS(as permanent store). Can we configure Flume as below ? * Source: HTTP (expecting JSON event as HTTP body, with a

KafkaStreams for counting/analytics use case

2016-03-19 Thread I PVP
Hi everyone, Is KafkaStream the way to go to allow a light real time counting/analytics that need to do things like ? : 1) Continuously update the total count of currently open purchase orders for each user as purchase orders updates are passing thought the broker. 2) Continuously update the

Re: How to implement use case

2017-04-27 Thread Steven Schlansker
eck if there are any reservations where remaining time before > pickup is less than x). > > Number of reservations we currently fetching is ~5000 and number of > notification/alerting rules is ~20 > > > Based on documentation and some blog posts I have impression that Kafka a

Re: Re : A specific use case

2016-08-04 Thread Guozhang Wang
ang Wang" > Pour : "users@kafka.apache.org" > Objet : A specific use case > Date : mer., août 3, 2016 23:38 > > Hello Hamza, > > By saying "broker" I think you are actually referring to a Kafka Streams > instance? > > > Guozhang > > On Mon, Au

RE: Re : A specific use case

2016-08-05 Thread Hamza HACHANI
Thanks Guozhang Wang. Hamza De : Guozhang Wang Envoyé : jeudi 4 août 2016 06:58:22 À : users@kafka.apache.org Objet : Re: Re : A specific use case Yeah, if you can buffer yourself in the process() function and then rely on punctuate() for generating the

Right Use Case For Kafka Streams?

2021-03-15 Thread Gareth Collins
Hi, We have a requirement to calculate metrics on a huge number of keys (could be hundreds of millions, perhaps billions of keys - attempting caching on individual keys in many cases will have almost a 0% cache hit rate). Is Kafka Streams with RocksDB and compacting topics the right tool for a tas

Kafka use case for my setup

2022-06-08 Thread Jay Linux
Hi there, I would like to automate some of my tasks using Apache Kafka. Previously i used to do the same using Apache Airflow and which worked fine. But i want to explore the same using Kafka whether this works better than Airflow or not. 1) Kafka runs on Server A 2) Kafka searches for a file na

Kafka use case for my setup

2022-06-08 Thread Jay Linux
Hi there, I would like to automate some of my tasks using Apache Kafka. Previously i used to do the same using Apache Airflow and which worked fine. But i want to explore the same using Kafka whether this works better than Airflow or not. 1) Kafka runs on Server A 2) Kafka searches for a file na

Re: Apache Kafka Use Case at WalmartLabs

2014-03-07 Thread Guozhang Wang
consumers. 5) Thanks! You can start by searching for JIRAs tagged with "newbie". Guozhang On Fri, Mar 7, 2014 at 11:39 AM, Bhavesh Mistry wrote: > We are planning to use Apache Kafka to replace Apache Fume for mostly as > log transport layer. Please see the attached image whic

Re: Apache Kafka Use Case at WalmartLabs

2014-03-07 Thread Neha Narkhede
ged with "newbie". > > Guozhang > > > On Fri, Mar 7, 2014 at 11:39 AM, Bhavesh Mistry > wrote: > > > We are planning to use Apache Kafka to replace Apache Fume for mostly as > > log transport layer. Please see the attached image which is similar use

Re: Flume use case for Kafka & HDFS

2015-09-24 Thread Tim Smith
Not out of the box, no - I don't think you can use an attribute of the posted JSON to specify topics for kafka or folder for HDFS. For dynamically creating topics in kafka, you would have to write some kind of custom kafka producer - the kafka channel or sink in flume requires a kafka topic to be

Re: KafkaStreams for counting/analytics use case

2016-03-21 Thread Guozhang Wang
Hello, >From your described use cases I think Kafka Streams would be a good fit, in that 1) it provides higher level DSLs for windowed aggregations, and 2) it is part of the Kafka open source (coming in the 0.10.0 release, which is being voted now), and if you have your data already in Kafka it is

Architecture recommendations for a tricky use case

2016-09-29 Thread Ali Akhtar
I have a somewhat tricky use case, and I'm looking for ideas. I have 5-6 Kafka producers, reading various APIs, and writing their raw data into Kafka. I need to: - Do ETL on the data, and standardize it. - Store the standardized data somewhere (HBase / Cassandra / Raw HDFS / ElasticS

Configuration guidelines for a specific use-case

2019-01-08 Thread Gioacchino Vino
Hi expert, I would ask you some guidelines, web-pages or comments regarding my use-case. *Requirements*: - 2000+ producers - input rate 600k messages/s - consumers must write in 3 different databases (so i assume 3 consumer groups) at 600k messages/s overall (200k messages/s/database

Re: Right Use Case For Kafka Streams?

2021-03-17 Thread Guozhang Wang
Hello Gareth, A common practice for rolling up aggregations with Kafka Streams is to do the finest granularity at processor (5 days in your case), and to coarse-grained rolling up upon query serving through the interactive query API -- i.e. whenever a query is issued for a 30 day aggregate you do

Re: Right Use Case For Kafka Streams?

2021-03-19 Thread Ning Zhang
just my 2 cents the best answer is always from the real-world practices :) RocksDB https://rocksdb.org/ is the implementation of "state store" in Kafka Stream and it is an "embedded" kv store (which is diff than the distributed kv store). The "state store" in Kafka Stream is also backed up by "

use case with high rate of duplicate messages

2013-10-01 Thread S Ahmed
I have a use case where thousands of servers send status type messages, which I am currently handling real-time w/o any kind of queueing system. So currently when I receive a message, and perform a md5 hash of the message, perform a lookup in my database to see if this is a duplicate, if not, I

Re: Architecture recommendations for a tricky use case

2016-09-29 Thread Ali Akhtar
Ali Akhtar" wrote: > >> I have a somewhat tricky use case, and I'm looking for ideas. >> >> I have 5-6 Kafka producers, reading various APIs, and writing their raw >> data into Kafka. >> >> I need to: >> >> - Do ETL on the data, and

Re: Architecture recommendations for a tricky use case

2016-09-29 Thread Mich Talebzadeh
es. > > On Thu, Sep 29, 2016 at 7:00 PM, Deepak Sharma > wrote: > >> What is the message inflow ? >> If it's really high , definitely spark will be of great use . >> >> Thanks >> Deepak >> >> On Sep 29, 2016 19:24, "Ali Akhtar" wr

Re: Architecture recommendations for a tricky use case

2016-09-29 Thread Deepak Sharma
What is the message inflow ? If it's really high , definitely spark will be of great use . Thanks Deepak On Sep 29, 2016 19:24, "Ali Akhtar" wrote: > I have a somewhat tricky use case, and I'm looking for ideas. > > I have 5-6 Kafka producers, reading various APIs,

Re: Architecture recommendations for a tricky use case

2016-09-29 Thread Ali Akhtar
29 September 2016 at 15:01, Ali Akhtar wrote: > >> It needs to be able to scale to a very large amount of data, yes. >> >> On Thu, Sep 29, 2016 at 7:00 PM, Deepak Sharma >> wrote: >> >>> What is the message inflow ? >>> If it's really hig

RE: Architecture recommendations for a tricky use case

2016-09-29 Thread Tauzell, Dave
recommendations for a tricky use case The web UI is actually the speed layer, it needs to be able to query the data online, and show the results in real-time. It also needs a custom front-end, so a system like Tableau can't be used, it must have a custom backend + front-end. Thanks for the recommend

Re: Architecture recommendations for a tricky use case

2016-09-29 Thread Deepak Sharma
ction of data or any other property which may >>> arise from relying on this email's technical content is explicitly >>> disclaimed. The author will in no case be liable for any monetary damages >>> arising from such loss, damage or destruction. >>> >>> >

Re: Architecture recommendations for a tricky use case

2016-09-29 Thread Mich Talebzadeh
ing on this email's technical content is explicitly >> disclaimed. The author will in no case be liable for any monetary damages >> arising from such loss, damage or destruction. >> >> >> >> On 29 September 2016 at 15:01, Ali Akhtar wrote: >> >>&g

Re: Architecture recommendations for a tricky use case

2016-09-29 Thread Ali Akhtar
;> any loss, damage or destruction of data or any other property which may >>> arise from relying on this email's technical content is explicitly >>> disclaimed. The author will in no case be liable for any monetary damages >>> arising from such loss, damage or destructi

Re: Architecture recommendations for a tricky use case

2016-09-29 Thread Deepak Sharma
>> any loss, damage or destruction of data or any other property which may >>> arise from relying on this email's technical content is explicitly >>> disclaimed. The author will in no case be liable for any monetary damages >>> arising from such loss, damage or destructi

Re: Architecture recommendations for a tricky use case

2016-09-29 Thread Deepak Sharma
CCdOABUrV8Pw >>>> <https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* >>>> >>>> >>>> >>>> http://talebzadehmich.wordpress.com >>>> >>>> >>>> *Disclaimer:* Use it at your own risk

Re: Architecture recommendations for a tricky use case

2016-09-29 Thread Cody Koeninger
;>>> Zeppelin to query data >>>>>> >>>>>> You will also need spark streaming to query data online for speed >>>>>> layer. That data could be stored in some transient fabric like ignite or >>>>>> even druid.

Re: Architecture recommendations for a tricky use case

2016-09-29 Thread Ali Akhtar
;>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> Dr Mich Talebzadeh >>>>> >>>>> >>>>> >>>>> LinkedIn * >>>>> https:/

Re: Architecture recommendations for a tricky use case

2016-09-29 Thread Deepak Sharma
Hi Cody Spark direct stream is just fine for this use case. But why postgres and not cassandra? Is there anything specific here that i may not be aware? Thanks Deepak On Thu, Sep 29, 2016 at 8:41 PM, Cody Koeninger wrote: > How are you going to handle etl failures? Do you care about l

Re: Architecture recommendations for a tricky use case

2016-09-29 Thread Cody Koeninger
I wouldn't give up the flexibility and maturity of a relational database, unless you have a very specific use case. I'm not trashing cassandra, I've used cassandra, but if all I know is that you're doing analytics, I wouldn't want to give up the ability to easily do ad-h

Re: Architecture recommendations for a tricky use case

2016-09-29 Thread Ali Akhtar
. What's the advantage of using Spark for reading Kafka instead of direct Kafka consumers? On Thu, Sep 29, 2016 at 8:28 PM, Cody Koeninger wrote: > I wouldn't give up the flexibility and maturity of a relational > database, unless you have a very specific use case. I'm not

Re: Architecture recommendations for a tricky use case

2016-09-29 Thread Mich Talebzadeh
as well and from Flume to Hbase I would have thought that if one wanted to do real time analytics with SS, then that would be a good fit with a real time dashboard. What is not so clear is the business use case for this. HTH Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/prof

Re: Architecture recommendations for a tricky use case

2016-09-29 Thread Cody Koeninger
t;> >> I wouldn't give up the flexibility and maturity of a relational >> database, unless you have a very specific use case. I'm not trashing >> cassandra, I've used cassandra, but if all I know is that you're doing >> analytics, I wouldn't w

Re: Architecture recommendations for a tricky use case

2016-09-29 Thread Deepak Sharma
Thu, Sep 29, 2016 at 8:28 PM, Cody Koeninger > wrote: > >> I wouldn't give up the flexibility and maturity of a relational >> database, unless you have a very specific use case. I'm not trashing >> cassandra, I've used cassandra, but if all I know is that yo

Re: Architecture recommendations for a tricky use case

2016-09-29 Thread Mich Talebzadeh
Hi Ali, What is the business use case for this? Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw <https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* http://talebzadehmich.wordpre

Re: Architecture recommendations for a tricky use case

2016-09-29 Thread Cody Koeninger
lt tolerant as >> possible. >> >> What's the advantage of using Spark for reading Kafka instead of direct >> Kafka consumers? >> >> On Thu, Sep 29, 2016 at 8:28 PM, Cody Koeninger >> wrote: >>> >>> I wouldn't give up the flexibility an

Re: Architecture recommendations for a tricky use case

2016-09-29 Thread Ali Akhtar
The business use case is to read a user's data from a variety of different services through their API, and then allowing the user to query that data, on a per service basis, as well as an aggregate across all services. The way I'm considering doing it, is to do some basic ETL (dr

Re: Architecture recommendations for a tricky use case

2016-09-29 Thread Mich Talebzadeh
ember 2016 at 16:49, Ali Akhtar wrote: > The business use case is to read a user's data from a variety of different > services through their API, and then allowing the user to query that data, > on a per service basis, as well as an aggregate across all services. > > The way I&#

Re: Architecture recommendations for a tricky use case

2016-09-29 Thread Cody Koeninger
nt and skill level. > > HTH > > -Mike > >> On Sep 29, 2016, at 8:54 AM, Ali Akhtar wrote: >> >> I have a somewhat tricky use case, and I'm looking for ideas. >> >> I have 5-6 Kafka producers, reading various APIs, and writing their raw data &

Re: Architecture recommendations for a tricky use case

2016-09-29 Thread Cody Koeninger
ther property which may arise > from relying on this email's technical content is explicitly disclaimed. The > author will in no case be liable for any monetary damages arising from such > loss, damage or destruction. > > > > > On 29 September 2016 at 16:49, Ali Akhta

Re: Architecture recommendations for a tricky use case

2016-09-29 Thread Gwen Shapira
.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw >> >> >> >> http://talebzadehmich.wordpress.com >> >> >> Disclaimer: Use it at your own risk. Any and all responsibility for any >> loss, damage or destruction of data or any other

Re: Architecture recommendations for a tricky use case

2016-09-29 Thread Cody Koeninger
;>> you want to write your own compaction code? Or use Hive 1.x+?) >>> >>> HBase? Depending on your admin… stability could be a problem. >>> Cassandra? That would be a separate cluster and that in itself could be a >>> problem… >>> >>>

Re: Architecture recommendations for a tricky use case

2016-09-29 Thread Avi Flax
> On Sep 29, 2016, at 09:54, Ali Akhtar wrote: > > I'd appreciate some thoughts / suggestions on which of these alternatives I > should go with (e.g, using raw Kafka consumers vs Spark for ETL, which > persistent data store to use, and how to query that data store in the > backend of the web UI,

Re: Architecture recommendations for a tricky use case

2016-09-29 Thread Ali Akhtar
Avi, Why did you choose Druid over Postgres / Cassandra / Elasticsearch? On Fri, Sep 30, 2016 at 1:09 AM, Avi Flax wrote: > > > On Sep 29, 2016, at 09:54, Ali Akhtar wrote: > > > > I'd appreciate some thoughts / suggestions on which of these > alternatives I > > should go with (e.g, using raw

Re: Architecture recommendations for a tricky use case

2016-09-29 Thread Michael Segel
e? Or use Hive 1.x+?) >>>> >>>> HBase? Depending on your admin… stability could be a problem. >>>> Cassandra? That would be a separate cluster and that in itself could be a >>>> problem… >>>> >>>> YMMV so you need to

Re: Architecture recommendations for a tricky use case

2016-09-29 Thread Michael Segel
>> HBase? Depending on your admin… stability could be a problem. >> Cassandra? That would be a separate cluster and that in itself could be a >> problem… >> >> YMMV so you need to address the pros/cons of each tool specific to your >> environment and skill level

Re: Architecture recommendations for a tricky use case

2016-09-29 Thread Michael Segel
. Cassandra? That would be a separate cluster and that in itself could be a problem… YMMV so you need to address the pros/cons of each tool specific to your environment and skill level. HTH -Mike > On Sep 29, 2016, at 8:54 AM, Ali Akhtar wrote: > > I have a somewhat tricky use case

Re: Architecture recommendations for a tricky use case

2016-09-29 Thread Alonso Isidoro Roman
y >> disclaimed. The author will in no case be liable for any monetary damages >> arising from such loss, damage or destruction. >> >> >> >> On 29 September 2016 at 15:01, Ali Akhtar wrote: >> >>> It needs to be able to scale to a very large amount of data,

Re: Architecture recommendations for a tricky use case

2016-09-30 Thread Andrew Stevenson
>>>> even druid. >>>>>> >>>>>> HTH >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >

Re: Architecture recommendations for a tricky use case

2016-10-05 Thread Avi Flax
> On Sep 29, 2016, at 16:39, Ali Akhtar wrote: > > Why did you choose Druid over Postgres / Cassandra / Elasticsearch? Well, to be clear, we haven’t chosen it yet — we’re evaluating it. That said, it is looking quite promising for our use case. The Druid docs say it well: > Drui

[KIP-94] SessionWindows - IndexOutOfBoundsException in simple use case

2017-02-22 Thread Marco Abitabile
Hello, I apologies with Matthias since I posted yesterday this issue on the wrong place on github :( I'm trying a simple use case of session windowing. TimeWindows works perfectly, however as I replace with SessionWindows, this exception is thrown: Exception in thread "Stre

Re: Configuration guidelines for a specific use-case

2019-01-08 Thread Ryanne Dolan
Latency sounds high to me, maybe your JVMs are GC'ing a lot? Ryanne On Tue, Jan 8, 2019, 11:45 AM Gioacchino Vino Hi expert, > > > I would ask you some guidelines, web-pages or comments regarding my > use-case. > > > *Requirements*: > > - 2000+ producers

Re: Configuration guidelines for a specific use-case

2019-01-09 Thread Gioacchino Vino
I would ask you some guidelines, web-pages or comments regarding my use-case. *Requirements*: - 2000+ producers - input rate 600k messages/s - consumers must write in 3 different databases (so i assume 3 consumer groups) at 600k messages/s overall (200k messages/s/database) - latency < 500m

Re: Configuration guidelines for a specific use-case

2019-01-11 Thread Ryanne Dolan
> > On Tue, Jan 8, 2019, 11:45 AM Gioacchino Vino > wrote: > > > >> Hi expert, > >> > >> > >> I would ask you some guidelines, web-pages or comments regarding my > >> use-case. > >> > >> > >> *Requirements*: > &

Re: use case with high rate of duplicate messages

2013-10-01 Thread Guozhang Wang
with the same key. All you need to do is setting the dedup frequency appropriately for your use case. Guozhang On Tue, Oct 1, 2013 at 8:19 AM, S Ahmed wrote: > I have a use case where thousands of servers send status type messages, > which I am currently handling real-time w/o any k

RE: use case with high rate of duplicate messages

2013-10-01 Thread Sybrandy, Casey
ation mechanism on the brokers that dedup messages with the same key. All you need to do is setting the dedup frequency appropriately for your use case. Guozhang On Tue, Oct 1, 2013 at 8:19 AM, S Ahmed wrote: > I have a use case where thousands of servers send status type > messages,

Re: use case with high rate of duplicate messages

2013-10-01 Thread Neha Narkhede
> > -Original Message- > From: Guozhang Wang [mailto:wangg...@gmail.com] > Sent: Tuesday, October 01, 2013 11:33 AM > To: users@kafka.apache.org > Subject: Re: use case with high rate of duplicate messages > > Batch processing will increase the throughput but also i

Is this a decent use case for Kafka Streams?

2017-07-12 Thread Stephen Powis
Hey! I was hoping I could get some input from people more experienced with Kafka Streams to determine if they'd be a good use case/solution for me. I have multi-tenant clients submitting data to a Kafka topic that they want ETL'd to a third party service. I'd like to batch and

Re: [KIP-94] SessionWindows - IndexOutOfBoundsException in simple use case

2017-02-22 Thread Damian Guy
wrote: > Hello, > > I apologies with Matthias since I posted yesterday this issue on the wrong > place on github :( > > I'm trying a simple use case of session windowing. TimeWindows works > perfectly, however as I replace with SessionWindows, this exception is > thr

Kafka Streams: finding a solution to a particular use case

2016-04-14 Thread Guillermo Lammers Corral
Hi, I am a newbie to Kafka Streams and I am using it trying to solve a particular use case. Let me explain. I have two sources of data both like that: Key (string) DateTime (hourly granularity) Value I need to join the two sources by key and date (hour of day) to obtain: Key (string) DateTime

Kafka Topics best practice for logging data pipeline use case

2017-03-14 Thread Ram Vittal
We are using latest Kafka and Logstash versions for ingesting several business apps logs(now few but eventually 100+) into ELK. We have a standardized logging structure for business apps to log data into Kafka topics and able to ingest into ELK via Kafka topics input plugin. Currently, we are usin

Re: Is this a decent use case for Kafka Streams?

2017-07-13 Thread Eno Thereska
ome input from people more experienced with > Kafka Streams to determine if they'd be a good use case/solution for me. > > I have multi-tenant clients submitting data to a Kafka topic that they want > ETL'd to a third party service. I'd like to batch and group these

Re: Is this a decent use case for Kafka Streams?

2017-07-13 Thread Jon Yeargers
created. On Wed, Jul 12, 2017 at 5:06 PM, Stephen Powis wrote: > Hey! I was hoping I could get some input from people more experienced with > Kafka Streams to determine if they'd be a good use case/solution for me. > > I have multi-tenant clients submitting data to a Kafka to

Is this a valid use case for reading local store ?

2020-09-30 Thread Parthasarathy, Mohan
Hi, A traditional kafka streams application (App1) reading data from a kafka topic, doing aggregations resulting in some local state. The output of this application is consumed by a different application(App2) for doing a different task. Under some conditions, there is an external trigger (asy

Re: Kafka Streams: finding a solution to a particular use case

2016-04-14 Thread Guozhang Wang
14, 2016 at 1:30 AM, Guillermo Lammers Corral < guillermo.lammers.cor...@tecsisa.com> wrote: > Hi, > > I am a newbie to Kafka Streams and I am using it trying to solve a > particular use case. Let me explain. > > I have two sources of data both like that: > > Key (string)

Re: Kafka Streams: finding a solution to a particular use case

2016-04-15 Thread Guillermo Lammers Corral
e streams are really a "changelog" stream, hence you > should create the stream as KTable, and do KTable-KTable join. > > 2) Could elaborate about "achieving this"? What behavior do require in the > application logic? > > > Guozhang > > > On Thu, Apr 14,

Re: Kafka Streams: finding a solution to a particular use case

2016-04-15 Thread Guozhang Wang
ble, and do KTable-KTable join. > > > > 2) Could elaborate about "achieving this"? What behavior do require in > the > > application logic? > > > > > > Guozhang > > > > > > On Thu, Apr 14, 2016 at 1:30 AM, Guillermo Lammers Corral

Re: Kafka Streams: finding a solution to a particular use case

2016-04-19 Thread Guillermo Lammers Corral
why > > is getting all the values from the topic even those with the same key? > > > > Thanks in advance! Great job answering community! > > > > 2016-04-14 20:00 GMT+02:00 Guozhang Wang : > > > > > Hi Guillermo, > > > > > > 1) Yes in your case, the streams a

Re: Kafka Streams: finding a solution to a particular use case

2016-04-19 Thread Henry Cai
pic). > > > > > > Q: On the other hand, just to try, I have a KTable that read messages > in > > > "resultTopic" and prints them. If the stream is a KTable I am wondering > > why > > > is getting all the values from the topic even those

Re: Kafka Streams: finding a solution to a particular use case

2016-04-19 Thread Guozhang Wang
; > > > > > > > Q: On the other hand, just to try, I have a KTable that read messages > > in > > > > "resultTopic" and prints them. If the stream is a KTable I am > wondering > > > why > > > > is getting all the values from the

Re: Kafka Streams: finding a solution to a particular use case

2016-04-19 Thread Henry Cai
of the KTable that does not have the > > > > > corresponding data by key in the other one, obtain null values in > the > > > > > result final topic is the expected behavior? > > > > > > > > > > My next step would be use Kafka Conn

Re: Kafka Streams: finding a solution to a particular use case

2016-04-19 Thread Guozhang Wang
see null values in topic "resultTopic", i.e. ("uniqueKey1", > > null) > > > > > > > > > > > > - If I send data to "topic1", for example a key/value like that > > > > > > ("uniqueKey1", "wor

Re: Kafka Streams: finding a solution to a particular use case

2016-04-19 Thread Henry Cai
- The streming is up & running without data in topics > > > > > > > > > > > > > > - I send data to "topic2", for example a key/value like that > > > > > > ("uniqueKey1", > > > > > > > "hello

Re: Kafka Streams: finding a solution to a particular use case

2016-04-20 Thread Matthias J. Sax
gt;>>>> question, >>>>>>> I'll >>>>>>>> try to explain with some pseudocode. >>>>>>>> >>>>>>>> I have two KTable with a join: >>>>>>>> >>>>>>&

Re: Kafka Streams: finding a solution to a particular use case

2016-04-20 Thread Guillermo Lammers Corral
orral < > >>>>>>> guillermo.lammers.cor...@tecsisa.com> wrote: > >>>>>>> > >>>>>>>> Hi Guozhang, > >>>>>>>> > >>>>>>>> Thank you very much for your reply and sorr

Re: Kafka Streams: finding a solution to a particular use case

2016-04-20 Thread Guozhang Wang
t; > > > > > > > ktable1.join(ktable2, (data1, data2) => new > ResultUnion(data1, > > > > > data2)) > > > > > > > > > > > > > > > > I send the result to a topic result.to("resultTopic"). > &g

Re: Kafka Streams: finding a solution to a particular use case

2016-04-20 Thread Henry Cai
neric > > > > > > question, > > > > > > > > I'll > > > > > > > > > try to explain with some pseudocode. > > > > > > > > > > > > > > > > > > I have two KTable with a join: > > > > > >

Re: Kafka Streams: finding a solution to a particular use case

2016-04-20 Thread Guozhang Wang
> > > > > > > > > > > > > > > > Guozhang > > > > > > > > > > > > > > > > > > On Fri, Apr 15, 2016 at 2:11 AM, Guillermo Lammers Corral < > > > > > > > > > guillermo.lam

  1   2   >