Re: Capturing and storing these Kafka events for query.
Another approach would be to create the query first (in something like KSQL) and then send the Kafka data through the pre-existing streaming query. In this case the results would be going into various result topics. Tools like KSQL also let you query historical data but you need to be sure that this won't create a performance impact if you have a lot of data and are continually query it over and over from the beginning to the end. It can work if the data set is small but as Manoj already said so well, it might be better to materialize your data stream into a repository designed with indexes beyond those in Kafka for faster or more complex interactive queries. -hans > On Jan 11, 2018, at 1:33 PM, Manoj Khangaonkar wrote: > > Hi, > > If I understood the question correctly , then the better approach is to > consume events from topic and store in > your favorite database. Then query the database as needed. > > Querying the topic for messages in kafka is not recommended as that will be > a linear search. > > regards > > On Thu, Jan 11, 2018 at 8:55 AM, Maria Pilar wrote: > >> Hi all, >> >> I have a requirement to be able to capture and store events for query, >> and I'm trying to choose the best option for that: >> >> 1) Capture the events from a separate topic, store events a state, in >> order to convert a stream to a table, that means materializing the >> stream. >> The option for it, that I'm thinking is to use an external state that >> is maintained in an external datastore, often a NoSQL (e.g Cassandra). >> That solution provides the advantage unlimited size and it can be >> accessed from multiple instances of the application or from different >> applications. >> >> 2) Other options could be query direct in local o internal state or in >> memory, however, to be able to query directly onto a topic, I would >> need to search a record by partition and offset, I don't know exactly >> how to implement that option if it's possible. >> >> Cheers. >> > > > > -- > http://khangaonkar.blogspot.com/
Re: Capturing and storing these Kafka events for query.
Hi, If I understood the question correctly , then the better approach is to consume events from topic and store in your favorite database. Then query the database as needed. Querying the topic for messages in kafka is not recommended as that will be a linear search. regards On Thu, Jan 11, 2018 at 8:55 AM, Maria Pilar wrote: > Hi all, > > I have a requirement to be able to capture and store events for query, > and I'm trying to choose the best option for that: > > 1) Capture the events from a separate topic, store events a state, in > order to convert a stream to a table, that means materializing the > stream. > The option for it, that I'm thinking is to use an external state that > is maintained in an external datastore, often a NoSQL (e.g Cassandra). > That solution provides the advantage unlimited size and it can be > accessed from multiple instances of the application or from different > applications. > > 2) Other options could be query direct in local o internal state or in > memory, however, to be able to query directly onto a topic, I would > need to search a record by partition and offset, I don't know exactly > how to implement that option if it's possible. > > Cheers. > -- http://khangaonkar.blogspot.com/
Capturing and storing these Kafka events for query.
Hi all, I have a requirement to be able to capture and store events for query, and I'm trying to choose the best option for that: 1) Capture the events from a separate topic, store events a state, in order to convert a stream to a table, that means materializing the stream. The option for it, that I'm thinking is to use an external state that is maintained in an external datastore, often a NoSQL (e.g Cassandra). That solution provides the advantage unlimited size and it can be accessed from multiple instances of the application or from different applications. 2) Other options could be query direct in local o internal state or in memory, however, to be able to query directly onto a topic, I would need to search a record by partition and offset, I don't know exactly how to implement that option if it's possible. Cheers.
