On Fri, Apr 28, 2017 at 4:52 PM, Shiva Ramagopal <tr.s...@gmail.com> wrote:
> Viktor, > > > On Fri, Apr 28, 2017 at 5:03 PM, Viktor Klang <viktor.kl...@gmail.com> > wrote: > >> >> >> On Fri, Apr 28, 2017 at 1:12 PM, Shiva Ramagopal <tr.s...@gmail.com> >> wrote: >> >>> Hi Viktor, >>> >>> On Fri, Apr 28, 2017 at 2:55 PM, Viktor Klang <viktor.kl...@gmail.com> >>> wrote: >>> >>>> Hi Shiva, >>>> >>>> On Fri, Apr 28, 2017 at 11:20 AM, Shiva Ramagopal <tr.s...@gmail.com> >>>> wrote: >>>> >>>>> I'm looking to compare Kafka Streams vs Akka Streams in two areas: >>>>> >>>>> 1. For ingesting between Kafka and HDFS/RDBMS >>>>> >>>>> Requirements are mainly around performance and latency. A Kafka topic >>>>> can have several million events, each corresponding to a database change >>>>> capture. When ingesting this topic into HDFS I'm also looking to partition >>>>> the data by day, typically based on a timestamp field in the event record, >>>>> aggregations on-the-fly (say by a userid field) and parquetizing >>>>> (preferably on the fly to optimize performance by reducing two I/O >>>>> operations). >>>>> >>>> >>>> Looking forward to see your benchmark! >>>> >>> >>> Hey, you wanted requirements! :) >>> >>>> >>>> >>>>> >>>>> 2. Low-latency processing >>>>> >>>>> Experiences around performances of Storm/Flink and Akka Streams would >>>>> be *very* nice. Typical use-cases are de-duping and enrichment with >>>>> metrics >>>>> computation (# duplicate events/records, aggregate metrics etc). Low >>>>> latency and scalability are main considerations >>>>> >>>> >>>> Low latency is not a metric, and scalability is not a profile. :) >>>> In other words: What latency distribution are you targeting and what >>>> scalability curve? >>>> >>> >>> I know latency is a nuanced topic. I'm just looking for broad >>> experiences on performance comparisons if anyone has done that. Currently >>> we have a Storm topology over 5 nodes doing enrichment of events from Kafka >>> that involves a lookup into a db per event. 90th percentile of latency of >>> this processing is under 200 ms and we are happy with this. While Storm is >>> mature, Akka Streams seems more general purpose than Storm. I'd like to use >>> Akka Streams for this reason if performance is comparable to Storm. >>> >> >> There's one important architectural difference here tho: Akka Streams are >> local-only (as in materialization). You can of course materialize Akka >> Streams on multiple nodes and use a transport to coordinate data >> processing. Interestingly it doesn't lock you in to a particular backend >> such that Kafka Streams or even Storm would do. >> > > The materialization part was something I was not aware of. Thanks for > pointing it out, really appreciate it. > You're most welcome > >> All of this boils down to requirements. Something like Flink or Google >> Beam could be viable options here as well. In your situation I'd look at >> the requirements and make a couple of prototypes before picking a winner. >> > > I was also looking for something to unify batch and streaming - which is > how I came to look at SMACK. I'd really like to keep the tech stack small > and have the parts integrate tightly with each other. Guess I have to pick > between Spark, Flink, Beam and Storm. > Sounds like a fun exercise :) > >> >>> >>> >>>> >>>> >>>>> >>>>> On Thu, Apr 27, 2017 at 8:31 PM, Viktor Klang <viktor.kl...@gmail.com> >>>>> wrote: >>>>> >>>>>> >>>>>> >>>>>> On Thu, Apr 27, 2017 at 10:39 AM, Shiva Ramagopal <tr.s...@gmail.com> >>>>>> wrote: >>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> I have read through multiple articles describing the SMACK stack but >>>>>>> I'm having difficulty understanding the role of Akka in the stack. How >>>>>>> does >>>>>>> Akka fit in? >>>>>>> >>>>>> >>>>>> Akka is for building the application itself. >>>>>> >>>>>> >>>>>>> >>>>>>> Also I would like to know of experiences in using Akka Streams vs >>>>>>> Kafka Connect for ingesting from Kafka into HDFS (Hive) and RDBMS. Has >>>>>>> anyone used Akka Streams for say, dynamic partitioning of events from a >>>>>>> Kafka topic into HDFS? >>>>>>> >>>>>> >>>>>> Based on what requirements? >>>>>> >>>>>> >>>>>>> >>>>>>> TIA >>>>>>> -Shiva >>>>>>> >>>>>>> -- >>>>>>> >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>> >>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/c >>>>>>> urrent/additional/faq.html >>>>>>> >>>>>>>>>> Search the archives: https://groups.google.com/grou >>>>>>> p/akka-user >>>>>>> --- >>>>>>> You received this message because you are subscribed to the Google >>>>>>> Groups "Akka User List" group. >>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>> send an email to akka-user+unsubscr...@googlegroups.com. >>>>>>> To post to this group, send email to akka-user@googlegroups.com. >>>>>>> Visit this group at https://groups.google.com/group/akka-user. >>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Cheers, >>>>>> √ >>>>>> >>>>>> -- >>>>>> >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>> >>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/c >>>>>> urrent/additional/faq.html >>>>>> >>>>>>>>>> Search the archives: https://groups.google.com/grou >>>>>> p/akka-user >>>>>> --- >>>>>> You received this message because you are subscribed to the Google >>>>>> Groups "Akka User List" group. >>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>> send an email to akka-user+unsubscr...@googlegroups.com. >>>>>> To post to this group, send email to akka-user@googlegroups.com. >>>>>> Visit this group at https://groups.google.com/group/akka-user. >>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>> >>>>> >>>>> -- >>>>> >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>> >>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/c >>>>> urrent/additional/faq.html >>>>> >>>>>>>>>> Search the archives: https://groups.google.com/grou >>>>> p/akka-user >>>>> --- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "Akka User List" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to akka-user+unsubscr...@googlegroups.com. >>>>> To post to this group, send email to akka-user@googlegroups.com. >>>>> Visit this group at https://groups.google.com/group/akka-user. >>>>> For more options, visit https://groups.google.com/d/optout. >>>>> >>>> >>>> >>>> >>>> -- >>>> Cheers, >>>> √ >>>> >>>> -- >>>> >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>> >>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/c >>>> urrent/additional/faq.html >>>> >>>>>>>>>> Search the archives: https://groups.google.com/grou >>>> p/akka-user >>>> --- >>>> You received this message because you are subscribed to the Google >>>> Groups "Akka User List" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to akka-user+unsubscr...@googlegroups.com. >>>> To post to this group, send email to akka-user@googlegroups.com. >>>> Visit this group at https://groups.google.com/group/akka-user. >>>> For more options, visit https://groups.google.com/d/optout. >>>> >>> >>> -- >>> >>>>>>>>>> Read the docs: http://akka.io/docs/ >>> >>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/c >>> urrent/additional/faq.html >>> >>>>>>>>>> Search the archives: https://groups.google.com/grou >>> p/akka-user >>> --- >>> You received this message because you are subscribed to the Google >>> Groups "Akka User List" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to akka-user+unsubscr...@googlegroups.com. >>> To post to this group, send email to akka-user@googlegroups.com. >>> Visit this group at https://groups.google.com/group/akka-user. >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> >> >> -- >> Cheers, >> √ >> >> -- >> >>>>>>>>>> Read the docs: http://akka.io/docs/ >> >>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/c >> urrent/additional/faq.html >> >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user >> --- >> You received this message because you are subscribed to the Google Groups >> "Akka User List" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to akka-user+unsubscr...@googlegroups.com. >> To post to this group, send email to akka-user@googlegroups.com. >> Visit this group at https://groups.google.com/group/akka-user. >> For more options, visit https://groups.google.com/d/optout. >> > > -- > >>>>>>>>>> Read the docs: http://akka.io/docs/ > >>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/ > current/additional/faq.html > >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user > --- > You received this message because you are subscribed to the Google Groups > "Akka User List" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to akka-user+unsubscr...@googlegroups.com. > To post to this group, send email to akka-user@googlegroups.com. > Visit this group at https://groups.google.com/group/akka-user. > For more options, visit https://groups.google.com/d/optout. > -- Cheers, √ -- >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>>>> Check the FAQ: >>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups "Akka User List" group. To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+unsubscr...@googlegroups.com. To post to this group, send email to akka-user@googlegroups.com. Visit this group at https://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout.
