Akka Streams is an excellent choice for this use case. I would even say that it is the canonical use case for Akka Streams. We have probably not highlighted that enough, yet.
Cheers, Patrik tors 14 juli 2016 kl. 13:22 skrev James Matlik <james.mat...@gmail.com>: > Using Akka streams for ETL is our primary use case. The back pressure > support has been extremely useful in helping us maximize throughout while > at the same time avoid overwhelming the multiple external rest services we > query against. By maintaining dedicated, fixed sized dispatcher pools, we > can easily use our legacy blocking client SDKs over a fixed max number of > concurrent connections/requests. Then the ETL can process as fast as it > possibly can within those constraints. > > We found the learning curve to be on the steep side, but once it clicks, > the power and ease of use Streams provides is... impressive... > refreshing... exhilarating... addicting.. take your pick. > > On Jul 14, 2016 12:28 AM, "Beno" <andy.zelin...@gmail.com> wrote: > >> Ive been using Akka streaming for a use case which I dont see much about >> - that of a small/moderate scale ETL or simple processing pipeline. Im >> relatively new to it all, so I just wanted to see if I might be missing >> something that would change my opinion, which is that Akka Streams is among >> the best tools for data cleaning-- the graph dsl is so easy to code with >> and reason about. >> >> The details: Batch processing to clean and curate data, with external >> calls RESTful requests as part of the flow. >> >> Source[A] (read from file or DB) ~> Flow[A,B](some transformation >> function) ~> Flow[B,C] (by way of a RESTful request/response) ~> >> Flow[C,D](graph query) ~> Sink[D](to DB) >> >> Where Source might be 50,000 lines in a file or rows in a table. >> >> Thanks for any feedback >> >> -- >> >>>>>>>>>> Read the docs: http://akka.io/docs/ >> >>>>>>>>>> Check the FAQ: >> http://doc.akka.io/docs/akka/current/additional/faq.html >> >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user >> --- >> You received this message because you are subscribed to the Google Groups >> "Akka User List" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to akka-user+unsubscr...@googlegroups.com. >> To post to this group, send email to akka-user@googlegroups.com. >> Visit this group at https://groups.google.com/group/akka-user. >> For more options, visit https://groups.google.com/d/optout. >> > -- > >>>>>>>>>> Read the docs: http://akka.io/docs/ > >>>>>>>>>> Check the FAQ: > http://doc.akka.io/docs/akka/current/additional/faq.html > >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user > --- > You received this message because you are subscribed to the Google Groups > "Akka User List" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to akka-user+unsubscr...@googlegroups.com. > To post to this group, send email to akka-user@googlegroups.com. > Visit this group at https://groups.google.com/group/akka-user. > For more options, visit https://groups.google.com/d/optout. > -- >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>>>> Check the FAQ: >>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups "Akka User List" group. To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+unsubscr...@googlegroups.com. To post to this group, send email to akka-user@googlegroups.com. Visit this group at https://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout.
