Ok. I suppose I should examine the GroupBy or SubFlow source code, but if I
understand correctly different stages will run concurrently (if fusing is
off or async boundaries have been added), but there's not a separate actor
for each substream in a SubFlow?
On Wed, Feb 3, 2016 at 7:35 AM, Francesco Di Muccio <
francesco.dimuc...@gmail.com> wrote:
>
> Il giorno mercoledì 3 febbraio 2016 15:30:29 UTC+1, rrodseth ha scritto:
>>
>> Thanks very much. Actually, I would argue this is preferable to what's in
>> the template now, and both deserve a juicy comment!
>>
>> Does groupBy alone introduce any parallelism? With/without fusing? In
>> this example, if there were n log levels rather than 5, would more than 5
>> files be written concurrently?
>>
>> In this example the 5 (or n) files are written concurrently, but one
> problem of this approach is that if mapAsync parallelism is less than 5 it
> can deadlock the whole stream.
>
>
>> With the proposed new two-input stage in akka, would parallel file
>> writing be internal and configurable?
>>
>> This might be a good time to say that I am extremely impressed with the
>> thought and ingenuity that has gone into this library, and love using it.
>> Thanks!
>>
>> On Wed, Feb 3, 2016 at 2:56 AM, Roland Kuhn <goo...@rkuhn.info> wrote:
>>
>>> Yes, this is exactly what I was referring to, and I hope it is clear
>>> that we don’t want to show this approach to users in the Activator
>>> template—we need a better solution :-)
>>>
>>> Thanks Francesco!
>>>
>>> Regards,
>>>
>>> Roland
>>>
>>> 3 feb 2016 kl. 10:31 skrev Francesco Di Muccio <francesco...@gmail.com>:
>>>
>>>
>>>
>>> Il giorno mercoledì 3 febbraio 2016 01:54:55 UTC+1, rrodseth ha scritto:
>>>>
>>>> No worries. I wish I had the time and expertise to help.
>>>>
>>>> I don't mean to be a pest, but since my credibility with management is
>>>> at stake [ :) ] can anyone suggest any ways I can tackle the problem of
>>>> groupBy followed by write to file?
>>>>
>>>> I might be able to get away with draining each substream to a
>>>> collection and then using mapAsync to run a separate stream (per-key) to
>>>> write the file, returning a Future as required by mapAsync. Not unlike
>>>> where the activator template has ended up, but I'd really rather not.
>>>>
>>>> But I also find myself wondering if I could do something with fold,
>>>> where the initial value contains the Sink for that substream.
>>>>
>>>> Roland also alluded to a hack(?) using prefixAndTail(0), but I don't
>>>> quite see what that would look like.
>>>>
>>>
>>> Hi,
>>>
>>> I can try to help using prefixAndTail, I came up with this so far:
>>>
>>> FileIO.fromFile(logFile).
>>> // parse chunks of bytes into lines
>>> via(Framing.delimiter(ByteString(System.lineSeparator),
>>> maximumFrameLength = 512, allowTruncation = true)).
>>> map(_.utf8String).
>>> map {
>>> case line@LoglevelPattern(level) => (level, line)
>>> case line@other => ("OTHER", line)
>>> }.
>>> // group them by log level
>>> groupBy(5, _._1).
>>> prefixAndTail(1).
>>> // write lines of each group to a separate file
>>> mapAsync(5) {
>>> case (Seq((level, line)), source) =>
>>> Source.single(line)
>>> .concat(source.map(_._2))
>>> .map(line => ByteString(line + "\n"))
>>> .runWith(FileIO.toFile(new
>>> java.io.File(s"target/log-$level.txt")))
>>>
>>> // Not sure that this will ever happen
>>> case _ =>
>>> Future.successful(0)
>>> }.
>>> mergeSubstreams.
>>> runWith(Sink.onComplete { _ =>
>>> system.shutdown()
>>> })
>>>
>>> What prefixAndTail does (if I got it right) emits a pair where the first
>>> value is a sequence of n elements (prefix), and
>>> the second value is the rest (tail) lifted to a source. Here I'm using
>>> prefixAndTail(1) because I need to extract the level
>>> from the first element to calculate the filename.
>>>
>>> Hope it helps,
>>> Francesco
>>>
>>>
>>>> Clutching at straws!
>>>>
>>>> On Tue, Feb 2, 2016 at 12:27 AM, Roland Kuhn <goo...@rkuhn.info> wrote:
>>>>
>>>>> Yes, indeed, I didn’t have the time to get that link yesterday. If
>>>>> anyone wants to work on that: contributions are *always* welcome! :-)
>>>>>
>>>>> Regards,
>>>>>
>>>>> Roland
>>>>>
>>>>> 1 feb 2016 kl. 23:59 skrev Richard Rodseth <rrod...@gmail.com>:
>>>>>
>>>>> For anyone following along, I believe this is the issue Roland refers
>>>>> to
>>>>>
>>>>> https://github.com/akka/akka/issues/18969
>>>>>
>>>>> On Mon, Feb 1, 2016 at 2:28 PM, Richard Rodseth <rrod...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Ouch. Thanks.
>>>>>>
>>>>>> On Mon, Feb 1, 2016 at 1:49 PM, Roland Kuhn <goo...@rkuhn.info>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi Richard,
>>>>>>>
>>>>>>> this is not yet solved, and we have an issue tracking this in
>>>>>>> akka/akka as well. It is not certain that we will be able to fix this
>>>>>>> before 2.4.2 comes out or whether the API addition that is necessary
>>>>>>> will
>>>>>>> have to wait until 2.4.3.
>>>>>>>
>>>>>>> Regards,
>>>>>>>
>>>>>>> Roland
>>>>>>>
>>>>>>> 1 feb 2016 kl. 22:34 skrev Richard Rodseth <rrod...@gmail.com>:
>>>>>>>
>>>>>>> I'm concerned that this might fall through the cracks since the
>>>>>>> GitHub issue is written against the Activator template rather than akka
>>>>>>> itself.
>>>>>>> If it's already solved in 2.4.x please let me know.
>>>>>>> If anyone can offer any guidance for a workaround, I'd appreciate
>>>>>>> it. In the linked discussion Roland mentioned something about
>>>>>>> prefixAndTail
>>>>>>> but said you wouldn't want that in a tutorial.
>>>>>>> I also think it would be helpful for those who have seen both the
>>>>>>> pre-SubFlow and post-SubFlow templates if the reason for this
>>>>>>> unfortunate
>>>>>>> regression was stated in the activator template.
>>>>>>>
>>>>>>> On Sun, Jan 31, 2016 at 6:23 PM, Richard Rodseth <rrod...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> I have run into this issue
>>>>>>>> https://github.com/typesafehub/activator-akka-stream-scala/issues/37
>>>>>>>>
>>>>>>>> I want to group a stream and write each substream to a separate
>>>>>>>> file. A pretty common use case, I'd imagine.
>>>>>>>>
>>>>>>>> The old version of the GroupLog example showed a groupBy() followed
>>>>>>>> by a to()
>>>>>>>>
>>>>>>>> Because of the introduction of SubFlow, the new version had to
>>>>>>>> introduce an unacceptable fold() into a strict collection
>>>>>>>>
>>>>>>>>
>>>>>>>> https://github.com/typesafehub/activator-akka-stream-scala/blob/c63afb5eecf50726c4417157efd26d697b621e99/src/main/scala/sample/stream/GroupLogFile.scala
>>>>>>>>
>>>>>>>> What are my options? ETA for a fix?
>>>>>>>> Thanks
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> >>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>> >>>>>>>>>> Check the FAQ:
>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>> >>>>>>>>>> Search the archives:
>>>>>>> https://groups.google.com/group/akka-user
>>>>>>> ---
>>>>>>> You received this message because you are subscribed to the Google
>>>>>>> Groups "Akka User List" group.
>>>>>>> To unsubscribe from this group and stop receiving emails from it,
>>>>>>> send an email to akka-user+...@googlegroups.com.
>>>>>>> To post to this group, send email to akka...@googlegroups.com.
>>>>>>> Visit this group at https://groups.google.com/group/akka-user.
>>>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> *Dr. Roland Kuhn*
>>>>>>> *Akka Tech Lead*
>>>>>>> Typesafe <http://typesafe.com/> – Reactive apps on the JVM.
>>>>>>> twitter: @rolandkuhn
>>>>>>> <http://twitter.com/#!/rolandkuhn>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> >>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>> >>>>>>>>>> Check the FAQ:
>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>> >>>>>>>>>> Search the archives:
>>>>>>> https://groups.google.com/group/akka-user
>>>>>>> ---
>>>>>>> You received this message because you are subscribed to the Google
>>>>>>> Groups "Akka User List" group.
>>>>>>> To unsubscribe from this group and stop receiving emails from it,
>>>>>>> send an email to akka-user+...@googlegroups.com.
>>>>>>> To post to this group, send email to akka...@googlegroups.com.
>>>>>>> Visit this group at https://groups.google.com/group/akka-user.
>>>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>> --
>>>>> >>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>> >>>>>>>>>> Check the FAQ:
>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>> >>>>>>>>>> Search the archives:
>>>>> https://groups.google.com/group/akka-user
>>>>> ---
>>>>> You received this message because you are subscribed to the Google
>>>>> Groups "Akka User List" group.
>>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>>> an email to akka-user+...@googlegroups.com.
>>>>> To post to this group, send email to akka...@googlegroups.com.
>>>>> Visit this group at https://groups.google.com/group/akka-user.
>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> *Dr. Roland Kuhn*
>>>>> *Akka Tech Lead*
>>>>> Typesafe <http://typesafe.com/> – Reactive apps on the JVM.
>>>>> twitter: @rolandkuhn
>>>>> <http://twitter.com/#!/rolandkuhn>
>>>>>
>>>>>
>>>>> --
>>>>> >>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>> >>>>>>>>>> Check the FAQ:
>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>> >>>>>>>>>> Search the archives:
>>>>> https://groups.google.com/group/akka-user
>>>>> ---
>>>>> You received this message because you are subscribed to the Google
>>>>> Groups "Akka User List" group.
>>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>>> an email to akka-user+...@googlegroups.com.
>>>>> To post to this group, send email to akka...@googlegroups.com.
>>>>> Visit this group at https://groups.google.com/group/akka-user.
>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>>
>>>>
>>>>
>>> --
>>> >>>>>>>>>> Read the docs: http://akka.io/docs/
>>> >>>>>>>>>> Check the FAQ:
>>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>> >>>>>>>>>> Search the archives:
>>> https://groups.google.com/group/akka-user
>>> ---
>>> You received this message because you are subscribed to the Google
>>> Groups "Akka User List" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to akka-user+...@googlegroups.com.
>>> To post to this group, send email to akka...@googlegroups.com.
>>> Visit this group at https://groups.google.com/group/akka-user.
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>>
>>>
>>>
>>> *Dr. Roland Kuhn*
>>> *Akka Tech Lead*
>>> Typesafe <http://typesafe.com/> – Reactive apps on the JVM.
>>> twitter: @rolandkuhn
>>> <http://twitter.com/#!/rolandkuhn>
>>>
>>> --
>>> >>>>>>>>>> Read the docs: http://akka.io/docs/
>>> >>>>>>>>>> Check the FAQ:
>>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>> >>>>>>>>>> Search the archives:
>>> https://groups.google.com/group/akka-user
>>> ---
>>> You received this message because you are subscribed to the Google
>>> Groups "Akka User List" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to akka-user+...@googlegroups.com.
>>> To post to this group, send email to akka...@googlegroups.com.
>>> Visit this group at https://groups.google.com/group/akka-user.
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>> --
> >>>>>>>>>> Read the docs: http://akka.io/docs/
> >>>>>>>>>> Check the FAQ:
> http://doc.akka.io/docs/akka/current/additional/faq.html
> >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
> ---
> You received this message because you are subscribed to the Google Groups
> "Akka User List" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to akka-user+unsubscr...@googlegroups.com.
> To post to this group, send email to akka-user@googlegroups.com.
> Visit this group at https://groups.google.com/group/akka-user.
> For more options, visit https://groups.google.com/d/optout.
>
--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ:
>>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka
User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to akka-user+unsubscr...@googlegroups.com.
To post to this group, send email to akka-user@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.
