Thanks very much. Actually, I would argue this is preferable to what's in
the template now, and both deserve a juicy comment!
Does groupBy alone introduce any parallelism? With/without fusing? In this
example, if there were n log levels rather than 5, would more than 5 files
be written concurrently?
With the proposed new two-input stage in akka, would parallel file writing
be internal and configurable?
This might be a good time to say that I am extremely impressed with the
thought and ingenuity that has gone into this library, and love using it.
Thanks!
On Wed, Feb 3, 2016 at 2:56 AM, Roland Kuhn <goo...@rkuhn.info> wrote:
> Yes, this is exactly what I was referring to, and I hope it is clear that
> we don’t want to show this approach to users in the Activator template—we
> need a better solution :-)
>
> Thanks Francesco!
>
> Regards,
>
> Roland
>
> 3 feb 2016 kl. 10:31 skrev Francesco Di Muccio <
> francesco.dimuc...@gmail.com>:
>
>
>
> Il giorno mercoledì 3 febbraio 2016 01:54:55 UTC+1, rrodseth ha scritto:
>>
>> No worries. I wish I had the time and expertise to help.
>>
>> I don't mean to be a pest, but since my credibility with management is at
>> stake [ :) ] can anyone suggest any ways I can tackle the problem of
>> groupBy followed by write to file?
>>
>> I might be able to get away with draining each substream to a collection
>> and then using mapAsync to run a separate stream (per-key) to write the
>> file, returning a Future as required by mapAsync. Not unlike where the
>> activator template has ended up, but I'd really rather not.
>>
>> But I also find myself wondering if I could do something with fold, where
>> the initial value contains the Sink for that substream.
>>
>> Roland also alluded to a hack(?) using prefixAndTail(0), but I don't
>> quite see what that would look like.
>>
>
> Hi,
>
> I can try to help using prefixAndTail, I came up with this so far:
>
> FileIO.fromFile(logFile).
> // parse chunks of bytes into lines
> via(Framing.delimiter(ByteString(System.lineSeparator),
> maximumFrameLength = 512, allowTruncation = true)).
> map(_.utf8String).
> map {
> case line@LoglevelPattern(level) => (level, line)
> case line@other => ("OTHER", line)
> }.
> // group them by log level
> groupBy(5, _._1).
> prefixAndTail(1).
> // write lines of each group to a separate file
> mapAsync(5) {
> case (Seq((level, line)), source) =>
> Source.single(line)
> .concat(source.map(_._2))
> .map(line => ByteString(line + "\n"))
> .runWith(FileIO.toFile(new java.io.File(s"target/log-$level.txt")))
>
> // Not sure that this will ever happen
> case _ =>
> Future.successful(0)
> }.
> mergeSubstreams.
> runWith(Sink.onComplete { _ =>
> system.shutdown()
> })
>
> What prefixAndTail does (if I got it right) emits a pair where the first
> value is a sequence of n elements (prefix), and
> the second value is the rest (tail) lifted to a source. Here I'm using
> prefixAndTail(1) because I need to extract the level
> from the first element to calculate the filename.
>
> Hope it helps,
> Francesco
>
>
>> Clutching at straws!
>>
>> On Tue, Feb 2, 2016 at 12:27 AM, Roland Kuhn <goo...@rkuhn.info> wrote:
>>
>>> Yes, indeed, I didn’t have the time to get that link yesterday. If
>>> anyone wants to work on that: contributions are *always* welcome! :-)
>>>
>>> Regards,
>>>
>>> Roland
>>>
>>> 1 feb 2016 kl. 23:59 skrev Richard Rodseth <rrod...@gmail.com>:
>>>
>>> For anyone following along, I believe this is the issue Roland refers to
>>>
>>> https://github.com/akka/akka/issues/18969
>>>
>>> On Mon, Feb 1, 2016 at 2:28 PM, Richard Rodseth <rrod...@gmail.com>
>>> wrote:
>>>
>>>> Ouch. Thanks.
>>>>
>>>> On Mon, Feb 1, 2016 at 1:49 PM, Roland Kuhn <goo...@rkuhn.info> wrote:
>>>>
>>>>> Hi Richard,
>>>>>
>>>>> this is not yet solved, and we have an issue tracking this in
>>>>> akka/akka as well. It is not certain that we will be able to fix this
>>>>> before 2.4.2 comes out or whether the API addition that is necessary will
>>>>> have to wait until 2.4.3.
>>>>>
>>>>> Regards,
>>>>>
>>>>> Roland
>>>>>
>>>>> 1 feb 2016 kl. 22:34 skrev Richard Rodseth <rrod...@gmail.com>:
>>>>>
>>>>> I'm concerned that this might fall through the cracks since the GitHub
>>>>> issue is written against the Activator template rather than akka itself.
>>>>> If it's already solved in 2.4.x please let me know.
>>>>> If anyone can offer any guidance for a workaround, I'd appreciate it.
>>>>> In the linked discussion Roland mentioned something about prefixAndTail
>>>>> but
>>>>> said you wouldn't want that in a tutorial.
>>>>> I also think it would be helpful for those who have seen both the
>>>>> pre-SubFlow and post-SubFlow templates if the reason for this unfortunate
>>>>> regression was stated in the activator template.
>>>>>
>>>>> On Sun, Jan 31, 2016 at 6:23 PM, Richard Rodseth <rrod...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> I have run into this issue
>>>>>> https://github.com/typesafehub/activator-akka-stream-scala/issues/37
>>>>>>
>>>>>> I want to group a stream and write each substream to a separate file.
>>>>>> A pretty common use case, I'd imagine.
>>>>>>
>>>>>> The old version of the GroupLog example showed a groupBy() followed
>>>>>> by a to()
>>>>>>
>>>>>> Because of the introduction of SubFlow, the new version had to
>>>>>> introduce an unacceptable fold() into a strict collection
>>>>>>
>>>>>>
>>>>>> https://github.com/typesafehub/activator-akka-stream-scala/blob/c63afb5eecf50726c4417157efd26d697b621e99/src/main/scala/sample/stream/GroupLogFile.scala
>>>>>>
>>>>>> What are my options? ETA for a fix?
>>>>>> Thanks
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>> --
>>>>> >>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>> >>>>>>>>>> Check the FAQ:
>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>> >>>>>>>>>> Search the archives:
>>>>> https://groups.google.com/group/akka-user
>>>>> ---
>>>>> You received this message because you are subscribed to the Google
>>>>> Groups "Akka User List" group.
>>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>>> an email to akka-user+...@googlegroups.com.
>>>>> To post to this group, send email to akka...@googlegroups.com.
>>>>> Visit this group at https://groups.google.com/group/akka-user.
>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> *Dr. Roland Kuhn*
>>>>> *Akka Tech Lead*
>>>>> Typesafe <http://typesafe.com/> – Reactive apps on the JVM.
>>>>> twitter: @rolandkuhn
>>>>> <http://twitter.com/#!/rolandkuhn>
>>>>>
>>>>>
>>>>> --
>>>>> >>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>> >>>>>>>>>> Check the FAQ:
>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>> >>>>>>>>>> Search the archives:
>>>>> https://groups.google.com/group/akka-user
>>>>> ---
>>>>> You received this message because you are subscribed to the Google
>>>>> Groups "Akka User List" group.
>>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>>> an email to akka-user+...@googlegroups.com.
>>>>> To post to this group, send email to akka...@googlegroups.com.
>>>>> Visit this group at https://groups.google.com/group/akka-user.
>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>>
>>>>
>>>>
>>>
>>> --
>>> >>>>>>>>>> Read the docs: http://akka.io/docs/
>>> >>>>>>>>>> Check the FAQ:
>>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>> >>>>>>>>>> Search the archives:
>>> https://groups.google.com/group/akka-user
>>> ---
>>> You received this message because you are subscribed to the Google
>>> Groups "Akka User List" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to akka-user+...@googlegroups.com.
>>> To post to this group, send email to akka...@googlegroups.com.
>>> Visit this group at https://groups.google.com/group/akka-user.
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>>
>>>
>>>
>>> *Dr. Roland Kuhn*
>>> *Akka Tech Lead*
>>> Typesafe <http://typesafe.com/> – Reactive apps on the JVM.
>>> twitter: @rolandkuhn
>>> <http://twitter.com/#!/rolandkuhn>
>>>
>>>
>>> --
>>> >>>>>>>>>> Read the docs: http://akka.io/docs/
>>> >>>>>>>>>> Check the FAQ:
>>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>> >>>>>>>>>> Search the archives:
>>> https://groups.google.com/group/akka-user
>>> ---
>>> You received this message because you are subscribed to the Google
>>> Groups "Akka User List" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to akka-user+...@googlegroups.com.
>>> To post to this group, send email to akka...@googlegroups.com.
>>> Visit this group at https://groups.google.com/group/akka-user.
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>>
> --
> >>>>>>>>>> Read the docs: http://akka.io/docs/
> >>>>>>>>>> Check the FAQ:
> http://doc.akka.io/docs/akka/current/additional/faq.html
> >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
> ---
> You received this message because you are subscribed to the Google Groups
> "Akka User List" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to akka-user+unsubscr...@googlegroups.com.
> To post to this group, send email to akka-user@googlegroups.com.
> Visit this group at https://groups.google.com/group/akka-user.
> For more options, visit https://groups.google.com/d/optout.
>
>
>
>
> *Dr. Roland Kuhn*
> *Akka Tech Lead*
> Typesafe <http://typesafe.com/> – Reactive apps on the JVM.
> twitter: @rolandkuhn
> <http://twitter.com/#!/rolandkuhn>
>
> --
> >>>>>>>>>> Read the docs: http://akka.io/docs/
> >>>>>>>>>> Check the FAQ:
> http://doc.akka.io/docs/akka/current/additional/faq.html
> >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
> ---
> You received this message because you are subscribed to the Google Groups
> "Akka User List" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to akka-user+unsubscr...@googlegroups.com.
> To post to this group, send email to akka-user@googlegroups.com.
> Visit this group at https://groups.google.com/group/akka-user.
> For more options, visit https://groups.google.com/d/optout.
>
--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ:
>>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka
User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to akka-user+unsubscr...@googlegroups.com.
To post to this group, send email to akka-user@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.
