On Fri, Dec 1, 2017 at 8:03 AM, kraythe <kray...@gmail.com> wrote: > What about using something like Kafka to distribute the matrix? >
Sounds like overkill (a lot of infra) if that is the only thing you are using Kafka for. I think Roland's suggestion is excellent. > > > On Thursday, November 30, 2017 at 12:54:59 PM UTC-6, rklaehn wrote: >> >> As Roland said, it is probably best to use some kind of side channel for >> large blobs. >> >> One thing you could do that is a bit out of the left field is to run ipfs >> <https://ipfs.io/> on each cluster node. Then you could save the 10MB >> data block in ipfs and just link to the data from the message. The actual >> data transfer works via ipfs, which is very efficient especially if you >> want to broadcast some data in a private network. And the actor >> communication does not get bogged down with huge blobs. >> >> A nice property of this approach is that the message remains immutable. >> Whereas the data behind an url can change, the content you get for an ipfs >> hash is always the same. >> >> Cheers, >> >> Rüdiger >> >> On Thu, Nov 30, 2017 at 7:37 PM Roland Kuhn <goo...@rkuhn.info> wrote: >> >>> Yes, I meant an Akka HTTP server for this purpose: spin it up on the >>> node where the map is created, can even be on a random port, and send the >>> URL to this server around. That server can offer the last ten versions (or >>> whatever), could use the hash of the preserialized map as identifier. That >>> way you can explicitly manage the dissemination and its resource usage. >>> >>> Regards, >>> >>> Roland >>> >>> Sent from my iPhone >>> >>> On 30. Nov 2017, at 18:43, kraythe <kra...@gmail.com> wrote: >>> >>> The issue here is that the matrices are recalculated quote frequently, >>> every minute is common so its not that it can be deployed to a static HTML >>> server. I suppose I could put an HTTP endpoint for server to server >>> communication and have that be outside of the Akka communication channels. >>> Is that what you are suggesting? I would have to know which node to send >>> the http request to of course and make sure the nodes can talk to each >>> other directly without using the load balancer. >>> >>> -- Robert >>> >>> On Thursday, November 30, 2017 at 9:15:03 AM UTC-6, rkuhn wrote: >>>> >>>> May I suggest serving the large blobs via a different mechanisms like >>>> HTTP? Sending around URL (and putting them inside events) seems much better >>>> than putting the data bytes all over the place. >>>> >>>> Regards, >>>> >>>> Roland >>>> >>>> 30 nov. 2017 kl. 16:11 skrev Patrik Nordwall <patrik....@gmail.com>: >>>> >>>> Sending 10 MB messages is indeed recipie for trouble. >>>> >>>> I’d suggest to split the 10 MB into 100 smaller messages and collect, >>>> reassembe, them on the receiving side. >>>> >>>> Perhaps you are fine with loosing such messages since they are >>>> periodically published again. Otherwise you could use DistributedData to >>>> share them. >>>> >>>> /Patrik >>>> tors 30 nov. 2017 kl. 14:35 skrev kraythe <kra...@gmail.com>: >>>> >>>>> Greetings, >>>>> >>>>> We have an actor structure where there is a cluster sharded actor that >>>>> calculates a parametric matrix of about 7 meg and has to distribute it to >>>>> 5 >>>>> other nodes for consumption. First of all there are the following >>>>> constraints. >>>>> >>>>> 1. The matrix has to be generated in one place. >>>>> 2. The matrix is needed on all nodes to handle user load. >>>>> 3. The matrix will be periodically generated again to handle changing >>>>> variables and then sent off to all nodes to handle user load. >>>>> 4. Saving the matrix in a database is probably not viable as it merely >>>>> shifts the networking load and the database would get very large very >>>>> fast. >>>>> The Database only saves the input params. >>>>> >>>>> We changed the akka max message size to 10 meg to accomplish this but >>>>> that feels a bit odd, but we didn't see another choice. Normally this >>>>> works >>>>> fine even though passing 10 meg messages around a DistributedPubSub seems >>>>> odd to me. However on startup the system has to stat up 2000 of these all >>>>> at once. As a result the sharding coordinators scream at us about buffered >>>>> messages. It eventually calms down and life resumes but I would love to be >>>>> able to do this without the bloodbath in the logs. >>>>> >>>>> Can someone recommend an alternative strategy for handling the >>>>> distribution of the parametric matrix that gets it to every node but >>>>> doesn't cause shard coordinator complaint bloodbath? >>>>> >>>>> Thanks in advance. >>>>> Robert SImmons Jr. MSc. >>>>> >>>>> -- >>>>> >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>> >>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/c >>>>> urrent/additional/faq.html >>>>> >>>>>>>>>> Search the archives: https://groups.google.com/grou >>>>> p/akka-user >>>>> --- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "Akka User List" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to akka-user+...@googlegroups.com. >>>>> To post to this group, send email to akka...@googlegroups.com. >>>>> Visit this group at https://groups.google.com/group/akka-user. >>>>> For more options, visit https://groups.google.com/d/optout. >>>>> >>>> >>>> -- >>>> >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>> >>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/c >>>> urrent/additional/faq.html >>>> >>>>>>>>>> Search the archives: https://groups.google.com/grou >>>> p/akka-user >>>> --- >>>> You received this message because you are subscribed to the Google >>>> Groups "Akka User List" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to akka-user+...@googlegroups.com. >>>> To post to this group, send email to akka...@googlegroups.com. >>>> Visit this group at https://groups.google.com/group/akka-user. >>>> For more options, visit https://groups.google.com/d/optout. >>>> >>>> >>>> -- >>> >>>>>>>>>> Read the docs: http://akka.io/docs/ >>> >>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/c >>> urrent/additional/faq.html >>> >>>>>>>>>> Search the archives: https://groups.google.com/grou >>> p/akka-user >>> --- >>> You received this message because you are subscribed to the Google >>> Groups "Akka User List" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to akka-user+...@googlegroups.com. >>> To post to this group, send email to akka...@googlegroups.com. >>> Visit this group at https://groups.google.com/group/akka-user. >>> For more options, visit https://groups.google.com/d/optout. >>> >>> -- >>> >>>>>>>>>> Read the docs: http://akka.io/docs/ >>> >>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/c >>> urrent/additional/faq.html >>> >>>>>>>>>> Search the archives: https://groups.google.com/grou >>> p/akka-user >>> --- >>> You received this message because you are subscribed to the Google >>> Groups "Akka User List" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to akka-user+...@googlegroups.com. >>> To post to this group, send email to akka...@googlegroups.com. >>> Visit this group at https://groups.google.com/group/akka-user. >>> For more options, visit https://groups.google.com/d/optout. >>> >> -- > >>>>>>>>>> Read the docs: http://akka.io/docs/ > >>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/ > current/additional/faq.html > >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user > --- > You received this message because you are subscribed to the Google Groups > "Akka User List" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to akka-user+unsubscr...@googlegroups.com. > To post to this group, send email to akka-user@googlegroups.com. > Visit this group at https://groups.google.com/group/akka-user. > For more options, visit https://groups.google.com/d/optout. > -- Patrik Nordwall Akka Tech Lead Lightbend <http://www.lightbend.com/> - Reactive apps on the JVM Twitter: @patriknw -- >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>>>> Check the FAQ: >>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups "Akka User List" group. To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+unsubscr...@googlegroups.com. To post to this group, send email to akka-user@googlegroups.com. Visit this group at https://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout.
