Re: [akka-user] Distributing lots of large messages to a cluster on startup issue.

[kraythe]

We considered that but the problem is that would be the only thing we use 
that cache for. We also considered putting it in the Cassandra database but 
that seemed wasteful for what is very transient data. 

-- Robert

On Thursday, November 30, 2017 at 11:45:11 AM UTC-6, Rob Crawford wrote:
>
> Maybe a shared cache would be a better way to deal with it?
>
> On Thursday, November 30, 2017 at 12:41:28 PM UTC-5, kraythe wrote:
>>
>> Yeah the problem is it cant be broken up, its sort of all or nothing. I 
>> cant be too specific beyond it being a parametric matrix. Believe me I 
>> would prefer not shipping it around. 
>>
>> On Thursday, November 30, 2017 at 9:11:51 AM UTC-6, Patrik Nordwall wrote:
>>>
>>> Sending 10 MB messages is indeed recipie for trouble. 
>>>
>>> I’d suggest to split the 10 MB into 100 smaller messages and collect, 
>>> reassembe, them on the receiving side.
>>>
>>> Perhaps you are fine with loosing such messages since they are 
>>> periodically published again. Otherwise you could use DistributedData to 
>>> share them.
>>>
>>> /Patrik
>>> tors 30 nov. 2017 kl. 14:35 skrev kraythe :
>>>
 Greetings, 

 We have an actor structure where there is a cluster sharded actor that 
 calculates a parametric matrix of about 7 meg and has to distribute it to 
 5 
 other nodes for consumption. First of all there are the following 
 constraints. 

 1. The matrix has to be generated in one place.
 2. The matrix is needed on all nodes to handle user load. 
 3. The matrix will be periodically generated again to handle changing 
 variables and then sent off to all nodes to handle user load. 
 4. Saving the matrix in a database is probably not viable as it merely 
 shifts the networking load and the database would get very large very 
 fast. 
 The Database only saves the input params. 

 We changed the akka max message size to 10 meg to accomplish this but 
 that feels a bit odd, but we didn't see another choice. Normally this 
 works 
 fine even though passing 10 meg messages around a DistributedPubSub seems 
 odd to me. However on startup the system has to stat up 2000 of these all 
 at once. As a result the sharding coordinators scream at us about buffered 
 messages. It eventually calms down and life resumes but I would love to be 
 able to do this without the bloodbath in the logs. 

 Can someone recommend an alternative strategy for handling the 
 distribution of the parametric matrix that gets it to every node but 
 doesn't cause shard coordinator complaint bloodbath? 

 Thanks in advance. 
 Robert SImmons Jr. MSc.

 -- 
 >> Read the docs: http://akka.io/docs/
 >> Check the FAQ: 
 http://doc.akka.io/docs/akka/current/additional/faq.html
 >> Search the archives: 
 https://groups.google.com/group/akka-user
 --- 
 You received this message because you are subscribed to the Google 
 Groups "Akka User List" group.
 To unsubscribe from this group and stop receiving emails from it, send 
 an email to akka-user+...@googlegroups.com.
 To post to this group, send email to akka...@googlegroups.com.
 Visit this group at https://groups.google.com/group/akka-user.
 For more options, visit https://groups.google.com/d/optout.

>>>

-- 
>>  Read the docs: http://akka.io/docs/
>>  Check the FAQ: 
>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>  Search the archives: https://groups.google.com/group/akka-user
--- 
You received this message because you are subscribed to the Google Groups "Akka 
User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to akka-user+unsubscr...@googlegroups.com.
To post to this group, send email to akka-user@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

Re: [akka-user] Distributing lots of large messages to a cluster on startup issue.

[kraythe]

Well we did have plans to use kafka for much more. Distributing real life 
events to a dozen different apps for consumption seemed to be a good use 
case for Kafka.  But I will consider both. Is there any reason I SHOULDNT 
use Kafka other than the infrastructure work? 

-- Robert

On Friday, December 1, 2017 at 1:53:52 AM UTC-6, Patrik Nordwall wrote:
>
>
>
> On Fri, Dec 1, 2017 at 8:03 AM, kraythe > 
> wrote:
>
>> What about using something like Kafka to distribute the matrix?
>>
>
> Sounds like overkill (a lot of infra) if that is the only thing you are 
> using Kafka for. I think Roland's suggestion is excellent.
>  
>
>>
>>
>> On Thursday, November 30, 2017 at 12:54:59 PM UTC-6, rklaehn wrote:
>>>
>>> As Roland said, it is probably best to use some kind of side channel for 
>>> large blobs.
>>>
>>> One thing you could do that is a bit out of the left field is to run 
>>> ipfs  on each cluster node. Then you could save the 
>>> 10MB data block in ipfs and just link to the data from the message. The 
>>> actual data transfer works via ipfs, which is very efficient especially if 
>>> you want to broadcast some data in a private network. And the actor 
>>> communication does not get bogged down with huge blobs.
>>>
>>> A nice property of this approach is that the message remains immutable. 
>>> Whereas the data behind an url can change, the content you get for an ipfs 
>>> hash is always the same.
>>>
>>> Cheers,
>>>
>>> Rüdiger
>>>
>>> On Thu, Nov 30, 2017 at 7:37 PM Roland Kuhn  wrote:
>>>
 Yes, I meant an Akka HTTP server for this purpose: spin it up on the 
 node where the map is created, can even be on a random port, and send the 
 URL to this server around. That server can offer the last ten versions (or 
 whatever), could use the hash of the preserialized map as identifier. That 
 way you can explicitly manage the dissemination and its resource usage.

 Regards,

 Roland 

 Sent from my iPhone

 On 30. Nov 2017, at 18:43, kraythe  wrote:

 The issue here is that the matrices are recalculated quote frequently, 
 every minute is common so its not that it can be deployed to a static HTML 
 server. I suppose I could put an HTTP endpoint for server to server 
 communication and have that be outside of the Akka communication channels. 
 Is that what you are suggesting? I would have to know which node to send 
 the http request to of course and make sure the nodes can talk to each 
 other directly without using the load balancer. 

 -- Robert

 On Thursday, November 30, 2017 at 9:15:03 AM UTC-6, rkuhn wrote:
>
> May I suggest serving the large blobs via a different mechanisms like 
> HTTP? Sending around URL (and putting them inside events) seems much 
> better 
> than putting the data bytes all over the place.
>
> Regards,
>
> Roland
>
> 30 nov. 2017 kl. 16:11 skrev Patrik Nordwall :
>
> Sending 10 MB messages is indeed recipie for trouble. 
>
> I’d suggest to split the 10 MB into 100 smaller messages and collect, 
> reassembe, them on the receiving side.
>
> Perhaps you are fine with loosing such messages since they are 
> periodically published again. Otherwise you could use DistributedData to 
> share them.
>
> /Patrik
> tors 30 nov. 2017 kl. 14:35 skrev kraythe :
>
>> Greetings, 
>>
>> We have an actor structure where there is a cluster sharded actor 
>> that calculates a parametric matrix of about 7 meg and has to distribute 
>> it 
>> to 5 other nodes for consumption. First of all there are the following 
>> constraints. 
>>
>> 1. The matrix has to be generated in one place.
>> 2. The matrix is needed on all nodes to handle user load. 
>> 3. The matrix will be periodically generated again to handle changing 
>> variables and then sent off to all nodes to handle user load. 
>> 4. Saving the matrix in a database is probably not viable as it 
>> merely shifts the networking load and the database would get very large 
>> very fast. The Database only saves the input params. 
>>
>> We changed the akka max message size to 10 meg to accomplish this but 
>> that feels a bit odd, but we didn't see another choice. Normally this 
>> works 
>> fine even though passing 10 meg messages around a DistributedPubSub 
>> seems 
>> odd to me. However on startup the system has to stat up 2000 of these 
>> all 
>> at once. As a result the sharding coordinators scream at us about 
>> buffered 
>> messages. It eventually calms down and life resumes but I would love to 
>> be 
>> able to do this without the bloodbath in the logs. 
>>
>> Can someone recommend an alternative strategy for handling the 
>> distribution of the parametric matrix that gets it to every node but 
>> doesn't cause 

Re: [akka-user] Distributing lots of large messages to a cluster on startup issue.

[Patrik Nordwall]

On Fri, Dec 1, 2017 at 8:03 AM, kraythe  wrote:

> What about using something like Kafka to distribute the matrix?
>

Sounds like overkill (a lot of infra) if that is the only thing you are
using Kafka for. I think Roland's suggestion is excellent.


>
>
> On Thursday, November 30, 2017 at 12:54:59 PM UTC-6, rklaehn wrote:
>>
>> As Roland said, it is probably best to use some kind of side channel for
>> large blobs.
>>
>> One thing you could do that is a bit out of the left field is to run ipfs
>>  on each cluster node. Then you could save the 10MB
>> data block in ipfs and just link to the data from the message. The actual
>> data transfer works via ipfs, which is very efficient especially if you
>> want to broadcast some data in a private network. And the actor
>> communication does not get bogged down with huge blobs.
>>
>> A nice property of this approach is that the message remains immutable.
>> Whereas the data behind an url can change, the content you get for an ipfs
>> hash is always the same.
>>
>> Cheers,
>>
>> Rüdiger
>>
>> On Thu, Nov 30, 2017 at 7:37 PM Roland Kuhn  wrote:
>>
>>> Yes, I meant an Akka HTTP server for this purpose: spin it up on the
>>> node where the map is created, can even be on a random port, and send the
>>> URL to this server around. That server can offer the last ten versions (or
>>> whatever), could use the hash of the preserialized map as identifier. That
>>> way you can explicitly manage the dissemination and its resource usage.
>>>
>>> Regards,
>>>
>>> Roland
>>>
>>> Sent from my iPhone
>>>
>>> On 30. Nov 2017, at 18:43, kraythe  wrote:
>>>
>>> The issue here is that the matrices are recalculated quote frequently,
>>> every minute is common so its not that it can be deployed to a static HTML
>>> server. I suppose I could put an HTTP endpoint for server to server
>>> communication and have that be outside of the Akka communication channels.
>>> Is that what you are suggesting? I would have to know which node to send
>>> the http request to of course and make sure the nodes can talk to each
>>> other directly without using the load balancer.
>>>
>>> -- Robert
>>>
>>> On Thursday, November 30, 2017 at 9:15:03 AM UTC-6, rkuhn wrote:

 May I suggest serving the large blobs via a different mechanisms like
 HTTP? Sending around URL (and putting them inside events) seems much better
 than putting the data bytes all over the place.

 Regards,

 Roland

 30 nov. 2017 kl. 16:11 skrev Patrik Nordwall :

 Sending 10 MB messages is indeed recipie for trouble.

 I’d suggest to split the 10 MB into 100 smaller messages and collect,
 reassembe, them on the receiving side.

 Perhaps you are fine with loosing such messages since they are
 periodically published again. Otherwise you could use DistributedData to
 share them.

 /Patrik
 tors 30 nov. 2017 kl. 14:35 skrev kraythe :

> Greetings,
>
> We have an actor structure where there is a cluster sharded actor that
> calculates a parametric matrix of about 7 meg and has to distribute it to 
> 5
> other nodes for consumption. First of all there are the following
> constraints.
>
> 1. The matrix has to be generated in one place.
> 2. The matrix is needed on all nodes to handle user load.
> 3. The matrix will be periodically generated again to handle changing
> variables and then sent off to all nodes to handle user load.
> 4. Saving the matrix in a database is probably not viable as it merely
> shifts the networking load and the database would get very large very 
> fast.
> The Database only saves the input params.
>
> We changed the akka max message size to 10 meg to accomplish this but
> that feels a bit odd, but we didn't see another choice. Normally this 
> works
> fine even though passing 10 meg messages around a DistributedPubSub seems
> odd to me. However on startup the system has to stat up 2000 of these all
> at once. As a result the sharding coordinators scream at us about buffered
> messages. It eventually calms down and life resumes but I would love to be
> able to do this without the bloodbath in the logs.
>
> Can someone recommend an alternative strategy for handling the
> distribution of the parametric matrix that gets it to every node but
> doesn't cause shard coordinator complaint bloodbath?
>
> Thanks in advance.
> Robert SImmons Jr. MSc.
>
> --
> >> Read the docs: http://akka.io/docs/
> >> Check the FAQ: http://doc.akka.io/docs/akka/c
> urrent/additional/faq.html
> >> Search the archives: https://groups.google.com/grou
> p/akka-user
> ---
> You received this message because you are subscribed to the Google
> Groups "Akka User List" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an e

Re: [akka-user] Distributing lots of large messages to a cluster on startup issue.

[kraythe]

What about using something like Kafka to distribute the matrix?

On Thursday, November 30, 2017 at 12:54:59 PM UTC-6, rklaehn wrote:
>
> As Roland said, it is probably best to use some kind of side channel for 
> large blobs.
>
> One thing you could do that is a bit out of the left field is to run ipfs 
>  on each cluster node. Then you could save the 10MB 
> data block in ipfs and just link to the data from the message. The actual 
> data transfer works via ipfs, which is very efficient especially if you 
> want to broadcast some data in a private network. And the actor 
> communication does not get bogged down with huge blobs.
>
> A nice property of this approach is that the message remains immutable. 
> Whereas the data behind an url can change, the content you get for an ipfs 
> hash is always the same.
>
> Cheers,
>
> Rüdiger
>
> On Thu, Nov 30, 2017 at 7:37 PM Roland Kuhn  > wrote:
>
>> Yes, I meant an Akka HTTP server for this purpose: spin it up on the node 
>> where the map is created, can even be on a random port, and send the URL to 
>> this server around. That server can offer the last ten versions (or 
>> whatever), could use the hash of the preserialized map as identifier. That 
>> way you can explicitly manage the dissemination and its resource usage.
>>
>> Regards,
>>
>> Roland 
>>
>> Sent from my iPhone
>>
>> On 30. Nov 2017, at 18:43, kraythe > 
>> wrote:
>>
>> The issue here is that the matrices are recalculated quote frequently, 
>> every minute is common so its not that it can be deployed to a static HTML 
>> server. I suppose I could put an HTTP endpoint for server to server 
>> communication and have that be outside of the Akka communication channels. 
>> Is that what you are suggesting? I would have to know which node to send 
>> the http request to of course and make sure the nodes can talk to each 
>> other directly without using the load balancer. 
>>
>> -- Robert
>>
>> On Thursday, November 30, 2017 at 9:15:03 AM UTC-6, rkuhn wrote:
>>>
>>> May I suggest serving the large blobs via a different mechanisms like 
>>> HTTP? Sending around URL (and putting them inside events) seems much better 
>>> than putting the data bytes all over the place.
>>>
>>> Regards,
>>>
>>> Roland
>>>
>>> 30 nov. 2017 kl. 16:11 skrev Patrik Nordwall :
>>>
>>> Sending 10 MB messages is indeed recipie for trouble. 
>>>
>>> I’d suggest to split the 10 MB into 100 smaller messages and collect, 
>>> reassembe, them on the receiving side.
>>>
>>> Perhaps you are fine with loosing such messages since they are 
>>> periodically published again. Otherwise you could use DistributedData to 
>>> share them.
>>>
>>> /Patrik
>>> tors 30 nov. 2017 kl. 14:35 skrev kraythe :
>>>
 Greetings, 

 We have an actor structure where there is a cluster sharded actor that 
 calculates a parametric matrix of about 7 meg and has to distribute it to 
 5 
 other nodes for consumption. First of all there are the following 
 constraints. 

 1. The matrix has to be generated in one place.
 2. The matrix is needed on all nodes to handle user load. 
 3. The matrix will be periodically generated again to handle changing 
 variables and then sent off to all nodes to handle user load. 
 4. Saving the matrix in a database is probably not viable as it merely 
 shifts the networking load and the database would get very large very 
 fast. 
 The Database only saves the input params. 

 We changed the akka max message size to 10 meg to accomplish this but 
 that feels a bit odd, but we didn't see another choice. Normally this 
 works 
 fine even though passing 10 meg messages around a DistributedPubSub seems 
 odd to me. However on startup the system has to stat up 2000 of these all 
 at once. As a result the sharding coordinators scream at us about buffered 
 messages. It eventually calms down and life resumes but I would love to be 
 able to do this without the bloodbath in the logs. 

 Can someone recommend an alternative strategy for handling the 
 distribution of the parametric matrix that gets it to every node but 
 doesn't cause shard coordinator complaint bloodbath? 

 Thanks in advance. 
 Robert SImmons Jr. MSc.

 -- 
 >> Read the docs: http://akka.io/docs/
 >> Check the FAQ: 
 http://doc.akka.io/docs/akka/current/additional/faq.html
 >> Search the archives: 
 https://groups.google.com/group/akka-user
 --- 
 You received this message because you are subscribed to the Google 
 Groups "Akka User List" group.
 To unsubscribe from this group and stop receiving emails from it, send 
 an email to akka-user+...@googlegroups.com.
 To post to this group, send email to akka...@googlegroups.com.
 Visit this group at https://groups.google.com/group/akka-user.
 For more options, visit https://groups.google.com/d/optout.

Re: [akka-user] Distributing lots of large messages to a cluster on startup issue.

[Rüdiger Klaehn]

As Roland said, it is probably best to use some kind of side channel for
large blobs.

One thing you could do that is a bit out of the left field is to run ipfs
 on each cluster node. Then you could save the 10MB data
block in ipfs and just link to the data from the message. The actual data
transfer works via ipfs, which is very efficient especially if you want to
broadcast some data in a private network. And the actor communication does
not get bogged down with huge blobs.

A nice property of this approach is that the message remains immutable.
Whereas the data behind an url can change, the content you get for an ipfs
hash is always the same.

Cheers,

Rüdiger

On Thu, Nov 30, 2017 at 7:37 PM Roland Kuhn  wrote:

> Yes, I meant an Akka HTTP server for this purpose: spin it up on the node
> where the map is created, can even be on a random port, and send the URL to
> this server around. That server can offer the last ten versions (or
> whatever), could use the hash of the preserialized map as identifier. That
> way you can explicitly manage the dissemination and its resource usage.
>
> Regards,
>
> Roland
>
> Sent from my iPhone
>
> On 30. Nov 2017, at 18:43, kraythe  wrote:
>
> The issue here is that the matrices are recalculated quote frequently,
> every minute is common so its not that it can be deployed to a static HTML
> server. I suppose I could put an HTTP endpoint for server to server
> communication and have that be outside of the Akka communication channels.
> Is that what you are suggesting? I would have to know which node to send
> the http request to of course and make sure the nodes can talk to each
> other directly without using the load balancer.
>
> -- Robert
>
> On Thursday, November 30, 2017 at 9:15:03 AM UTC-6, rkuhn wrote:
>>
>> May I suggest serving the large blobs via a different mechanisms like
>> HTTP? Sending around URL (and putting them inside events) seems much better
>> than putting the data bytes all over the place.
>>
>> Regards,
>>
>> Roland
>>
>> 30 nov. 2017 kl. 16:11 skrev Patrik Nordwall :
>>
>> Sending 10 MB messages is indeed recipie for trouble.
>>
>> I’d suggest to split the 10 MB into 100 smaller messages and collect,
>> reassembe, them on the receiving side.
>>
>> Perhaps you are fine with loosing such messages since they are
>> periodically published again. Otherwise you could use DistributedData to
>> share them.
>>
>> /Patrik
>> tors 30 nov. 2017 kl. 14:35 skrev kraythe :
>>
>>> Greetings,
>>>
>>> We have an actor structure where there is a cluster sharded actor that
>>> calculates a parametric matrix of about 7 meg and has to distribute it to 5
>>> other nodes for consumption. First of all there are the following
>>> constraints.
>>>
>>> 1. The matrix has to be generated in one place.
>>> 2. The matrix is needed on all nodes to handle user load.
>>> 3. The matrix will be periodically generated again to handle changing
>>> variables and then sent off to all nodes to handle user load.
>>> 4. Saving the matrix in a database is probably not viable as it merely
>>> shifts the networking load and the database would get very large very fast.
>>> The Database only saves the input params.
>>>
>>> We changed the akka max message size to 10 meg to accomplish this but
>>> that feels a bit odd, but we didn't see another choice. Normally this works
>>> fine even though passing 10 meg messages around a DistributedPubSub seems
>>> odd to me. However on startup the system has to stat up 2000 of these all
>>> at once. As a result the sharding coordinators scream at us about buffered
>>> messages. It eventually calms down and life resumes but I would love to be
>>> able to do this without the bloodbath in the logs.
>>>
>>> Can someone recommend an alternative strategy for handling the
>>> distribution of the parametric matrix that gets it to every node but
>>> doesn't cause shard coordinator complaint bloodbath?
>>>
>>> Thanks in advance.
>>> Robert SImmons Jr. MSc.
>>>
>>> --
>>> >> Read the docs: http://akka.io/docs/
>>> >> Check the FAQ:
>>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>> >> Search the archives:
>>> https://groups.google.com/group/akka-user
>>> ---
>>> You received this message because you are subscribed to the Google
>>> Groups "Akka User List" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to akka-user+...@googlegroups.com.
>>> To post to this group, send email to akka...@googlegroups.com.
>>> Visit this group at https://groups.google.com/group/akka-user.
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>> --
>> >> Read the docs: http://akka.io/docs/
>> >> Check the FAQ:
>> http://doc.akka.io/docs/akka/current/additional/faq.html
>> >> Search the archives: https://groups.google.com/group/akka-user
>> ---
>> You received this message because you are subscribed to the Google Groups
>> "Akka User List" group.
>> To

Re: [akka-user] Distributing lots of large messages to a cluster on startup issue.

[Roland Kuhn]

Yes, I meant an Akka HTTP server for this purpose: spin it up on the node where 
the map is created, can even be on a random port, and send the URL to this 
server around. That server can offer the last ten versions (or whatever), could 
use the hash of the preserialized map as identifier. That way you can 
explicitly manage the dissemination and its resource usage.

Regards,

Roland 

Sent from my iPhone

> On 30. Nov 2017, at 18:43, kraythe  wrote:
> 
> The issue here is that the matrices are recalculated quote frequently, every 
> minute is common so its not that it can be deployed to a static HTML server. 
> I suppose I could put an HTTP endpoint for server to server communication and 
> have that be outside of the Akka communication channels. Is that what you are 
> suggesting? I would have to know which node to send the http request to of 
> course and make sure the nodes can talk to each other directly without using 
> the load balancer. 
> 
> -- Robert
> 
>> On Thursday, November 30, 2017 at 9:15:03 AM UTC-6, rkuhn wrote:
>> May I suggest serving the large blobs via a different mechanisms like HTTP? 
>> Sending around URL (and putting them inside events) seems much better than 
>> putting the data bytes all over the place.
>> 
>> Regards,
>> 
>> Roland
>> 
>>> 30 nov. 2017 kl. 16:11 skrev Patrik Nordwall :
>>> 
>>> Sending 10 MB messages is indeed recipie for trouble. 
>>> 
>>> I’d suggest to split the 10 MB into 100 smaller messages and collect, 
>>> reassembe, them on the receiving side.
>>> 
>>> Perhaps you are fine with loosing such messages since they are periodically 
>>> published again. Otherwise you could use DistributedData to share them.
>>> 
>>> /Patrik
 tors 30 nov. 2017 kl. 14:35 skrev kraythe :
 Greetings, 
 
 We have an actor structure where there is a cluster sharded actor that 
 calculates a parametric matrix of about 7 meg and has to distribute it to 
 5 other nodes for consumption. First of all there are the following 
 constraints. 
 
 1. The matrix has to be generated in one place.
 2. The matrix is needed on all nodes to handle user load. 
 3. The matrix will be periodically generated again to handle changing 
 variables and then sent off to all nodes to handle user load. 
 4. Saving the matrix in a database is probably not viable as it merely 
 shifts the networking load and the database would get very large very 
 fast. The Database only saves the input params. 
 
 We changed the akka max message size to 10 meg to accomplish this but that 
 feels a bit odd, but we didn't see another choice. Normally this works 
 fine even though passing 10 meg messages around a DistributedPubSub seems 
 odd to me. However on startup the system has to stat up 2000 of these all 
 at once. As a result the sharding coordinators scream at us about buffered 
 messages. It eventually calms down and life resumes but I would love to be 
 able to do this without the bloodbath in the logs. 
 
 Can someone recommend an alternative strategy for handling the 
 distribution of the parametric matrix that gets it to every node but 
 doesn't cause shard coordinator complaint bloodbath? 
 
 Thanks in advance. 
 Robert SImmons Jr. MSc.
 
 -- 
 >> Read the docs: http://akka.io/docs/
 >> Check the FAQ: 
 >> http://doc.akka.io/docs/akka/current/additional/faq.html
 >> Search the archives: https://groups.google.com/group/akka-user
 --- 
 You received this message because you are subscribed to the Google Groups 
 "Akka User List" group.
 To unsubscribe from this group and stop receiving emails from it, send an 
 email to akka-user+...@googlegroups.com.
 To post to this group, send email to akka...@googlegroups.com.
 Visit this group at https://groups.google.com/group/akka-user.
 For more options, visit https://groups.google.com/d/optout.
>>> 
>>> 
>>> -- 
>>> >> Read the docs: http://akka.io/docs/
>>> >> Check the FAQ: 
>>> >> http://doc.akka.io/docs/akka/current/additional/faq.html
>>> >> Search the archives: https://groups.google.com/group/akka-user
>>> --- 
>>> You received this message because you are subscribed to the Google Groups 
>>> "Akka User List" group.
>>> To unsubscribe from this group and stop receiving emails from it, send an 
>>> email to akka-user+...@googlegroups.com.
>>> To post to this group, send email to akka...@googlegroups.com.
>>> Visit this group at https://groups.google.com/group/akka-user.
>>> For more options, visit https://groups.google.com/d/optout.
>> 
> 
> -- 
> >> Read the docs: http://akka.io/docs/
> >> Check the FAQ: 
> >> http://doc.akka.io/docs/akka/current/additional/faq.html
> >> Search the archives: https://groups.google.com/group/akka-user
> --- 
> You received this message because you are subscribed to th

Re: [akka-user] Distributing lots of large messages to a cluster on startup issue.

[Rob Crawford]

Maybe a shared cache would be a better way to deal with it?

On Thursday, November 30, 2017 at 12:41:28 PM UTC-5, kraythe wrote:
>
> Yeah the problem is it cant be broken up, its sort of all or nothing. I 
> cant be too specific beyond it being a parametric matrix. Believe me I 
> would prefer not shipping it around. 
>
> On Thursday, November 30, 2017 at 9:11:51 AM UTC-6, Patrik Nordwall wrote:
>>
>> Sending 10 MB messages is indeed recipie for trouble. 
>>
>> I’d suggest to split the 10 MB into 100 smaller messages and collect, 
>> reassembe, them on the receiving side.
>>
>> Perhaps you are fine with loosing such messages since they are 
>> periodically published again. Otherwise you could use DistributedData to 
>> share them.
>>
>> /Patrik
>> tors 30 nov. 2017 kl. 14:35 skrev kraythe :
>>
>>> Greetings, 
>>>
>>> We have an actor structure where there is a cluster sharded actor that 
>>> calculates a parametric matrix of about 7 meg and has to distribute it to 5 
>>> other nodes for consumption. First of all there are the following 
>>> constraints. 
>>>
>>> 1. The matrix has to be generated in one place.
>>> 2. The matrix is needed on all nodes to handle user load. 
>>> 3. The matrix will be periodically generated again to handle changing 
>>> variables and then sent off to all nodes to handle user load. 
>>> 4. Saving the matrix in a database is probably not viable as it merely 
>>> shifts the networking load and the database would get very large very fast. 
>>> The Database only saves the input params. 
>>>
>>> We changed the akka max message size to 10 meg to accomplish this but 
>>> that feels a bit odd, but we didn't see another choice. Normally this works 
>>> fine even though passing 10 meg messages around a DistributedPubSub seems 
>>> odd to me. However on startup the system has to stat up 2000 of these all 
>>> at once. As a result the sharding coordinators scream at us about buffered 
>>> messages. It eventually calms down and life resumes but I would love to be 
>>> able to do this without the bloodbath in the logs. 
>>>
>>> Can someone recommend an alternative strategy for handling the 
>>> distribution of the parametric matrix that gets it to every node but 
>>> doesn't cause shard coordinator complaint bloodbath? 
>>>
>>> Thanks in advance. 
>>> Robert SImmons Jr. MSc.
>>>
>>> -- 
>>> >> Read the docs: http://akka.io/docs/
>>> >> Check the FAQ: 
>>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>> >> Search the archives: 
>>> https://groups.google.com/group/akka-user
>>> --- 
>>> You received this message because you are subscribed to the Google 
>>> Groups "Akka User List" group.
>>> To unsubscribe from this group and stop receiving emails from it, send 
>>> an email to akka-user+...@googlegroups.com.
>>> To post to this group, send email to akka...@googlegroups.com.
>>> Visit this group at https://groups.google.com/group/akka-user.
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>

-- 
>>  Read the docs: http://akka.io/docs/
>>  Check the FAQ: 
>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>  Search the archives: https://groups.google.com/group/akka-user
--- 
You received this message because you are subscribed to the Google Groups "Akka 
User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to akka-user+unsubscr...@googlegroups.com.
To post to this group, send email to akka-user@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

Re: [akka-user] Distributing lots of large messages to a cluster on startup issue.

[kraythe]

The issue here is that the matrices are recalculated quote frequently, 
every minute is common so its not that it can be deployed to a static HTML 
server. I suppose I could put an HTTP endpoint for server to server 
communication and have that be outside of the Akka communication channels. 
Is that what you are suggesting? I would have to know which node to send 
the http request to of course and make sure the nodes can talk to each 
other directly without using the load balancer. 

-- Robert

On Thursday, November 30, 2017 at 9:15:03 AM UTC-6, rkuhn wrote:
>
> May I suggest serving the large blobs via a different mechanisms like 
> HTTP? Sending around URL (and putting them inside events) seems much better 
> than putting the data bytes all over the place.
>
> Regards,
>
> Roland
>
> 30 nov. 2017 kl. 16:11 skrev Patrik Nordwall  >:
>
> Sending 10 MB messages is indeed recipie for trouble. 
>
> I’d suggest to split the 10 MB into 100 smaller messages and collect, 
> reassembe, them on the receiving side.
>
> Perhaps you are fine with loosing such messages since they are 
> periodically published again. Otherwise you could use DistributedData to 
> share them.
>
> /Patrik
> tors 30 nov. 2017 kl. 14:35 skrev kraythe 
> >:
>
>> Greetings, 
>>
>> We have an actor structure where there is a cluster sharded actor that 
>> calculates a parametric matrix of about 7 meg and has to distribute it to 5 
>> other nodes for consumption. First of all there are the following 
>> constraints. 
>>
>> 1. The matrix has to be generated in one place.
>> 2. The matrix is needed on all nodes to handle user load. 
>> 3. The matrix will be periodically generated again to handle changing 
>> variables and then sent off to all nodes to handle user load. 
>> 4. Saving the matrix in a database is probably not viable as it merely 
>> shifts the networking load and the database would get very large very fast. 
>> The Database only saves the input params. 
>>
>> We changed the akka max message size to 10 meg to accomplish this but 
>> that feels a bit odd, but we didn't see another choice. Normally this works 
>> fine even though passing 10 meg messages around a DistributedPubSub seems 
>> odd to me. However on startup the system has to stat up 2000 of these all 
>> at once. As a result the sharding coordinators scream at us about buffered 
>> messages. It eventually calms down and life resumes but I would love to be 
>> able to do this without the bloodbath in the logs. 
>>
>> Can someone recommend an alternative strategy for handling the 
>> distribution of the parametric matrix that gets it to every node but 
>> doesn't cause shard coordinator complaint bloodbath? 
>>
>> Thanks in advance. 
>> Robert SImmons Jr. MSc.
>>
>> -- 
>> >> Read the docs: http://akka.io/docs/
>> >> Check the FAQ: 
>> http://doc.akka.io/docs/akka/current/additional/faq.html
>> >> Search the archives: https://groups.google.com/group/akka-user
>> --- 
>> You received this message because you are subscribed to the Google Groups 
>> "Akka User List" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to akka-user+...@googlegroups.com .
>> To post to this group, send email to akka...@googlegroups.com 
>> .
>> Visit this group at https://groups.google.com/group/akka-user.
>> For more options, visit https://groups.google.com/d/optout.
>>
>
> -- 
> >> Read the docs: http://akka.io/docs/
> >> Check the FAQ: 
> http://doc.akka.io/docs/akka/current/additional/faq.html
> >> Search the archives: https://groups.google.com/group/akka-user
> --- 
> You received this message because you are subscribed to the Google Groups 
> "Akka User List" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to akka-user+...@googlegroups.com .
> To post to this group, send email to akka...@googlegroups.com 
> .
> Visit this group at https://groups.google.com/group/akka-user.
> For more options, visit https://groups.google.com/d/optout.
>
>
>

-- 
>>  Read the docs: http://akka.io/docs/
>>  Check the FAQ: 
>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>  Search the archives: https://groups.google.com/group/akka-user
--- 
You received this message because you are subscribed to the Google Groups "Akka 
User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to akka-user+unsubscr...@googlegroups.com.
To post to this group, send email to akka-user@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

Re: [akka-user] Distributing lots of large messages to a cluster on startup issue.

[kraythe]

Yeah the problem is it cant be broken up, its sort of all or nothing. I 
cant be too specific beyond it being a parametric matrix. Believe me I 
would prefer not shipping it around. 

On Thursday, November 30, 2017 at 9:11:51 AM UTC-6, Patrik Nordwall wrote:
>
> Sending 10 MB messages is indeed recipie for trouble. 
>
> I’d suggest to split the 10 MB into 100 smaller messages and collect, 
> reassembe, them on the receiving side.
>
> Perhaps you are fine with loosing such messages since they are 
> periodically published again. Otherwise you could use DistributedData to 
> share them.
>
> /Patrik
> tors 30 nov. 2017 kl. 14:35 skrev kraythe 
> >:
>
>> Greetings, 
>>
>> We have an actor structure where there is a cluster sharded actor that 
>> calculates a parametric matrix of about 7 meg and has to distribute it to 5 
>> other nodes for consumption. First of all there are the following 
>> constraints. 
>>
>> 1. The matrix has to be generated in one place.
>> 2. The matrix is needed on all nodes to handle user load. 
>> 3. The matrix will be periodically generated again to handle changing 
>> variables and then sent off to all nodes to handle user load. 
>> 4. Saving the matrix in a database is probably not viable as it merely 
>> shifts the networking load and the database would get very large very fast. 
>> The Database only saves the input params. 
>>
>> We changed the akka max message size to 10 meg to accomplish this but 
>> that feels a bit odd, but we didn't see another choice. Normally this works 
>> fine even though passing 10 meg messages around a DistributedPubSub seems 
>> odd to me. However on startup the system has to stat up 2000 of these all 
>> at once. As a result the sharding coordinators scream at us about buffered 
>> messages. It eventually calms down and life resumes but I would love to be 
>> able to do this without the bloodbath in the logs. 
>>
>> Can someone recommend an alternative strategy for handling the 
>> distribution of the parametric matrix that gets it to every node but 
>> doesn't cause shard coordinator complaint bloodbath? 
>>
>> Thanks in advance. 
>> Robert SImmons Jr. MSc.
>>
>> -- 
>> >> Read the docs: http://akka.io/docs/
>> >> Check the FAQ: 
>> http://doc.akka.io/docs/akka/current/additional/faq.html
>> >> Search the archives: https://groups.google.com/group/akka-user
>> --- 
>> You received this message because you are subscribed to the Google Groups 
>> "Akka User List" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to akka-user+...@googlegroups.com .
>> To post to this group, send email to akka...@googlegroups.com 
>> .
>> Visit this group at https://groups.google.com/group/akka-user.
>> For more options, visit https://groups.google.com/d/optout.
>>
>

-- 
>>  Read the docs: http://akka.io/docs/
>>  Check the FAQ: 
>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>  Search the archives: https://groups.google.com/group/akka-user
--- 
You received this message because you are subscribed to the Google Groups "Akka 
User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to akka-user+unsubscr...@googlegroups.com.
To post to this group, send email to akka-user@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

Re: [akka-user] Distributing lots of large messages to a cluster on startup issue.

[Roland Kuhn]

May I suggest serving the large blobs via a different mechanisms like HTTP? 
Sending around URL (and putting them inside events) seems much better than 
putting the data bytes all over the place.

Regards,

Roland

> 30 nov. 2017 kl. 16:11 skrev Patrik Nordwall :
> 
> Sending 10 MB messages is indeed recipie for trouble. 
> 
> I’d suggest to split the 10 MB into 100 smaller messages and collect, 
> reassembe, them on the receiving side.
> 
> Perhaps you are fine with loosing such messages since they are periodically 
> published again. Otherwise you could use DistributedData to share them.
> 
> /Patrik
> tors 30 nov. 2017 kl. 14:35 skrev kraythe  >:
> Greetings, 
> 
> We have an actor structure where there is a cluster sharded actor that 
> calculates a parametric matrix of about 7 meg and has to distribute it to 5 
> other nodes for consumption. First of all there are the following 
> constraints. 
> 
> 1. The matrix has to be generated in one place.
> 2. The matrix is needed on all nodes to handle user load. 
> 3. The matrix will be periodically generated again to handle changing 
> variables and then sent off to all nodes to handle user load. 
> 4. Saving the matrix in a database is probably not viable as it merely shifts 
> the networking load and the database would get very large very fast. The 
> Database only saves the input params. 
> 
> We changed the akka max message size to 10 meg to accomplish this but that 
> feels a bit odd, but we didn't see another choice. Normally this works fine 
> even though passing 10 meg messages around a DistributedPubSub seems odd to 
> me. However on startup the system has to stat up 2000 of these all at once. 
> As a result the sharding coordinators scream at us about buffered messages. 
> It eventually calms down and life resumes but I would love to be able to do 
> this without the bloodbath in the logs. 
> 
> Can someone recommend an alternative strategy for handling the distribution 
> of the parametric matrix that gets it to every node but doesn't cause shard 
> coordinator complaint bloodbath? 
> 
> Thanks in advance. 
> Robert SImmons Jr. MSc.
> 
> -- 
> >> Read the docs: http://akka.io/docs/ 
> >> Check the FAQ: 
> >> http://doc.akka.io/docs/akka/current/additional/faq.html 
> >> 
> >> Search the archives: https://groups.google.com/group/akka-user 
> >> 
> --- 
> You received this message because you are subscribed to the Google Groups 
> "Akka User List" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to akka-user+unsubscr...@googlegroups.com 
> .
> To post to this group, send email to akka-user@googlegroups.com 
> .
> Visit this group at https://groups.google.com/group/akka-user 
> .
> For more options, visit https://groups.google.com/d/optout 
> .
> 
> -- 
> >> Read the docs: http://akka.io/docs/ 
> >> Check the FAQ: 
> >> http://doc.akka.io/docs/akka/current/additional/faq.html 
> >> 
> >> Search the archives: https://groups.google.com/group/akka-user 
> >> 
> --- 
> You received this message because you are subscribed to the Google Groups 
> "Akka User List" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to akka-user+unsubscr...@googlegroups.com 
> .
> To post to this group, send email to akka-user@googlegroups.com 
> .
> Visit this group at https://groups.google.com/group/akka-user 
> .
> For more options, visit https://groups.google.com/d/optout 
> .

-- 
>>  Read the docs: http://akka.io/docs/
>>  Check the FAQ: 
>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>  Search the archives: https://groups.google.com/group/akka-user
--- 
You received this message because you are subscribed to the Google Groups "Akka 
User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to akka-user+unsubscr...@googlegroups.com.
To post to this group, send email to akka-user@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

Re: [akka-user] Distributing lots of large messages to a cluster on startup issue.

[Patrik Nordwall]

Sending 10 MB messages is indeed recipie for trouble.

I’d suggest to split the 10 MB into 100 smaller messages and collect,
reassembe, them on the receiving side.

Perhaps you are fine with loosing such messages since they are periodically
published again. Otherwise you could use DistributedData to share them.

/Patrik
tors 30 nov. 2017 kl. 14:35 skrev kraythe :

> Greetings,
>
> We have an actor structure where there is a cluster sharded actor that
> calculates a parametric matrix of about 7 meg and has to distribute it to 5
> other nodes for consumption. First of all there are the following
> constraints.
>
> 1. The matrix has to be generated in one place.
> 2. The matrix is needed on all nodes to handle user load.
> 3. The matrix will be periodically generated again to handle changing
> variables and then sent off to all nodes to handle user load.
> 4. Saving the matrix in a database is probably not viable as it merely
> shifts the networking load and the database would get very large very fast.
> The Database only saves the input params.
>
> We changed the akka max message size to 10 meg to accomplish this but that
> feels a bit odd, but we didn't see another choice. Normally this works fine
> even though passing 10 meg messages around a DistributedPubSub seems odd to
> me. However on startup the system has to stat up 2000 of these all at once.
> As a result the sharding coordinators scream at us about buffered messages.
> It eventually calms down and life resumes but I would love to be able to do
> this without the bloodbath in the logs.
>
> Can someone recommend an alternative strategy for handling the
> distribution of the parametric matrix that gets it to every node but
> doesn't cause shard coordinator complaint bloodbath?
>
> Thanks in advance.
> Robert SImmons Jr. MSc.
>
> --
> >> Read the docs: http://akka.io/docs/
> >> Check the FAQ:
> http://doc.akka.io/docs/akka/current/additional/faq.html
> >> Search the archives: https://groups.google.com/group/akka-user
> ---
> You received this message because you are subscribed to the Google Groups
> "Akka User List" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to akka-user+unsubscr...@googlegroups.com.
> To post to this group, send email to akka-user@googlegroups.com.
> Visit this group at https://groups.google.com/group/akka-user.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
>>  Read the docs: http://akka.io/docs/
>>  Check the FAQ: 
>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>  Search the archives: https://groups.google.com/group/akka-user
--- 
You received this message because you are subscribed to the Google Groups "Akka 
User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to akka-user+unsubscr...@googlegroups.com.
To post to this group, send email to akka-user@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.