Cool, I'll try out doing the leave request from one of the stable members. I have updated this <https://github.com/akka/akka/issues/13226> ticket.
On Mon, Jul 21, 2014 at 12:43 PM, Patrik Nordwall <[email protected] > wrote: > > > > 21 jul 2014 kl. 06:45 skrev shikhar <[email protected]>: > > Thanks for the input Patrik. > > I've confirmed that the issue wasn't anything in the application by > isolating it in a simple project > <https://github.com/shikhar/akka-cluster-churn>, see Churn.scala > <https://github.com/shikhar/akka-cluster-churn/blob/master/src/main/scala/Churn.scala>. > I set the payload size limit to 32k in the hope of seeing the issue sooner. > It happened after a churn of ~750 nodes. > > I think for now I'll bump the limit a notch in my actual project. > > Really looking forward to pruning for 2.4, unbounded growth in gossip size > is pretty undesirable > > > I agree. That must be fixed. > I still think you can make it last much longer if you don't perform state > changes from the transient nodes. > > Thanks for verifying, please add a note to the pruning ticket. > /patrik > > > On Sun, Jul 20, 2014 at 7:18 PM, Patrik Nordwall < > [email protected]> wrote: > >> Yes, it can be because of the many join/leave cycles and lack of vector >> clock pruning. >> >> I agree that it is strange that the cluster is working after that, >> because the gossip message size will not shrink. >> >> The vector clocks only increase in size if the cluster membership state >> is modified by different nodes, so it would be interesting to understand >> how your transient nodes contribute to such changes. Which node performs >> the leave action? If that is done by a transient node, you could perhaps >> perform that from one of the stable nodes instead. >> >> Pruning is on our radar for 2.4. >> >> /Patrik >> >> 20 jul 2014 kl. 08:55 skrev shikhar <[email protected]>: >> >> After the cluster has been up for a longish period (> a day or so), and >> only then, I'm seeing: >> >> [2014-07-20 00:00:00,708][ERROR][akka.remote.EndpointWriter] >> [akka.tcp://[email protected]:ZZZZ/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2FXXXX%40YYYY%3AZZZZ-16/endpointWriter] >> Transient association error (association remains live) >> akka.remote.OversizedPayloadException: Discarding oversized payload sent >> to Actor[akka.tcp://XXXX@YYYY:ZZZZ/]: max allowed size 128000 bytes, >> actual size of encoded class akka.actor.ActorSelectionMessage was 147704 >> bytes. >> >> logged every second, for message sends to different cluster members. >> >> This is on a 21 node cluster, however there is one peculiarity that some >> some processes join the cluster via cron jobs (2 of them every minute) and >> exit shortly. >> >> Despite this OversizedPayloadException logging, the cluster seems to be >> functioning normally otherwise. New members are joining and leaving. I just >> shutdown the cluster singleton and failover happened smoothly. >> >> The number of bytes on that actor selection message keeps growing, BTW. >> It had gone from 147K to 178K in about 6 hours. >> >> I understand there is a setting for the payload size limit, " >> akka.remote.netty.tcp.maximum-frame-size", but before bumping that I'd >> like to understand what is going on here. >> >> Could this be a consequence of the fact that akka cluster does not >> perform vector clock pruning, and I have nodes constantly joining and >> leaving? If so, it's strange that the cluster can keep functioning despite >> what may be failed gossip sends? >> >> I don't think this is anything at the application-level, the messages >> sent using selections are quite small. >> >> -- >> >>>>>>>>>> Read the docs: http://akka.io/docs/ >> >>>>>>>>>> Check the FAQ: >> http://doc.akka.io/docs/akka/current/additional/faq.html >> >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user >> --- >> You received this message because you are subscribed to the Google Groups >> "Akka User List" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To post to this group, send email to [email protected]. >> Visit this group at http://groups.google.com/group/akka-user. >> For more options, visit https://groups.google.com/d/optout. >> >> -- >> >>>>>>>>>> Read the docs: http://akka.io/docs/ >> >>>>>>>>>> Check the FAQ: >> http://doc.akka.io/docs/akka/current/additional/faq.html >> >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user >> --- >> You received this message because you are subscribed to the Google Groups >> "Akka User List" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To post to this group, send email to [email protected]. >> Visit this group at http://groups.google.com/group/akka-user. >> For more options, visit https://groups.google.com/d/optout. >> > > -- > >>>>>>>>>> Read the docs: http://akka.io/docs/ > >>>>>>>>>> Check the FAQ: > http://doc.akka.io/docs/akka/current/additional/faq.html > >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user > --- > You received this message because you are subscribed to the Google Groups > "Akka User List" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > Visit this group at http://groups.google.com/group/akka-user. > For more options, visit https://groups.google.com/d/optout. > > -- > >>>>>>>>>> Read the docs: http://akka.io/docs/ > >>>>>>>>>> Check the FAQ: > http://doc.akka.io/docs/akka/current/additional/faq.html > >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user > --- > You received this message because you are subscribed to the Google Groups > "Akka User List" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > Visit this group at http://groups.google.com/group/akka-user. > For more options, visit https://groups.google.com/d/optout. > -- >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>>>> Check the FAQ: >>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups "Akka User List" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout.
