Re: [akka-user] Nodes becomes unreachable due to serialization-issues with one single message

[Patrik Nordwall]

Thanks for reporting., Fixed, will be in upcoming 2.4.10

/Patrik
tors 1 sep. 2016 kl. 20:58 skrev Moax76 :

> Issue created: https://github.com/akka/akka/issues/21343
>
> But I'll modify my serializer anyway.
>
> Thanks.
>
> -Morten
>
>
> On Thursday, September 1, 2016 at 8:33:04 PM UTC+2, Patrik Nordwall wrote:
>
>> I think we should handle exceptions from serializer.toBinary as transient.
>> Can you please create an issue.  I
>> can understand that the reason might be that it is a program bug and the
>> system should probably not run with the bug, but I think error logging
>> should be enough since the exception doesn't result in any undefined state.
>>
>> Deserialization is a another story, which is exhausted in that ticket
>> Martynas linked to.
>>
>
>> On Thu, Sep 1, 2016 at 7:46 PM, Moax76  wrote:
>>
>>> Thanks a lot for the info.
>>>
>>> I'll fix my serializer so that it wraps all exceptions
>>> in NotSerializableException.
>>> Am I understanding it correct that I should only do this wrapping when
>>> serializing, and not deserializing?
>>>
>>> Thanks again.
>>>
>>> -Morten
>>>
>>> On Thursday, September 1, 2016 at 4:14:43 PM UTC+2, drewhk wrote:

 The only reason serialization errors cause a connection to be dropped
 if the *deserialization* fails. This is because TCP is a streaming protocol
 and we cannot decide why the deserialization failed, because of some bug,
 or the TCP framing is out of sync. We just drop the connection to start
 from a clean state.

 -Endre

 On Thu, Sep 1, 2016 at 3:33 PM, Martynas Mickevičius <
 martynas.m...@lightbend.com> wrote:

> EndpointWriter is only treating NotSerializableException as a
> transient one. So wrapping all of the exceptions from your serializer
> into NotSerializableException will not get the association dropped.
>
> Here is the discussion on having all of the exceptions transient:
> https://github.com/akka/akka/issues/20641
>
> On Wed, Aug 31, 2016 at 4:56 PM, Viktor Klang 
> wrote:
>
>> Well, I agree that it shouldn't gate the connection due to a failure
>> to serialize.
>>
>> Do you have a minimal reproducer to share?
>>
>> On Wed, Aug 31, 2016 at 3:48 PM, Moax76  wrote:
>>
>>>
>>>
>>> On Wednesday, August 31, 2016 at 3:37:41 PM UTC+2, √ wrote:

 What happens if you fix the serializer?



>>> If I fix the serializer, then the problem off course goes away, but
>>> that is not the point.
>>>
>>> The problem is that instead of dropping the message that (for some
>>> 'unknown' reason) failed to be serialized, it "drops" the connection, 
>>> and
>>> GATES it so that it will not try to reconnect in 5 seconds.
>>>
>>> If it worked as described above, this serializing issue would be a
>>> regular bug the only effected parts of the application, instead of the
>>> entire application going "down" due to cluster-issues.
>>>
>>>
>>> -Morten
>>>
>>> --
>>> >> Read the docs: http://akka.io/docs/
>>> >> Check the FAQ:
>>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>> >> Search the archives:
>>> https://groups.google.com/group/akka-user
>>> ---
>>> You received this message because you are subscribed to the Google
>>> Groups "Akka User List" group.
>>> To unsubscribe from this group and stop receiving emails from it,
>>> send an email to akka-user+...@googlegroups.com.
>>> To post to this group, send email to akka...@googlegroups.com.
>>> Visit this group at https://groups.google.com/group/akka-user.
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>>
>>
>> --
>> Cheers,
>> √
>>
>> --
>> >> Read the docs: http://akka.io/docs/
>> >> Check the FAQ:
>> http://doc.akka.io/docs/akka/current/additional/faq.html
>> >> Search the archives:
>> https://groups.google.com/group/akka-user
>> ---
>> You received this message because you are subscribed to the Google
>> Groups "Akka User List" group.
>> To unsubscribe from this group and stop receiving emails from it,
>> send an email to akka-user+...@googlegroups.com.
>> To post to this group, send email to akka...@googlegroups.com.
>> Visit this group at https://groups.google.com/group/akka-user.
>> For more options, visit https://groups.google.com/d/optout.
>>
>
> --
> >> Read the docs: http://akka.io/docs/
> >> Check the FAQ:
> http://doc.akka.io/docs/akka/current/additional/faq.html
> >> Search the archives:
> https://groups.google.com/group/akka-user
> ---
> You received this message because you are subscribed to the Google
> Groups "Akka User List" group.
> To un

Re: [akka-user] Nodes becomes unreachable due to serialization-issues with one single message

[Moax76]

Issue created: https://github.com/akka/akka/issues/21343

But I'll modify my serializer anyway.

Thanks.

-Morten

On Thursday, September 1, 2016 at 8:33:04 PM UTC+2, Patrik Nordwall wrote:
>
> I think we should handle exceptions from serializer.toBinary as transient. 
> Can you please create an issue.  I 
> can understand that the reason might be that it is a program bug and the 
> system should probably not run with the bug, but I think error logging 
> should be enough since the exception doesn't result in any undefined state.
>
> Deserialization is a another story, which is exhausted in that ticket 
> Martynas linked to.
>
> On Thu, Sep 1, 2016 at 7:46 PM, Moax76 
> > wrote:
>
>> Thanks a lot for the info.
>>
>> I'll fix my serializer so that it wraps all exceptions 
>> in NotSerializableException.
>> Am I understanding it correct that I should only do this wrapping when 
>> serializing, and not deserializing?
>>
>> Thanks again.
>>
>> -Morten
>>
>> On Thursday, September 1, 2016 at 4:14:43 PM UTC+2, drewhk wrote:
>>>
>>> The only reason serialization errors cause a connection to be dropped if 
>>> the *deserialization* fails. This is because TCP is a streaming protocol 
>>> and we cannot decide why the deserialization failed, because of some bug, 
>>> or the TCP framing is out of sync. We just drop the connection to start 
>>> from a clean state. 
>>>
>>> -Endre
>>>
>>> On Thu, Sep 1, 2016 at 3:33 PM, Martynas Mickevičius <
>>> martynas.m...@lightbend.com> wrote:
>>>
 EndpointWriter is only treating NotSerializableException as a transient 
 one. So wrapping all of the exceptions from your serializer 
 into NotSerializableException will not get the association dropped.

 Here is the discussion on having all of the exceptions transient: 
 https://github.com/akka/akka/issues/20641

 On Wed, Aug 31, 2016 at 4:56 PM, Viktor Klang  
 wrote:

> Well, I agree that it shouldn't gate the connection due to a failure 
> to serialize.
>
> Do you have a minimal reproducer to share?
>
> On Wed, Aug 31, 2016 at 3:48 PM, Moax76  wrote:
>
>>
>>
>> On Wednesday, August 31, 2016 at 3:37:41 PM UTC+2, √ wrote:
>>>
>>> What happens if you fix the serializer?
>>>
>>>
>>>
>> If I fix the serializer, then the problem off course goes away, but 
>> that is not the point.
>>
>> The problem is that instead of dropping the message that (for some 
>> 'unknown' reason) failed to be serialized, it "drops" the connection, 
>> and 
>> GATES it so that it will not try to reconnect in 5 seconds.
>>
>> If it worked as described above, this serializing issue would be a 
>> regular bug the only effected parts of the application, instead of the 
>> entire application going "down" due to cluster-issues.
>>
>>
>> -Morten
>>
>> -- 
>> >> Read the docs: http://akka.io/docs/
>> >> Check the FAQ: 
>> http://doc.akka.io/docs/akka/current/additional/faq.html
>> >> Search the archives: 
>> https://groups.google.com/group/akka-user
>> --- 
>> You received this message because you are subscribed to the Google 
>> Groups "Akka User List" group.
>> To unsubscribe from this group and stop receiving emails from it, 
>> send an email to akka-user+...@googlegroups.com.
>> To post to this group, send email to akka...@googlegroups.com.
>> Visit this group at https://groups.google.com/group/akka-user.
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>
>
> -- 
> Cheers,
> √
>
> -- 
> >> Read the docs: http://akka.io/docs/
> >> Check the FAQ: 
> http://doc.akka.io/docs/akka/current/additional/faq.html
> >> Search the archives: 
> https://groups.google.com/group/akka-user
> --- 
> You received this message because you are subscribed to the Google 
> Groups "Akka User List" group.
> To unsubscribe from this group and stop receiving emails from it, send 
> an email to akka-user+...@googlegroups.com.
> To post to this group, send email to akka...@googlegroups.com.
> Visit this group at https://groups.google.com/group/akka-user.
> For more options, visit https://groups.google.com/d/optout.
>

 -- 
 >> Read the docs: http://akka.io/docs/
 >> Check the FAQ: 
 http://doc.akka.io/docs/akka/current/additional/faq.html
 >> Search the archives: 
 https://groups.google.com/group/akka-user
 --- 
 You received this message because you are subscribed to the Google 
 Groups "Akka User List" group.
 To unsubscribe from this group and stop receiving emails from it, send 
 an email to akka-user+...@googlegroups.com.
 To post to this group, send email to akka...@googlegroups.com.
 Visit this group 

Re: [akka-user] Nodes becomes unreachable due to serialization-issues with one single message

[Patrik Nordwall]

I think we should handle exceptions from serializer.toBinary as transient.
Can you please create an issue.  I
can understand that the reason might be that it is a program bug and the
system should probably not run with the bug, but I think error logging
should be enough since the exception doesn't result in any undefined state.

Deserialization is a another story, which is exhausted in that ticket
Martynas linked to.

On Thu, Sep 1, 2016 at 7:46 PM, Moax76  wrote:

> Thanks a lot for the info.
>
> I'll fix my serializer so that it wraps all exceptions
> in NotSerializableException.
> Am I understanding it correct that I should only do this wrapping when
> serializing, and not deserializing?
>
> Thanks again.
>
> -Morten
>
> On Thursday, September 1, 2016 at 4:14:43 PM UTC+2, drewhk wrote:
>>
>> The only reason serialization errors cause a connection to be dropped if
>> the *deserialization* fails. This is because TCP is a streaming protocol
>> and we cannot decide why the deserialization failed, because of some bug,
>> or the TCP framing is out of sync. We just drop the connection to start
>> from a clean state.
>>
>> -Endre
>>
>> On Thu, Sep 1, 2016 at 3:33 PM, Martynas Mickevičius <
>> martynas.m...@lightbend.com> wrote:
>>
>>> EndpointWriter is only treating NotSerializableException as a transient
>>> one. So wrapping all of the exceptions from your serializer
>>> into NotSerializableException will not get the association dropped.
>>>
>>> Here is the discussion on having all of the exceptions transient:
>>> https://github.com/akka/akka/issues/20641
>>>
>>> On Wed, Aug 31, 2016 at 4:56 PM, Viktor Klang 
>>> wrote:
>>>
 Well, I agree that it shouldn't gate the connection due to a failure to
 serialize.

 Do you have a minimal reproducer to share?

 On Wed, Aug 31, 2016 at 3:48 PM, Moax76  wrote:

>
>
> On Wednesday, August 31, 2016 at 3:37:41 PM UTC+2, √ wrote:
>>
>> What happens if you fix the serializer?
>>
>>
>>
> If I fix the serializer, then the problem off course goes away, but
> that is not the point.
>
> The problem is that instead of dropping the message that (for some
> 'unknown' reason) failed to be serialized, it "drops" the connection, and
> GATES it so that it will not try to reconnect in 5 seconds.
>
> If it worked as described above, this serializing issue would be a
> regular bug the only effected parts of the application, instead of the
> entire application going "down" due to cluster-issues.
>
>
> -Morten
>
> --
> >> Read the docs: http://akka.io/docs/
> >> Check the FAQ: http://doc.akka.io/docs/akka/c
> urrent/additional/faq.html
> >> Search the archives: https://groups.google.com/grou
> p/akka-user
> ---
> You received this message because you are subscribed to the Google
> Groups "Akka User List" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to akka-user+...@googlegroups.com.
> To post to this group, send email to akka...@googlegroups.com.
> Visit this group at https://groups.google.com/group/akka-user.
> For more options, visit https://groups.google.com/d/optout.
>



 --
 Cheers,
 √

 --
 >> Read the docs: http://akka.io/docs/
 >> Check the FAQ: http://doc.akka.io/docs/akka/c
 urrent/additional/faq.html
 >> Search the archives: https://groups.google.com/grou
 p/akka-user
 ---
 You received this message because you are subscribed to the Google
 Groups "Akka User List" group.
 To unsubscribe from this group and stop receiving emails from it, send
 an email to akka-user+...@googlegroups.com.
 To post to this group, send email to akka...@googlegroups.com.
 Visit this group at https://groups.google.com/group/akka-user.
 For more options, visit https://groups.google.com/d/optout.

>>>
>>> --
>>> >> Read the docs: http://akka.io/docs/
>>> >> Check the FAQ: http://doc.akka.io/docs/akka/c
>>> urrent/additional/faq.html
>>> >> Search the archives: https://groups.google.com/grou
>>> p/akka-user
>>> ---
>>> You received this message because you are subscribed to the Google
>>> Groups "Akka User List" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to akka-user+...@googlegroups.com.
>>> To post to this group, send email to akka...@googlegroups.com.
>>> Visit this group at https://groups.google.com/group/akka-user.
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>> --
> >> Read the docs: http://akka.io/docs/
> >> Check the FAQ: http://doc.akka.io/docs/akka/
> current/additional/faq.html
> >> Search the archives: https://groups.google.com/group/akka-user
> ---
> You received this message be

Re: [akka-user] Nodes becomes unreachable due to serialization-issues with one single message

[Moax76]

Thanks a lot for the info.

I'll fix my serializer so that it wraps all exceptions 
in NotSerializableException.
Am I understanding it correct that I should only do this wrapping when 
serializing, and not deserializing?

Thanks again.

-Morten

On Thursday, September 1, 2016 at 4:14:43 PM UTC+2, drewhk wrote:
>
> The only reason serialization errors cause a connection to be dropped if 
> the *deserialization* fails. This is because TCP is a streaming protocol 
> and we cannot decide why the deserialization failed, because of some bug, 
> or the TCP framing is out of sync. We just drop the connection to start 
> from a clean state. 
>
> -Endre
>
> On Thu, Sep 1, 2016 at 3:33 PM, Martynas Mickevičius <
> martynas.m...@lightbend.com > wrote:
>
>> EndpointWriter is only treating NotSerializableException as a transient 
>> one. So wrapping all of the exceptions from your serializer 
>> into NotSerializableException will not get the association dropped.
>>
>> Here is the discussion on having all of the exceptions transient: 
>> https://github.com/akka/akka/issues/20641
>>
>> On Wed, Aug 31, 2016 at 4:56 PM, Viktor Klang > > wrote:
>>
>>> Well, I agree that it shouldn't gate the connection due to a failure to 
>>> serialize.
>>>
>>> Do you have a minimal reproducer to share?
>>>
>>> On Wed, Aug 31, 2016 at 3:48 PM, Moax76 >> > wrote:
>>>


 On Wednesday, August 31, 2016 at 3:37:41 PM UTC+2, √ wrote:
>
> What happens if you fix the serializer?
>
>
>
 If I fix the serializer, then the problem off course goes away, but 
 that is not the point.

 The problem is that instead of dropping the message that (for some 
 'unknown' reason) failed to be serialized, it "drops" the connection, and 
 GATES it so that it will not try to reconnect in 5 seconds.

 If it worked as described above, this serializing issue would be a 
 regular bug the only effected parts of the application, instead of the 
 entire application going "down" due to cluster-issues.


 -Morten

 -- 
 >> Read the docs: http://akka.io/docs/
 >> Check the FAQ: 
 http://doc.akka.io/docs/akka/current/additional/faq.html
 >> Search the archives: 
 https://groups.google.com/group/akka-user
 --- 
 You received this message because you are subscribed to the Google 
 Groups "Akka User List" group.
 To unsubscribe from this group and stop receiving emails from it, send 
 an email to akka-user+...@googlegroups.com .
 To post to this group, send email to akka...@googlegroups.com 
 .
 Visit this group at https://groups.google.com/group/akka-user.
 For more options, visit https://groups.google.com/d/optout.

>>>
>>>
>>>
>>> -- 
>>> Cheers,
>>> √
>>>
>>> -- 
>>> >> Read the docs: http://akka.io/docs/
>>> >> Check the FAQ: 
>>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>> >> Search the archives: 
>>> https://groups.google.com/group/akka-user
>>> --- 
>>> You received this message because you are subscribed to the Google 
>>> Groups "Akka User List" group.
>>> To unsubscribe from this group and stop receiving emails from it, send 
>>> an email to akka-user+...@googlegroups.com .
>>> To post to this group, send email to akka...@googlegroups.com 
>>> .
>>> Visit this group at https://groups.google.com/group/akka-user.
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>> -- 
>> >> Read the docs: http://akka.io/docs/
>> >> Check the FAQ: 
>> http://doc.akka.io/docs/akka/current/additional/faq.html
>> >> Search the archives: https://groups.google.com/group/akka-user
>> --- 
>> You received this message because you are subscribed to the Google Groups 
>> "Akka User List" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to akka-user+...@googlegroups.com .
>> To post to this group, send email to akka...@googlegroups.com 
>> .
>> Visit this group at https://groups.google.com/group/akka-user.
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>

-- 
>>  Read the docs: http://akka.io/docs/
>>  Check the FAQ: 
>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>  Search the archives: https://groups.google.com/group/akka-user
--- 
You received this message because you are subscribed to the Google Groups "Akka 
User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to akka-user+unsubscr...@googlegroups.com.
To post to this group, send email to akka-user@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

Re: [akka-user] Nodes becomes unreachable due to serialization-issues with one single message

[Endre Varga]

The only reason serialization errors cause a connection to be dropped if
the *deserialization* fails. This is because TCP is a streaming protocol
and we cannot decide why the deserialization failed, because of some bug,
or the TCP framing is out of sync. We just drop the connection to start
from a clean state.

-Endre

On Thu, Sep 1, 2016 at 3:33 PM, Martynas Mickevičius <
martynas.mickevic...@lightbend.com> wrote:

> EndpointWriter is only treating NotSerializableException as a transient
> one. So wrapping all of the exceptions from your serializer
> into NotSerializableException will not get the association dropped.
>
> Here is the discussion on having all of the exceptions transient:
> https://github.com/akka/akka/issues/20641
>
> On Wed, Aug 31, 2016 at 4:56 PM, Viktor Klang 
> wrote:
>
>> Well, I agree that it shouldn't gate the connection due to a failure to
>> serialize.
>>
>> Do you have a minimal reproducer to share?
>>
>> On Wed, Aug 31, 2016 at 3:48 PM, Moax76 
>> wrote:
>>
>>>
>>>
>>> On Wednesday, August 31, 2016 at 3:37:41 PM UTC+2, √ wrote:

 What happens if you fix the serializer?



>>> If I fix the serializer, then the problem off course goes away, but that
>>> is not the point.
>>>
>>> The problem is that instead of dropping the message that (for some
>>> 'unknown' reason) failed to be serialized, it "drops" the connection, and
>>> GATES it so that it will not try to reconnect in 5 seconds.
>>>
>>> If it worked as described above, this serializing issue would be a
>>> regular bug the only effected parts of the application, instead of the
>>> entire application going "down" due to cluster-issues.
>>>
>>>
>>> -Morten
>>>
>>> --
>>> >> Read the docs: http://akka.io/docs/
>>> >> Check the FAQ: http://doc.akka.io/docs/akka/c
>>> urrent/additional/faq.html
>>> >> Search the archives: https://groups.google.com/grou
>>> p/akka-user
>>> ---
>>> You received this message because you are subscribed to the Google
>>> Groups "Akka User List" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to akka-user+unsubscr...@googlegroups.com.
>>> To post to this group, send email to akka-user@googlegroups.com.
>>> Visit this group at https://groups.google.com/group/akka-user.
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>>
>>
>> --
>> Cheers,
>> √
>>
>> --
>> >> Read the docs: http://akka.io/docs/
>> >> Check the FAQ: http://doc.akka.io/docs/akka/c
>> urrent/additional/faq.html
>> >> Search the archives: https://groups.google.com/group/akka-user
>> ---
>> You received this message because you are subscribed to the Google Groups
>> "Akka User List" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to akka-user+unsubscr...@googlegroups.com.
>> To post to this group, send email to akka-user@googlegroups.com.
>> Visit this group at https://groups.google.com/group/akka-user.
>> For more options, visit https://groups.google.com/d/optout.
>>
>
> --
> >> Read the docs: http://akka.io/docs/
> >> Check the FAQ: http://doc.akka.io/docs/akka/
> current/additional/faq.html
> >> Search the archives: https://groups.google.com/group/akka-user
> ---
> You received this message because you are subscribed to the Google Groups
> "Akka User List" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to akka-user+unsubscr...@googlegroups.com.
> To post to this group, send email to akka-user@googlegroups.com.
> Visit this group at https://groups.google.com/group/akka-user.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
>>  Read the docs: http://akka.io/docs/
>>  Check the FAQ: 
>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>  Search the archives: https://groups.google.com/group/akka-user
--- 
You received this message because you are subscribed to the Google Groups "Akka 
User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to akka-user+unsubscr...@googlegroups.com.
To post to this group, send email to akka-user@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

Re: [akka-user] Nodes becomes unreachable due to serialization-issues with one single message

[Martynas Mickevičius]

EndpointWriter is only treating NotSerializableException as a transient
one. So wrapping all of the exceptions from your serializer
into NotSerializableException will not get the association dropped.

Here is the discussion on having all of the exceptions transient:
https://github.com/akka/akka/issues/20641

On Wed, Aug 31, 2016 at 4:56 PM, Viktor Klang 
wrote:

> Well, I agree that it shouldn't gate the connection due to a failure to
> serialize.
>
> Do you have a minimal reproducer to share?
>
> On Wed, Aug 31, 2016 at 3:48 PM, Moax76  wrote:
>
>>
>>
>> On Wednesday, August 31, 2016 at 3:37:41 PM UTC+2, √ wrote:
>>>
>>> What happens if you fix the serializer?
>>>
>>>
>>>
>> If I fix the serializer, then the problem off course goes away, but that
>> is not the point.
>>
>> The problem is that instead of dropping the message that (for some
>> 'unknown' reason) failed to be serialized, it "drops" the connection, and
>> GATES it so that it will not try to reconnect in 5 seconds.
>>
>> If it worked as described above, this serializing issue would be a
>> regular bug the only effected parts of the application, instead of the
>> entire application going "down" due to cluster-issues.
>>
>>
>> -Morten
>>
>> --
>> >> Read the docs: http://akka.io/docs/
>> >> Check the FAQ: http://doc.akka.io/docs/akka/c
>> urrent/additional/faq.html
>> >> Search the archives: https://groups.google.com/group/akka-user
>> ---
>> You received this message because you are subscribed to the Google Groups
>> "Akka User List" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to akka-user+unsubscr...@googlegroups.com.
>> To post to this group, send email to akka-user@googlegroups.com.
>> Visit this group at https://groups.google.com/group/akka-user.
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>
>
> --
> Cheers,
> √
>
> --
> >> Read the docs: http://akka.io/docs/
> >> Check the FAQ: http://doc.akka.io/docs/akka/
> current/additional/faq.html
> >> Search the archives: https://groups.google.com/group/akka-user
> ---
> You received this message because you are subscribed to the Google Groups
> "Akka User List" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to akka-user+unsubscr...@googlegroups.com.
> To post to this group, send email to akka-user@googlegroups.com.
> Visit this group at https://groups.google.com/group/akka-user.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
>>  Read the docs: http://akka.io/docs/
>>  Check the FAQ: 
>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>  Search the archives: https://groups.google.com/group/akka-user
--- 
You received this message because you are subscribed to the Google Groups "Akka 
User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to akka-user+unsubscr...@googlegroups.com.
To post to this group, send email to akka-user@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

Re: [akka-user] Nodes becomes unreachable due to serialization-issues with one single message

[Viktor Klang]

Well, I agree that it shouldn't gate the connection due to a failure to
serialize.

Do you have a minimal reproducer to share?

On Wed, Aug 31, 2016 at 3:48 PM, Moax76  wrote:

>
>
> On Wednesday, August 31, 2016 at 3:37:41 PM UTC+2, √ wrote:
>>
>> What happens if you fix the serializer?
>>
>>
>>
> If I fix the serializer, then the problem off course goes away, but that
> is not the point.
>
> The problem is that instead of dropping the message that (for some
> 'unknown' reason) failed to be serialized, it "drops" the connection, and
> GATES it so that it will not try to reconnect in 5 seconds.
>
> If it worked as described above, this serializing issue would be a regular
> bug the only effected parts of the application, instead of the entire
> application going "down" due to cluster-issues.
>
>
> -Morten
>
> --
> >> Read the docs: http://akka.io/docs/
> >> Check the FAQ: http://doc.akka.io/docs/akka/
> current/additional/faq.html
> >> Search the archives: https://groups.google.com/group/akka-user
> ---
> You received this message because you are subscribed to the Google Groups
> "Akka User List" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to akka-user+unsubscr...@googlegroups.com.
> To post to this group, send email to akka-user@googlegroups.com.
> Visit this group at https://groups.google.com/group/akka-user.
> For more options, visit https://groups.google.com/d/optout.
>



-- 
Cheers,
√

-- 
>>  Read the docs: http://akka.io/docs/
>>  Check the FAQ: 
>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>  Search the archives: https://groups.google.com/group/akka-user
--- 
You received this message because you are subscribed to the Google Groups "Akka 
User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to akka-user+unsubscr...@googlegroups.com.
To post to this group, send email to akka-user@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

Re: [akka-user] Nodes becomes unreachable due to serialization-issues with one single message

[Moax76]

On Wednesday, August 31, 2016 at 3:37:41 PM UTC+2, √ wrote:
>
> What happens if you fix the serializer?
>
>
>
If I fix the serializer, then the problem off course goes away, but that is 
not the point.

The problem is that instead of dropping the message that (for some 
'unknown' reason) failed to be serialized, it "drops" the connection, and 
GATES it so that it will not try to reconnect in 5 seconds.

If it worked as described above, this serializing issue would be a regular 
bug the only effected parts of the application, instead of the entire 
application going "down" due to cluster-issues.


-Morten

-- 
>>  Read the docs: http://akka.io/docs/
>>  Check the FAQ: 
>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>  Search the archives: https://groups.google.com/group/akka-user
--- 
You received this message because you are subscribed to the Google Groups "Akka 
User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to akka-user+unsubscr...@googlegroups.com.
To post to this group, send email to akka-user@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

Re: [akka-user] Nodes becomes unreachable due to serialization-issues with one single message

[Viktor Klang]

What happens if you fix the serializer?

On Wed, Aug 31, 2016 at 1:39 PM, Moax76  wrote:

> Hi,
>
> I am using Akka 2.4.9 with persistence, cluster and so on.
>
> I have two nodes and everything works fine.
>
> I was sending a "custom" message from node1 to node2, but the message
> failed to be serialized (Due to a bug in my code).
> This resulted in the akka-remoting connection being GATED (
> http://doc.akka.io/docs/akka/current/scala/remoting.html#
> Lifecycle_and_Failure_Recovery_Model) which in turn resulted in the
> cluster nodes being UNREACHABLE.
>
> So to me it looks like the serialization issue with this one single
> message brought the cluster "down"..
>
> Logs from node2
>
> 2016-08-31 12:11:59,948 ERROR  akkaSrc=akka.tcp://x@node2/
> system/endpointManager/reliableEndpointWriter-akka.
> tcp%3A%2F%2Fx%40node1-0/endpointWriter  AssociationError
> [akka.tcp://x@node2] -> [akka.tcp://x@node1]: Error [Failed to write
> message to the transport] [ akka.remote.EndpointException: *Failed to
> write message to the transport* Caused by: 
> com.fasterxml.jackson.databind.JsonMappingException:
> Instantiation of [simple type, class z.x.engine.BBDestEHInstance] value
> failed (z.x.engine.BBError): id cannot be null  -
> akka.remote.EndpointWriter, 
> akkaThrd=x-akka.remote.default-remote-dispatcher-6,
> thrd=x-akka.actor.default-dispatcher-4, lgr=akka.remote.EndpointWriter
>
> callstack-snipit:
> .
> .
> at no.nextgentel.oss.akkatools.serializing.JacksonJsonSerializer.
> toBinary(JacksonJsonSerializer.scala:84)
> at akka.remote.MessageSerializer$.*serialize*(MessageSerializer.
> scala:37)
> at akka.remote.*EndpointWriter*$$anonfun$serializeMessage$1.
> apply(Endpoint.scala:886)
> at akka.remote.EndpointWriter$$anonfun$serializeMessage$1.
> apply(Endpoint.scala:886)
> at scala.util.DynamicVariable.withValue(DynamicVariable.scala:58)
> at akka.remote.EndpointWriter.serializeMessage(Endpoint.scala:885)
> at akka.remote.EndpointWriter.writeSend(Endpoint.scala:780)
> at akka.remote.EndpointWriter$$anonfun$4.applyOrElse(
> Endpoint.scala:755)
> at akka.actor.Actor$class.aroundReceive(Actor.scala:484)
> at akka.remote.EndpointActor.aroundReceive(Endpoint.scala:447)
> at akka.actor.ActorCell.receiveMessage(ActorCell.scala:526)
> at akka.actor.ActorCell.invoke(ActorCell.scala:495)
> at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257)
> at akka.dispatch.Mailbox.run(Mailbox.scala:224)
> at akka.dispatch.Mailbox.exec(Mailbox.scala:234)
> at scala.concurrent.forkjoin.ForkJoinTask.doExec(
> ForkJoinTask.java:260)
> at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.
> runTask(ForkJoinPool.java:1339)
> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(
> ForkJoinPool.java:1979)
> at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(
> ForkJoinWorkerThread.java:107)
>
>
>
>
> 2016-08-31 12:11:59,967 WARN   akkaSrc=akka.tcp://x@node2/
> system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fx%40node1-0
>  Association with remote system [akka.tcp://x@node1] *has failed, address
> is now gated for [5000] ms*. Reason: [Failed to write message to the
> transport] Caused by: [Instantiation of [simple type, class
> z.x.engine.BBDestEHInstance] value failed (z.x.engine.BBError): id cannot
> be null - akka.remote.ReliableDeliverySupervisor, 
> akkaThrd=x-akka.remote.default-remote-dispatcher-6,
> thrd=x-akka.actor.default-dispatcher-4, lgr=akka.remote.
> ReliableDeliverySupervisor
> 2016-08-31 12:12:03,802 WARN   
> akkaSrc=akka.tcp://x@node2/system/cluster/core/daemon
>  Cluster Node [akka.tcp://x@node2] - Marking node(s) as *UNREACHABLE 
> *[Member(address
> = akka.tcp://x@node1, status = Up)]. Node roles [] - 
> akka.cluster.ClusterCoreDaemon,
> akkaThrd=x-akka.actor.default-dispatcher-19, 
> thrd=x-akka.actor.default-dispatcher-28,
> lgr=akka.cluster.ClusterCoreDaemon
> 2016-08-31 12:12:03,810 WARN   akkaSrc=akka.tcp://x@node2/user/$a  Member
> detected as *unreachable*: Member(address = akka.tcp://x@node1, status =
> Up) - memberCount: 2 - no.nextgentel.oss.akkatools.cluster.ClusterListener,
> akkaThrd=x-akka.actor.default-dispatcher-78, 
> thrd=x-akka.actor.default-dispatcher-20,
> lgr=no.nextgentel.oss.akkatools.cluster.ClusterListener
>
> Logs from node1
> 2016-08-31 12:12:04,708 WARN   akkaSrc=akka.tcp://x@node1/user/$a  Member
> detected as *unreachable*: Member(address = akka.tcp://x@node2, status =
> Up) - memberCount: 2 - no.nextgentel.oss.akkatools.cluster.ClusterListener,
> akkaThrd=x-akka.actor.default-dispatcher-28, 
> thrd=x-akka.actor.default-dispatcher-28,
> lgr=no.nextgentel.oss.akkatools.cluster.ClusterListener
>
>
> Is my understanding of the problem correct? Is this the correct behavior?
> Can I do something different to prevent it from happening?
>
> -Morten
>
> --
> >> Read the docs: http://akka.io/docs/
>