Re: [Mercury] Mercury restarting (was persistence policy)

Wed, 18 Jun 2008 08:08:44 -0700 

Hi Paul,
Yes RM store have the information. But the problem is there is no way for the RM implementation to differentiate an invocation from a previous invocation. 

Let's take the example that Amila has mentioned.

The application program send 100 messages everyday to an endpoint X.
Let's assume today the application crashes after it sends the 25th message.


Now the application recovers from the crash (somehow depending on the 
recovery logic of the application) and wanted to resume the same sequence.
If it starts a new sequence then the RM implementation has no way to know 
that this is the previously crashed sequance.
The endpoint details are the same, but since application sends messages 
everyday, the RM implementation cannot differentiate this from a typical 
daily messages.



So there should be a mechanism for the application client to inform RM about 
the sequence (using some unique ID) that it wants to proceed after a crash.


Let me know your thoughts about this.

Thanks,
Jaliya









----- Original Message ----- 
From: "Paul Fremantle" <[EMAIL PROTECTED]>

To: "Jaliya Ekanayake" <[EMAIL PROTECTED]>

Cc: "Amila Suriarachchi" <[EMAIL PROTECTED]>; "Chamikara Jayalath" 
<[EMAIL PROTECTED]>; <[email protected]>

Sent: Wednesday, June 18, 2008 10:30 AM
Subject: Re: [Mercury] Mercury restarting (was persistence policy)



Jaliya

I don't understand your example.

In the RM persistent store there is enough information to recover from
a crash without any extra requirement from the client. What is
missing?

Paul


On Wed, Jun 18, 2008 at 2:30 PM, Jaliya Ekanayake <[EMAIL PROTECTED]> 
wrote:

Hi Amila,

I think the discussion is mixing up the application recovery and the
persistance requirements of RM.
We need to separate the two cases.

In your example, how the application client determines whether it has
crashed or not is a separate problem to RM.

From RM point of view, this is what should happen.


The application client should be given a simple API to let RM know that 
this

is the same sequance that it needs to continue before crashing.
Something like "resumeSequance(seqID)"


We need this information from the application client simply because RM 
does
not have a way to differentiate the different invocations of the 
application

client.


In RM database, we only need to record the messages and the propety to 
know

whether the sequance is completed or not.
When the application client recovers, it can simply call
"resumeSequance(seqID) and probably start sending the messages again from
the begining.

However, in the RM database our records shows that we have successfully 
sent

up to 25h message. So RM will continue only after 26th message.

Thanks,
Jaliya



----- Original Message -----
From: Amila Suriarachchi
To: Jaliya Ekanayake
Cc: Chamikara Jayalath ; Paul Fremantle ; [email protected]
Sent: Wednesday, June 18, 2008 1:41 AM
Subject: Re: [Mercury] Mercury restarting (was persistence policy)
Here is what I thought when writing the persistence.

Lets take this scenario,

There is an application client which process transaction logs and send

messages to server. From another application it gets log files to a 
folder

called *input* folder and then this application  moves this file to

*processing* folder and start reading log entries and send messages. 
After

finish it move the file to *finish* folder.

Then say this Application client crashes when it processing the file. 
When
at the start up the application can understand it has crashed by looking 
at

the files in the *processing* folder.

Then it has to determine how many messages it has send. This can be done 
by
checking the RM persistence database. Then it can restart the sequence 
and

start sending messages from the stop point.

The main point here is that this application should have a recovery 
process
and it should be able to understand something has gone wrong. (in this 
case

looking at the files in the *processing* folder).

This logic can only be written in the Application level recovery process.
That also can be written in a demon but I don't see any advantage.

thanks,
Amila.



On Wed, Jun 18, 2008 at 10:26 AM, Amila Suriarachchi
<[EMAIL PROTECTED]> wrote:




On Tue, Jun 17, 2008 at 9:53 PM, Jaliya Ekanayake 
<[EMAIL PROTECTED]>

wrote:



Don't know if somebody has done this earlier, but as Chamikara 
mentioned,

we can have a daemon process in the client side which acts as the:

1. Endpoint for the dual channel communications originating from that
machine.
2. manager for the reliable communications.


The communications between the RMHandler (not the client application) 
and

the daemon process should happen via a persistence storage mechanism.



isn't this mean the user has to write this logic in the deamon. So I 
think
writing this straight away in the Application client would make it 
easier.



The daemon will keep trying to send the messages, and if something 
fail,

once in a while bug the user with a pop up window :)


Then anyway application client has to handle this isn't it? This means
some recovery part should goes to application client.

thanks,
Amila.


Thanks,
Jaliya


----- Original Message -----
From: Chamikara Jayalath
To: Amila Suriarachchi
Cc: Paul Fremantle ; [email protected]
Sent: Tuesday, June 17, 2008 11:05 AM
Subject: Re: [Mercury] Mercury restarting (was persistence policy)
Amila, Paul,

I don't think I said that application client should have any logic to

manage recovery. I simply said that one of following two has to be done 
to

manage recovery in the client side.


1. There should be a permanent RM agent running in the client side, 
which

manages recovery.

2. RM client should check for crashed sequences whenever it is invoked 
in

the client side, if it find any they should be resumed.


In both cases work should be done by the RM client transparently, not 
by

the application client.

Chamikara


On Tue, Jun 17, 2008 at 10:42 AM, Amila Suriarachchi
<[EMAIL PROTECTED]> wrote:



On Tue, Jun 17, 2008 at 7:09 PM, Paul Fremantle <[EMAIL PROTECTED]>
wrote:


Amila

The Mercury design is wrong.

The application client should not be responsible for understanding if

the system needs to restart. There isn't a single successful 
messaging

or transaction system in the world that requires the application to
incorporate logic to handle recovery. Sandesha2 manages automatic
recovery and so should Mercury.



Here the question is that when a client dies. (i.e. system crashed) 
how

it automatically restart?
Actually I had a chat with Chamikara regarding this and he told that
sandesha2 also can not
automatically restart.
Chamikara could you please comment on this?

thanks,
Amila.



Paul

On Tue, Jun 17, 2008 at 2:36 PM, Amila Suriarachchi
<[EMAIL PROTECTED]> wrote:

> +1 to add a policy to declare whether a service supports 
> Persistence

> or not.
>
> IMO if RM provides persistence then it should happen in both client
> and
> server side.

> But at the client side Application layer has to do some part as 
> well.

> In Mercury if a client dies when sending a sequence it can be
> restarted by

> sending a message by setting a property called 
> MercuryResumeSequence

> to
> true.
> But in this case Application client have to check how many messages
> it had

> send to RM either by keeping a track of messages or by examining 
> the

> Mercury
> data base store.
>
> thanks,
> Amila.
>
> On Tue, Jun 17, 2008 at 6:50 AM, Jaliya Ekanayake
> <[EMAIL PROTECTED]>
> wrote:
>>
>> Hi Paul,
>>
>> I agree with the two scenarios you mentioned.
>> So, if the majority of the use cases for RM are of the above type
>> with
>> long running communications, then we should stick to the persisted
>> reliability in every usecase.
>> Otherwise, I think we'd better keep the client side free from the
>> mandatory database integration.
>>
>> Thanks,
>> Jaliya
>>
>>
>> ----- Original Message ----- From: "Paul Fremantle"
>> <[EMAIL PROTECTED]>
>> To: "Jaliya Ekanayake" <[EMAIL PROTECTED]>
>> Cc: <[email protected]>
>> Sent: Monday, June 16, 2008 1:05 PM
>> Subject: Re: Policy or other extensions to indicate persistent
>> messaging
>>
>>
>>> Jaliya
>>>
>>> I agree that it is the responsibility of the client. But I'm
>>> worried
>>> about two scenarios.
>>>
>>> 1) The client crashes and restarts - the server has missing
>>> messages

>>> and without them it cannot deliver the later messages that it 
>>> has.

>>> 2) The client crashes and restarts - the server has responses
>>> stored,
>>> ready to send, but the client has lost the sequence state and
>>> cannot
>>> accept the stored messages.
>>>
>>> I still think that there is a requirement to ENSURE that a
>>> particular
>>> communication is completely reliable.
>>>
>>> Paul
>>>
>>> On Mon, Jun 16, 2008 at 5:54 PM, Jaliya Ekanayake
>>> <[EMAIL PROTECTED]>
>>> wrote:
>>>>
>>>> Hi Paul,
>>>>
>>>> Please see my comments below.
>>>>
>>>> Thanks,
>>>> Jaliya
>>>> ----- Original Message ----- From: "Paul Fremantle"
>>>> <[EMAIL PROTECTED]>
>>>> To: "Jaliya Ekanayake" <[EMAIL PROTECTED]>
>>>> Sent: Monday, June 16, 2008 11:40 AM
>>>> Subject: Re: Policy or other extensions to indicate persistent
>>>> messaging
>>>>
>>>>
>>>>> Jaliya
>>>>>

>>>>>> My idea is that the persistence is a feature that the admin 
>>>>>> can

>>>>>> turn
>>>>>> on
>>>>>> or
>>>>>> off per service.
>>>>>
>>>>> So I agree that it is a good feature to turn it on and off per

>>>>> service, and that we support today. So for this model, what I 
>>>>> am
>>>>> proposing adding is the ability for the server to advertize 
>>>>> that

>>>>> it
>>>>> supports that.
>>>>>
>>>>>> If it is to be per sequence, I think it will over
>>>>>> complicate the handshakes.
>>>>>

>>>>> It will *complicate* the handshakes :) I agree. I think to say 
>>>>> it

>>>>> over-complicates the handshake is a subjective idea.
>>>>>
>>>> I agree.
>>>>
>>>>>> Also, Sandesha should not impose restrictions to the clients
>>>>>> based on
>>>>>> their
>>>>>> persistence settings.
>>>>>

>>>>> I don't agree. WSRM imposes plenty of restrictions on the 
>>>>> client

>>>>> and
>>>>> server. It is perfectly possible for a server to refuse to
>>>>> communicate
>>>>> with a client that does not support RM. With WSRM 1.1 It is
>>>>> possible
>>>>> for clients to demand that the server creates an association
>>>>> between
>>>>> the sequence and a security session.
>>>>>
>>>>> If the server supports persistence but the client doesn't, then
>>>>> there
>>>>> is no overall guarantee of reliability. So I believe that it
>>>>> ought to
>>>>> be *possible* for a server to send CreateSequenceRefused if it
>>>>> *requires* persistence and the client cannot provide it. Of
>>>>> course
>>>>> that should be configured by the server. Similarly the client
>>>>> should
>>>>> be able to demand (like a mustUnderstand) that the server
>>>>> provides a
>>>>> persistent capability.
>>>>>
>>>>> Why do I want this? Almost every customer I talk to says that
>>>>> WSRM
>>>>> without persistence is basically pointless.
>>>>

>>>> I totally agree with this. Without persistance, I can almost 
>>>> trust

>>>> TCP
>>>> for
>>>> the reliability. (not for the multi-transport multi-hop cases)
>>>> So we need persistance for *real* use cases.
>>>>
>>>> The blogging from people
>>>>>

>>>>> who believe that WSRM should not support persistence has - in 
>>>>> my

>>>>> view
>>>>> - been very harmful to the adoption of the spec. Now the normal
>>>>> argument is that "you can support persistence if you need it".
>>>>> But
>>>>> frankly, that is a weak argument, because in a real distributed
>>>>> SOA

>>>>> you cannot (at the moment) know if there is persistence 
>>>>> involved,

>>>>> except maybe by phoning up the sysadmin at the other end. So if
>>>>> you
>>>>> require proper persistent reliability then you need a way of
>>>>> agreeing
>>>>> between both parties that it exists. Now WSRM has the perfect
>>>>> model
>>>>> for this negotiation - the CreateSequence/CSResponse. This
>>>>> ability to
>>>>> negotiate details is perfect for this, and I don't see any
>>>>> problem
>>>>> extending it to support this.
>>>>>
>>>>

>>>> I agree that the Client should be able to request persistence 
>>>> from

>>>> a
>>>> Service, but I still don' t understand why we need the server to
>>>> reject
>>>> a
>>>> client if it does not support persistence.
>>>> Here is a scenario.
>>>>

>>>> Say we have two companies A and B that perform some 
>>>> communications

>>>> over
>>>> snail mail.

>>>> All the mails received by the company B first go through its 
>>>> mail

>>>> processing
>>>> system (MPS) which keeps information on all the mails received,
>>>> and also
>>>> keeps a copy of them.

>>>> All the mails sent by the company B also go through the same 
>>>> MPS.

>>>>

>>>> Company A is not that sophisticated in its operations. They 
>>>> don't

>>>> simply
>>>> have a system as above.
>>>>
>>>> Now consider the communications that could happen.
>>>>
>>>> 1.  A sends a mail to B
>>>> If this reach B, things are fine.

>>>> If this does not reach B then it is a problem of A. A should 
>>>> keep

>>>> a copy
>>>> so
>>>> that it can send it again.
>>>>
>>>>
>>>> 2. B sends a mail to A as a response to A's request
>>>> If this reach A and processed by A things are fine

>>>> If this reach A but not processed by A, then B can send it 
>>>> again.

>>>> (up to
>>>> some number of times)
>>>> If this does not reach A, B can still send it again.
>>>>

>>>> 3. B needs to send a mail to A requesting something. (Now A is 
>>>> the

>>>> service
>>>> provider)
>>>> In this case B can request that A provides the necessary
>>>> reliability to
>>>> its
>>>> requests.
>>>> If A does not support it, B should not proceed.
>>>>

>>>> Therefore, IMO the client does not need to have a real 
>>>> persistence

>>>> to
>>>> use a
>>>> service offered with persisted reliability, but it can request
>>>> this
>>>> feature
>>>> from the service.
>>>>
>>>> Let me know your thoughts. probably I have missed some use case.
>>>>
>>>> Thanks,
>>>> Jaliya
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> Say, the server supports persistance. In that case, any request
>>>> sent by
>>>> the
>>>> client is guranteed to be served.
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>> Paul
>>>>>
>>>>> , but we can advertise that we have persistence for this
>>>>>>
>>>>>> service.
>>>>>
>>>>>
>>>>>>
>>>>>> Thanks,
>>>>>> Jaliya
>>>>>>
>>>>>> ----- Original Message ----- From: "Paul Fremantle"
>>>>>> <[EMAIL PROTECTED]>
>>>>>> To: "Danushka Menikkumbura" <[EMAIL PROTECTED]>
>>>>>> Cc: <[email protected]>
>>>>>> Sent: Monday, June 16, 2008 8:12 AM
>>>>>> Subject: Re: Policy or other extensions to indicate persistent
>>>>>> messaging
>>>>>>
>>>>>>
>>>>>>> Danushka
>>>>>>>
>>>>>>> I'm not clear I understand your point.
>>>>>>>
>>>>>>> Firstly, I was just using AMQP as an example - I didn't mean
>>>>>>> that we

>>>>>>> wanted to be exactly the same as AMQP or do anything with 
>>>>>>> AMQP.

>>>>>>>
>>>>>>> So firstly, in general, I don't believe that Sandesha2 can
>>>>>>> modify the
>>>>>>> persistence on a sequence by sequence basis - either it is on
>>>>>>> for a

>>>>>>> service or off. However, logically the sequence *is* the 
>>>>>>> level

>>>>>>> at
>>>>>>> which persistence can be defined. So these are the options I
>>>>>>> see:
>>>>>>>

>>>>>>> * The server has persistence set permanently on for a 
>>>>>>> service.

>>>>>>> It
>>>>>>> does
>>>>>>> not demand persistence from the client. It publishes a policy
>>>>>>> saying

>>>>>>> persistence is optional for the client. It accepts any 
>>>>>>> sequence
>>>>>>> creation, and persistently stores messages. If the client 
>>>>>>> "asks

>>>>>>> for

>>>>>>> persistence" during the create sequence, then it says "yes 
>>>>>>> I'm
>>>>>>> persistent in reply" (by some yet to be determined 
>>>>>>> mechanisms).

>>>>>>> This

>>>>>>> is how we operate today, except with the addition of the 
>>>>>>> policy

>>>>>>> and
>>>>>>> the optional information passing during create sequence.
>>>>>>>

>>>>>>> * The server has persistence set permanently on for a 
>>>>>>> service.

>>>>>>>  It
>>>>>>> demands persistence from the client. Therefore it publishes a
>>>>>>> policy
>>>>>>> saying that client's must be persistent. During the
>>>>>>> CreateSequence

>>>>>>> exchange, it can refuse any client that doesn't agree (by 
>>>>>>> some
>>>>>>> yet-to-be-defined mechanism) to be persistent. This would be 
>>>>>>> a

>>>>>>> new
>>>>>>> capability.
>>>>>>>

>>>>>>> * The server has some clever ability to turn persistence on 
>>>>>>> or

>>>>>>> off
>>>>>>> per-sequence based on the create sequence. In this model, the
>>>>>>> server
>>>>>>> picks up the preference from the client. So the same service
>>>>>>> might
>>>>>>> have some persistent and some non-persistent sequences. Maybe
>>>>>>> this is
>>>>>>> overkill and beyond the basic requirements. I'm not clear.
>>>>>>> However,

>>>>>>> this seems to be a model that other systems allow. Of course, 
>>>>>>> a

>>>>>>> server
>>>>>>> could respond that it doesn't support this capability.
>>>>>>>
>>>>>>> Ideally, all of this would be designed to allow backwards

>>>>>>> compatibility. So even in the cases where the server refuses 
>>>>>>> a
>>>>>>> sequence because it requires persistence and the client 
>>>>>>> doesn't

>>>>>>> support this extension, the failure is explained in the error
>>>>>>> and the
>>>>>>> administrator can see why.
>>>>>>>
>>>>>>> Paul
>>>>>>>
>>>>>>> On Mon, Jun 16, 2008 at 12:20 PM, Danushka Menikkumbura
>>>>>>> <[EMAIL PROTECTED]> wrote:
>>>>>>>>
>>>>>>>>>>> 1) A policy element to indicate whether this endpoint
>>>>>>>>>>> supports
>>>>>>>>>>> and/or
>>>>>>>>>>> requires persistence
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> (b) Even if we go for transport level persistence, the
>>>>>>>>>> endpoint
>>>>>>>>>> does
>>>>>>>>>> not
>>>>>>>>>> have a say in it, because in AMQP we can have either queue
>>>>>>>>>> level
>>>>>>>>>> persistence

>>>>>>>>>> (i.e. transport receiver level abstraction) or message 
>>>>>>>>>> level

>>>>>>>>>> persistence
>>>>>>>>>> (i.e. message sender level abstraction).
>>>>>>>>>>
>>>>>>>>
>>>>>>>> Hi Paul,
>>>>>>>>  But still this is not doable with AMQP.
>>>>>>>>
>>>>>>>> Danushka
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Paul Fremantle
>>>>>>> Co-Founder and CTO, WSO2
>>>>>>> Apache Synapse PMC Chair
>>>>>>> OASIS WS-RX TC Co-chair
>>>>>>>
>>>>>>> blog: http://pzf.fremantle.org
>>>>>>> [EMAIL PROTECTED]
>>>>>>>
>>>>>>> "Oxygenating the Web Service Platform", www.wso2.com
>>>>>>>
>>>>>>>
>>>>>>> ---------------------------------------------------------------------

>>>>>>> To unsubscribe, e-mail: 
>>>>>>> [EMAIL PROTECTED]

>>>>>>> For additional commands, e-mail:
>>>>>>> [EMAIL PROTECTED]
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Paul Fremantle
>>>>> Co-Founder and CTO, WSO2
>>>>> Apache Synapse PMC Chair
>>>>> OASIS WS-RX TC Co-chair
>>>>>
>>>>> blog: http://pzf.fremantle.org
>>>>> [EMAIL PROTECTED]
>>>>>
>>>>> "Oxygenating the Web Service Platform", www.wso2.com
>>>>
>>>>
>>>
>>>
>>>
>>> --
>>> Paul Fremantle
>>> Co-Founder and CTO, WSO2
>>> Apache Synapse PMC Chair
>>> OASIS WS-RX TC Co-chair
>>>
>>> blog: http://pzf.fremantle.org
>>> [EMAIL PROTECTED]
>>>
>>> "Oxygenating the Web Service Platform", www.wso2.com
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: [EMAIL PROTECTED]
>>> For additional commands, e-mail: [EMAIL PROTECTED]
>>>
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: [EMAIL PROTECTED]
>> For additional commands, e-mail: [EMAIL PROTECTED]
>>
>
>
>
> --
> Amila Suriarachchi,
> WSO2 Inc.



--
Paul Fremantle
Co-Founder and CTO, WSO2
Apache Synapse PMC Chair
OASIS WS-RX TC Co-chair

blog: http://pzf.fremantle.org
[EMAIL PROTECTED]

"Oxygenating the Web Service Platform", www.wso2.com




--
Amila Suriarachchi,
WSO2 Inc.




--
Amila Suriarachchi,
WSO2 Inc.



--
Amila Suriarachchi,
WSO2 Inc.




--
Paul Fremantle
Co-Founder and CTO, WSO2
Apache Synapse PMC Chair
OASIS WS-RX TC Co-chair

blog: http://pzf.fremantle.org
[EMAIL PROTECTED]

"Oxygenating the Web Service Platform", www.wso2.com

---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]




---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]























					Reply via email to