Re: Write behind and eventual consistency

[vkulichenko]

If it's this kind of batch processing, and if you're still waiting for a
batch to be processed completely before the next one can be processed, why
do you want to use write-behind? What is wrong with sync write-through in
this case?

-Val



--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/Write-behind-and-eventual-consistency-tp12242p12438.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Write behind and eventual consistency

[Gaurav Bajaj]

Any updates on the two use cases mentioned above?
On Apr 27, 2017 10:53 AM, "Gaurav Bajaj"  wrote:

> Hi Val,
>
> Our use case :
>
> 1. Read records from file
> 2. Do computations on each record
> 3. Put them in the cache and persistence using write behind.
> 4. When all the records from file are processed, updated in Cache and also
> persisted to DB, we want to
>  trigger some other process which will do next set of operations on these
> records from cache.
> 5. We want to trigger this next process and mark original file as
> processed, only when we are sure data is persisted, so that in case of Node
> failure we need not process that file again.
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> On Thu, Apr 27, 2017 at 10:26 AM, steve.hostettler <
> steve.hostett...@gmail.com> wrote:
>
>> Hi Val,
>>
>> the use case is the following
>>
>> 1) Load data into the database from an external system
>> 2) Once ready load it into the grid
>> 3) Process something that does massive write behinds
>> 4) Take a snapshot of the results (or) Do a backup of the tables   <<---
>> At
>> this point I need the eventual consistency to ...eventually be
>>
>> At step 4 I cannot afford to have some update still in progress. This is
>> even more important since because of write behind I cannot maintain
>> referential integrity (since the insert/update are done in a random order)
>>
>>
>>
>> --
>> View this message in context: http://apache-ignite-users.705
>> 18.x6.nabble.com/Write-behind-and-eventual-consistency-tp12242p12287.html
>> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>>
>
>

Re: Write behind and eventual consistency

[Gaurav Bajaj]

Hi Val,

Our use case :

1. Read records from file
2. Do computations on each record
3. Put them in the cache and persistence using write behind.
4. When all the records from file are processed, updated in Cache and also
persisted to DB, we want to
 trigger some other process which will do next set of operations on these
records from cache.
5. We want to trigger this next process and mark original file as
processed, only when we are sure data is persisted, so that in case of Node
failure we need not process that file again.




















On Thu, Apr 27, 2017 at 10:26 AM, steve.hostettler <
steve.hostett...@gmail.com> wrote:

> Hi Val,
>
> the use case is the following
>
> 1) Load data into the database from an external system
> 2) Once ready load it into the grid
> 3) Process something that does massive write behinds
> 4) Take a snapshot of the results (or) Do a backup of the tables   <<--- At
> this point I need the eventual consistency to ...eventually be
>
> At step 4 I cannot afford to have some update still in progress. This is
> even more important since because of write behind I cannot maintain
> referential integrity (since the insert/update are done in a random order)
>
>
>
> --
> View this message in context: http://apache-ignite-users.
> 70518.x6.nabble.com/Write-behind-and-eventual-
> consistency-tp12242p12287.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>

RE: Write behind and eventual consistency

[steve.hostettler]

Hi Val,

the use case is the following

1) Load data into the database from an external system
2) Once ready load it into the grid
3) Process something that does massive write behinds
4) Take a snapshot of the results (or) Do a backup of the tables   <<--- At
this point I need the eventual consistency to ...eventually be 

At step 4 I cannot afford to have some update still in progress. This is
even more important since because of write behind I cannot maintain
referential integrity (since the insert/update are done in a random order)



--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/Write-behind-and-eventual-consistency-tp12242p12287.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

RE: Write behind and eventual consistency

[vkulichenko]

Hi Steve,

What is the business use case behind this?

-Val



--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/Write-behind-and-eventual-consistency-tp12242p12285.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

RE: Write behind and eventual consistency

[Steve Hostettler]

Hello Val,

Thanks for your help. Don't you think that this would be an interesting
functionality?

Steve

-Original Message-
From: vkulichenko [mailto:valentin.kuliche...@gmail.com] 
Sent: Wednesday, April 26, 2017 9:57 AM
To: user@ignite.apache.org
Subject: Re: Write behind and eventual consistency

There is no way to do this. However, you can take a look at
GridCacheWriteBehindStore which implements this functionality and try
tweaking it so that the queue size is exposed somehow.

-Val



--
View this message in context:
http://apache-ignite-users.70518.x6.nabble.com/Write-behind-and-eventual-con
sistency-tp12242p12256.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Write behind and eventual consistency

[vkulichenko]

There is no way to do this. However, you can take a look at
GridCacheWriteBehindStore which implements this functionality and try
tweaking it so that the queue size is exposed somehow.

-Val



--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/Write-behind-and-eventual-consistency-tp12242p12256.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Write behind and eventual consistency

[steve.hostett...@gmail.com]

Hello,Assuming there is no more jobs or tasks going (that I csn comtrol from an application perspective) I would like to know when the database is in sync with the caches. Otherwise I cannot get a coherent snapshot. Knowing that there is no jobs ongoing and the queue is empty would be enough. How do I check the status of that queue?Steve Original Message Subject: Re: Write behind and eventual consistencyFrom: vkulichenko To: user@ignite.apache.orgCC: Hi Steve,I don't think it's currently possible and frankly I'm not sure I understandwhat it actually means. Can you clarify what is implied in "no more writebehind operations waiting for completion"? We could probably check if thequeue is empty, but what if new updates happen right after or concurrentlywith this check?-Val--View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Write-behind-and-eventual-consistency-tp12242p12247.htmlSent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Write behind and eventual consistency

[vkulichenko]

Hi Steve,

I don't think it's currently possible and frankly I'm not sure I understand
what it actually means. Can you clarify what is implied in "no more write
behind operations waiting for completion"? We could probably check if the
queue is empty, but what if new updates happen right after or concurrently
with this check?

-Val



--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/Write-behind-and-eventual-consistency-tp12242p12247.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.