Re: [akka-user] akka.persistence, snapshots and deleting messages from journal

2017-12-18 Thread Michael Zhong 
case SaveSnapshotSuccess(metadata) =>
  println(s"snapshot saved. seqNum:${metadata.sequenceNr}, 
timeStamp:${metadata.timestamp}")
  deleteMessages(metadata.sequenceNr - eventDeleteOffset)

Isn't this falsey? event is about snapshot, deleteMessages(..) is about 
snapshot, they are essentially different stores, Or their Nr lays in same 
Nr-series?


在 2015年12月15日星期二 UTC+8上午2:25:25，Patrik Nordwall写道：
>
> Thanks for sharing, Brice. That is an important finding.
>
> On Wed, Dec 2, 2015 at 2:23 PM, Brice Figureau  > wrote:
>
>> On 18/02/2014 21:39, dpen...@good-cloud.com  wrote:
>> > I would assume that a common scenario is to periodically snapshot state
>> > and keep 1 or more snapshots.  In this case, I assume that as you delete
>> > older snapshots, you would also delete messages older than the snapshot
>> > from the journal in order to keep the journal from growing indefinitely.
>> >
>> > Is this a typical usage or am I missing something?  I don't see anything
>> > in the code that automatically does this (and I didn't really expect to
>> > given the general flexibility theme of the framework).  However, I don't
>> > see anything in the documentation that alludes to any other use of
>> > deleteMessage other than in dealing with exceptions.
>>
>> I thought it would be the good idea to do so, and I was doing this
>> beforehand.
>>
>> But if you're using the Cassandra journal as I do, be aware that
>> deleting past events produces tombstones that are removed only after
>> gc_grace_seconds and a major compaction happens (or a minor one that
>> removes tombstones). Until that time, those tombstones will hurt your
>> reads quite hard from what I've found (and it's more than certainly
>> related to the static column). This was the issue for me, because I had
>> several PersistentView that were doing a constant replay of some of the
>> more prolific PersistentActor, up to the point where the replaying
>> cassandra query was taking more time than the view refresh rate, and
>> thus the cassandra journal was sending more and more read requests, up
>> to the point where the cassandra cluster couldn't follow anymore and was
>> using more 100% CPU, dropping writes.
>>
>> I fixed the problem by not doing any deletion at the expense of using
>> more and more disk space. I will probably do the deletion in a
>> controlled way (and making sure we can run a user compaction just after).
>>
>> HTH,
>> --
>> Brice Figureau
>> My Blog: http://www.masterzen.fr/
>>
>> --
>> >>  Read the docs: http://akka.io/docs/
>> >>  Check the FAQ: 
>> http://doc.akka.io/docs/akka/current/additional/faq.html
>> >>  Search the archives: 
>> https://groups.google.com/group/akka-user
>> ---
>> You received this message because you are subscribed to the Google Groups 
>> "Akka User List" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to akka-user+...@googlegroups.com .
>> To post to this group, send email to akka...@googlegroups.com 
>> .
>> Visit this group at http://groups.google.com/group/akka-user.
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>
>
> -- 
>
> Patrik Nordwall
> Typesafe  -  Reactive apps on the JVM
> Twitter: @patriknw
>
>

-- 
>>  Read the docs: http://akka.io/docs/
>>  Check the FAQ: 
>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>  Search the archives: https://groups.google.com/group/akka-user
--- 
You received this message because you are subscribed to the Google Groups "Akka 
User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to akka-user+unsubscr...@googlegroups.com.
To post to this group, send email to akka-user@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

Re: [akka-user] akka.persistence, snapshots and deleting messages from journal

2014-09-18 Thread Brian McGee 
Hey,

I'm implementing a PersistentActor which batches writes for a db. After 
successfully writing a batch I can't figure out how to determine the latest 
message number I should use to pass to deleteMessages to prevent the events 
I just persisted to the db from appearing in the recovery. Any ideas?

Brian

On Wednesday, February 19, 2014 3:08:37 PM UTC+1, dpen...@good-cloud.com 
wrote:
>
> Agreed - in our legacy system, we have a storage manager that removes old 
> journals and snapshots as space is needed.  The old journals are sometimes 
> very useful for postmortem analysis.
>
> -david
>
> On Wednesday, February 19, 2014 2:45:17 AM UTC-5, Patrik Nordwall wrote:
>>
>> You can use deleteMessages to delete all messages up to a specified 
>> sequence number. I would not use it too eagerly, since the information in 
>> the history can be valuable in itself, but that is of course depending on 
>> the application.
>>
>> /Patrik
>>
>>
>> On Tue, Feb 18, 2014 at 9:39 PM,  wrote:
>>
>>> I would assume that a common scenario is to periodically snapshot state 
>>> and keep 1 or more snapshots.  In this case, I assume that as you delete 
>>> older snapshots, you would also delete messages older than the snapshot 
>>> from the journal in order to keep the journal from growing indefinitely.
>>>
>>> Is this a typical usage or am I missing something?  I don't see anything 
>>> in the code that automatically does this (and I didn't really expect to 
>>> given the general flexibility theme of the framework).  However, I don't 
>>> see anything in the documentation that alludes to any other use of 
>>> deleteMessage other than in dealing with exceptions.
>>>
>>> -david
>>>
>>> -- 
>>> >> Read the docs: http://akka.io/docs/
>>> >> Check the FAQ: http://akka.io/faq/
>>> >> Search the archives: 
>>> https://groups.google.com/group/akka-user
>>> --- 
>>> You received this message because you are subscribed to the Google 
>>> Groups "Akka User List" group.
>>> To unsubscribe from this group and stop receiving emails from it, send 
>>> an email to akka-user+...@googlegroups.com.
>>> To post to this group, send email to akka...@googlegroups.com.
>>> Visit this group at http://groups.google.com/group/akka-user.
>>> For more options, visit https://groups.google.com/groups/opt_out.
>>>
>>
>>
>>
>> -- 
>>
>> Patrik Nordwall
>> Typesafe  -  Reactive apps on the JVM
>> Twitter: @patriknw
>>
>>  

-- 
>>  Read the docs: http://akka.io/docs/
>>  Check the FAQ: 
>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>  Search the archives: https://groups.google.com/group/akka-user
--- 
You received this message because you are subscribed to the Google Groups "Akka 
User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to akka-user+unsubscr...@googlegroups.com.
To post to this group, send email to akka-user@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

Re: [akka-user] akka.persistence, snapshots and deleting messages from journal

2014-09-18 Thread danny . browning 
Make sure you're receiving the snapshot success message.

case SaveSnapshotSuccess(metadata) => {
  deleteMessages(metadata.sequenceNr, true)
}

You can also save that metadata in case you want to clean up snapshots. Per 
Patrik's comment, you may want to verify that some interval has passed 
before you deleteMessages. If you want, you can also track the last 
snapshot for easy deletion.

lastSnapshot.foreach( m => deleteSnapshot(m.sequenceNr, m.timestamp))

On Thursday, September 18, 2014 5:06:55 AM UTC-6, Brian McGee wrote:
>
> Hey,
>
> I'm implementing a PersistentActor which batches writes for a db. After 
> successfully writing a batch I can't figure out how to determine the latest 
> message number I should use to pass to deleteMessages to prevent the events 
> I just persisted to the db from appearing in the recovery. Any ideas?
>
> Brian
>
> On Wednesday, February 19, 2014 3:08:37 PM UTC+1, dpen...@good-cloud.com 
> wrote:
>>
>> Agreed - in our legacy system, we have a storage manager that removes old 
>> journals and snapshots as space is needed.  The old journals are sometimes 
>> very useful for postmortem analysis.
>>
>> -david
>>
>> On Wednesday, February 19, 2014 2:45:17 AM UTC-5, Patrik Nordwall wrote:
>>>
>>> You can use deleteMessages to delete all messages up to a specified 
>>> sequence number. I would not use it too eagerly, since the information in 
>>> the history can be valuable in itself, but that is of course depending on 
>>> the application.
>>>
>>> /Patrik
>>>
>>>
>>> On Tue, Feb 18, 2014 at 9:39 PM,  wrote:
>>>
 I would assume that a common scenario is to periodically snapshot state 
 and keep 1 or more snapshots.  In this case, I assume that as you delete 
 older snapshots, you would also delete messages older than the snapshot 
 from the journal in order to keep the journal from growing indefinitely.

 Is this a typical usage or am I missing something?  I don't see 
 anything in the code that automatically does this (and I didn't really 
 expect to given the general flexibility theme of the framework).  However, 
 I don't see anything in the documentation that alludes to any other use of 
 deleteMessage other than in dealing with exceptions.

 -david

 -- 
 >> Read the docs: http://akka.io/docs/
 >> Check the FAQ: http://akka.io/faq/
 >> Search the archives: 
 https://groups.google.com/group/akka-user
 --- 
 You received this message because you are subscribed to the Google 
 Groups "Akka User List" group.
 To unsubscribe from this group and stop receiving emails from it, send 
 an email to akka-user+...@googlegroups.com.
 To post to this group, send email to akka...@googlegroups.com.
 Visit this group at http://groups.google.com/group/akka-user.
 For more options, visit https://groups.google.com/groups/opt_out.

>>>
>>>
>>>
>>> -- 
>>>
>>> Patrik Nordwall
>>> Typesafe  -  Reactive apps on the JVM
>>> Twitter: @patriknw
>>>
>>>  

-- 
>>  Read the docs: http://akka.io/docs/
>>  Check the FAQ: 
>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>  Search the archives: https://groups.google.com/group/akka-user
--- 
You received this message because you are subscribed to the Google Groups "Akka 
User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to akka-user+unsubscr...@googlegroups.com.
To post to this group, send email to akka-user@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

Re: [akka-user] akka.persistence, snapshots and deleting messages from journal

2014-02-18 Thread Patrik Nordwall 
You can use deleteMessages to delete all messages up to a specified
sequence number. I would not use it too eagerly, since the information in
the history can be valuable in itself, but that is of course depending on
the application.

/Patrik


On Tue, Feb 18, 2014 at 9:39 PM,  wrote:

> I would assume that a common scenario is to periodically snapshot state
> and keep 1 or more snapshots.  In this case, I assume that as you delete
> older snapshots, you would also delete messages older than the snapshot
> from the journal in order to keep the journal from growing indefinitely.
>
> Is this a typical usage or am I missing something?  I don't see anything
> in the code that automatically does this (and I didn't really expect to
> given the general flexibility theme of the framework).  However, I don't
> see anything in the documentation that alludes to any other use of
> deleteMessage other than in dealing with exceptions.
>
> -david
>
> --
> >> Read the docs: http://akka.io/docs/
> >> Check the FAQ: http://akka.io/faq/
> >> Search the archives: https://groups.google.com/group/akka-user
> ---
> You received this message because you are subscribed to the Google Groups
> "Akka User List" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to akka-user+unsubscr...@googlegroups.com.
> To post to this group, send email to akka-user@googlegroups.com.
> Visit this group at http://groups.google.com/group/akka-user.
> For more options, visit https://groups.google.com/groups/opt_out.
>



-- 

Patrik Nordwall
Typesafe  -  Reactive apps on the JVM
Twitter: @patriknw

-- 
>>  Read the docs: http://akka.io/docs/
>>  Check the FAQ: http://akka.io/faq/
>>  Search the archives: https://groups.google.com/group/akka-user
--- 
You received this message because you are subscribed to the Google Groups "Akka 
User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to akka-user+unsubscr...@googlegroups.com.
To post to this group, send email to akka-user@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/groups/opt_out.

Re: [akka-user] akka.persistence, snapshots and deleting messages from journal

2014-02-19 Thread dpennell 
Agreed - in our legacy system, we have a storage manager that removes old 
journals and snapshots as space is needed.  The old journals are sometimes 
very useful for postmortem analysis.

-david

On Wednesday, February 19, 2014 2:45:17 AM UTC-5, Patrik Nordwall wrote:
>
> You can use deleteMessages to delete all messages up to a specified 
> sequence number. I would not use it too eagerly, since the information in 
> the history can be valuable in itself, but that is of course depending on 
> the application.
>
> /Patrik
>
>
> On Tue, Feb 18, 2014 at 9:39 PM, >wrote:
>
>> I would assume that a common scenario is to periodically snapshot state 
>> and keep 1 or more snapshots.  In this case, I assume that as you delete 
>> older snapshots, you would also delete messages older than the snapshot 
>> from the journal in order to keep the journal from growing indefinitely.
>>
>> Is this a typical usage or am I missing something?  I don't see anything 
>> in the code that automatically does this (and I didn't really expect to 
>> given the general flexibility theme of the framework).  However, I don't 
>> see anything in the documentation that alludes to any other use of 
>> deleteMessage other than in dealing with exceptions.
>>
>> -david
>>
>> -- 
>> >> Read the docs: http://akka.io/docs/
>> >> Check the FAQ: http://akka.io/faq/
>> >> Search the archives: https://groups.google.com/group/akka-user
>> --- 
>> You received this message because you are subscribed to the Google Groups 
>> "Akka User List" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to akka-user+...@googlegroups.com .
>> To post to this group, send email to akka...@googlegroups.com
>> .
>> Visit this group at http://groups.google.com/group/akka-user.
>> For more options, visit https://groups.google.com/groups/opt_out.
>>
>
>
>
> -- 
>
> Patrik Nordwall
> Typesafe  -  Reactive apps on the JVM
> Twitter: @patriknw
>
>  

-- 
>>  Read the docs: http://akka.io/docs/
>>  Check the FAQ: http://akka.io/faq/
>>  Search the archives: https://groups.google.com/group/akka-user
--- 
You received this message because you are subscribed to the Google Groups "Akka 
User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to akka-user+unsubscr...@googlegroups.com.
To post to this group, send email to akka-user@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/groups/opt_out.

Re: [akka-user] akka.persistence, snapshots and deleting messages from journal

2015-12-02 Thread Brice Figureau 
On 18/02/2014 21:39, dpenn...@good-cloud.com wrote:
> I would assume that a common scenario is to periodically snapshot state
> and keep 1 or more snapshots.  In this case, I assume that as you delete
> older snapshots, you would also delete messages older than the snapshot
> from the journal in order to keep the journal from growing indefinitely.
> 
> Is this a typical usage or am I missing something?  I don't see anything
> in the code that automatically does this (and I didn't really expect to
> given the general flexibility theme of the framework).  However, I don't
> see anything in the documentation that alludes to any other use of
> deleteMessage other than in dealing with exceptions.

I thought it would be the good idea to do so, and I was doing this
beforehand.

But if you're using the Cassandra journal as I do, be aware that
deleting past events produces tombstones that are removed only after
gc_grace_seconds and a major compaction happens (or a minor one that
removes tombstones). Until that time, those tombstones will hurt your
reads quite hard from what I've found (and it's more than certainly
related to the static column). This was the issue for me, because I had
several PersistentView that were doing a constant replay of some of the
more prolific PersistentActor, up to the point where the replaying
cassandra query was taking more time than the view refresh rate, and
thus the cassandra journal was sending more and more read requests, up
to the point where the cassandra cluster couldn't follow anymore and was
using more 100% CPU, dropping writes.

I fixed the problem by not doing any deletion at the expense of using
more and more disk space. I will probably do the deletion in a
controlled way (and making sure we can run a user compaction just after).

HTH,
-- 
Brice Figureau
My Blog: http://www.masterzen.fr/

-- 
>>  Read the docs: http://akka.io/docs/
>>  Check the FAQ: 
>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>  Search the archives: https://groups.google.com/group/akka-user
--- 
You received this message because you are subscribed to the Google Groups "Akka 
User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to akka-user+unsubscr...@googlegroups.com.
To post to this group, send email to akka-user@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

Re: [akka-user] akka.persistence, snapshots and deleting messages from journal

2015-12-14 Thread Patrik Nordwall 
Thanks for sharing, Brice. That is an important finding.

On Wed, Dec 2, 2015 at 2:23 PM, Brice Figureau 
wrote:

> On 18/02/2014 21:39, dpenn...@good-cloud.com wrote:
> > I would assume that a common scenario is to periodically snapshot state
> > and keep 1 or more snapshots.  In this case, I assume that as you delete
> > older snapshots, you would also delete messages older than the snapshot
> > from the journal in order to keep the journal from growing indefinitely.
> >
> > Is this a typical usage or am I missing something?  I don't see anything
> > in the code that automatically does this (and I didn't really expect to
> > given the general flexibility theme of the framework).  However, I don't
> > see anything in the documentation that alludes to any other use of
> > deleteMessage other than in dealing with exceptions.
>
> I thought it would be the good idea to do so, and I was doing this
> beforehand.
>
> But if you're using the Cassandra journal as I do, be aware that
> deleting past events produces tombstones that are removed only after
> gc_grace_seconds and a major compaction happens (or a minor one that
> removes tombstones). Until that time, those tombstones will hurt your
> reads quite hard from what I've found (and it's more than certainly
> related to the static column). This was the issue for me, because I had
> several PersistentView that were doing a constant replay of some of the
> more prolific PersistentActor, up to the point where the replaying
> cassandra query was taking more time than the view refresh rate, and
> thus the cassandra journal was sending more and more read requests, up
> to the point where the cassandra cluster couldn't follow anymore and was
> using more 100% CPU, dropping writes.
>
> I fixed the problem by not doing any deletion at the expense of using
> more and more disk space. I will probably do the deletion in a
> controlled way (and making sure we can run a user compaction just after).
>
> HTH,
> --
> Brice Figureau
> My Blog: http://www.masterzen.fr/
>
> --
> >>  Read the docs: http://akka.io/docs/
> >>  Check the FAQ:
> http://doc.akka.io/docs/akka/current/additional/faq.html
> >>  Search the archives:
> https://groups.google.com/group/akka-user
> ---
> You received this message because you are subscribed to the Google Groups
> "Akka User List" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to akka-user+unsubscr...@googlegroups.com.
> To post to this group, send email to akka-user@googlegroups.com.
> Visit this group at http://groups.google.com/group/akka-user.
> For more options, visit https://groups.google.com/d/optout.
>



-- 

Patrik Nordwall
Typesafe  -  Reactive apps on the JVM
Twitter: @patriknw

-- 
>>  Read the docs: http://akka.io/docs/
>>  Check the FAQ: 
>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>  Search the archives: https://groups.google.com/group/akka-user
--- 
You received this message because you are subscribed to the Google Groups "Akka 
User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to akka-user+unsubscr...@googlegroups.com.
To post to this group, send email to akka-user@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.