Hi Andrzej,
On 26.08.14 09:15, Andrzej Dębski wrote:
Hello
Lately I have been reading about a possibility of using Apache Kafka
as journal/snapshot store for akka-persistence.
I am aware of the plugin created by Martin
Krasser: https://github.com/krasserm/akka-persistence-kafka/ and also
I read other topic about Kafka as
journal https://groups.google.com/forum/#!searchin/akka-user/kakfka/akka-user/iIHmvC6bVrI/zeZJtW0_6FwJ.
In both sources I linked two ideas were presented:
1. Set log retention to 7 days, take snapshots every 3 days (example
values)
2. Set log retention to unlimited.
Here is the first question: in first case wouldn't it mean that
persistent views would receive skewed view of the PersistentActor
state (only events from 7 days) - is it really viable solution? As far
as I know PersistentView can only receive events - it can't receive
snapshots from corresponding PersistentActor (which is good in general
case).
PersistentViews can create their own snapshots which are isolated from
the corresponding PersistentActor's snapshots.
Second question (more directed to Martin): in the thread I linked you
wrote:
I don't go into Kafka partitioning details here but it is
possible to implement the journal driver in a way that both a
single persistent actor's data are partitioned *and* kept in order
I am very interested in this idea. AFAIK it is not yet implemented in
current plugin but I was wondering if you could share high level idea
how would you achieve that (one persistent actor, multiple partitions,
ordering ensured)?
The idea is to
- first write events 1 to n to partition 1
- then write events n+1 to 2n to partition 2
- then write events 2n+1 to 3n to partition 3
- ... and so on
This works because a PersistentActor is the only writer to a partitioned
journal topic. During replay, you first replay partition 1, then
partition 2 and so on. This should be rather easy to implement in the
Kafka journal, just didn't have time so far; pull requests are welcome
:) Btw, the Cassandra journal
<https://github.com/krasserm/akka-persistence-cassandra> follows the
very same strategy for scaling with data volume (by using different
partition keys).
Cheers,
Martin
--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ:
http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google
Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to akka-user+unsubscr...@googlegroups.com
<mailto:akka-user+unsubscr...@googlegroups.com>.
To post to this group, send email to akka-user@googlegroups.com
<mailto:akka-user@googlegroups.com>.
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.
--
Martin Krasser
blog: http://krasserm.blogspot.com
code: http://github.com/krasserm
twitter: http://twitter.com/mrt1nz
--
Read the docs: http://akka.io/docs/
Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to akka-user+unsubscr...@googlegroups.com.
To post to this group, send email to akka-user@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.
