Re: Point in Time Recovery

[Jonathan Lacefield]

Hello,

  Have you tried the procedure documented here:
http://www.datastax.com/documentation/cassandra/1.2/cassandra/configuration/configLogArchive_t.html

Thanks,

Jonathan

Jonathan Lacefield
Solutions Architect, DataStax
(404) 822 3487






On Thu, Apr 10, 2014 at 1:19 AM, Dennis Schwan wrote:

> Hey there,
>
> do you know any description how to perform a point-in-time recovery
> using the archived commitlogs?
> We have already tried several things but it just did not work.
> We have a 20 Node Cluster (10 in each DC).
>
> Thanks in Advance,
> Dennis
>
> --
> Dennis Schwan
>
> Oracle DBA
> Mail Core
>
> 1&1 Internet AG | Brauerstraße 48 | 76135 Karlsruhe | Germany
> Phone: +49 721 91374-8738
> E-Mail: dennis.sch...@1und1.de | Web: www.1und1.de
>
> Hauptsitz Montabaur, Amtsgericht Montabaur, HRB 6484
>
> Vorstand: Ralph Dommermuth, Frank Einhellinger, Robert Hoffmann, Andreas
> Hofmann, Markus Huhn, Hans-Henning Kettler, Uwe Lamnek, Jan Oetjen,
> Christian Würst
> Aufsichtsratsvorsitzender: Michael Scheeren
>
> Member of United Internet
>
> Diese E-Mail kann vertrauliche und/oder gesetzlich geschützte
> Informationen enthalten. Wenn Sie nicht der bestimmungsgemäße Adressat sind
> oder diese E-Mail irrtümlich erhalten haben, unterrichten Sie bitte den
> Absender und vernichten Sie diese Email. Anderen als dem bestimmungsgemäßen
> Adressaten ist untersagt, diese E-Mail zu speichern, weiterzuleiten oder
> ihren Inhalt auf welche Weise auch immer zu verwenden.
>
> This E-Mail may contain confidential and/or privileged information. If you
> are not the intended recipient of this E-Mail, you are hereby notified that
> saving, distribution or use of the content of this E-Mail in any way is
> prohibited. If you have received this E-Mail in error, please notify the
> sender and delete the E-Mail.
>

Re: Point in Time Recovery

[Robert Coli]

On Thu, Apr 10, 2014 at 1:19 AM, Dennis Schwan wrote:

> do you know any description how to perform a point-in-time recovery
> using the archived commitlogs?
> We have already tried several things but it just did not work.
>

Are you restoring the entire *cluster* to a point in time, or a given node?
And why?

The only people who are likely to have any experience/expertise with that
archived commitlog stuff are the people from Netflix who contributed it.

=Rob

Re: Point in Time Recovery

[Dennis Schwan]

Hi Rob,

we need this for the worst case scenario, so our intention is to restore the 
entire cluster, not a single node.
I am really not sure what the correct procedure would be. I think we have 
configured everything properly so the nodes are archiving the commitlogs (even 
though I am not sure when exactly a single commitlog gets archived) but we 
could not manage to recover a database from those archived commitlogs. What we 
did was:

1. Insert "good" data into the cluster
2. Take a snapshot
3. Insert "good" data into the cluster
4. Note Timestamp
5. Insert "bad" data into the cluster
6. Shut down whole cluster
7. Delete SSTables and restore from snapshot
8. Set Timestamp in commitlog_archiving.properties and restart the nodes

The archived commitlogs are copied to the restore directory and afterwards 
cassandra is replaying those commitlogs but still we only see the data from the 
snapshot, not the commitlogs.

Regards,
Dennis

P.S.: Cassandra 2.0.6

Am 10.04.2014 23:17, schrieb Robert Coli:
On Thu, Apr 10, 2014 at 1:19 AM, Dennis Schwan 
mailto:dennis.sch...@1und1.de>> wrote:
do you know any description how to perform a point-in-time recovery
using the archived commitlogs?
We have already tried several things but it just did not work.

Are you restoring the entire *cluster* to a point in time, or a given node? And 
why?

The only people who are likely to have any experience/expertise with that 
archived commitlog stuff are the people from Netflix who contributed it.

=Rob






--
Dennis Schwan

Oracle DBA
Mail Core

1&1 Internet AG | Brauerstraße 48 | 76135 Karlsruhe | Germany
Phone: +49 721 91374-8738
E-Mail: dennis.sch...@1und1.de | Web: 
www.1und1.de

Hauptsitz Montabaur, Amtsgericht Montabaur, HRB 6484

Vorstand: Ralph Dommermuth, Frank Einhellinger, Robert Hoffmann, Andreas 
Hofmann, Markus Huhn, Hans-Henning Kettler, Uwe Lamnek, Jan Oetjen, Christian 
Würst
Aufsichtsratsvorsitzender: Michael Scheeren

Member of United Internet

Diese E-Mail kann vertrauliche und/oder gesetzlich geschützte Informationen 
enthalten. Wenn Sie nicht der bestimmungsgemäße Adressat sind oder diese E-Mail 
irrtümlich erhalten haben, unterrichten Sie bitte den Absender und vernichten 
Sie diese Email. Anderen als dem bestimmungsgemäßen Adressaten ist untersagt, 
diese E-Mail zu speichern, weiterzuleiten oder ihren Inhalt auf welche Weise 
auch immer zu verwenden.

This E-Mail may contain confidential and/or privileged information. If you are 
not the intended recipient of this E-Mail, you are hereby notified that saving, 
distribution or use of the content of this E-Mail in any way is prohibited. If 
you have received this E-Mail in error, please notify the sender and delete the 
E-Mail.

Re: Point in Time Recovery

[Robert Coli]

On Fri, Apr 11, 2014 at 1:21 AM, Dennis Schwan wrote:

>  The archived commitlogs are copied to the restore directory and
> afterwards cassandra is replaying those commitlogs but still we only see
> the data from the snapshot, not the commitlogs.
>

If you turn up debug log4j settings, you should be able to see whether the
replay is correctly applying mutations to memtables.

Do you see a flush of memtables to sstables at the end of commitlog replay?
If not, memtables are not being created by commitlog replay.

=Rob

Re: Point in Time Recovery

[Dennis Schwan]

Hi Rob,

I know it has been a while but we managed to perform a point-in-time recovery.
I am not really sure what the problem was but I guess it has to do with not 
reading exactly (use GMT and not local time zone, copying archivelogs to the 
wrong place, etc.).

So everything should work as described but I think there should be a little 
more automation in it.

Thanks all,
Dennis

Am 11.04.2014 21:11, schrieb Robert Coli:
On Fri, Apr 11, 2014 at 1:21 AM, Dennis Schwan 
mailto:dennis.sch...@1und1.de>> wrote:
The archived commitlogs are copied to the restore directory and afterwards 
cassandra is replaying those commitlogs but still we only see the data from the 
snapshot, not the commitlogs.

If you turn up debug log4j settings, you should be able to see whether the 
replay is correctly applying mutations to memtables.

Do you see a flush of memtables to sstables at the end of commitlog replay? If 
not, memtables are not being created by commitlog replay.

=Rob



--
Dennis Schwan

Oracle DBA
Mail Core

1&1 Internet AG | Brauerstraße 48 | 76135 Karlsruhe | Germany
Phone: +49 721 91374-8738
E-Mail: dennis.sch...@1und1.de | Web: 
www.1und1.de

Hauptsitz Montabaur, Amtsgericht Montabaur, HRB 6484

Vorstand: Ralph Dommermuth, Frank Einhellinger, Robert Hoffmann, Andreas 
Hofmann, Markus Huhn, Hans-Henning Kettler, Uwe Lamnek, Jan Oetjen, Christian 
Würst
Aufsichtsratsvorsitzender: Michael Scheeren

Member of United Internet

Diese E-Mail kann vertrauliche und/oder gesetzlich geschützte Informationen 
enthalten. Wenn Sie nicht der bestimmungsgemäße Adressat sind oder diese E-Mail 
irrtümlich erhalten haben, unterrichten Sie bitte den Absender und vernichten 
Sie diese Email. Anderen als dem bestimmungsgemäßen Adressaten ist untersagt, 
diese E-Mail zu speichern, weiterzuleiten oder ihren Inhalt auf welche Weise 
auch immer zu verwenden.

This E-Mail may contain confidential and/or privileged information. If you are 
not the intended recipient of this E-Mail, you are hereby notified that saving, 
distribution or use of the content of this E-Mail in any way is prohibited. If 
you have received this E-Mail in error, please notify the sender and delete the 
E-Mail.

Re: Point in Time Recovery

[Robert Coli]

On Tue, Apr 29, 2014 at 7:46 AM, Dennis Schwan wrote:

>  I know it has been a while but we managed to perform a point-in-time
> recovery.
> I am not really sure what the problem was but I guess it has to do with
> not reading exactly (use GMT and not local time zone, copying archivelogs
> to the wrong place, etc.).
>

Glad to hear things are working, thank you for sharing your experience back
with the list community. :)

=Rob