> De : Sylvain Lebresne [mailto:sylv...@datastax.com]
> > 2) copy the snapshot sstable in the right place and call the JMX
> > method
> > loadNewSSTables() (in the column family MBean, which mean you need to
> > do that per-CF).
>
> How does this affect the contents of the CommitLogs? I mean, I
> imagine using this for rollback'ing errors in the app side, like
> some/many/all objects deleted. How will updates/deletes saved in
> CommitLogs impact the data restored this way? Also: while I'm restoring
> a KS of CF, is it possible to cut write requests but not reads? I hope
> I'm explaining myself clearly...
In particular, I'm thinking on a restore like this:
* the app does something stupid.
* (if possible) I stop writes to the KS or CF.
* remove the db files with the wrong data.
* I put the db files from the restore.
* nodetool refresh
* profit!
But on Friday I did it slightly different (too tired, I was confused),
and only did the refresh on one node and then a repair on the same, and
it failed like this:
* state just after the app did something stupid:
root@pnscassandra04:~# tree /var/opt/hosting/db/cassandra/data/one_cf/cf_1/
/var/opt/hosting/db/cassandra/data/one_cf/cf_1/
|-- one_cf-cf_1-hd-13-CompressionInfo.db
|-- one_cf-cf_1-hd-13-Data.db
|-- one_cf-cf_1-hd-13-Filter.db
|-- one_cf-cf_1-hd-13-Index.db
|-- one_cf-cf_1-hd-13-Statistics.db
|-- one_cf-cf_1-hd-14-CompressionInfo.db
|-- one_cf-cf_1-hd-14-Data.db
|-- one_cf-cf_1-hd-14-Filter.db
|-- one_cf-cf_1-hd-14-Index.db
|-- one_cf-cf_1-hd-14-Statistics.db
|-- one_cf-cf_1-hd-15-CompressionInfo.db
|-- one_cf-cf_1-hd-15-Data.db
|-- one_cf-cf_1-hd-15-Filter.db
|-- one_cf-cf_1-hd-15-Index.db
|-- one_cf-cf_1-hd-15-Statistics.db
`-- snapshots
`-- 1344609231532
|-- one_cf-cf_1-hd-10-CompressionInfo.db
|-- one_cf-cf_1-hd-10-Data.db
|-- one_cf-cf_1-hd-10-Filter.db
|-- one_cf-cf_1-hd-10-Index.db
|-- one_cf-cf_1-hd-10-Statistics.db
|-- one_cf-cf_1-hd-11-CompressionInfo.db
|-- one_cf-cf_1-hd-11-Data.db
|-- one_cf-cf_1-hd-11-Filter.db
|-- one_cf-cf_1-hd-11-Index.db
|-- one_cf-cf_1-hd-11-Statistics.db
|-- one_cf-cf_1-hd-12-CompressionInfo.db
|-- one_cf-cf_1-hd-12-Data.db
|-- one_cf-cf_1-hd-12-Filter.db
|-- one_cf-cf_1-hd-12-Index.db
|-- one_cf-cf_1-hd-12-Statistics.db
|-- one_cf-cf_1-hd-9-CompressionInfo.db
|-- one_cf-cf_1-hd-9-Data.db
|-- one_cf-cf_1-hd-9-Filter.db
|-- one_cf-cf_1-hd-9-Index.db
`-- one_cf-cf_1-hd-9-Statistics.db
* I remove the 'wrong' databases
root@pnscassandra04:~# rm -v /var/opt/hosting/db/cassandra/data/one_cf/cf_1/*.db
removed
`/var/opt/hosting/db/cassandra/data/one_cf/cf_1/one_cf-cf_1-hd-13-CompressionInfo.db'
removed
`/var/opt/hosting/db/cassandra/data/one_cf/cf_1/one_cf-cf_1-hd-13-Data.db'
removed
`/var/opt/hosting/db/cassandra/data/one_cf/cf_1/one_cf-cf_1-hd-13-Filter.db'
removed
`/var/opt/hosting/db/cassandra/data/one_cf/cf_1/one_cf-cf_1-hd-13-Index.db'
removed
`/var/opt/hosting/db/cassandra/data/one_cf/cf_1/one_cf-cf_1-hd-13-Statistics.db'
removed
`/var/opt/hosting/db/cassandra/data/one_cf/cf_1/one_cf-cf_1-hd-14-CompressionInfo.db'
removed
`/var/opt/hosting/db/cassandra/data/one_cf/cf_1/one_cf-cf_1-hd-14-Data.db'
removed
`/var/opt/hosting/db/cassandra/data/one_cf/cf_1/one_cf-cf_1-hd-14-Filter.db'
removed
`/var/opt/hosting/db/cassandra/data/one_cf/cf_1/one_cf-cf_1-hd-14-Index.db'
removed
`/var/opt/hosting/db/cassandra/data/one_cf/cf_1/one_cf-cf_1-hd-14-Statistics.db'
removed
`/var/opt/hosting/db/cassandra/data/one_cf/cf_1/one_cf-cf_1-hd-15-CompressionInfo.db'
removed
`/var/opt/hosting/db/cassandra/data/one_cf/cf_1/one_cf-cf_1-hd-15-Data.db'
removed
`/var/opt/hosting/db/cassandra/data/one_cf/cf_1/one_cf-cf_1-hd-15-Filter.db'
removed
`/var/opt/hosting/db/cassandra/data/one_cf/cf_1/one_cf-cf_1-hd-15-Index.db'
removed
`/var/opt/hosting/db/cassandra/data/one_cf/cf_1/one_cf-cf_1-hd-15-Statistics.db'
* restore
root@pnscassandra04:~# for i in
/var/opt/hosting/db/cassandra/data/one_cf/cf_1/snapshots/1344609231532/*; do
ln -v $i /var/opt/hosting/db/cassandra/data/one_cf/cf_1/;
done
`/var/opt/hosting/db/cassandra/data/one_cf/cf_1/one_cf-cf_1-hd-10-CompressionInfo.db'
=>
`/var/opt/hosting/db/cassandra/data/one_cf/cf_1/snapshots/1344609231532/one_cf-cf_1-hd-10-CompressionInfo.db'
`/var/opt/hosting/db/cassandra/data/one_cf/cf_1/one_cf-cf_1-hd-10-Data.db' =>
`/var/opt/hosting/db/cassandra/data/one_cf/cf_1/snapshots/1344609231532/one_cf-cf_1-hd-10-Data.db'
`/var/opt/hosting/db/cassandra/data/one_cf/cf_1/one_cf-cf_1-hd-10-Filter.db' =>
`/var/opt/hosting/db/cassandra/data/one_cf/cf_1/snapshots/1344609231532/one_cf-cf_1-hd-10-Filter.db'
`/var/opt/hosting/db/cassandra/data/one_cf/cf_1/one_cf-cf_1-hd-10-Index.db' =>
`/var/opt/hosting/db/cassandra/data/one_cf/cf_1/snapshots/1344609231532/one_cf-cf_1-hd-10-Index.db'
`/var/opt/hosting/db/cassandra/data/one_cf/cf_1/one_cf-cf_1-hd-10-Statistics.db'
=>
`/var/opt/hosting/db/cassandra/data/one_cf/cf_1/snapshots/1344609231532/one_cf-cf_1-hd-10-Statistics.db'
`/var/opt/hosting/db/cassandra/data/one_cf/cf_1/one_cf-cf_1-hd-11-CompressionInfo.db'
=>
`/var/opt/hosting/db/cassandra/data/one_cf/cf_1/snapshots/1344609231532/one_cf-cf_1-hd-11-CompressionInfo.db'
`/var/opt/hosting/db/cassandra/data/one_cf/cf_1/one_cf-cf_1-hd-11-Data.db' =>
`/var/opt/hosting/db/cassandra/data/one_cf/cf_1/snapshots/1344609231532/one_cf-cf_1-hd-11-Data.db'
`/var/opt/hosting/db/cassandra/data/one_cf/cf_1/one_cf-cf_1-hd-11-Filter.db' =>
`/var/opt/hosting/db/cassandra/data/one_cf/cf_1/snapshots/1344609231532/one_cf-cf_1-hd-11-Filter.db'
`/var/opt/hosting/db/cassandra/data/one_cf/cf_1/one_cf-cf_1-hd-11-Index.db' =>
`/var/opt/hosting/db/cassandra/data/one_cf/cf_1/snapshots/1344609231532/one_cf-cf_1-hd-11-Index.db'
`/var/opt/hosting/db/cassandra/data/one_cf/cf_1/one_cf-cf_1-hd-11-Statistics.db'
=>
`/var/opt/hosting/db/cassandra/data/one_cf/cf_1/snapshots/1344609231532/one_cf-cf_1-hd-11-Statistics.db'
`/var/opt/hosting/db/cassandra/data/one_cf/cf_1/one_cf-cf_1-hd-12-CompressionInfo.db'
=>
`/var/opt/hosting/db/cassandra/data/one_cf/cf_1/snapshots/1344609231532/one_cf-cf_1-hd-12-CompressionInfo.db'
`/var/opt/hosting/db/cassandra/data/one_cf/cf_1/one_cf-cf_1-hd-12-Data.db' =>
`/var/opt/hosting/db/cassandra/data/one_cf/cf_1/snapshots/1344609231532/one_cf-cf_1-hd-12-Data.db'
`/var/opt/hosting/db/cassandra/data/one_cf/cf_1/one_cf-cf_1-hd-12-Filter.db' =>
`/var/opt/hosting/db/cassandra/data/one_cf/cf_1/snapshots/1344609231532/one_cf-cf_1-hd-12-Filter.db'
`/var/opt/hosting/db/cassandra/data/one_cf/cf_1/one_cf-cf_1-hd-12-Index.db' =>
`/var/opt/hosting/db/cassandra/data/one_cf/cf_1/snapshots/1344609231532/one_cf-cf_1-hd-12-Index.db'
`/var/opt/hosting/db/cassandra/data/one_cf/cf_1/one_cf-cf_1-hd-12-Statistics.db'
=>
`/var/opt/hosting/db/cassandra/data/one_cf/cf_1/snapshots/1344609231532/one_cf-cf_1-hd-12-Statistics.db'
`/var/opt/hosting/db/cassandra/data/one_cf/cf_1/one_cf-cf_1-hd-9-CompressionInfo.db'
=>
`/var/opt/hosting/db/cassandra/data/one_cf/cf_1/snapshots/1344609231532/one_cf-cf_1-hd-9-CompressionInfo.db'
`/var/opt/hosting/db/cassandra/data/one_cf/cf_1/one_cf-cf_1-hd-9-Data.db' =>
`/var/opt/hosting/db/cassandra/data/one_cf/cf_1/snapshots/1344609231532/one_cf-cf_1-hd-9-Data.db'
`/var/opt/hosting/db/cassandra/data/one_cf/cf_1/one_cf-cf_1-hd-9-Filter.db' =>
`/var/opt/hosting/db/cassandra/data/one_cf/cf_1/snapshots/1344609231532/one_cf-cf_1-hd-9-Filter.db'
`/var/opt/hosting/db/cassandra/data/one_cf/cf_1/one_cf-cf_1-hd-9-Index.db' =>
`/var/opt/hosting/db/cassandra/data/one_cf/cf_1/snapshots/1344609231532/one_cf-cf_1-hd-9-Index.db'
`/var/opt/hosting/db/cassandra/data/one_cf/cf_1/one_cf-cf_1-hd-9-Statistics.db'
=>
`/var/opt/hosting/db/cassandra/data/one_cf/cf_1/snapshots/1344609231532/one_cf-cf_1-hd-9-Statistics.db'
* and refresh
root@pnscassandra04:~# nodetool refresh one_cf cf_1
INFO [RMI TCP Connection(54)-10.234.169.244] 2012-08-10 17:00:54,426
ColumnFamilyStore.java (line 460) Loading new SSTables for one_cf/cf_1...
INFO [RMI TCP Connection(54)-10.234.169.244] 2012-08-10 17:00:54,432
ColumnFamilyStore.java (line 488) Renaming new SSTable
/var/opt/hosting/db/cassandra/data/one_cf/cf_1/one_cf-cf_1-hd-9 to
/var/opt/hosting/db/cassandra/data/one_cf/cf_1/one_cf-cf_1-hd-16
INFO [RMI TCP Connection(54)-10.234.169.244] 2012-08-10 17:00:54,432
SSTableReader.java (line 170) Opening
/var/opt/hosting/db/cassandra/data/one_cf/cf_1/one_cf-cf_1-hd-16 (2263898 bytes)
INFO [RMI TCP Connection(54)-10.234.169.244] 2012-08-10 17:00:54,488
ColumnFamilyStore.java (line 488) Renaming new SSTable
/var/opt/hosting/db/cassandra/data/one_cf/cf_1/one_cf-cf_1-hd-11 to
/var/opt/hosting/db/cassandra/data/one_cf/cf_1/one_cf-cf_1-hd-17
INFO [RMI TCP Connection(54)-10.234.169.244] 2012-08-10 17:00:54,489
SSTableReader.java (line 170) Opening
/var/opt/hosting/db/cassandra/data/one_cf/cf_1/one_cf-cf_1-hd-17 (2124847 bytes)
INFO [RMI TCP Connection(54)-10.234.169.244] 2012-08-10 17:00:54,517
ColumnFamilyStore.java (line 488) Renaming new SSTable
/var/opt/hosting/db/cassandra/data/one_cf/cf_1/one_cf-cf_1-hd-10 to
/var/opt/hosting/db/cassandra/data/one_cf/cf_1/one_cf-cf_1-hd-18
INFO [RMI TCP Connection(54)-10.234.169.244] 2012-08-10 17:00:54,517
SSTableReader.java (line 170) Opening
/var/opt/hosting/db/cassandra/data/one_cf/cf_1/one_cf-cf_1-hd-18 (2526508 bytes)
INFO [RMI TCP Connection(54)-10.234.169.244] 2012-08-10 17:00:54,551
ColumnFamilyStore.java (line 488) Renaming new SSTable
/var/opt/hosting/db/cassandra/data/one_cf/cf_1/one_cf-cf_1-hd-12 to
/var/opt/hosting/db/cassandra/data/one_cf/cf_1/one_cf-cf_1-hd-19
INFO [RMI TCP Connection(54)-10.234.169.244] 2012-08-10 17:00:54,551
SSTableReader.java (line 170) Opening
/var/opt/hosting/db/cassandra/data/one_cf/cf_1/one_cf-cf_1-hd-19 (146340 bytes)
INFO [RMI TCP Connection(54)-10.234.169.244] 2012-08-10 17:00:54,555
ColumnFamilyStore.java (line 510) Loading new SSTables and building secondary
indexes for one_cf/cf_1:
[SSTableReader(path='/var/opt/hosting/db/cassandra/data/one_cf/cf_1/one_cf-cf_1-hd-17-Data.db'),
SSTableReader(path='/var/opt/hosting/db/cassandra/data/one_cf/cf_1/one_cf-cf_1-hd-19-Data.db'),
SSTableReader(path='/var/opt/hosting/db/cassandra/data/one_cf/cf_1/one_cf-cf_1-hd-16-Data.db'),
SSTableReader(path='/var/opt/hosting/db/cassandra/data/one_cf/cf_1/one_cf-cf_1-hd-18-Data.db')]
INFO [RMI TCP Connection(54)-10.234.169.244] 2012-08-10 17:00:54,555
ColumnFamilyStore.java (line 526) Done loading load new SSTables for one_cf/cf_1
* and here's the error at repair time:
root@pnscassandra04:~# nodetool repair one_cf cf_1
INFO [RMI TCP Connection(56)-10.234.169.244] 2012-08-10 17:03:12,354
StorageService.java (line 1925) Starting repair command #3, repairing 5 ranges.
INFO [AntiEntropySessions:5] 2012-08-10 17:03:12,358 AntiEntropyService.java
(line 666) [repair #811be330-e2fc-11e1-0000-b64817724cbd] new session: will
sync pnscassandra04.vdev.bas.s1.p.fti.net/10.234.169.244, /10.234.169.245,
/10.234.169.241, /10.234.169.242, /10.234.169.243 on range
(34028236692093846346337460743176821145,68056473384187692692674921486353642291]
for one_cf.[cf_1]
INFO [AntiEntropySessions:5] 2012-08-10 17:03:12,359 AntiEntropyService.java
(line 871) [repair #811be330-e2fc-11e1-0000-b64817724cbd] requesting merkle
trees for cf_1 (to [/10.234.169.245, /10.234.169.241, /10.234.169.242,
/10.234.169.243, pnscassandra04.vdev.bas.s1.p.fti.net/10.234.169.244])
ERROR [ValidationExecutor:2] 2012-08-10 17:03:12,361
AbstractCassandraDaemon.java (line 134) Exception in thread
Thread[ValidationExecutor:2,1,main]
java.io.IOError: java.io.FileNotFoundException:
/var/opt/hosting/db/cassandra/data/one_cf/cf_1/one_cf-cf_1-hd-13-Data.db (No
such file or directory)
[...]
I understand that this procedure (refresh+repair in one node) makes
absolutely
no sense, but the error leads me to think that after refresh, the 'wrong'
databases
are still taken in account. Is it that the whole procedure is bad from the
start and
I should use SSTableLoader instead?
--
Marcos Dione
SysAdmin
Astek Sud-Est
pour FT/TGPF/OPF/PORTAIL/DOP/HEBEX @ Marco Polo
04 97 12 62 45 - mdione....@orange.com
_________________________________________________________________________________________________________________________
Ce message et ses pieces jointes peuvent contenir des informations
confidentielles ou privilegiees et ne doivent donc
pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce
message par erreur, veuillez le signaler
a l'expediteur et le detruire ainsi que les pieces jointes. Les messages
electroniques etant susceptibles d'alteration,
France Telecom - Orange decline toute responsabilite si ce message a ete
altere, deforme ou falsifie. Merci.
This message and its attachments may contain confidential or privileged
information that may be protected by law;
they should not be distributed, used or copied without authorisation.
If you have received this email in error, please notify the sender and delete
this message and its attachments.
As emails may be altered, France Telecom - Orange is not liable for messages
that have been modified, changed or falsified.
Thank you.
