date:20160426

Re: Inconsistent Reads after Restoring Snapshot

2016-04-26 Thread Romain Hardouin

You can make a restore on the new node A (don't forget to set the token(s) in 
cassandra.yaml), start the node with -Dcassandra.join_ring=false and then run a 
repair on it. Have a look at 
https://issues.apache.org/jira/browse/CASSANDRA-6961
Best,
Romain 

Le Mardi 26 avril 2016 4h26, Anuj Wadehra  a écrit :
 

 Hi,
We have 2.0.14. We use RF=3 and read/write at Quorum. Moreover, we dont use 
incremental backups. As per the documentation at 
https://docs.datastax.com/en/cassandra/2.0/cassandra/operations/ops_backup_snapshot_restore_t.html
 , if i need to restore a Snapshot on SINGLE node in a cluster, I would run 
repair at the end. But while the repair is going on, reads may get inconsistent.

Consider following scenario:10 AM Daily Snapshot taken of node A and moved to 
backup location11 AM A record is inserted such that node A and B insert the 
record but there is a mutation drop on node C.1 PM Node A crashes and data is 
restored from latest 10 AM snapshot. Now, only Node B has the record.
Now, my question is:
Till the repair is completed on node A,a read at Quorum may return inconsistent 
result based on the nodes from which data is read.If data is read from node A 
and node C, nothing is returned and if data is read from node A and node B, 
record is returned. This is a vital point which is not highlighted anywhere.

Please confirm my understanding.If my understanding is right, how to make sure 
that my reads are not inconsistent while a node is being repair after restoring 
a snapshot.
I think, autobootstrapping the node without joining the ring till the repair is 
completed, is an alternative option. But snapshots save lot of streaming as 
compared to bootstrap.
Will incremental backups guarantee that 
ThanksAnuj

Sent from Yahoo Mail on Android

Re: Inconsistent Reads after Restoring Snapshot

2016-04-26 Thread Anuj Wadehra

Thanks Romain !! So just to clarify, you are suggesting following steps:

10 AM Daily Snapshot taken of node A and moved to backup location
11 AM A record is inserted such that node A and B insert the record but there
is a mutation drop on node C.
1 PM Node A crashes
1 PM Follow following steps to restore the 10 AM snapshot on node A:
1. Restore the data as mentioned in
https://docs.datastax.com/en/cassandra/2.0/cassandra/operations/ops_backup_snapshot_restore_t.html

with ONE EXCEPTION >> start node A with -Dcassandra.join_ring=false
.
2. Run repair
3. Retstart the node A with -Dcassandra.join_ring=true

Please confirm.

I was not aware that join_ring can also be used using a normal reboot. I
thought it was only an option during autobootstrap :)

Thanks
Anuj

On Tue, 26/4/16, Romain Hardouin wrote:

Subject: Re: Inconsistent Reads after Restoring Snapshot
To: "user@cassandra.apache.org"
Date: Tuesday, 26 April, 2016, 12:47 PM

You can make a restore on the new node A (don't
forget to set the token(s) in cassandra.yaml), start the
node with -Dcassandra.join_ring=false and then run a repair
on it. Have a look at https://issues.apache.org/jira/browse/CASSANDRA-6961
Best,
Romain

Le Mardi 26 avril
2016 4h26, Anuj Wadehra a
écrit :

Hi,
We
have 2.0.14. We use RF=3 and read/write at Quorum. Moreover,
we dont use incremental backups. As per the documentation
at ,
if i need to restore a Snapshot on SINGLE node in a cluster,
I would run repair at the end. But while the repair is going
on, reads may get inconsistent.

Consider
following scenario:10
AM Daily Snapshot taken of node A and moved to backup
location11
AM A record is inserted such that node A and B insert the
record but there is a mutation drop on node C.1
PM Node A crashes and data is restored from latest 10 AM
snapshot. Now, only Node B has the record.
Now,
my question is:
Till
the repair is completed on node A,a read at Quorum may
return inconsistent result based on the nodes from which
data is read.If data is read from node A and node C, nothing
is returned and if data is read from node A and node B,
record is returned. This is a vital point which is not
highlighted anywhere.

Please
confirm my understanding.If my understanding is right, how
to make sure that my reads are not inconsistent while a node
is being repair after restoring a snapshot.
I
think, autobootstrapping the node without joining the ring
till the repair is completed, is an alternative option. But
snapshots save lot of streaming as compared to
bootstrap.
Will
incremental backups guarantee that
ThanksAnuj

Sent
from Yahoo Mail on Android

Re: Inconsistent Reads after Restoring Snapshot

2016-04-26 Thread Romain Hardouin

Yes the "Node restart method" with -Dcassandra.join_ring=false. Note that they
advise to make a repair anyway. But thanks to join_ring=false the node will
hibernate and not serve stale data.Tell me if I'm wrong you assume that server
A is still OK, therefore system keyspace still exist? If not (disk KO) it's not
the same procedure (hence the tokens in cassandra.yaml that I mentioned).
Actually I'm not sure of what you assume by "node A crashes". You should try on
a test cluster or with CCM (https://github.com/pcmanus/ccm) in order to
familiarize yourself with the procedure.
Romain

Le Mardi 26 avril 2016 11h02, Anuj Wadehra a écrit
:

Thanks Romain !! So just to clarify, you are suggesting following steps:

with ONE EXCEPTION >> start node A with -Dcassandra.join_ring=false
.
2. Run repair
3. Retstart the node A with -Dcassandra.join_ring=true

Please confirm.

I was not aware that join_ring can also be used using a normal reboot. I
thought it was only an option during autobootstrap :)

Thanks
Anuj

On Tue, 26/4/16, Romain Hardouin wrote:

Subject: Re: Inconsistent Reads after Restoring Snapshot
To: "user@cassandra.apache.org"
Date: Tuesday, 26 April, 2016, 12:47 PM

Le Mardi 26 avril
2016 4h26, Anuj Wadehra a
écrit :

Sent
from Yahoo Mail on Android

Re: Upgrading to SSD

2016-04-26 Thread Jérôme Mainaud

Hello,

Maybe you should call « nodetool drain » just before stoping the node.
As it flush the memtables, the commitlog will be empty and so easier to
move.

https://docs.datastax.com/en/cassandra/2.0/cassandra/tools/toolsDrain.html


-- 
Jérôme Mainaud
jer...@mainaud.com

2016-04-26 8:44 GMT+02:00 Anuj Wadehra :

> Thanks All !!
>
> Anuj
> 
> On Mon, 25/4/16, Alain RODRIGUEZ  wrote:
>
>  Subject: Re: Upgrading to SSD
>  To: user@cassandra.apache.org
>  Date: Monday, 25 April, 2016, 2:45 PM
>
>  Hi
>  Anuj, You could do the following instead
>  to minimize server downtime:
>  1. rsync while the server is
>  running2. rsync again
>  to get any new files3.
>  shut server down4.
>  rsync for the 3rd time 5. change directory in yaml and
>  start back up
>  +1
>  Here
>  are some more details about that process and a script doing
>  most of the job:
> thelastpickle.com/blog/2016/02/25/removing-a-disk-mapping-from-cassandra.html
>  Hope it will be useful to
>  you
>  C*heers,---Alain
>  Rodriguez - alain@thelastpickle.comFrance
>  The Last Pickle - Apache Cassandra
>  Consultinghttp://www.thelastpickle.com
>  2016-04-23 21:47 GMT+02:00
>  Jonathan Haddad :
>  You could do the
>  following instead to minimize server downtime:
>  1. rsync while the server is
>  running2. rsync again to get any new
>  files3. shut server down4. rsync for
>  the 3rd time5. change directory in yaml and start
>  back up
>
>
>  On Sat, Apr 23, 2016 at 12:23 PM Clint Martin
>  
>  wrote:
>  As long as you shut down the node before you start
>  copying and moving stuff around it shouldn't matter if
>  you take backups or snapshots or whatever.
>  When you add the filesystem for the ssd will
>  you be removing the existing filesystem? Or will you be able
>  to keep both filesystems mounted at the same time for the
>  migration?  If you can keep them both at the same time then
>  an of system backup isn't strictly necessary.  Just
>  change your data dir config in your yaml. Copy the data and
>  commit from the old dir to the new ssd and restart the node.
>
>  If you can't keep both filesystems mounted
>  concurrently then a remote system is necessary to copy the
>  data to. But the steps and procedure are the same.
>  Running repair before you do the migration
>  isn't strictly necessary. Not a bad idea if you
>  don't mind spending the time. Definitely run repair
>  after you restart the node, especially if you take longer
>  than the hint interval to perform the work.
>  As for your filesystems, there is really
>  nothing special to worry about. Depending on which
>  filesystem you use there are recommendations for tuning and
>  configuration that you should probably follow.
>  (Datastax's recommendations as well as AL tobey's
>  tuning guide are great resources.
> https://tobert.github.io/pages/als-cassandra-21-tuning-guide.html
>  )
>
>
>  Clint
>  On Apr 23, 2016 3:05
>  PM, "Anuj Wadehra" 
>  wrote:
>  Hi
>  We have a 3 node cluster of 2.0.14.
>  We use Read/Write Qorum and RF is 3.  We want to move data
>  and commitlog directory from a SATA HDD to SSD. We have
>  planned to do a rolling upgrade.
>  We plan to run repair -pr on all
>  nodes  to sync data upfront and then execute following
>  steps on each server one by one:
>  1. Take backup of data/commitlog
>  directory to external server.2. Change mount
>  points so that Cassandra data/commitlog directory now points
>  to SSD.3. Copy files from external backup to
>  SSD.4. Start Cassandra.5. Run full
>  repair on the node before starting step 1 on next
>  node.
>  Questions:1. Is copying
>  commitlog and data directory good enough or we should go for
>  taking snapshot of each node and restoring data from that
>  snapshot?
>  2. Any
>  precautions we need to take while moving data to new SSD?
>  File system format of two disks etc.
>  3. Should we drain data before
>  taking backup? We are also restoring commitlog directory
>  from backup.
>  4. I have
>  added repair to sync full data upfront and a repair after
>  restoring data on each node. Sounds safe and
>  logical?
>  5. Any
>  problems you see with mentioned approach? Any better
>  approach?
>  ThanksAnuj
>
>  Sent
>  from Yahoo Mail on
>  Android
>
>
>

Re: Data repairing on one node questionably affects data on others

2016-04-26 Thread Jean Carlo

Did you use a backup of the keyspace system?

If not, you might do removenode of that node and re added to the cluster to
re generate new tokens.


Saludos

Jean Carlo

"The best way to predict the future is to invent it" Alan Kay

On Tue, Apr 26, 2016 at 12:06 AM, ssiv...@gmail.com 
wrote:

> Hi All,
>
> I have cluster of 7 nodes completely balanced (each node owns ~500GB of
> data).
> And I have one keyspace and one table and three replicas. Than, I just
> failed one node's disk, replace it with a new one and started repairing.
> During that process I noticed that additional two nodes have started
> getting data, and at the end of the repairing three nodes have twice more
> data than at the beginning.
> I'm curious, is it a normal behavior for Cassandra? Why not only one node,
> but three, have gotten data during repairing? May be it's because of clocks
> skew?
>
> Thanks!
>
> --
> best regards,
> Sergey
>
>

Re: Data repairing on one node questionably affects data on others

2016-04-26 Thread ssiv...@gmail.com

No, I didn't. I just want to understand how C* handle such case and what 
is a predictable behavior.


On 04/26/2016 02:51 PM, Jean Carlo wrote:

Did you use a backup of the keyspace system?

If not, you might do removenode of that node and re added to the 
cluster to re generate new tokens.



Saludos

Jean Carlo

"The best way to predict the future is to invent it" Alan Kay

On Tue, Apr 26, 2016 at 12:06 AM, ssiv...@gmail.com 
 > wrote:


Hi All,

I have cluster of 7 nodes completely balanced (each node owns
~500GB of data).
And I have one keyspace and one table and three replicas. Than, I
just failed one node's disk, replace it with a new one and started
repairing.
During that process I noticed that additional two nodes have
started getting data, and at the end of the repairing three nodes
have twice more data than at the beginning.
I'm curious, is it a normal behavior for Cassandra? Why not only
one node, but three, have gotten data during repairing? May be
it's because of clocks skew?

Thanks!

-- 
best regards,

Sergey




--
Thanks,
Serj

Re: Data repairing on one node questionably affects data on others

2016-04-26 Thread Jean Carlo

Well that may explain why you do have an unbalanced distribution of data.
It is not cassandra normal behaivor. If you lose your disk, and you don't
have a backup of your data (at least system because it is not replicated),
this is what I think should be the normal procedure to recover your node.

- stop cassandra on the node down
- nodetool removenode id
- update de conf of your cluster to remove the ip of the node removed
(cassandra.yaml, cassandra-topology.properties). You might want to make a
copy of your original conf
- restart your cluster
- check the node does not appear in nodetool status
- restore your previous conf of your cluster. including the node that was
down
- on the node down, if its ip address is in the list of seeds, remove it in
order to do the boobstrap
- restart cassandra in every node one by one.

It is what we do in our cluster.



Saludos

Jean Carlo

"The best way to predict the future is to invent it" Alan Kay

On Tue, Apr 26, 2016 at 2:25 PM, ssiv...@gmail.com 
wrote:

> No, I didn't. I just want to understand how C* handle such case and what
> is a predictable behavior.
>
>
> On 04/26/2016 02:51 PM, Jean Carlo wrote:
>
> Did you use a backup of the keyspace system?
>
> If not, you might do removenode of that node and re added to the cluster
> to re generate new tokens.
>
>
> Saludos
>
> Jean Carlo
>
> "The best way to predict the future is to invent it" Alan Kay
>
> On Tue, Apr 26, 2016 at 12:06 AM, ssiv...@gmail.com <
> ssiv...@gmail.com> wrote:
>
>> Hi All,
>>
>> I have cluster of 7 nodes completely balanced (each node owns ~500GB of
>> data).
>> And I have one keyspace and one table and three replicas. Than, I just
>> failed one node's disk, replace it with a new one and started repairing.
>> During that process I noticed that additional two nodes have started
>> getting data, and at the end of the repairing three nodes have twice more
>> data than at the beginning.
>> I'm curious, is it a normal behavior for Cassandra? Why not only one
>> node, but three, have gotten data during repairing? May be it's because of
>> clocks skew?
>>
>> Thanks!
>>
>> --
>> best regards,
>> Sergey
>>
>>
>
> --
> Thanks,
> Serj
>
>

[RELEASE] Apache Cassandra 2.1.14 released

2016-04-26 Thread Jake Luciani

The Cassandra team is pleased to announce the release of Apache Cassandra
version 2.1.14.

Apache Cassandra is a fully distributed database. It is the right choice
when you need scalability and high availability without compromising
performance.

 http://cassandra.apache.org/

Downloads of source and binary distributions are listed in our download
section:

 http://cassandra.apache.org/download/

This version is a bug fix release[1] on the 2.1 series. As always, please
pay
attention to the release notes[2] and Let us know[3] if you were to
encounter
any problem.

Enjoy!

[1]: http://goo.gl/7lm5sY (CHANGES.txt)
[2]: http://goo.gl/SUIzT9 (NEWS.txt)
[3]: https://issues.apache.org/jira/browse/CASSANDRA

[RELEASE] Apache Cassandra 2.2.6 released

2016-04-26 Thread Jake Luciani

The Cassandra team is pleased to announce the release of Apache Cassandra
version 2.2.6.

Apache Cassandra is a fully distributed database. It is the right choice
when you need scalability and high availability without compromising
performance.

 http://cassandra.apache.org/

Downloads of source and binary distributions are listed in our download
section:

 http://cassandra.apache.org/download/

This version is a bug fix release[1] on the 2.2 series. As always, please
pay
attention to the release notes[2] and Let us know[3] if you were to
encounter
any problem.

Enjoy!

[1]: http://goo.gl/yCpWu7 (CHANGES.txt)
[2]: http://goo.gl/qktJUS (NEWS.txt)
[3]: https://issues.apache.org/jira/browse/CASSANDRA

Re: Security assessment of Cassandra

2016-04-26 Thread Jack Krupansky

Just following up... Oleg, have you gotten a satisfactory level of feedback
from the community on the security assessment issues?

And if there is any sort of final assessment that can be publicly accessed,
that would be great.

-- Jack Krupansky

On Thu, Feb 11, 2016 at 3:29 PM, oleg yusim  wrote:

> Greetings,
>
> Performing security assessment of Cassandra with the goal of generating
> STIG for Cassandra (iase.disa.mil/stigs/Pages/a-z.aspx) I ran across some
> questions regarding the way certain security features are implemented (or
> not) in Cassandra.
>
> I composed the list of questions on these topics, which I wasn't able to
> find definitive answer to anywhere else and posted it here:
>
> https://drive.google.com/open?id=0B2L9nW4Cyj41YWd1UkI4ZXVPYmM
>
> It is shared with all the members of that list, and any of the members of
> this list is welcome to comment on this document (there is a place for
> community comments specially reserved near each of the questions and my
> take on it).
>
> I would greatly appreciate Cassandra community help here.
>
> Thanks,
>
> Oleg
>

Re: Inconsistent Reads after Restoring Snapshot

Re: Inconsistent Reads after Restoring Snapshot

Re: Inconsistent Reads after Restoring Snapshot

Re: Upgrading to SSD

Re: Data repairing on one node questionably affects data on others

Re: Data repairing on one node questionably affects data on others

Re: Data repairing on one node questionably affects data on others

[RELEASE] Apache Cassandra 2.1.14 released

[RELEASE] Apache Cassandra 2.2.6 released

Re: Security assessment of Cassandra

10 matches

Site Navigation

Mail list logo

Footer information