Re: what's the difference between repair CF separately and repair the entire node?

2011-09-13 Thread Sylvain Lebresne 
On Wed, Sep 14, 2011 at 2:38 AM, Yan Chunlu  wrote:
> me neither don't want to repair one CF at the time.
> the "node repair" took a week and still running, compactionstats and
> netstream shows nothing is running on every node,  and also no error
> message, no exception, really no idea what was it doing,

To add to the list of things repair does wrong in 0.7, we'll have to add that
if one of the node participating in the repair (so any node that share a range
with the node on which repair was started) goes down (even for a short time),
then the repair will simply hang forever doing nothing. And no specific
error message will be logged. That could be what happened. Again, recent
releases of 0.8 fix that too.

--
Sylvain

> I stopped yesterday.  maybe I should run repair again while disable
> compaction on all nodes?
> thanks!
>
> On Wed, Sep 14, 2011 at 6:57 AM, Peter Schuller
>  wrote:
>>
>> > I think it is a serious problem since I can not "repair".  I am
>> > using cassandra on production servers. is there some way to fix it
>> > without upgrade?  I heard of that 0.8.x is still not quite ready in
>> > production environment.
>>
>> It is a serious issue if you really need to repair one CF at the time.
>> However, looking at your original post it seems this is not
>> necessarily your issue. Do you need to, or was your concern rather the
>> overall time repair took?
>>
>> There are other things that are improved in 0.8 that affect 0.7. In
>> particular, (1) in 0.7 compaction, including validating compactions
>> that are part of repair, is non-concurrent so if your repair starts
>> while there is a long-running compaction going it will have to wait,
>> and (2) semi-related is that the merkle tree calculation that is part
>> of repair/anti-entropy may happen "out of synch" if one of the nodes
>> participating happen to be busy with compaction. This in turns causes
>> additional data to be sent as part of repair.
>>
>> That might be why your immediately following repair took a long time,
>> but it's difficult to tell.
>>
>> If you're having issues with repair and large data sets, I would
>> generally say that upgrading to 0.8 is recommended. However, if you're
>> on 0.7.4, beware of
>> https://issues.apache.org/jira/browse/CASSANDRA-3166
>>
>> --
>> / Peter Schuller (@scode on twitter)
>
>

Re: what's the difference between repair CF separately and repair the entire node?

2011-09-13 Thread Yan Chunlu 
me neither don't want to repair one CF at the time.

the "node repair" took a week and still running, compactionstats and
netstream shows nothing is running on every node,  and also no error
message, no exception, really no idea what was it doing,  I stopped
yesterday.  maybe I should run repair again while disable  compaction on all
nodes?

thanks!


On Wed, Sep 14, 2011 at 6:57 AM, Peter Schuller  wrote:

> > I think it is a serious problem since I can not "repair".  I am
> > using cassandra on production servers. is there some way to fix it
> > without upgrade?  I heard of that 0.8.x is still not quite ready in
> > production environment.
>
> It is a serious issue if you really need to repair one CF at the time.
> However, looking at your original post it seems this is not
> necessarily your issue. Do you need to, or was your concern rather the
> overall time repair took?
>
> There are other things that are improved in 0.8 that affect 0.7. In
> particular, (1) in 0.7 compaction, including validating compactions
> that are part of repair, is non-concurrent so if your repair starts
> while there is a long-running compaction going it will have to wait,
> and (2) semi-related is that the merkle tree calculation that is part
> of repair/anti-entropy may happen "out of synch" if one of the nodes
> participating happen to be busy with compaction. This in turns causes
> additional data to be sent as part of repair.
>
> That might be why your immediately following repair took a long time,
> but it's difficult to tell.
>
> If you're having issues with repair and large data sets, I would
> generally say that upgrading to 0.8 is recommended. However, if you're
> on 0.7.4, beware of
> https://issues.apache.org/jira/browse/CASSANDRA-3166
>
> --
> / Peter Schuller (@scode on twitter)
>

Re: what's the difference between repair CF separately and repair the entire node?

2011-09-13 Thread Peter Schuller 
> I think it is a serious problem since I can not "repair".  I am
> using cassandra on production servers. is there some way to fix it
> without upgrade?  I heard of that 0.8.x is still not quite ready in
> production environment.

It is a serious issue if you really need to repair one CF at the time.
However, looking at your original post it seems this is not
necessarily your issue. Do you need to, or was your concern rather the
overall time repair took?

There are other things that are improved in 0.8 that affect 0.7. In
particular, (1) in 0.7 compaction, including validating compactions
that are part of repair, is non-concurrent so if your repair starts
while there is a long-running compaction going it will have to wait,
and (2) semi-related is that the merkle tree calculation that is part
of repair/anti-entropy may happen "out of synch" if one of the nodes
participating happen to be busy with compaction. This in turns causes
additional data to be sent as part of repair.

That might be why your immediately following repair took a long time,
but it's difficult to tell.

If you're having issues with repair and large data sets, I would
generally say that upgrading to 0.8 is recommended. However, if you're
on 0.7.4, beware of
https://issues.apache.org/jira/browse/CASSANDRA-3166

-- 
/ Peter Schuller (@scode on twitter)

Re: what's the difference between repair CF separately and repair the entire node?

2011-09-13 Thread Peter Schuller 
>> It's "okay" but won't do what you want; due to a bug you'll see
>> streaming of data for other column families than the one you're trying
>> to repair. This will be fixed in 1.0.
>
> I think we might be running into this. Is CASSANDRA-2280 the issue
> you're referring to?

Yes. Sorry for not providing the reference.

-- 
/ Peter Schuller (@scode on twitter)

Re: Configuring multi DC cluster

2011-09-13 Thread aaron morton 
Looks good to me. Last time I checked the Partitioner did not take the DC into 
consideration https://issues.apache.org/jira/browse/CASSANDRA-3047

Good luck. 

-
Aaron Morton
Freelance Cassandra Developer
@aaronmorton
http://www.thelastpickle.com

On 14/09/2011, at 8:41 AM, Anand Somani wrote:

> Hi,
> 
> Just trying to setup a cluster of 4 nodes for multiDC scenario - with 2 nodes 
> in each DC. This is all on the same box just for testing the configuration 
> aspect. I have configured things as
> PropertyFile
> 127.0.0.4=SC:rack1
> 127.0.0.5=SC:rack2
> 127.0.0.6=AT:rack1
> 127.0.0.7=AT:rack2
> # default for unknown nodes
> default=SC:rack1
> Setup initial tokens as - advised 
> configured keyspace with SC:2, AT:2
> ring looks like
> Address Status State   LoadOwnsToken  
>  
>
> 85070591730234615865843651857942052865  
> 127.0.0.4   Up Normal  464.98 KB   50.00%  0  
>  
> 127.0.0.5   Up Normal  464.98 KB   0.00%   1  
>  
> 127.0.0.6   Up Normal  464.99 KB   50.00%  
> 85070591730234615865843651857942052864  
> 127.0.0.7   Up Normal  464.99 KB   0.00%   
> 85070591730234615865843651857942052865
> Is that what I should expect the ring to look like? Is there anything else I 
> should be testing/validating to make sure that things are configured 
> correctly for NTS?
> 
> Thanks
> Anand

Re: Index search in provided list of rows (list of rowKeys).

2011-09-13 Thread aaron morton 
Not sure it's a feature cassandra needs, it would radically change the meaning 
of get_indexes_slices(). If you already know the row keys the assumption would 
be you know they are the rows you want to get. 

Feel free to add a Jira though. 

IMHO this sounds more like Sphinx not supporting all the features you need, 
rather than cassandra. Can you use a different search engine such as Solr, 
Solandra or Elastic Search? Or 

Cheers
-
Aaron Morton
Freelance Cassandra Developer
@aaronmorton
http://www.thelastpickle.com

On 13/09/2011, at 10:27 AM, Evgeniy Ryabitskiy wrote:

> Something like this.
> 
> Actually I think it's better to extend get_indexed_slice() API instead of 
> creating new one thrift method.
> I wish to have something like this:
> 
> //here we run query to external search engine
> List keys = performSphinxQuery(someFullTextSearchQuery);
> IndexClause indexClause = new IndexClause();
> 
> //required API to set list of keys
> indexClause.setKeys(keys);
> indexClause.setExpressions(someFilteringExpressions);
> List finalResult = get_indexed_slices(colParent, indexClause, colPredicate, 
> cLevel);
> 
> 
> 
> I can't solve my issue with single get_indexed_slice().
> Here is issue in more details: 
> 1) have ~ 6 millions records, in feature could be much more
> 2) have  > 10k different properties (stored as column values in Cassandra), 
> in feature could be much more
> 3) properties are text descriptions , int/float values, string values 
> 4) need to implement search over all properties. For text descriptions: full 
> text search. for int/float properties: range search.
> 5) Search query could use any combination of property descriptions. Like full 
> text search description and some range expression for int/float field.
> 6) have external search engine (Sphinx) that indexed all string and text 
> properties
> 7) still need to perform range search for int, float fields.
> 
> So now I split my query expressions in 2 groups:
> 1) expressions that can be handled by search engine
> 2) others (additional filters)
> 
> For example I run first query to Sphinx and got list of rowKeys, with length 
> of 100k.  (mark as RESULT1)
> Now I need to filter it by second group of expressions. For example I have 
> simple expression: "age > 25".
> So imagine I would run get_indexed_slice() with this query and could possibly 
> get half of my records in result. (mark as RESULT2)
> Then I would need to get intersection between RESULT1 and RESULT2 on client 
> side, which could take a lot of time and memory.
> That is why I can't use single get_indexed_slice here.
> 
> For me is better to iterate RESULT1 (with 100k records) at client side to 
> filter by age and got 10-50k record as final result. Disadvantage here is 
> that I have to fetch all 100k records.
> 
> Evgeny.
> 
> 
> 
> 
> 
> 
> 
> 
>

Configuring multi DC cluster

2011-09-13 Thread Anand Somani 
Hi,

Just trying to setup a cluster of 4 nodes for multiDC scenario - with 2
nodes in each DC. This is all on the same box just for testing the
configuration aspect. I have configured things as

   - PropertyFile
  - 127.0.0.4=SC:rack1
  127.0.0.5=SC:rack2
  127.0.0.6=AT:rack1
  127.0.0.7=AT:rack2
  # default for unknown nodes
  default=SC:rack1
   - Setup initial tokens as - advised
   - configured keyspace with SC:2, AT:2
   - ring looks like
  - Address Status State   LoadOwns
  Token

  85070591730234615865843651857942052865
  127.0.0.4   Up Normal  464.98 KB   50.00%
  0
  127.0.0.5   Up Normal  464.98 KB   0.00%
  1
  127.0.0.6   Up Normal  464.99 KB   50.00%
  85070591730234615865843651857942052864
  127.0.0.7   Up Normal  464.99 KB   0.00%
  85070591730234615865843651857942052865

Is that what I should expect the ring to look like? Is there anything else I
should be testing/validating to make sure that things are configured
correctly for NTS?

Thanks
Anand

Re: StorageProxy Mbean not exposed in 0.7.8 anymore

2011-09-13 Thread Anand Somani 
yes, I see it now.

Thx

On Tue, Sep 13, 2011 at 10:11 AM, Nick Bailey  wrote:

> The StorageProxyMBean should still be exposed. It won't exist until some
> reads or writes are performed on the cluster though. You may need to
> actually do some reads/writes to see it show up.
>
>
> On Tue, Sep 13, 2011 at 11:53 AM, Anand Somani wrote:
>
>> Hi,
>>
>> Upgraded from 7.4 to 7.8, noticed that StorageProxy (under cassandra.db)
>> is no longer exposed, is that intentional? So the question are these covered
>> somewhere else?
>>
>> Thanks
>> Anand
>>
>>
>>
>

Re: StorageProxy Mbean not exposed in 0.7.8 anymore

2011-09-13 Thread Nick Bailey 
The StorageProxyMBean should still be exposed. It won't exist until some
reads or writes are performed on the cluster though. You may need to
actually do some reads/writes to see it show up.

On Tue, Sep 13, 2011 at 11:53 AM, Anand Somani  wrote:

> Hi,
>
> Upgraded from 7.4 to 7.8, noticed that StorageProxy (under cassandra.db) is
> no longer exposed, is that intentional? So the question are these covered
> somewhere else?
>
> Thanks
> Anand
>
>
>

StorageProxy Mbean not exposed in 0.7.8 anymore

2011-09-13 Thread Anand Somani 
Hi,

Upgraded from 7.4 to 7.8, noticed that StorageProxy (under cassandra.db) is
no longer exposed, is that intentional? So the question are these covered
somewhere else?

Thanks
Anand

Re: Error in upgrading cassandra to 0.8.5

2011-09-13 Thread Roshan Dawrani 
On Tue, Sep 13, 2011 at 7:03 PM, Jonathan Ellis  wrote:

> Just remove the row cache files.
>

Thanks a lot. The 0.8.5 Cassandra started just fine after getting rid of
those *KeyCache files.

-- 
Roshan
Blog: http://roshandawrani.wordpress.com/
Twitter: @roshandawrani 
Skype: roshandawrani

Re: memtable flush thresholds

2011-09-13 Thread Jonathan Ellis 
Because for small columns the Java "container" objects are larger than
the actual column data.

On Tue, Sep 13, 2011 at 9:38 AM, Radim Kolar  wrote:
>> flush of Memtable-idx_graphable@915643571
> (4671498/96780112 serialized/live bytes, 59891 ops)
> can somebody explain why difference between live and serialized is so big?
> Serialized bytes are actual bytes written to sstable?
>
>



-- 
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of DataStax, the source for professional Cassandra support
http://www.datastax.com

Re: Error in upgrading cassandra to 0.8.5

2011-09-13 Thread Jonathan Ellis 
More or less.  NEWS.txt explains upgrade procedure in more detail.

On Tue, Sep 13, 2011 at 8:43 AM, Roshan Dawrani  wrote:
> Hi.
> Thanks for replying back. Just a quick cross-confirmation, if you don't
> mind:
> So, I can do the following without fearing any loss of data, etc?
> * "nodetool drain" and bring down the 0.7.0 node (it's a one node setup that
> I am upgrading)
> * remove all files from saved_caches directory (both system*-KeyCache as
> well as application-specific*-KeyCache files)
> * bring up the new 0.8.5 node (pointing to same "data" and "commitlog"
> directories)
> The *KeyCache files can be removed safely in between, right?
> Cheers.
> On Tue, Sep 13, 2011 at 7:03 PM, Jonathan Ellis  wrote:
>>
>> Just remove the row cache files.
>>
>> On Tue, Sep 13, 2011 at 8:23 AM, Roshan Dawrani 
>> wrote:
>> > Hi,
>> > I am in the process of upgrading Cassandra to the recently released
>> > v0.8.5
>> > and facing an issue.
>> > We had two Cassandra environments - one having 0.7.0 and another 0.7.2.
>> > The
>> > upgrade on 0.7.2 has happened smoothly, but on the environment having
>> > 0.7.0,
>> > when I make the switch, the DB fails to come up with the error below.
>> > Can someone help please?
>> >
>> > -
>> >  INFO 13:03:46,025 JNA not found. Native methods will be disabled.
>> >  INFO 13:03:46,034 Loading settings from
>> > file:/apache-cassandra-0.8.5/conf/cassandra.yaml
>> >  INFO 13:03:46,195 DiskAccessMode 'auto' determined to be mmap,
>> > indexAccessMode is mmap
>> >  ...
>> >  INFO 13:03:46,307 Removing compacted SSTable files (see
>> > http://wiki.apache.org/cassandra/MemtableSSTable)
>> >  INFO 13:03:46,379 reading saved cache
>> > /cassandra/saved_caches/system-IndexInfo-KeyCache
>> > ERROR 13:03:46,380 Exception encountered during startup.
>> > java.lang.NegativeArraySizeException
>> >         at
>> >
>> > org.apache.cassandra.cache.AutoSavingCache.readSaved(AutoSavingCache.java:130)
>> >         at
>> >
>> > org.apache.cassandra.db.ColumnFamilyStore.(ColumnFamilyStore.java:273)
>> >         at
>> >
>> > org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:465)
>> >         at
>> >
>> > org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:435)
>> >         at org.apache.cassandra.db.Table.initCf(Table.java:369)
>> >         at org.apache.cassandra.db.Table.(Table.java:306)
>> >         at org.apache.cassandra.db.Table.open(Table.java:111)
>> >         at
>> > org.apache.cassandra.db.SystemTable.checkHealth(SystemTable.java:217)
>> >         at
>> >
>> > org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassandraDaemon.java:154)
>> >         at
>> >
>> > org.apache.cassandra.service.AbstractCassandraDaemon.activate(AbstractCassandraDaemon.java:341)
>> >         at
>> >
>> > org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:97)
>> >
>> > -
>> > --
>> > Roshan
>> > Blog: http://roshandawrani.wordpress.com/
>> > Twitter: @roshandawrani
>> > Skype: roshandawrani
>> >
>> >
>>
>>
>>
>> --
>> Jonathan Ellis
>> Project Chair, Apache Cassandra
>> co-founder of DataStax, the source for professional Cassandra support
>> http://www.datastax.com
>
>
>
> --
> Roshan
> Blog: http://roshandawrani.wordpress.com/
> Twitter: @roshandawrani
> Skype: roshandawrani
>
>



-- 
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of DataStax, the source for professional Cassandra support
http://www.datastax.com

Re: Can initial_token be decimal or hexadecimal format?

2011-09-13 Thread Jonathan Ellis 
Parsing is done by TokenFactory.fromString, so yes, it matters per partitioner.

On Tue, Sep 13, 2011 at 4:09 AM, Mck  wrote:
> And does it matter when using different partitoners?
>
> In the config it seems only strings are used.
> In RP it parses this string into a BigInteger so it needs to be in
> decimal format,
> but for ByteOrderPartitioner it uses FBUtilities.hexToBytes(..) when
> translating a string to a token (BytesToken).
>
> More to the point...
> For a 3 node cluster using BOP where my largest token will be
> 0x8000 (coincidently 2**127)
> should i write out initial_tokens like
>
> node0: 0
> node1: 2AAA
> node2: 5554
>
> or like
>
> node0: 0
> node1: 56713727820156410577229101238628035242
> node2: 113427455640312821154458202477256070484
>
>
> If it is the former there's some important documentation missing.
>
> ~mck
>
>
> ps CASSANDRA-1006 seems to be of some relation.
>
>
>



-- 
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of DataStax, the source for professional Cassandra support
http://www.datastax.com

Re: adding node without bootstrap

2011-09-13 Thread Edward Capriolo 
2011/9/13 Radim Kolar 

> If i add node without bootstrap and run repair on both nodes then no data
> are moved to new node, just new writes are going to new node. This is
> expected behaviour or it is some kind of user or program error?
>

No. This is exactly what you are asking the node to do by setting false. The
goal of auto_bootstrap=true is to migrate data to the node automatically.
You can turn this method off if you have moved data their yourself out of
band (maybe rsync).

If you join a node with auto_bootstrap=false you had better be working at
quorum or higher to avoid stale/not found reads. You should then repair the
node right away to get all the missing data back on the node. This is not
suggested. It is best to leave auto_boostrap=true and let Cassandra handle
this on the front end.

Edward

Re: memtable flush thresholds

2011-09-13 Thread Radim Kolar 
 flush of Memtable-idx_graphable@915643571*(4671498/96780112 

serialized/live bytes, 59891 ops)
can somebody explain why difference between live and serialized is so 
big? Serialized bytes are actual bytes written to sstable?

*

Re: balancing issue with Random partitioner

2011-09-13 Thread David McNelis 
So we tried decommissionning the 100.5 node, then re added it to the ring.
 It now  appears to be  streaming data properly to that node.

On Tue, Sep 13, 2011 at 6:56 AM, David McNelis
wrote:

> I ran a repair on 100.5.  It returned back almost immediately and netstats
> and tpstats don't show any additional activity.  Nor does the Java process
> start using more juice.
>
> When I look at the cfstats for the main keyspace, number of keys is 0 in
> all cases.  When I look at other nodes in the cluster there is a more
> realistic distribution.
>
>
> On Mon, Sep 12, 2011 at 7:18 PM, aaron morton wrote:
>
>> Try a reapir on 100.5 , it will then request the data from the existing
>> nodes.
>>
>> You will then need to clean on the existing three nodes once the repair
>> has completed.
>>
>> Cheers
>>
>>  -
>> Aaron Morton
>> Freelance Cassandra Developer
>> @aaronmorton
>> http://www.thelastpickle.com
>>
>> On 13/09/2011, at 9:32 AM, David McNelis wrote:
>>
>> Auto-bootstrapping is turned on and the node had  been started several
>> hours ago.   Since the node already shows up as part of the ring I would
>> imagine that nodetool join wouldn't do anything.Is there a command to
>> jumpstart bootstrapping?
>>
>> On Mon, Sep 12, 2011 at 4:22 PM, Jonathan Ellis wrote:
>>
>>> Looks kind of like the 4th node was added to the cluster w/o
>>> bootstrapping.
>>>
>>> On Mon, Sep 12, 2011 at 3:59 PM, David McNelis
>>>  wrote:
>>> > We are running the datastax .8 rpm distro.  We have a situation where
>>> we
>>> > have 4 nodes and each owns 25% of the keys.  However the last node in
>>> the
>>> > ring does not seem to be  getting much of a load at all.
>>> > We are using the random partitioner, we have a total of about 20k keys
>>> that
>>> > are sequential...
>>> > Our nodetool ring  output is currently:
>>> > Address DC  RackStatus State   Load
>>>  Owns
>>> >Token
>>> >
>>> >127605887595351923798765477786913079296
>>> > 10.181.138.167  datacenter1 rack1   Up Normal  99.37 GB
>>> >  25.00%  0
>>> > 192.168.100.6   datacenter1 rack1   Up Normal  106.25 GB
>>> > 25.00%  42535295865117307932921825928971026432
>>> > 10.181.137.37   datacenter1 rack1   Up Normal  77.7 GB
>>> > 25.00%  85070591730234615865843651857942052863
>>> > 192.168.100.5   datacenter1 rack1   Up Normal  494.67 KB
>>> > 25.00%  127605887595351923798765477786913079296
>>> >
>>> > Nothing is running on netstats on .37 or .5.
>>> > I understand that the nature of the beast would cause the load to
>>> differ
>>> > between the nodes...but I wouldn't expect it to be so drastic.  We had
>>> the
>>> > token for .37 set to 85070591730234615865843651857942052864, and I
>>> > decremented and moved it to try to kickstart some streaming on the
>>> thought
>>> > that something may have failed, but that didn't yield any appreciable
>>> > results.
>>> > Are we seeing completely abnormal behavior?  Should I consider making
>>> the
>>> > token for the fourth node considerably smaller?  We calculated the
>>> node's
>>> > tokens using the standard python script.
>>> > --
>>> > David McNelis
>>> > Lead Software Engineer
>>> > Agentis Energy
>>> > www.agentisenergy.com
>>> > o: 630.359.6395
>>> > c: 219.384.5143
>>> > A Smart Grid technology company focused on helping consumers of energy
>>> > control an often under-managed resource.
>>> >
>>> >
>>>
>>>
>>>
>>> --
>>> Jonathan Ellis
>>> Project Chair, Apache Cassandra
>>> co-founder of DataStax, the source for professional Cassandra support
>>> http://www.datastax.com
>>>
>>
>>
>>
>> --
>> *David McNelis*
>> Lead Software Engineer
>> Agentis Energy
>> www.agentisenergy.com
>> o: 630.359.6395
>>  c: 219.384.5143
>>
>> *A Smart Grid technology company focused on helping consumers of energy
>> control an often under-managed resource.*
>>
>>
>>
>>
>
>
> --
> *David McNelis*
> Lead Software Engineer
> Agentis Energy
> www.agentisenergy.com
> o: 630.359.6395
> c: 219.384.5143
>
> *A Smart Grid technology company focused on helping consumers of energy
> control an often under-managed resource.*
>
>
>


-- 
*David McNelis*
Lead Software Engineer
Agentis Energy
www.agentisenergy.com
o: 630.359.6395
c: 219.384.5143

*A Smart Grid technology company focused on helping consumers of energy
control an often under-managed resource.*

adding node without bootstrap

2011-09-13 Thread Radim Kolar 
If i add node without bootstrap and run repair on both nodes then no 
data are moved to new node, just new writes are going to new node. This 
is expected behaviour or it is some kind of user or program error?

Re: Replace Live Node

2011-09-13 Thread Kyle Gibson 
Just wanted to update and say the move was completed without any
issues in v 0.7.8. Thanks for all the help everyone.

On Mon, Sep 12, 2011 at 4:31 PM, Jeremy Hanna
 wrote:
>> So to move data from node with token 0, the new node needs to have
>> initial token set to 170141183460469231731687303715884105727 ?
>
> I would do this route.
>
>> Another idea: could I move token to 1, and then use token 0 on the new node?
>
> nodetool move prior to 0.8 is a very heavy operation.

Re: Error in upgrading cassandra to 0.8.5

2011-09-13 Thread Roshan Dawrani 
Hi.

Thanks for replying back. Just a quick cross-confirmation, if you don't
mind:

So, I can do the following without fearing any loss of data, etc?

* "nodetool drain" and bring down the 0.7.0 node (it's a one node setup that
I am upgrading)
* remove all files from saved_caches directory (both system*-KeyCache as
well as application-specific*-KeyCache files)
* bring up the new 0.8.5 node (pointing to same "data" and "commitlog"
directories)

The *KeyCache files can be removed safely in between, right?

Cheers.

On Tue, Sep 13, 2011 at 7:03 PM, Jonathan Ellis  wrote:

> Just remove the row cache files.
>
> On Tue, Sep 13, 2011 at 8:23 AM, Roshan Dawrani 
> wrote:
> > Hi,
> > I am in the process of upgrading Cassandra to the recently released
> v0.8.5
> > and facing an issue.
> > We had two Cassandra environments - one having 0.7.0 and another 0.7.2.
> The
> > upgrade on 0.7.2 has happened smoothly, but on the environment having
> 0.7.0,
> > when I make the switch, the DB fails to come up with the error below.
> > Can someone help please?
> >
> -
> >  INFO 13:03:46,025 JNA not found. Native methods will be disabled.
> >  INFO 13:03:46,034 Loading settings from
> > file:/apache-cassandra-0.8.5/conf/cassandra.yaml
> >  INFO 13:03:46,195 DiskAccessMode 'auto' determined to be mmap,
> > indexAccessMode is mmap
> >  ...
> >  INFO 13:03:46,307 Removing compacted SSTable files (see
> > http://wiki.apache.org/cassandra/MemtableSSTable)
> >  INFO 13:03:46,379 reading saved cache
> > /cassandra/saved_caches/system-IndexInfo-KeyCache
> > ERROR 13:03:46,380 Exception encountered during startup.
> > java.lang.NegativeArraySizeException
> > at
> >
> org.apache.cassandra.cache.AutoSavingCache.readSaved(AutoSavingCache.java:130)
> > at
> >
> org.apache.cassandra.db.ColumnFamilyStore.(ColumnFamilyStore.java:273)
> > at
> >
> org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:465)
> > at
> >
> org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:435)
> > at org.apache.cassandra.db.Table.initCf(Table.java:369)
> > at org.apache.cassandra.db.Table.(Table.java:306)
> > at org.apache.cassandra.db.Table.open(Table.java:111)
> > at
> > org.apache.cassandra.db.SystemTable.checkHealth(SystemTable.java:217)
> > at
> >
> org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassandraDaemon.java:154)
> > at
> >
> org.apache.cassandra.service.AbstractCassandraDaemon.activate(AbstractCassandraDaemon.java:341)
> > at
> > org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:97)
> >
> -
> > --
> > Roshan
> > Blog: http://roshandawrani.wordpress.com/
> > Twitter: @roshandawrani
> > Skype: roshandawrani
> >
> >
>
>
>
> --
> Jonathan Ellis
> Project Chair, Apache Cassandra
> co-founder of DataStax, the source for professional Cassandra support
> http://www.datastax.com
>



-- 
Roshan
Blog: http://roshandawrani.wordpress.com/
Twitter: @roshandawrani 
Skype: roshandawrani

Re: Error in upgrading cassandra to 0.8.5

2011-09-13 Thread Jonathan Ellis 
Just remove the row cache files.

On Tue, Sep 13, 2011 at 8:23 AM, Roshan Dawrani  wrote:
> Hi,
> I am in the process of upgrading Cassandra to the recently released v0.8.5
> and facing an issue.
> We had two Cassandra environments - one having 0.7.0 and another 0.7.2. The
> upgrade on 0.7.2 has happened smoothly, but on the environment having 0.7.0,
> when I make the switch, the DB fails to come up with the error below.
> Can someone help please?
> -
>  INFO 13:03:46,025 JNA not found. Native methods will be disabled.
>  INFO 13:03:46,034 Loading settings from
> file:/apache-cassandra-0.8.5/conf/cassandra.yaml
>  INFO 13:03:46,195 DiskAccessMode 'auto' determined to be mmap,
> indexAccessMode is mmap
>  ...
>  INFO 13:03:46,307 Removing compacted SSTable files (see
> http://wiki.apache.org/cassandra/MemtableSSTable)
>  INFO 13:03:46,379 reading saved cache
> /cassandra/saved_caches/system-IndexInfo-KeyCache
> ERROR 13:03:46,380 Exception encountered during startup.
> java.lang.NegativeArraySizeException
>         at
> org.apache.cassandra.cache.AutoSavingCache.readSaved(AutoSavingCache.java:130)
>         at
> org.apache.cassandra.db.ColumnFamilyStore.(ColumnFamilyStore.java:273)
>         at
> org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:465)
>         at
> org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:435)
>         at org.apache.cassandra.db.Table.initCf(Table.java:369)
>         at org.apache.cassandra.db.Table.(Table.java:306)
>         at org.apache.cassandra.db.Table.open(Table.java:111)
>         at
> org.apache.cassandra.db.SystemTable.checkHealth(SystemTable.java:217)
>         at
> org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassandraDaemon.java:154)
>         at
> org.apache.cassandra.service.AbstractCassandraDaemon.activate(AbstractCassandraDaemon.java:341)
>         at
> org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:97)
> -
> --
> Roshan
> Blog: http://roshandawrani.wordpress.com/
> Twitter: @roshandawrani
> Skype: roshandawrani
>
>



-- 
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of DataStax, the source for professional Cassandra support
http://www.datastax.com

Error in upgrading cassandra to 0.8.5

2011-09-13 Thread Roshan Dawrani 
Hi,

I am in the process of upgrading Cassandra to the recently released v0.8.5
and facing an issue.

We had two Cassandra environments - one having 0.7.0 and another 0.7.2. The
upgrade on 0.7.2 has happened smoothly, but on the environment having 0.7.0,
when I make the switch, the DB fails to come up with the error below.

Can someone help please?

-
 INFO 13:03:46,025 JNA not found. Native methods will be disabled.
 INFO 13:03:46,034 Loading settings from
file:/apache-cassandra-0.8.5/conf/cassandra.yaml
 INFO 13:03:46,195 DiskAccessMode 'auto' determined to be mmap,
indexAccessMode is mmap
 ...
 INFO 13:03:46,307 Removing compacted SSTable files (see
http://wiki.apache.org/cassandra/MemtableSSTable)
 INFO 13:03:46,379 reading saved cache
/cassandra/saved_caches/system-IndexInfo-KeyCache
ERROR 13:03:46,380 Exception encountered during startup.
java.lang.NegativeArraySizeException
at
org.apache.cassandra.cache.AutoSavingCache.readSaved(AutoSavingCache.java:130)
at
org.apache.cassandra.db.ColumnFamilyStore.(ColumnFamilyStore.java:273)
at
org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:465)
at
org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:435)
at org.apache.cassandra.db.Table.initCf(Table.java:369)
at org.apache.cassandra.db.Table.(Table.java:306)
at org.apache.cassandra.db.Table.open(Table.java:111)
at
org.apache.cassandra.db.SystemTable.checkHealth(SystemTable.java:217)
at
org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassandraDaemon.java:154)
at
org.apache.cassandra.service.AbstractCassandraDaemon.activate(AbstractCassandraDaemon.java:341)
at
org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:97)
-

-- 
Roshan
Blog: http://roshandawrani.wordpress.com/
Twitter: @roshandawrani 
Skype: roshandawrani

Re: balancing issue with Random partitioner

2011-09-13 Thread David McNelis 
I ran a repair on 100.5.  It returned back almost immediately and netstats
and tpstats don't show any additional activity.  Nor does the Java process
start using more juice.

When I look at the cfstats for the main keyspace, number of keys is 0 in all
cases.  When I look at other nodes in the cluster there is a more realistic
distribution.


On Mon, Sep 12, 2011 at 7:18 PM, aaron morton wrote:

> Try a reapir on 100.5 , it will then request the data from the existing
> nodes.
>
> You will then need to clean on the existing three nodes once the repair has
> completed.
>
> Cheers
>
> -
> Aaron Morton
> Freelance Cassandra Developer
> @aaronmorton
> http://www.thelastpickle.com
>
> On 13/09/2011, at 9:32 AM, David McNelis wrote:
>
> Auto-bootstrapping is turned on and the node had  been started several
> hours ago.   Since the node already shows up as part of the ring I would
> imagine that nodetool join wouldn't do anything.Is there a command to
> jumpstart bootstrapping?
>
> On Mon, Sep 12, 2011 at 4:22 PM, Jonathan Ellis  wrote:
>
>> Looks kind of like the 4th node was added to the cluster w/o
>> bootstrapping.
>>
>> On Mon, Sep 12, 2011 at 3:59 PM, David McNelis
>>  wrote:
>> > We are running the datastax .8 rpm distro.  We have a situation where we
>> > have 4 nodes and each owns 25% of the keys.  However the last node in
>> the
>> > ring does not seem to be  getting much of a load at all.
>> > We are using the random partitioner, we have a total of about 20k keys
>> that
>> > are sequential...
>> > Our nodetool ring  output is currently:
>> > Address DC  RackStatus State   Load
>>  Owns
>> >Token
>> >
>> >127605887595351923798765477786913079296
>> > 10.181.138.167  datacenter1 rack1   Up Normal  99.37 GB
>> >  25.00%  0
>> > 192.168.100.6   datacenter1 rack1   Up Normal  106.25 GB
>> > 25.00%  42535295865117307932921825928971026432
>> > 10.181.137.37   datacenter1 rack1   Up Normal  77.7 GB
>> > 25.00%  85070591730234615865843651857942052863
>> > 192.168.100.5   datacenter1 rack1   Up Normal  494.67 KB
>> > 25.00%  127605887595351923798765477786913079296
>> >
>> > Nothing is running on netstats on .37 or .5.
>> > I understand that the nature of the beast would cause the load to differ
>> > between the nodes...but I wouldn't expect it to be so drastic.  We had
>> the
>> > token for .37 set to 85070591730234615865843651857942052864, and I
>> > decremented and moved it to try to kickstart some streaming on the
>> thought
>> > that something may have failed, but that didn't yield any appreciable
>> > results.
>> > Are we seeing completely abnormal behavior?  Should I consider making
>> the
>> > token for the fourth node considerably smaller?  We calculated the
>> node's
>> > tokens using the standard python script.
>> > --
>> > David McNelis
>> > Lead Software Engineer
>> > Agentis Energy
>> > www.agentisenergy.com
>> > o: 630.359.6395
>> > c: 219.384.5143
>> > A Smart Grid technology company focused on helping consumers of energy
>> > control an often under-managed resource.
>> >
>> >
>>
>>
>>
>> --
>> Jonathan Ellis
>> Project Chair, Apache Cassandra
>> co-founder of DataStax, the source for professional Cassandra support
>> http://www.datastax.com
>>
>
>
>
> --
> *David McNelis*
> Lead Software Engineer
> Agentis Energy
> www.agentisenergy.com
> o: 630.359.6395
> c: 219.384.5143
>
> *A Smart Grid technology company focused on helping consumers of energy
> control an often under-managed resource.*
>
>
>
>


-- 
*David McNelis*
Lead Software Engineer
Agentis Energy
www.agentisenergy.com
o: 630.359.6395
c: 219.384.5143

*A Smart Grid technology company focused on helping consumers of energy
control an often under-managed resource.*

Re: NullPointerException doing cleanup

2011-09-13 Thread Fredrik Stigbäck 
Thanks.
Will upgrade to 0.8.5.

Regards
/Fredrik

2011/9/13 Sylvain Lebresne 

> This should be fixed in 0.8.5 (more precisely by
> https://issues.apache.org/jira/browse/CASSANDRA-3039)
>
> --
> Sylvain
>
> On Tue, Sep 13, 2011 at 12:53 PM, Fredrik Stigbäck
>  wrote:
> > Trying to do a nodetool cleanup but gets a NullPointerException.
> > I've done some debugging and found out that the property
> > org.apache.cassandra.db.compaction.PrecompactedRow.compactedCf is
> > initilaized to null since
> > ColumnFamilyStore.removeDeleted(ColumnFamily, int) returns null which I
> > interpret as the row is marked for delete and has no columns.
> > This will cause org.apache.cassandra.db.ColumnIndexer.serializeInternal
> to
> > get a IIterableColumns == null.
> > Any ideas? Corrupted SStable or a bug or something else?
> > Running Cassandra 0.8.4 on a Windows 7 but have seen the same behaviour
> on
> > Linux.
> >
> > 09:09:50 ERROR [AbstractCassandraDaemon] [] Fatal exception in thread
> > Thread[CompactionExecutor:8,1,main]
> > java.lang.NullPointerException
> > at
> >
> org.apache.cassandra.db.ColumnIndexer.serializeInternal(ColumnIndexer.java:60)
> > at
> > org.apache.cassandra.db.ColumnIndexer.serialize(ColumnIndexer.java:50)
> > at
> >
> org.apache.cassandra.db.compaction.PrecompactedRow.write(PrecompactedRow.java:110)
> > at
> >
> org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:132)
> > at
> >
> org.apache.cassandra.db.compaction.CompactionManager.doCleanupCompaction(CompactionManager.java:866)
> > at
> >
> org.apache.cassandra.db.compaction.CompactionManager.access$500(CompactionManager.java:65)
> > at
> >
> org.apache.cassandra.db.compaction.CompactionManager$2.call(CompactionManager.java:204)
> > at
> > java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> > at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > at java.lang.Thread.run(Thread.java:662)
> >
> > /Regards Fredrik
> >
> >
>



-- 
Fredrik Larsson Stigbäck
SiteVision AB Vasagatan 10, 107 10 Örebro
019-17 30 30

Re: NullPointerException doing cleanup

2011-09-13 Thread Sylvain Lebresne 
This should be fixed in 0.8.5 (more precisely by
https://issues.apache.org/jira/browse/CASSANDRA-3039)

--
Sylvain

On Tue, Sep 13, 2011 at 12:53 PM, Fredrik Stigbäck
 wrote:
> Trying to do a nodetool cleanup but gets a NullPointerException.
> I've done some debugging and found out that the property
> org.apache.cassandra.db.compaction.PrecompactedRow.compactedCf is
> initilaized to null since
> ColumnFamilyStore.removeDeleted(ColumnFamily, int) returns null which I
> interpret as the row is marked for delete and has no columns.
> This will cause org.apache.cassandra.db.ColumnIndexer.serializeInternal  to
> get a IIterableColumns == null.
> Any ideas? Corrupted SStable or a bug or something else?
> Running Cassandra 0.8.4 on a Windows 7 but have seen the same behaviour on
> Linux.
>
> 09:09:50 ERROR [AbstractCassandraDaemon] [] Fatal exception in thread
> Thread[CompactionExecutor:8,1,main]
> java.lang.NullPointerException
>     at
> org.apache.cassandra.db.ColumnIndexer.serializeInternal(ColumnIndexer.java:60)
>     at
> org.apache.cassandra.db.ColumnIndexer.serialize(ColumnIndexer.java:50)
>     at
> org.apache.cassandra.db.compaction.PrecompactedRow.write(PrecompactedRow.java:110)
>     at
> org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:132)
>     at
> org.apache.cassandra.db.compaction.CompactionManager.doCleanupCompaction(CompactionManager.java:866)
>     at
> org.apache.cassandra.db.compaction.CompactionManager.access$500(CompactionManager.java:65)
>     at
> org.apache.cassandra.db.compaction.CompactionManager$2.call(CompactionManager.java:204)
>     at
> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>     at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>     at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>     at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>     at java.lang.Thread.run(Thread.java:662)
>
> /Regards Fredrik
>
>

NullPointerException doing cleanup

2011-09-13 Thread Fredrik Stigbäck 
Trying to do a nodetool cleanup but gets a NullPointerException.
I've done some debugging and found out that the property
org.apache.cassandra.db.compaction.PrecompactedRow.compactedCf is
initilaized to null since
ColumnFamilyStore.removeDeleted(ColumnFamily, int) returns null which I
interpret as the row is marked for delete and has no columns.
This will cause org.apache.cassandra.db.ColumnIndexer.serializeInternal  to
get a IIterableColumns == null.
Any ideas? Corrupted SStable or a bug or something else?
Running Cassandra 0.8.4 on a Windows 7 but have seen the same behaviour on
Linux.

09:09:50 ERROR [AbstractCassandraDaemon] [] Fatal exception in thread
Thread[CompactionExecutor:8,1,main]
java.lang.NullPointerException
at
org.apache.cassandra.db.ColumnIndexer.serializeInternal(ColumnIndexer.java:60)
at
org.apache.cassandra.db.ColumnIndexer.serialize(ColumnIndexer.java:50)
at
org.apache.cassandra.db.compaction.PrecompactedRow.write(PrecompactedRow.java:110)
at
org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:132)
at
org.apache.cassandra.db.compaction.CompactionManager.doCleanupCompaction(CompactionManager.java:866)
at
org.apache.cassandra.db.compaction.CompactionManager.access$500(CompactionManager.java:65)
at
org.apache.cassandra.db.compaction.CompactionManager$2.call(CompactionManager.java:204)
at
java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

/Regards Fredrik

AUTO: Manoj Chaudhary is out of the office (returning 09/15/2011)

2011-09-13 Thread Manoj Chaudhary 


I am out of the office until 09/15/2011.

I will be out office starting 09/13/2011 for WebSphere F&C conference ,
returning back on 09/16/2011.

 I will respond to your message when I return to the office on 09/16//2011.

For anything urgent please contact Clara Liang( Clara C Liang/Silicon
Valley/IBM).


Note: This is an automated response to your message  "Can initial_token be
decimal or hexadecimal format?" sent on 9/13/11 3:09:15.

This is the only notification you will receive while this person is away.

Can initial_token be decimal or hexadecimal format?

2011-09-13 Thread Mck 
And does it matter when using different partitoners?

In the config it seems only strings are used.
In RP it parses this string into a BigInteger so it needs to be in
decimal format,
but for ByteOrderPartitioner it uses FBUtilities.hexToBytes(..) when
translating a string to a token (BytesToken).

More to the point...
For a 3 node cluster using BOP where my largest token will be
0x8000 (coincidently 2**127)
should i write out initial_tokens like

node0: 0
node1: 2AAA
node2: 5554

or like

node0: 0
node1: 56713727820156410577229101238628035242
node2: 113427455640312821154458202477256070484


If it is the former there's some important documentation missing.

~mck


ps CASSANDRA-1006 seems to be of some relation.