Question about seeds in tow node cluster.

2011-02-13 Thread Xiaobo Gu 
Hi,
If the cluster only have tow nodes, should they both in the seeds list?

Regards,

Xiaobo Gu

Re: Do supercolumns have a purpose?

2011-02-13 Thread David Boxenhorn 
I agree, that is the way to go. Then each piece of new functionality will
not have to be implemented twice.

On Sat, Feb 12, 2011 at 9:41 AM, Stu Hood  wrote:

> I would like to continue to support super columns, but to slowly convert
> them into "compound column names", since that is really all they really are.
>
>
> On Thu, Feb 10, 2011 at 10:16 AM, Frank LoVecchio wrote:
>
>> I've found super column families quite useful when using
>> RandomOrderedPartioner on a low-maintenance cluster (as opposed to
>> Byte/Ordered), e.g. returning ordered data from a TimeUUID comparator type;
>> try doing that with one regular column family and secondary indexes (you
>> could obviously sort on the client side, but that is tedious and not logical
>> for older data).
>>
>> On Thu, Feb 10, 2011 at 12:32 AM, David Boxenhorn wrote:
>>
>>> Mike, my problem is that I have an database and codebase that already
>>> uses supercolumns. If I had to do it over, it wouldn't use them, for the
>>> reasons you point out. In fact, I have a feeling that over time supercolumns
>>> will become deprecated de facto, if not de jure. That's why I would like to
>>> see them represented internally as regular columns, with an upgrade path for
>>> backward compatibility.
>>>
>>> I would love to do it myself! (I haven't looked at the code base, but I
>>> don't understand why it should be so hard.) But my employer has other
>>> ideas...
>>>
>>>
>>> On Wed, Feb 9, 2011 at 8:14 PM, Mike Malone  wrote:
>>>
 On Tue, Feb 8, 2011 at 2:03 AM, David Boxenhorn wrote:

> Shaun, I agree with you, but marking them as deprecated is not good
> enough for me. I can't easily stop using supercolumns. I need an upgrade
> path.
>

 David,

 Cassandra is open source and community developed. The right thing to do
 is what's best for the community, which sometimes conflicts with what's 
 best
 for individual users. Such strife should be minimized, it will never be
 eliminated. Luckily, because this is an open source, liberal licensed
 project, if you feel strongly about something you should feel free to add
 whatever features you want yourself. I'm sure other people in your 
 situation
 will thank you for it.

 At a minimum I think it would behoove you to re-read some of the
 comments here re: why super columns aren't really needed and take another
 look at your data model and code. I would actually be quite surprised to
 find a use of super columns that could not be trivially converted to normal
 columns. In fact, it should be possible to do at the framework/client
 library layer - you probably wouldn't even need to change any application
 code.

 Mike

 On Tue, Feb 8, 2011 at 3:53 AM, Shaun Cutts wrote:
>
>>
>> I'm a newbie here, but, with apologies for my presumptuousness, I
>> think you should deprecate SuperColumns. They are already distracting 
>> you,
>> and as the years go by the cost of supporting them as you add more and 
>> more
>> functionality is only likely to get worse. It would be better to 
>> concentrate
>> on making the "core" column families better (and I'm sure we can all 
>> think
>> of lots of things we'd like).
>>
>> Just dropping SuperColumns would be bad for your reputation -- and for
>> users like David who are currently using them. But if you mark them 
>> clearly
>> as deprecated and explain why and what to do instead (perhaps putting a 
>> bit
>> of effort into migration tools... or even a "virtual" layer supporting
>> arbitrary hierarchical data), then you can drop them in a few years (when
>> you get to 1.0, say), without people feeling betrayed.
>>
>> -- Shaun
>>
>> On Feb 6, 2011, at 3:48 AM, David Boxenhorn wrote:
>>
>> "My main point was to say that it's think it is better to create
>> tickets for what you want, rather than for something else completely
>> different that would, as a by-product, give you what you want."
>>
>> Then let me say what I want: I want supercolumn families to have any
>> feature that regular column families have.
>>
>> My data model is full of supercolumns. I used them, even though I knew
>> it didn't *have to*, "because they were there", which implied to me that 
>> I
>> was supposed to use them for some good reason. Now I suspect that they 
>> will
>> gradually become less and less functional, as features are added to 
>> regular
>> column families and not supported for supercolumn families.
>>
>>
>> On Fri, Feb 4, 2011 at 10:58 AM, Sylvain Lebresne <
>> sylv...@datastax.com> wrote:
>>
>>> On Fri, Feb 4, 2011 at 12:35 AM, Mike Malone wrote:
>>>
 On Thu, Feb 3, 2011 at 6:44 AM, Sylvain Lebresne <
 sylv...@datastax.com> wrote:

> On Thu, Feb 3, 2011 at 3:00 PM, David Boxen

Re: Limit on amount of CFs

2011-02-13 Thread Peter Schuller 
> Reading in the documentation (specially on the tuning section) is clear the
> the number of Column Families affects the performance, in particular the
> amount of memory assigned to the heap.
> My question is: What's the hard limit on the number of CFs?
> Does anybody implemented an application with large number of CFs? if so,
> whats that number?

There's no meaningful hard limit as such that I am aware of, but the
idea is that you have few of them (meaning usually below 10 or so,
maybe more sometimes, but not hundreds or thousands).

The overhead associated is indirect in the sense that for a fixed heap
size, you'd need lower memtable flush thresholds per cf, leading to
smaller sstablers being flushed and thus leading to more compaction.

If you have so many memtables that you are worried about a hard limit,
you probably have too many of them already for reasons other than any
hard limits :)

-- 
/ Peter Schuller

Re: Partioning and Sorting is it CF Key or Column Key?

2011-02-13 Thread Peter Schuller 
> Some questions I have:

Answering two of them independently of your Java snippet; not sure
what you intend to be read into it.

> 1) Is partitioning based on CF.KEY or KEY of Column? From what I read it's
> based on column keys and not the CF keys but want to confirm.

Partitioning is based on row key.

> 2) When data is entered is it sorted based on CF.KEY or KEY of column or
> both?

It is always sorted on column name as per the settings of the column
family. It may be sorted on row key if the OrderPreservingPartitioner
is used, but usually one uses the RandomPartitioner in which case it
will not be sorted on key (it will be sorted on ring token).

--
/ Peter Schuller

Re: Limit on amount of CFs

2011-02-13 Thread Filip Nguyen 


On 13.2.2011 11:40, Peter Schuller wrote:

Reading in the documentation (specially on the tuning section) is clear the
the number of Column Families affects the performance, in particular the
amount of memory assigned to the heap.
My question is: What's the hard limit on the number of CFs?
Does anybody implemented an application with large number of CFs? if so,
whats that number?

There's no meaningful hard limit as such that I am aware of, but the
idea is that you have few of them (meaning usually below 10 or so,
maybe more sometimes, but not hundreds or thousands).
But when modeling the application I understand so far that ColumnFamily is sort 
of "table with objects". In typical application there are lot of tables so why 
is the mindset set towards having more or less 10 ColumnFamilies?
Even in this trivial example there are already 7 CFs 
http://www.rackspace.com/cloud/blog/2010/05/12/cassandra-by-example/.
So what is best practice to create applications using Cassandra? Divide 
application to more parts and create Keyspace for each one of them?

Re: Limit on amount of CFs

2011-02-13 Thread Peter Schuller 
> But when modeling the application I understand so far that ColumnFamily is
> sort of "table with objects". In typical application there are lot of tables
> so why is the mindset set towards having more or less 10 ColumnFamilies?
> Even in this trivial example there are already 7 CFs
> http://www.rackspace.com/cloud/blog/2010/05/12/cassandra-by-example/.
> So what is best practice to create applications using Cassandra? Divide
> application to more parts and create Keyspace for each one of them?

Keyspaces don't really help. You can have 100 column families if you
want, but if you're worried about overhead then whatever overhead you
do get will tend to be indirect in nature. Smaller memtables, more
files on disk, etc.

If you have a legitimate use case for N column families, then that's
the way to do it. It is just that the tendency is towards fewer CF:s;
for data access together the idea is often to put it into a single CF
under the same row key, rather than doing the RDBMS style. Instead of
a fully normalized system with foreign keys, you tend to group data
together and keep them in fewer column families - often with the same
row key across multiple cf:s instead of foreign keys.

I guess to re-phrase: In Cassandra the idea is that you model your
data after the expected read and write behavior rather than optimizing
for normalization. This tends to mean fewer CF:s rather than lots and
lots of CF:s, but it does depend on use-case. Sometimes it can mean
more CF:s instead of feer (if you split data into different CF:s to
separate out often read data from seldom read data, or if you
de-normalize to provide multiple materialized views of the same data).

So I think the right approach is to look at what the correct data
model would be. If that somehow results in an extreme amounts of CF:s,
then re-evaluate based on the specific use-case. Maybe that is truly
what you need, maybe not. But in any case, the primary concern should
not be any potential hard limit but rather the performance
implications of how data is stored. If after reaching a conclusion the
number of CF:s is high enough that there is a concern that you may hit
some kind of artificial limit or unintended side-effect, one can look
at the situation then.

Suppose there is a hard limit. Suppose there is a piece of code
somewhere that says "you can only have up to 100 column families".
Even if that were the case, it would be pretty useless to respond to
the OP's question with that information. If the use case *truly* calls
for 100+ CF:s, then you have to look at the actual effects of that.
What was the hard limt, and why is it there? Is it just a matter of
removing the hard limit which had no real purpose? The answer to
whether or not the limit is truly "hard" in any given situation, is
probably going to be at least as dependent on that situation as it is
on the code that enforces the limit...

(Obviously though it doesn't hurt to know of actual hard artificial
limits, and as I said I'm not aware of any.)

-- 
/ Peter Schuller

Explaining the Replication Factor, N and W and R

2011-02-13 Thread Janne Jalkanen 
Folks,

as it seems that wrapping the brain around the R+W>N concept is a big hurdle 
for a lot of users, I made a simple web page that allows you to try out the 
different parameters and see how they affect the system.

http://www.ecyrd.com/cassandracalculator/

Let me know if you have any suggestions to improve the wording, or you spot a 
bug. Trying to go for simplicity and clarity over absolute correctness here, as 
this is meant to help newbies.

(App is completely self-contained HTML and Javascript.)

/Janne

Secondary index - keys only.

2011-02-13 Thread Shay Assulin 
HI,

Is there a way to get only the keys of indexed rows (without getting 
columns) using get_indexed_slices method?

I am using Hector to access Cassandra and I want to count rows with a 
specific index - so i need to get only the keys.

I am doing the following:


n = 0

while (true) {

indexedSliceQuery.setStartKey(lastFetchedKey  )   
indexedSliceQuery.setReturnedKeysOnly();
indexedSliceQuery.addFilter(  the indexed column EQ value  )

res = indexedSliceQuery.execute();
n += res.size();
update lastFetchedKey
if (size == 0) break;
} 

It looks like the time it takes to get keys are just like the time it 
takes to get the entire row.

delete key permanently

2011-02-13 Thread Mark Zitnik 
Hi,

I would like to delete a key permanently in cassandra 0.7 and not receive it
in get range api.

Is it possible.

Thanks

Re: Confused about get_slice SliceRange behavior with bloom filter

2011-02-13 Thread Jonathan Ellis 
On Sun, Feb 13, 2011 at 12:37 AM, E S  wrote:
> I've gotten myself really confused by
> http://wiki.apache.org/cassandra/ArchitectureInternals and am hoping someone 
> can
> help me understand what the io behavior of this operation would be.
>
> When I do a get_slice for a column range, will it seek to every SSTable?  I 
> had
> thought that it would use the bloom filter on the row key so that it would 
> only
> do a seek to SSTables that have a very high probability of containing columns
> for that row.

Yes.

> In the linked doc above, it seems to say that it is only used for
> exact column names.  Am I misunderstanding this?

Yes.  You may be confusing multi-row behavior with multi-column.

> On a related note, if instead of using a SliceRange I provide an explicit list
> of columns, will I have to read all SSTables that have values for the columns

Yes.

> or is it smart enough to stop after finding a value from the most recent
> SSTable?

There is no way to know which value is most recent without having to
read it first.

-- 
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of DataStax, the source for professional Cassandra support
http://www.datastax.com

Re: delete key permanently

2011-02-13 Thread Jonathan Ellis 
http://wiki.apache.org/cassandra/FAQ#range_ghosts

On Sun, Feb 13, 2011 at 9:08 AM, Mark Zitnik  wrote:
> Hi,
>
> I would like to delete a key permanently in cassandra 0.7 and not receive it
> in get range api.
> Is it possible.
> Thanks
>
>
>



-- 
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of DataStax, the source for professional Cassandra support
http://www.datastax.com

Re: Secondary index - keys only.

2011-02-13 Thread Jonathan Ellis 
No.

On Sun, Feb 13, 2011 at 8:48 AM, Shay Assulin  wrote:
> HI,
>
> Is there a way to get only the keys of indexed rows (without getting
> columns) using get_indexed_slices method?
>
> I am using Hector to access Cassandra and I want to count rows with a
> specific index - so i need to get only the keys.
>
> I am doing the following:
>
>
> n = 0
>
> while (true) {
>
>         indexedSliceQuery.setStartKey(    lastFetchedKey  )
>
>         indexedSliceQuery.setReturnedKeysOnly();
>         indexedSliceQuery.addFilter(  the indexed column EQ value  )
>
>         res = indexedSliceQuery.execute();
>         n += res.size();
>         update lastFetchedKey
>         if (size == 0) break;
> }
>
> It looks like the time it takes to get keys are just like the time it takes
> to get the entire row.
>
>
>



-- 
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of DataStax, the source for professional Cassandra support
http://www.datastax.com

Re: Does Cassandra support multiple listen_address and rpc_address?

2011-02-13 Thread Edward Capriolo 
On Sun, Feb 13, 2011 at 1:39 AM, Xiaobo Gu  wrote:
> multiple network paths for inner-cluster communication will boost performance
>
> Thanks.
>
> Xiaobo Gu
>

No. Each node has a single IP. You can boost performance in a similar
way with Ethernet bonding, or 10G

Re: Explaining the Replication Factor, N and W and R

2011-02-13 Thread Eric Evans 
On Sun, 2011-02-13 at 15:49 +0200, Janne Jalkanen wrote:
> as it seems that wrapping the brain around the R+W>N concept is a big
> hurdle for a lot of users, I made a simple web page that allows you to
> try out the different parameters and see how they affect the system.
> 
> http://www.ecyrd.com/cassandracalculator/
> 
> Let me know if you have any suggestions to improve the wording, or you
> spot a bug. Trying to go for simplicity and clarity over absolute
> correctness here, as this is meant to help newbies.
> 
> (App is completely self-contained HTML and Javascript.) 

Neat.

-- 
Eric Evans
eev...@rackspace.com

Re: Explaining the Replication Factor, N and W and R

2011-02-13 Thread Rustam Aliyev 

On 13/02/2011 13:49, Janne Jalkanen wrote:

Folks,

as it seems that wrapping the brain around the R+W>N concept is a big hurdle 
for a lot of users, I made a simple web page that allows you to try out the 
different parameters and see how they affect the system.

http://www.ecyrd.com/cassandracalculator/

Let me know if you have any suggestions to improve the wording, or you spot a 
bug. Trying to go for simplicity and clarity over absolute correctness here, as 
this is meant to help newbies.

(App is completely self-contained HTML and Javascript.)

/Janne


Excellent! How about adding Hinted Handoff enabled/disabled option?

--
Rustam

Re: Explaining the Replication Factor, N and W and R

2011-02-13 Thread Janne Jalkanen 
> Excellent! How about adding Hinted Handoff enabled/disabled option?

Sure, once I understand it ;-)

/Janne

Re: rename column family

2011-02-13 Thread Aaron Morton 
There are functions on the Cassandra API to rename and drop column families, 
see 
http://wiki.apache.org/cassandra/API dropping a CF does not immediately free up 
the disk space, see the docs.

AFAIK the rename is not atomic across the cluster (that would require locks) so 
you best bet would be to switch to a new CF in your code

Read and writes in Cassandra compete for resources (CPU and disk) but they will 
not block each other as there is no locking system. You may find the 
performance acceptable, if not just add more machines :)

Switching CF's may be a valid way to handle meta data bulk deletes, like 
horizontal partitions in MS SQL and My SQL. Obviously it will deep end on how 
much data you have and how much capacity you have.

Let us know how you get on.

Cheers
Aaron
 
On 11/02/2011, at 11:33 AM, Karl Hiramoto  wrote:

> On 02/10/11 22:19, Aaron Morton wrote:
>> That should read "Without more information"
>> 
>> A
>> On 11 Feb, 2011,at 10:15 AM, Aaron Morton  wrote:
>> 
>>> With more information I'd say this is not a good idea.
>>> 
>>> I would suggest looking at why you do the table switch in the MySql
>>> version and consider if it's still necessary in the Cassandra version.
>>> 
> I do the table switch because it's the fastest way to rebuild an entire
> dataset, Say your importing a flat CSV file, you have various cases.
> 
> 1.  Exact same data loaded, only update timestamp.
> 2.  new data that was not in previous dataset.
> 3.  changed data from previous dataset (update)
> 4.   Data that is not in new data, but is in old.  (delete) Rebuilding
> the entire table saves millions of search/delete operations.
> 
> 
> In mysql reading/writing the table at the same time  (Many millions of
> rows,  many GB of data) slows things down beyond my strict performance
> requirements, doing the rename table, makes both the reads/writes much
> faster..  Yes, I know this probably doesn't apply to Cassandra.
> 
> If Cassandra could do something like the mysql rename it would avoid
> having to do the deletes on individual rows, or the repair/compaction of
> the column family to remove all the stale data.  Disk space usage is
> also very important.   I know after a new import is complete, all the
> old data is stale.
> 
>>> Could you use prefixes in your keys that the app knows about and
>>> switch those?
> Yes, but makes the app more complex, and needs to know when the data is
> consistent after the import.I think I would have to do a range scan
> to delete all the stale data.
> 
> A TTL would be risky as a TTL too high would waste disk space, and stale
> data would be around longer than wanted.A TTL too low would risk not
> having data available if a new import should fail, or be delayed.
> 
> 
> --
> Karl

Re: How to store news lists in optimal way?

2011-02-13 Thread Aaron Morton 
The best way to store things depends on how you want to read them back.

You could use a compound key such as user/listtype and then store the items in 
the lists as column were the col name is a timestamp and the col value is a 
packed data structure like json.

As bill says, don't create a CF per user.

Aaron

On 12/02/2011, at 4:14 AM, Bill Speirs  wrote:

> I don't know enough about Lucene to comment, but option #2 seems like
> a bad idea. You shouldn't grow your database by the number of Column
> Families as there are bad implications to doing this. Option #1 or #3
> seems plausible depending upon how much data you have.
> 
> Hope this helps...
> 
> Bill-
> 
> On Fri, Feb 11, 2011 at 4:26 AM, Nikolay Fominyh  wrote:
>> Hello.
>> 
>> We have a problem with finding way to store user news list.
>> Our task - we have 2 group of filters:
>> 1) News Types(photo, video, audio, etc... )
>> 2) User Lists(user, friends, friends friends, etc..)
>> 
>> Our solution ways:
>> 1) Store matrix of filters for each user.
>> UserNews:
>>   user_id1:
>>: (date, uuid)
>>_: list(date, news_uuid)
>>_: list(date, news_uuid)
>>
>>: (date, uuid)
>>_: list(date, news_uuid)
>>_: list(date, news_uuid)
>>...
>>   ...
>>   user_idN:
>>: (date, uuid)
>>_: list(date, news_uuid)
>>_: list(date, news_uuid)
>>
>>: (date, uuid)
>>_: list(date, news_uuid)
>>_: list(date, news_uuid)
>>...
>> 
>> Numbers of filters in this way more than 100..
>> 
>> 2) Store news lists for each user in personal column family.
>> UserNews_:
>>date_uuid:
>> : indexed
>> : indexed
>>
>> 
>> In this case we have huge number of column families.
>> 
>> 3) Store news lists in SQL Database and News in Cassandra.
>> 
>> 4) Use Apache Lucene for indexes on filters.
>> 
>> Question: What of this ways is most optimal? Is there another ways of 
>> solutuion?
>> 
>>

Re: Column name size

2011-02-13 Thread Aaron Morton 
FWIW I would first try to reduce the number of columns, before reducing their 
name length. If you always pull back the same columns (e.g. User details) 
consider packing them in json dict and storing them in one column.

Aaron

On 12/02/2011, at 5:22 AM, Chris Burroughs  wrote:

> On 02/11/2011 05:06 AM, Patrik Modesto wrote:
>> Hi all!
>> 
>> I'm thinking if size of a column name could matter for a large dataset
>> in Cassandra  (I mean lots of rows). For example what if I have a row
>> with 10 columns each has 10 bytes value and 10 bytes name. Do I have
>> half the row size just of the column names and the other half of the
>> data (not counting storage overhead)?  What if I have 10M of these
>> rows? Is there a difference? Should I use some 3bytes codes for a
>> column name to save memory/bandwidth?
>> 
>> Thanks,
>> Patrik
> 
> You are correct that you can for small row/column key values they key
> itself can represent a large proportion of the total size.  I think you
> will find the consensus on  this list is that trying to be clever with
> names is usually not worth the additional complexity.
> 
> The right solution to this is
> https://issues.apache.org/jira/browse/CASSANDRA-47.

Re: Extra Large Memtables

2011-02-13 Thread Tyler Hobbs 
I should note up front that the JVM simply does not handle heap sizes above
20G very well because the GC starts to become problematic.

Do you read rows in a uniformly random way?  If not, caching is your best
bet for reducing read latencies.  You should have enough space to cache all
of your keys, and you may be able to have an effective row cache, depending
on your row sizes and access patterns.  Obviously the OS buffer cache will
help out tremendously once warm.

Some people do run Cassandra with really high memory boxes like yours, but
it's not optimal for Cassandra.  More nodes with less hardware have many
advantages.  I suppose splitting your CF is an attempt to do something
similar.

-- 
Tyler Hobbs
Software Engineer, DataStax 
Maintainer of the pycassa  Cassandra
Python client library

Re: Basic Cassandra Architecture questions

2011-02-13 Thread Aaron Morton 
You can get consistency by using Quorum, or write at All and read at one, or 
write at one and read at All

Start with quorum.

If you read at one, then read repair will work in the background to fix the 
data. But the result returned to your client may be inconsistent.

Aaron

On 12/02/2011, at 7:17 AM, mcasandra  wrote:

> 
> What's the best practice in terms of consistency? I am assuming R+W > N
> should be the best practice.
> 
> I thought even if R+W=N then there is some version level reconciliation that
> kicks off if in case older version of the key/value is read. But to think of
> it may not be possible. But then if R+W <= N it means there is a high chance
> of getting wrong version of the key/value. Is that true?
> -- 
> View this message in context: 
> http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Basic-Cassandra-Architecture-questions-tp6014218p6016731.html
> Sent from the cassandra-u...@incubator.apache.org mailing list archive at 
> Nabble.com.

Re: Question about seeds in tow node cluster.

2011-02-13 Thread Aaron Morton 
The can be, but it's not necessary.

Aaron


On 13/02/2011, at 9:04 PM, Xiaobo Gu  wrote:

> Hi,
> If the cluster only have tow nodes, should they both in the seeds list?
> 
> Regards,
> 
> Xiaobo Gu

Re: Confused about get_slice SliceRange behavior with bloom filter

2011-02-13 Thread Aditya Narayan 
Jonathan,
If I ask for around 150-200 columns (totally random not sequential) from a
very wide row that contains more than a million or even more columns then,
is the read performance of the SliceQuery operation affected by or "depends
on the length of the row" ?? (For my use case, I would use the column names
list for this SliceQuery operation).


Thanks
Aditya

On Sun, Feb 13, 2011 at 8:41 PM, Jonathan Ellis  wrote:

> On Sun, Feb 13, 2011 at 12:37 AM, E S  wrote:
> > I've gotten myself really confused by
> > http://wiki.apache.org/cassandra/ArchitectureInternals and am hoping
> someone can
> > help me understand what the io behavior of this operation would be.
> >
> > When I do a get_slice for a column range, will it seek to every SSTable?
>  I had
> > thought that it would use the bloom filter on the row key so that it
> would only
> > do a seek to SSTables that have a very high probability of containing
> columns
> > for that row.
>
> Yes.
>
> > In the linked doc above, it seems to say that it is only used for
> > exact column names.  Am I misunderstanding this?
>
> Yes.  You may be confusing multi-row behavior with multi-column.
>
> > On a related note, if instead of using a SliceRange I provide an explicit
> list
> > of columns, will I have to read all SSTables that have values for the
> columns
>
> Yes.
>
> > or is it smart enough to stop after finding a value from the most recent
> > SSTable?
>
> There is no way to know which value is most recent without having to
> read it first.
>
> --
> Jonathan Ellis
> Project Chair, Apache Cassandra
> co-founder of DataStax, the source for professional Cassandra support
> http://www.datastax.com
>

NFS instead of local storage

2011-02-13 Thread mcasandra 

I just now watched some videos about performance tunning. And it looks like
most of the bottleneck could be on reads. Also, it looks like it's advisable
to put commit logs on separate drive.

I was wondering if it makes sense to use NFS (if we can) with netapp array
which provides it's own read and write caching.
-- 
View this message in context: 
http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/NFS-instead-of-local-storage-tp6021959p6021959.html
Sent from the cassandra-u...@incubator.apache.org mailing list archive at 
Nabble.com.

Re: Confused about get_slice SliceRange behavior with bloom filter

2011-02-13 Thread aaron morton 
AFAIK yes. 

Until your row is column_index_size_in_kb in size (and in some circumstances a 
compaction must have run) the code has to scan through all of the columns in 
the row to find the 150-200 you want.

From the help in cassandra.yaml

# Add column indexes to a row after its contents reach this size.
# Increase if your column values are large, or if you have a very large
# number of columns.  The competing causes are, Cassandra has to
# deserialize this much of the row to read a single column, so you want
# it to be small - at least if you do many partial-row reads - but all
# the index data is read for each access, so you don't want to generate
# that wastefully either.
column_index_size_in_kb: 64

If you are making a lot of value less columns, it may pay work out how many it 
would take before they reach this size. Note, I've not checked but I assume the 
column name is included in determining the row size. 

Hope that helps.
Aaron

On 14 Feb 2011, at 10:19, Aditya Narayan wrote:

> Jonathan,
> If I ask for around 150-200 columns (totally random not sequential) from a 
> very wide row that contains more than a million or even more columns then, is 
> the read performance of the SliceQuery operation affected by or "depends on 
> the length of the row" ?? (For my use case, I would use the column names list 
> for this SliceQuery operation).
> 
> 
> Thanks
> Aditya
> 
> On Sun, Feb 13, 2011 at 8:41 PM, Jonathan Ellis  wrote:
> On Sun, Feb 13, 2011 at 12:37 AM, E S  wrote:
> > I've gotten myself really confused by
> > http://wiki.apache.org/cassandra/ArchitectureInternals and am hoping 
> > someone can
> > help me understand what the io behavior of this operation would be.
> >
> > When I do a get_slice for a column range, will it seek to every SSTable?  I 
> > had
> > thought that it would use the bloom filter on the row key so that it would 
> > only
> > do a seek to SSTables that have a very high probability of containing 
> > columns
> > for that row.
> 
> Yes.
> 
> > In the linked doc above, it seems to say that it is only used for
> > exact column names.  Am I misunderstanding this?
> 
> Yes.  You may be confusing multi-row behavior with multi-column.
> 
> > On a related note, if instead of using a SliceRange I provide an explicit 
> > list
> > of columns, will I have to read all SSTables that have values for the 
> > columns
> 
> Yes.
> 
> > or is it smart enough to stop after finding a value from the most recent
> > SSTable?
> 
> There is no way to know which value is most recent without having to
> read it first.
> 
> --
> Jonathan Ellis
> Project Chair, Apache Cassandra
> co-founder of DataStax, the source for professional Cassandra support
> http://www.datastax.com
>

Re: rename column family

2011-02-13 Thread Aaron Morton 
Forgot to put on the end of this, you could take that approach but it's not what CF's are designed for. Delete's are relatively cheap compared to MySql etc because most of the work is done in the compaction. My first approach would be to use row keys with prefixes, switch at the application level, run deletions jobs in the background and tune GCGraceSeconds and repairs to manage disk space. Depending on what your data is, you may find the ingest process is easier. e.g. - there is no update call in cassandra, so you may be able to do no look inserts- overwriting existing data with the same value will mean it is stored twice, but this will be removed during compactions- you can issue a no look delete for a row, if the row is not there there will be some overhead (tombstone will be kept). Again during compaction it will be removed. Also, if you know the maximum timestamp of the previous load job you could try this trick to do a bulk delete http://www.mail-archive.com/user@cassandra.apache.org/msg09576.htmlAgain, data is cleaned up as part of compaction. You mileage may vary. Aaron On 14 Feb, 2011,at 08:40 AM, Aaron Morton  wrote:There are functions on the Cassandra API to rename and drop column families, see 
http://wiki.apache.org/cassandra/API dropping a CF does not immediately free up the disk space, see the docs.

AFAIK the rename is not atomic across the cluster (that would require locks) so you best bet would be to switch to a new CF in your code

Read and writes in Cassandra compete for resources (CPU and disk) but they will not block each other as there is no locking system. You may find the performance acceptable, if not just add more machines :)

Switching CF's may be a valid way to handle meta data bulk deletes, like horizontal partitions in MS SQL and My SQL. Obviously it will deep end on how much data you have and how much capacity you have.

Let us know how you get on.

Cheers
Aaron
 
On 11/02/2011, at 11:33 AM, Karl Hiramoto  wrote:

> On 02/10/11 22:19, Aaron Morton wrote:
>> That should read "Without more information"
>> 
>> A
>> On 11 Feb, 2011,at 10:15 AM, Aaron Morton  wrote:
>> 
>>> With more information I'd say this is not a good idea.
>>> 
>>> I would suggest looking at why you do the table switch in the MySql
>>> version and consider if it's still necessary in the Cassandra version.
>>> 
> I do the table switch because it's the fastest way to rebuild an entire
> dataset, Say your importing a flat CSV file, you have various cases.
> 
> 1.  Exact same data loaded, only update timestamp.
> 2.  new data that was not in previous dataset.
> 3.  changed data from previous dataset (update)
> 4.   Data that is not in new data, but is in old.  (delete) Rebuilding
> the entire table saves millions of search/delete operations.
> 
> 
> In mysql reading/writing the table at the same time  (Many millions of
> rows,  many GB of data) slows things down beyond my strict performance
> requirements, doing the rename table, makes both the reads/writes much
> faster..  Yes, I know this probably doesn't apply to Cassandra.
> 
> If Cassandra could do something like the mysql rename it would avoid
> having to do the deletes on individual rows, or the repair/compaction of
> the column family to remove all the stale data.  Disk space usage is
> also very important.   I know after a new import is complete, all the
> old data is stale.
> 
>>> Could you use prefixes in your keys that the app knows about and
>>> switch those?
> Yes, but makes the app more complex, and needs to know when the data is
> consistent after the import.I think I would have to do a range scan
> to delete all the stale data.
> 
> A TTL would be risky as a TTL too high would waste disk space, and stale
> data would be around longer than wanted.A TTL too low would risk not
> having data available if a new import should fail, or be delayed.
> 
> 
> --
> Karl