I'm not a Cassandra dev, so take what I say with a lot of salt, but AFAICT, there is a certain amount of overhead in maintaining a CF, so when you have large numbers of CFs, this adds up. From a layperson's perspective, this observation sounds reasonable, since zero-cost CFs would be tantamount to being able to implement secondary indexes by just adding CFs. So instead of paying the for the overhead (or ineffectiveness of high-cardinaility secondary indexes, which ever way you want to look at it), you are expecting a free lunch by just scaling out in terms on new CFs. I would imagine that under the covers, the layout of Cassandra has a sweet spot of a smallish number of CFs (i.e. 10s), but these can practically have as many rows as you like.
On Mon, Oct 8, 2012 at 11:02 AM, Vanger <disc...@gmail.com> wrote: > So what solution should be for cassandra architecture when we need to make > Hadoop M\R jobs and not be restricted by number of CF? > What we have now is fair amount of CFs (> 2K) and this number is slowly > growing so we already planing to merge partitioned CFs. But our next goal is > to run hadoop tasks on those CFs. All we have is plain Hector and custom ORM > on top of it. As far as i understand VirtualKeyspace doesn't help in our > case. > Also i dont understand why not implement support for many CF ( or build-in > partitioning ) on cassandra side. Anybody can explain why this can or cannot > be done in cassandra? > > Just in case: > We're using cassandra 1.0.11 on 30 nodes (planning upgrade on 1.1.* soon). > > -- > W/ best regards, > Sergey. > > On 04.10.2012 0:10, Hiller, Dean wrote: > > Okay, so it only took me two solid days not a week. PlayOrm in master > branch now supports virtual CF's or virtual tables in ONE CF, so you can > have 1000's or millions of virtual CF's in one CF now. It works with all > the Scalable-SQL, works with the joins, and works with the PlayOrm command > line tool. > > Two ways to do it, if you are using the ORM half, you just annotate > > @NoSqlEntity("MyVirtualCfName") > @NoSqlVirtualCf(storedInCf="sharedCf") > > So it's stored in sharedCf with the table name of MyVirtualCfName(in command > line tool, use MyVirtualCfName to query the table). > > Then if you don't know your meta data ahead of time, you need to create > DboTableMeta and DboColumnMeta objects and save them for every table you > create and can use TypedRow to read and persist (which is what we have a > project doing). > > If you try it out let me know. We usually get bug fixes in pretty fast if > you run into anything. (more and more questions are forming on stack > overflow as well ;) ). > > Later, > Dean > > > >