Re: Random Distribution, yet Order Preserving Partitioner

2013-08-27 Thread Manoj Mainali
Hi Takenori, I can't tell for sure without knowing what kind of data you have and how much you have.You can use the random partitioner and use the concept of metadata row that stores the row key, as for example like below {metadata_row}: key1 | key2 | key3 key1:column1 | column2 When you do

Re: Random Distribution, yet Order Preserving Partitioner

2013-08-27 Thread Takenori Sato
Hi Manoj, Thanks for your advise. More or less, basically we do the same. As you pointed out, we now face with many cases that can not be solved by data modeling, and which are reaching to 100 millions of columns. We can split them down to multiple pieces of metadata rows, but that will bring

Re: Random Distribution, yet Order Preserving Partitioner

2013-08-23 Thread Nikolay Mihaylov
It can handle some millions of columns, but not more like 10M. I mean, a request for such a row concentrates on a particular node, so the performance degrades. I also had idea for semi-ordered partitioner - instead of single MD5, to have two MD5's. works for us with wide row with about 40-50 M,

Re: Random Distribution, yet Order Preserving Partitioner

2013-08-22 Thread Nikolay Mihaylov
my five cents - token and key are not same. it was like this long time ago (single MD5 assumed single key) if you want ordered, you probably can arrange your data in a way so you can get it in ordered fashion. for example long ago, i had single column family with single key and about 2-3 M

Re: Random Distribution, yet Order Preserving Partitioner

2013-08-22 Thread Takenori Sato
Hi Nick, token and key are not same. it was like this long time ago (single MD5 assumed single key) True. That reminds me of making a test with the latest 1.2 instead of our current 1.0! if you want ordered, you probably can arrange your data in a way so you can get it in ordered fashion.