problem when trying to get_range_slice()
Hi all,
my env
6 servers with about 200GB data.
data structure,
64B rowkey + (5B column)*20,
rowkey and column.value are all random bytes from a-z,A-Z,0-9
problem
when I tried iterate over the data in the storage, I always get
org::apache::cassandra::TimedOutException
(RpcTimeout = 2minutes)
questions
1.How could I iterate over my data then?
2.In the code below, I gave key_start/end/count twice, one for
get_range_slice() and the other for SlicePreditor. Are they the same?
Thanks!
scan-data application
CassandraFactory factory(argv[1], 9160);
tr1::shared_ptrCassandra client(factory.create());
Keyspace *key_space = client-getKeyspace(campaign);
mapstring, vectorColumn mResult;
ColumnParent cp;
SlicePredicate sp;
string szStart, szEnd;
uint32_t uCount = 100; //try to retrieve 100 keys?
szStart =
;
szEnd =
1000;
cp.column_family = test_col;
sp.__isset.slice_range = true;
sp.slice_range.start =
;
sp.slice_range.finish =
1000;
sp.slice_range.count = 100;
try {
mResult = key_space-getRangeSlice(cp, sp, szStart, szEnd,
uCount); //by libcassandra, mainly invoking get_range_slice()
/* then iterate the result and output */
}
}
} catch (org::apache::cassandra::InvalidRequestException ire) {
cout ire.why endl;
delete key_space;
return 1;
}
return 0;
--
Kevin Yuan
www.yuan-shuai.info
Re: [***SPAM*** ] problem when trying to get_range_slice()
more info:
CL = ONE,
replica = 2,
and when I tried to monitor the disk_io with iostat I get almost 0MB/s
read 0% CPU on the machine the scan-data app started on.
Thanks!
??: Shuai Yuan yuansh...@supertool.net.cn
??: user@cassandra.apache.org
: [***SPAM*** ] problem when trying to get_range_slice()
: Thu, 03 Jun 2010 15:57:14 +0800
Hi all,
my env
6 servers with about 200GB data.
data structure,
64B rowkey + (5B column)*20,
rowkey and column.value are all random bytes from a-z,A-Z,0-9
problem
when I tried iterate over the data in the storage, I always get
org::apache::cassandra::TimedOutException
(RpcTimeout = 2minutes)
questions
1.How could I iterate over my data then?
2.In the code below, I gave key_start/end/count twice, one for
get_range_slice() and the other for SlicePreditor. Are they the same?
Thanks!
scan-data application
CassandraFactory factory(argv[1], 9160);
tr1::shared_ptrCassandra client(factory.create());
Keyspace *key_space = client-getKeyspace(campaign);
mapstring, vectorColumn mResult;
ColumnParent cp;
SlicePredicate sp;
string szStart, szEnd;
uint32_t uCount = 100; //try to retrieve 100 keys?
szStart =
;
szEnd =
1000;
cp.column_family = test_col;
sp.__isset.slice_range = true;
sp.slice_range.start =
;
sp.slice_range.finish =
1000;
sp.slice_range.count = 100;
try {
mResult = key_space-getRangeSlice(cp, sp, szStart, szEnd,
uCount); //by libcassandra, mainly invoking get_range_slice()
/* then iterate the result and output */
}
}
} catch (org::apache::cassandra::InvalidRequestException ire) {
cout ire.why endl;
delete key_space;
return 1;
}
return 0;
--
Kevin Yuan
www.yuan-shuai.info
Re: [***SPAM*** ] Re: problem when trying to get_range_slice()
Thanks for the hint.
I found out it was too many opened files error and the server side
just lost response to the get_range_slice() request by throwing out an
exception.
Now works with ulimit -n 32768.
Kevin
??: Jonathan Ellis jbel...@gmail.com
??: user@cassandra.apache.org
: [***SPAM*** ] Re: problem when trying to get_range_slice()
: Thu, 3 Jun 2010 19:07:28 -0700
use smaller slices and page through the data
2010/6/3 Shuai Yuan yuansh...@supertool.net.cn:
Hi all,
my env
6 servers with about 200GB data.
data structure,
64B rowkey + (5B column)*20,
rowkey and column.value are all random bytes from a-z,A-Z,0-9
problem
when I tried iterate over the data in the storage, I always get
org::apache::cassandra::TimedOutException
(RpcTimeout = 2minutes)
questions
1.How could I iterate over my data then?
2.In the code below, I gave key_start/end/count twice, one for
get_range_slice() and the other for SlicePreditor. Are they the same?
Thanks!
scan-data application
CassandraFactory factory(argv[1], 9160);
tr1::shared_ptrCassandra client(factory.create());
Keyspace *key_space = client-getKeyspace(campaign);
mapstring, vectorColumn mResult;
ColumnParent cp;
SlicePredicate sp;
string szStart, szEnd;
uint32_t uCount = 100; //try to retrieve 100 keys?
szStart =
;
szEnd =
1000;
cp.column_family = test_col;
sp.__isset.slice_range = true;
sp.slice_range.start =
;
sp.slice_range.finish =
1000;
sp.slice_range.count = 100;
try {
mResult = key_space-getRangeSlice(cp, sp, szStart, szEnd,
uCount); //by libcassandra, mainly invoking get_range_slice()
/* then iterate the result and output */
}
}
} catch (org::apache::cassandra::InvalidRequestException ire) {
cout ire.why endl;
delete key_space;
return 1;
}
return 0;
--
Kevin Yuan
www.yuan-shuai.info
Re: [***SPAM*** ] Re: [***SPAM*** ] Re: [***SPAM*** ] Re: [***SPAM*** ] Re: writing speed test
Still seems MEM. However it's hard to convince that constantly writing(even great amount of data) needs so much MEM(16GB). The process is quite simple, input_data - memtable - flush to disk right? What does cassandra need so much MEM for? Thanks! ?? 2010-06-02 16:24 +0800??lwl?? No. But I did some capacity tests about another distributed system. Your former test cost too much MEM, it was the bottleneck. caches and JVM cost MEM, so I suggested to decrease them. What is the bottleneck of your current test now? ?? 2010??6??2?? 4:13??Shuai Yuan yuansh...@supertool.net.cn?? Hi, I tried, 1-consistency level ZERO 2-JVM heap 4GB 3-normal Memtable cache and now I have about 30% improvment. However I want to know if you have also done w/r benchmark and what's the result? ?? 2010-06-02 11:35 +0800??lwl?? and, why did you set JVM has 8G heap? 8g, seems too big. ?? 2010??6??2?? 11:20??lwl lwl.ro...@gmail.com?? 3.32 concurrent read 128 write in storage-conf.xml, other cache enlarged as well. maybe you can try to decrease the size of caches. ?? 2010??6??2?? 11:14??Shuai Yuan yuansh...@supertool.net.cn?? ?? 2010-06-02 10:37 +0800??lwl?? is all the 4 servers' MEM almost 100%? Yes ?? 2010??6??2?? 10:12??Shuai Yuan yuansh...@supertool.net.cn?? Thanks lwl. Then is there anyway of tuning this, faster flush to disk or else? Cheers, Kevin ?? 2010-06-02 09:57 +0800??lwl ?? MEM: almost 100% (16GB) - maybe this is the bottleneck. writing concerns Memtable and SSTable in memory. ?? 2010??6??2?? 9:48??Shuai Yuan yuansh...@supertool.net.cn?? ?? 2010-06-01 15:00 -0500?? Jonathan Shook?? Also, what are you meaning specifically by 'slow'? Which measurements are you looking at. What are your baseline constraints for your test system? Actually, the problem is the utilizaton of resources(for a single machine): CPU: 700% / 1600% (16 cores) MEM: almost 100% (16GB) Swap: almost 0% Disk IO(write): 20~30MB / 200MB (7.2k raid5, benchmarked previously) NET: up to 100Mbps / 950Mbps (1Gbps, tuned and benchmarked previously) So the speed of generating load, about 15M/s as reported before seems quite slow to me. I assume
Re: [***SPAM*** ] Re: writing speed test
Thanks Peter! In my test application, for each record, rowkey - rand() * 4, about 64B column * 20 - rand() * 20, about 320B I use batch_insert(rowkey, col*20) in thrift. Kevin Yuan ??: Peter Sch??ller sc...@spotify.com ??: user@cassandra.apache.org : [***SPAM*** ] Re: writing speed test : Wed, 2 Jun 2010 10:44:52 +0200 Since this thread has now gone on for a while... As far as I can tell you never specify the characteristics of your writes. Evaluating expected write throughput in terms of MB/s to disk is pretty impossible if one does not know anything about the nature of the writes. If you're expecting 50 MB, is that reasonable? I don't know; if you're writing a gazillion one-byte values with shortish keys, 50 MB/seconds translates to a *huge* amounts of writes per second and you're likely to be CPU bound even in the most efficient implementation reasonably possible. If on the other hand you're writing large values (say slabs of 128k) you might more reasonably be expecting higher disk throughput. I don't have enough hands-on experience with cassandra to have a feel for the CPU vs. disk in terms of bottlenecking, and when we expect to bottleneck on what, but I can say that it's definitely going to matter quite a lot what *kind* of writes you're doing. This tends to be the case regardless of the database system.
writing speed test
Hi all, I'm testing writing speed of cassandra with 4 servers. I'm confused by the behavior of cassandra. ---env--- load-data app written in c++, using libcassandra (w/ modified batch insert) 20 writing threads in 2 processes running on 2 servers ---optimization--- 1.turn log level to INFO 2.JVM has 8G heap 3.32 concurrent read 128 write in storage-conf.xml, other cache enlarged as well. ---result--- 1-monitoring by `date;nodetool -h host ring` I add all load together and measure the writing speed by (load_difference / time_difference), and I get about 15MB/s for the whole cluster. 2-monitoring by `iostat -m 10` I can watch the disk_io from the system level and have about 10MB/s - 65MB/s for a single machine. Very big variance over time. 3-monitoring by `iptraf -g` In this way I watch the communication between servers and get about 10MB/s for a single machine. ---opinion--- So, have you checked the writing speed of cassandra? I feel it's quite slow currently. Could anyone confirm this is the normal writing speed of cassandra, or please provide someway of improving it? -- Shuai Yuan Supertool Corp. ?? 13810436859 yuan-sh...@yuan-shuai.info
Re: [***SPAM*** ] access a multinode cluster
?? 2010-06-01 15:00 +0800??huajun qi?? If you have a multinode cluster, which node you should connect to fetch data? any one. Is there a master node in a cluster which accepts data request and dispatch it? Or every node in the cluster is completely same? no master. all the same. If all nodes are same in a cluster, should client connect to random node to reduce cassandra's load? I think so. But I guess if you're sure where the data is, you can connect the target machine directly. -- Location: Kevin Yuan
question about class SlicePredicate
Hi all, I don't quite understand the usage of 'class SlicePredicate' when trying to retrieve a ranged slice. How should it be initialized? Thanks! -- Kevin Yuan www.yuan-shuai.info
Re: writing speed test
?? 2010-06-01 15:00 -0500??Jonathan Shook?? Also, what are you meaning specifically by 'slow'? Which measurements are you looking at. What are your baseline constraints for your test system? Actually, the problem is the utilizaton of resources(for a single machine): CPU: 700% / 1600% (16 cores) MEM: almost 100% (16GB) Swap: almost 0% Disk IO(write): 20~30MB / 200MB (7.2k raid5, benchmarked previously) NET: up to 100Mbps / 950Mbps (1Gbps, tuned and benchmarked previously) So the speed of generating load, about 15M/s as reported before seems quite slow to me. I assume the system should get at least about 50MB/s of Disk IO speed. MEM? I don't think it plays a major role in this writing game. What's the bottleneck of the system? P.S about Consistency Level, I've tried ONE/DCQUORUM and found ONE is about 10-15% faster. However that's neither a promising result. Thanks! Kevin 2010/6/1 ?? shiyingjie1...@gmail.com: Hi, It would be better if we know which Consistency Level did you choose, and what is the schema of test data? ?? 2010??6??1?? 4:48??Shuai Yuan yuansh...@supertool.net.cn?? Hi all, I'm testing writing speed of cassandra with 4 servers. I'm confused by the behavior of cassandra. ---env--- load-data app written in c++, using libcassandra (w/ modified batch insert) 20 writing threads in 2 processes running on 2 servers ---optimization--- 1.turn log level to INFO 2.JVM has 8G heap 3.32 concurrent read 128 write in storage-conf.xml, other cache enlarged as well. ---result--- 1-monitoring by `date;nodetool -h host ring` I add all load together and measure the writing speed by (load_difference / time_difference), and I get about 15MB/s for the whole cluster. 2-monitoring by `iostat -m 10` I can watch the disk_io from the system level and have about 10MB/s - 65MB/s for a single machine. Very big variance over time. 3-monitoring by `iptraf -g` In this way I watch the communication between servers and get about 10MB/s for a single machine. ---opinion--- So, have you checked the writing speed of cassandra? I feel it's quite slow currently. Could anyone confirm this is the normal writing speed of cassandra, or please provide someway of improving it? -- Kevin Yuan www.yuan-shuai.info -- Kevin Yuan www.yuan-shuai.info
Re: [***SPAM*** ] Re: writing speed test
Thanks lwl. Then is there anyway of tuning this, faster flush to disk or else? Cheers, Kevin ?? 2010-06-02 09:57 +0800??lwl?? MEM: almost 100% (16GB) - maybe this is the bottleneck. writing concerns Memtable and SSTable in memory. ?? 2010??6??2?? 9:48??Shuai Yuan yuansh...@supertool.net.cn?? ?? 2010-06-01 15:00 -0500??Jonathan Shook?? Also, what are you meaning specifically by 'slow'? Which measurements are you looking at. What are your baseline constraints for your test system? Actually, the problem is the utilizaton of resources(for a single machine): CPU: 700% / 1600% (16 cores) MEM: almost 100% (16GB) Swap: almost 0% Disk IO(write): 20~30MB / 200MB (7.2k raid5, benchmarked previously) NET: up to 100Mbps / 950Mbps (1Gbps, tuned and benchmarked previously) So the speed of generating load, about 15M/s as reported before seems quite slow to me. I assume the system should get at least about 50MB/s of Disk IO speed. MEM? I don't think it plays a major role in this writing game. What's the bottleneck of the system? P.S about Consistency Level, I've tried ONE/DCQUORUM and found ONE is about 10-15% faster. However that's neither a promising result. Thanks! Kevin 2010/6/1 ?? shiyingjie1...@gmail.com: Hi, It would be better if we know which Consistency Level did you choose, and what is the schema of test data? ?? 2010??6??1?? 4:48??Shuai Yuan yuansh...@supertool.net.cn?? Hi all, I'm testing writing speed of cassandra with 4 servers. I'm confused by the behavior of cassandra. ---env--- load-data app written in c++, using libcassandra (w/ modified batch insert) 20 writing threads in 2 processes running on 2 servers ---optimization--- 1.turn log level to INFO 2.JVM has 8G heap 3.32 concurrent read 128 write in storage-conf.xml, other cache enlarged as well. ---result--- 1-monitoring by `date;nodetool -h host ring` I add all load together and measure the writing speed by (load_difference / time_difference), and I get about 15MB/s for the whole cluster. 2-monitoring by `iostat -m 10` I can watch the disk_io from the system level and have about 10MB/s - 65MB/s for a single machine. Very big variance over time. 3-monitoring by `iptraf -g` In this way I watch the communication between servers and get about 10MB/s for a single machine. ---opinion--- So, have you checked the writing speed of cassandra? I feel it's quite slow currently. Could anyone confirm this is the normal writing speed of cassandra, or please provide someway of improving it? -- Kevin Yuan www.yuan-shuai.info -- Kevin Yuan www.yuan-shuai.info -- Shuai Yuan Supertool Corp. ?? www.yuan-shuai.info
Re: [***SPAM*** ] Re: [***SPAM*** ] Re: writing speed test
?? 2010-06-02 10:37 +0800??lwl?? is all the 4 servers' MEM almost 100%? Yes ?? 2010??6??2?? 10:12??Shuai Yuan yuansh...@supertool.net.cn?? Thanks lwl. Then is there anyway of tuning this, faster flush to disk or else? Cheers, Kevin ?? 2010-06-02 09:57 +0800??lwl?? MEM: almost 100% (16GB) - maybe this is the bottleneck. writing concerns Memtable and SSTable in memory. ?? 2010??6??2?? 9:48??Shuai Yuan yuansh...@supertool.net.cn?? ?? 2010-06-01 15:00 -0500??Jonathan Shook?? Also, what are you meaning specifically by 'slow'? Which measurements are you looking at. What are your baseline constraints for your test system? Actually, the problem is the utilizaton of resources(for a single machine): CPU: 700% / 1600% (16 cores) MEM: almost 100% (16GB) Swap: almost 0% Disk IO(write): 20~30MB / 200MB (7.2k raid5, benchmarked previously) NET: up to 100Mbps / 950Mbps (1Gbps, tuned and benchmarked previously) So the speed of generating load, about 15M/s as reported before seems quite slow to me. I assume the system should get at least about 50MB/s of Disk IO speed. MEM? I don't think it plays a major role in this writing game. What's the bottleneck of the system? P.S about Consistency Level, I've tried ONE/DCQUORUM and found ONE is about 10-15% faster. However that's neither a promising result. Thanks! Kevin 2010/6/1 ?? shiyingjie1...@gmail.com: Hi, It would be better if we know which Consistency Level did you choose, and what is the schema of test data? ?? 2010??6??1?? 4:48??Shuai Yuan yuansh...@supertool.net.cn?? Hi all, I'm testing writing speed of cassandra with 4 servers. I'm confused by the behavior of cassandra. ---env--- load-data app written in c++, using libcassandra (w/ modified batch insert) 20 writing threads in 2 processes running on 2 servers ---optimization--- 1.turn log level to INFO 2.JVM has 8G heap 3.32 concurrent read 128 write in storage-conf.xml, other cache enlarged as well. ---result--- 1-monitoring by `date;nodetool -h host ring` I add all load together and measure the writing speed by (load_difference / time_difference), and I get about 15MB/s for the whole cluster. 2-monitoring by `iostat -m 10` I can watch the disk_io from the system level and have about 10MB/s - 65MB/s for a single machine. Very big variance over time. 3-monitoring by `iptraf -g` In this way I watch the communication between servers and get about 10MB/s for a single machine. ---opinion--- So, have you checked the writing speed of cassandra? I feel it's quite slow currently. Could anyone confirm this is the normal writing speed of cassandra, or please provide someway of improving it? -- Kevin Yuan www.yuan-shuai.info
