Re: Writes slower then reads
I turned off 1 large cronjob which caused the CPU not to get used for ~ 60% once every 10 minutes. Both write and read are fast now. Just think I was overloading the node. Weird though that shutting down the node did not improve the speed. Thank you all for your time! Robin 2012/1/5 aaron morton > What happens when you turn off the cron jobs ? > > Cheers > > - > Aaron Morton > Freelance Developer > @aaronmorton > http://www.thelastpickle.com > > On 6/01/2012, at 6:57 AM, Philippe wrote: > > Unless you are doing huge batches no... don't have any other idea for > now... > > 2012/1/5 R. Verlangen > >> The write and read load is very minimal the moment. Roughly 10 writes + >> 10 reads / second. So 20 operations per second. Don't think that overloads >> my cluster, does it? >> >> >> 2012/1/5 Philippe >> >>> You may be overloading the cluster though... >>> >>> My hypothesis is that your traffic is being spread across your node and >>> that one slow node is slowing down the fraction of traffic that goes to >>> that node (when it's acting as coordinator). >>> So what I would do is reduce the read load a lot to make sure I don't >>> overload the cluster and measure if I see a 1/RF improvement in response >>> time which would validate my hypothesis. >>> >>> >>> 2012/1/5 R. Verlangen >>> >>> It does not appear to affect the response time, certainly not in a positive way. 2012/1/5 Philippe > What if you shutdown the cassandra service on the slow node, does that > improve your read performance ? > If it does then that sole node is responsible for the slow down > because it can't act as a coordinator fast enough. > > 2012/1/5 R. Verlangen > > I'm also reading with CL = ONE >> >> >> 2012/1/5 Philippe >> >>> Depending on the CL you're reading at it will yes : if the CL >>> requires that the "slow" node create a digest of the data and send it to >>> the coordinator then it might explain the poor performance on reads. >>> What >>> is your read CL ? >>> >>> 2012/1/5 R. Verlangen >>> >>> As I posted this I noticed that the other node's CPU is running high on some other cronjobs (every couple of minutes to 60% usage). Is the lack of more CPU cycles a problem in this case? Robin 2012/1/5 R. Verlangen CPU is idle (< 10% usage). Disk reads occasionally blocks over > 32/64K. Writes around 0-5MB per second. Network traffic 0.1 / 0.1 > MB/s (in > / out). Paging 0. System int ~ 1300, csw ~ 2500. > > > 2012/1/5 Philippe > >> What can you see in vmstat/dstat ? >> Le 5 janv. 2012 11:58, "R. Verlangen" a écrit : >> >> Hi there, >>> >>> I'm running a cassandra 0.8.6 cluster with 2 nodes (in 2 DC's), >>> RF = 2. Actual data on the nodes is only 1GB. Disk latency < 1ms. >>> Disk >>> throughput ~ 0.4MB/s. OS load always below 1 (on a 8 core machine >>> with 16GB >>> ram). >>> >>> When I'm running my writes against the cluster with cl = ONE all >>> reads appear to be faster then the writes. >>> >>> Average write speed = 1600us/operation >>> Average read speed = 200us/operation >>> >>> I'm really wondering why this is the case. Anyone got a clue? >>> >>> With kind regards, >>> Robin >>> >> > >>> >> > >>> >> > >
Re: Writes slower then reads
What happens when you turn off the cron jobs ? Cheers - Aaron Morton Freelance Developer @aaronmorton http://www.thelastpickle.com On 6/01/2012, at 6:57 AM, Philippe wrote: > Unless you are doing huge batches no... don't have any other idea for now... > > 2012/1/5 R. Verlangen > The write and read load is very minimal the moment. Roughly 10 writes + 10 > reads / second. So 20 operations per second. Don't think that overloads my > cluster, does it? > > > 2012/1/5 Philippe > You may be overloading the cluster though... > > My hypothesis is that your traffic is being spread across your node and that > one slow node is slowing down the fraction of traffic that goes to that node > (when it's acting as coordinator). > So what I would do is reduce the read load a lot to make sure I don't > overload the cluster and measure if I see a 1/RF improvement in response time > which would validate my hypothesis. > > > 2012/1/5 R. Verlangen > > It does not appear to affect the response time, certainly not in a positive > way. > > > 2012/1/5 Philippe > What if you shutdown the cassandra service on the slow node, does that > improve your read performance ? > If it does then that sole node is responsible for the slow down because it > can't act as a coordinator fast enough. > > 2012/1/5 R. Verlangen > > I'm also reading with CL = ONE > > > 2012/1/5 Philippe > Depending on the CL you're reading at it will yes : if the CL requires that > the "slow" node create a digest of the data and send it to the coordinator > then it might explain the poor performance on reads. What is your read CL ? > > 2012/1/5 R. Verlangen > > As I posted this I noticed that the other node's CPU is running high on some > other cronjobs (every couple of minutes to 60% usage). Is the lack of more > CPU cycles a problem in this case? > > Robin > > 2012/1/5 R. Verlangen > > CPU is idle (< 10% usage). Disk reads occasionally blocks over 32/64K. Writes > around 0-5MB per second. Network traffic 0.1 / 0.1 MB/s (in / out). Paging 0. > System int ~ 1300, csw ~ 2500. > > > 2012/1/5 Philippe > What can you see in vmstat/dstat ? > > Le 5 janv. 2012 11:58, "R. Verlangen" a écrit : > > Hi there, > > I'm running a cassandra 0.8.6 cluster with 2 nodes (in 2 DC's), RF = 2. > Actual data on the nodes is only 1GB. Disk latency < 1ms. Disk throughput ~ > 0.4MB/s. OS load always below 1 (on a 8 core machine with 16GB ram). > > When I'm running my writes against the cluster with cl = ONE all reads appear > to be faster then the writes. > > Average write speed = 1600us/operation > Average read speed = 200us/operation > > I'm really wondering why this is the case. Anyone got a clue? > > With kind regards, > Robin > > > > > > > > >
Re: Writes slower then reads
Unless you are doing huge batches no... don't have any other idea for now... 2012/1/5 R. Verlangen > The write and read load is very minimal the moment. Roughly 10 writes + 10 > reads / second. So 20 operations per second. Don't think that overloads my > cluster, does it? > > > 2012/1/5 Philippe > >> You may be overloading the cluster though... >> >> My hypothesis is that your traffic is being spread across your node and >> that one slow node is slowing down the fraction of traffic that goes to >> that node (when it's acting as coordinator). >> So what I would do is reduce the read load a lot to make sure I don't >> overload the cluster and measure if I see a 1/RF improvement in response >> time which would validate my hypothesis. >> >> >> 2012/1/5 R. Verlangen >> >> It does not appear to affect the response time, certainly not in a >>> positive way. >>> >>> >>> 2012/1/5 Philippe >>> What if you shutdown the cassandra service on the slow node, does that improve your read performance ? If it does then that sole node is responsible for the slow down because it can't act as a coordinator fast enough. 2012/1/5 R. Verlangen I'm also reading with CL = ONE > > > 2012/1/5 Philippe > >> Depending on the CL you're reading at it will yes : if the CL >> requires that the "slow" node create a digest of the data and send it to >> the coordinator then it might explain the poor performance on reads. What >> is your read CL ? >> >> 2012/1/5 R. Verlangen >> >> As I posted this I noticed that the other node's CPU is running high >>> on some other cronjobs (every couple of minutes to 60% usage). Is the >>> lack >>> of more CPU cycles a problem in this case? >>> >>> Robin >>> >>> 2012/1/5 R. Verlangen >>> >>> CPU is idle (< 10% usage). Disk reads occasionally blocks over 32/64K. Writes around 0-5MB per second. Network traffic 0.1 / 0.1 MB/s (in / out). Paging 0. System int ~ 1300, csw ~ 2500. 2012/1/5 Philippe > What can you see in vmstat/dstat ? > Le 5 janv. 2012 11:58, "R. Verlangen" a écrit : > > Hi there, >> >> I'm running a cassandra 0.8.6 cluster with 2 nodes (in 2 DC's), >> RF = 2. Actual data on the nodes is only 1GB. Disk latency < 1ms. >> Disk >> throughput ~ 0.4MB/s. OS load always below 1 (on a 8 core machine >> with 16GB >> ram). >> >> When I'm running my writes against the cluster with cl = ONE all >> reads appear to be faster then the writes. >> >> Average write speed = 1600us/operation >> Average read speed = 200us/operation >> >> I'm really wondering why this is the case. Anyone got a clue? >> >> With kind regards, >> Robin >> > >>> >> > >>> >> >
Re: Writes slower then reads
The write and read load is very minimal the moment. Roughly 10 writes + 10 reads / second. So 20 operations per second. Don't think that overloads my cluster, does it? 2012/1/5 Philippe > You may be overloading the cluster though... > > My hypothesis is that your traffic is being spread across your node and > that one slow node is slowing down the fraction of traffic that goes to > that node (when it's acting as coordinator). > So what I would do is reduce the read load a lot to make sure I don't > overload the cluster and measure if I see a 1/RF improvement in response > time which would validate my hypothesis. > > > 2012/1/5 R. Verlangen > > It does not appear to affect the response time, certainly not in a >> positive way. >> >> >> 2012/1/5 Philippe >> >>> What if you shutdown the cassandra service on the slow node, does that >>> improve your read performance ? >>> If it does then that sole node is responsible for the slow down because >>> it can't act as a coordinator fast enough. >>> >>> 2012/1/5 R. Verlangen >>> >>> I'm also reading with CL = ONE 2012/1/5 Philippe > Depending on the CL you're reading at it will yes : if the CL requires > that the "slow" node create a digest of the data and send it to the > coordinator then it might explain the poor performance on reads. What is > your read CL ? > > 2012/1/5 R. Verlangen > > As I posted this I noticed that the other node's CPU is running high >> on some other cronjobs (every couple of minutes to 60% usage). Is the >> lack >> of more CPU cycles a problem in this case? >> >> Robin >> >> 2012/1/5 R. Verlangen >> >> CPU is idle (< 10% usage). Disk reads occasionally blocks over >>> 32/64K. Writes around 0-5MB per second. Network traffic 0.1 / 0.1 MB/s >>> (in >>> / out). Paging 0. System int ~ 1300, csw ~ 2500. >>> >>> >>> 2012/1/5 Philippe >>> What can you see in vmstat/dstat ? Le 5 janv. 2012 11:58, "R. Verlangen" a écrit : Hi there, > > I'm running a cassandra 0.8.6 cluster with 2 nodes (in 2 DC's), RF > = 2. Actual data on the nodes is only 1GB. Disk latency < 1ms. Disk > throughput ~ 0.4MB/s. OS load always below 1 (on a 8 core machine > with 16GB > ram). > > When I'm running my writes against the cluster with cl = ONE all > reads appear to be faster then the writes. > > Average write speed = 1600us/operation > Average read speed = 200us/operation > > I'm really wondering why this is the case. Anyone got a clue? > > With kind regards, > Robin > >>> >> > >>> >> >
Re: Writes slower then reads
You may be overloading the cluster though... My hypothesis is that your traffic is being spread across your node and that one slow node is slowing down the fraction of traffic that goes to that node (when it's acting as coordinator). So what I would do is reduce the read load a lot to make sure I don't overload the cluster and measure if I see a 1/RF improvement in response time which would validate my hypothesis. 2012/1/5 R. Verlangen > It does not appear to affect the response time, certainly not in a > positive way. > > > 2012/1/5 Philippe > >> What if you shutdown the cassandra service on the slow node, does that >> improve your read performance ? >> If it does then that sole node is responsible for the slow down because >> it can't act as a coordinator fast enough. >> >> 2012/1/5 R. Verlangen >> >> I'm also reading with CL = ONE >>> >>> >>> 2012/1/5 Philippe >>> Depending on the CL you're reading at it will yes : if the CL requires that the "slow" node create a digest of the data and send it to the coordinator then it might explain the poor performance on reads. What is your read CL ? 2012/1/5 R. Verlangen As I posted this I noticed that the other node's CPU is running high on > some other cronjobs (every couple of minutes to 60% usage). Is the lack of > more CPU cycles a problem in this case? > > Robin > > 2012/1/5 R. Verlangen > > CPU is idle (< 10% usage). Disk reads occasionally blocks over 32/64K. >> Writes around 0-5MB per second. Network traffic 0.1 / 0.1 MB/s (in / >> out). >> Paging 0. System int ~ 1300, csw ~ 2500. >> >> >> 2012/1/5 Philippe >> >>> What can you see in vmstat/dstat ? >>> Le 5 janv. 2012 11:58, "R. Verlangen" a écrit : >>> >>> Hi there, I'm running a cassandra 0.8.6 cluster with 2 nodes (in 2 DC's), RF = 2. Actual data on the nodes is only 1GB. Disk latency < 1ms. Disk throughput ~ 0.4MB/s. OS load always below 1 (on a 8 core machine with 16GB ram). When I'm running my writes against the cluster with cl = ONE all reads appear to be faster then the writes. Average write speed = 1600us/operation Average read speed = 200us/operation I'm really wondering why this is the case. Anyone got a clue? With kind regards, Robin >>> >> > >>> >> >
Re: Writes slower then reads
It does not appear to affect the response time, certainly not in a positive way. 2012/1/5 Philippe > What if you shutdown the cassandra service on the slow node, does that > improve your read performance ? > If it does then that sole node is responsible for the slow down because it > can't act as a coordinator fast enough. > > 2012/1/5 R. Verlangen > > I'm also reading with CL = ONE >> >> >> 2012/1/5 Philippe >> >>> Depending on the CL you're reading at it will yes : if the CL requires >>> that the "slow" node create a digest of the data and send it to the >>> coordinator then it might explain the poor performance on reads. What is >>> your read CL ? >>> >>> 2012/1/5 R. Verlangen >>> >>> As I posted this I noticed that the other node's CPU is running high on some other cronjobs (every couple of minutes to 60% usage). Is the lack of more CPU cycles a problem in this case? Robin 2012/1/5 R. Verlangen CPU is idle (< 10% usage). Disk reads occasionally blocks over 32/64K. > Writes around 0-5MB per second. Network traffic 0.1 / 0.1 MB/s (in / out). > Paging 0. System int ~ 1300, csw ~ 2500. > > > 2012/1/5 Philippe > >> What can you see in vmstat/dstat ? >> Le 5 janv. 2012 11:58, "R. Verlangen" a écrit : >> >> Hi there, >>> >>> I'm running a cassandra 0.8.6 cluster with 2 nodes (in 2 DC's), RF = >>> 2. Actual data on the nodes is only 1GB. Disk latency < 1ms. Disk >>> throughput ~ 0.4MB/s. OS load always below 1 (on a 8 core machine with >>> 16GB >>> ram). >>> >>> When I'm running my writes against the cluster with cl = ONE all >>> reads appear to be faster then the writes. >>> >>> Average write speed = 1600us/operation >>> Average read speed = 200us/operation >>> >>> I'm really wondering why this is the case. Anyone got a clue? >>> >>> With kind regards, >>> Robin >>> >> > >>> >> >
Re: Writes slower then reads
What if you shutdown the cassandra service on the slow node, does that improve your read performance ? If it does then that sole node is responsible for the slow down because it can't act as a coordinator fast enough. 2012/1/5 R. Verlangen > I'm also reading with CL = ONE > > > 2012/1/5 Philippe > >> Depending on the CL you're reading at it will yes : if the CL requires >> that the "slow" node create a digest of the data and send it to the >> coordinator then it might explain the poor performance on reads. What is >> your read CL ? >> >> 2012/1/5 R. Verlangen >> >> As I posted this I noticed that the other node's CPU is running high on >>> some other cronjobs (every couple of minutes to 60% usage). Is the lack of >>> more CPU cycles a problem in this case? >>> >>> Robin >>> >>> 2012/1/5 R. Verlangen >>> >>> CPU is idle (< 10% usage). Disk reads occasionally blocks over 32/64K. Writes around 0-5MB per second. Network traffic 0.1 / 0.1 MB/s (in / out). Paging 0. System int ~ 1300, csw ~ 2500. 2012/1/5 Philippe > What can you see in vmstat/dstat ? > Le 5 janv. 2012 11:58, "R. Verlangen" a écrit : > > Hi there, >> >> I'm running a cassandra 0.8.6 cluster with 2 nodes (in 2 DC's), RF = >> 2. Actual data on the nodes is only 1GB. Disk latency < 1ms. Disk >> throughput ~ 0.4MB/s. OS load always below 1 (on a 8 core machine with >> 16GB >> ram). >> >> When I'm running my writes against the cluster with cl = ONE all >> reads appear to be faster then the writes. >> >> Average write speed = 1600us/operation >> Average read speed = 200us/operation >> >> I'm really wondering why this is the case. Anyone got a clue? >> >> With kind regards, >> Robin >> > >>> >> >
Re: Writes slower then reads
I'm also reading with CL = ONE 2012/1/5 Philippe > Depending on the CL you're reading at it will yes : if the CL requires > that the "slow" node create a digest of the data and send it to the > coordinator then it might explain the poor performance on reads. What is > your read CL ? > > 2012/1/5 R. Verlangen > > As I posted this I noticed that the other node's CPU is running high on >> some other cronjobs (every couple of minutes to 60% usage). Is the lack of >> more CPU cycles a problem in this case? >> >> Robin >> >> 2012/1/5 R. Verlangen >> >> CPU is idle (< 10% usage). Disk reads occasionally blocks over 32/64K. >>> Writes around 0-5MB per second. Network traffic 0.1 / 0.1 MB/s (in / out). >>> Paging 0. System int ~ 1300, csw ~ 2500. >>> >>> >>> 2012/1/5 Philippe >>> What can you see in vmstat/dstat ? Le 5 janv. 2012 11:58, "R. Verlangen" a écrit : Hi there, > > I'm running a cassandra 0.8.6 cluster with 2 nodes (in 2 DC's), RF = > 2. Actual data on the nodes is only 1GB. Disk latency < 1ms. Disk > throughput ~ 0.4MB/s. OS load always below 1 (on a 8 core machine with > 16GB > ram). > > When I'm running my writes against the cluster with cl = ONE all reads > appear to be faster then the writes. > > Average write speed = 1600us/operation > Average read speed = 200us/operation > > I'm really wondering why this is the case. Anyone got a clue? > > With kind regards, > Robin > >>> >> >
Re: Writes slower then reads
Depending on the CL you're reading at it will yes : if the CL requires that the "slow" node create a digest of the data and send it to the coordinator then it might explain the poor performance on reads. What is your read CL ? 2012/1/5 R. Verlangen > As I posted this I noticed that the other node's CPU is running high on > some other cronjobs (every couple of minutes to 60% usage). Is the lack of > more CPU cycles a problem in this case? > > Robin > > 2012/1/5 R. Verlangen > > CPU is idle (< 10% usage). Disk reads occasionally blocks over 32/64K. >> Writes around 0-5MB per second. Network traffic 0.1 / 0.1 MB/s (in / out). >> Paging 0. System int ~ 1300, csw ~ 2500. >> >> >> 2012/1/5 Philippe >> >>> What can you see in vmstat/dstat ? >>> Le 5 janv. 2012 11:58, "R. Verlangen" a écrit : >>> >>> Hi there, I'm running a cassandra 0.8.6 cluster with 2 nodes (in 2 DC's), RF = 2. Actual data on the nodes is only 1GB. Disk latency < 1ms. Disk throughput ~ 0.4MB/s. OS load always below 1 (on a 8 core machine with 16GB ram). When I'm running my writes against the cluster with cl = ONE all reads appear to be faster then the writes. Average write speed = 1600us/operation Average read speed = 200us/operation I'm really wondering why this is the case. Anyone got a clue? With kind regards, Robin >>> >> >
Re: Writes slower then reads
As I posted this I noticed that the other node's CPU is running high on some other cronjobs (every couple of minutes to 60% usage). Is the lack of more CPU cycles a problem in this case? Robin 2012/1/5 R. Verlangen > CPU is idle (< 10% usage). Disk reads occasionally blocks over 32/64K. > Writes around 0-5MB per second. Network traffic 0.1 / 0.1 MB/s (in / out). > Paging 0. System int ~ 1300, csw ~ 2500. > > > 2012/1/5 Philippe > >> What can you see in vmstat/dstat ? >> Le 5 janv. 2012 11:58, "R. Verlangen" a écrit : >> >> Hi there, >>> >>> I'm running a cassandra 0.8.6 cluster with 2 nodes (in 2 DC's), RF = 2. >>> Actual data on the nodes is only 1GB. Disk latency < 1ms. Disk throughput ~ >>> 0.4MB/s. OS load always below 1 (on a 8 core machine with 16GB ram). >>> >>> When I'm running my writes against the cluster with cl = ONE all reads >>> appear to be faster then the writes. >>> >>> Average write speed = 1600us/operation >>> Average read speed = 200us/operation >>> >>> I'm really wondering why this is the case. Anyone got a clue? >>> >>> With kind regards, >>> Robin >>> >> >
Re: Writes slower then reads
CPU is idle (< 10% usage). Disk reads occasionally blocks over 32/64K. Writes around 0-5MB per second. Network traffic 0.1 / 0.1 MB/s (in / out). Paging 0. System int ~ 1300, csw ~ 2500. 2012/1/5 Philippe > What can you see in vmstat/dstat ? > Le 5 janv. 2012 11:58, "R. Verlangen" a écrit : > > Hi there, >> >> I'm running a cassandra 0.8.6 cluster with 2 nodes (in 2 DC's), RF = 2. >> Actual data on the nodes is only 1GB. Disk latency < 1ms. Disk throughput ~ >> 0.4MB/s. OS load always below 1 (on a 8 core machine with 16GB ram). >> >> When I'm running my writes against the cluster with cl = ONE all reads >> appear to be faster then the writes. >> >> Average write speed = 1600us/operation >> Average read speed = 200us/operation >> >> I'm really wondering why this is the case. Anyone got a clue? >> >> With kind regards, >> Robin >> >
Re: Writes slower then reads
What can you see in vmstat/dstat ? Le 5 janv. 2012 11:58, "R. Verlangen" a écrit : > Hi there, > > I'm running a cassandra 0.8.6 cluster with 2 nodes (in 2 DC's), RF = 2. > Actual data on the nodes is only 1GB. Disk latency < 1ms. Disk throughput ~ > 0.4MB/s. OS load always below 1 (on a 8 core machine with 16GB ram). > > When I'm running my writes against the cluster with cl = ONE all reads > appear to be faster then the writes. > > Average write speed = 1600us/operation > Average read speed = 200us/operation > > I'm really wondering why this is the case. Anyone got a clue? > > With kind regards, > Robin >
Writes slower then reads
Hi there, I'm running a cassandra 0.8.6 cluster with 2 nodes (in 2 DC's), RF = 2. Actual data on the nodes is only 1GB. Disk latency < 1ms. Disk throughput ~ 0.4MB/s. OS load always below 1 (on a 8 core machine with 16GB ram). When I'm running my writes against the cluster with cl = ONE all reads appear to be faster then the writes. Average write speed = 1600us/operation Average read speed = 200us/operation I'm really wondering why this is the case. Anyone got a clue? With kind regards, Robin
