Re: Read after Write inconsistent at times
Hi, Check the tombstone count, If is it to high, your query will be impacted. If tombstone is a problem, you can try to reduce your "gc_grace_seconds" to reduce tombstone count(Carefully because you use cross data centers). Tchau, Petrus Silva On Fri, Feb 24, 2017 at 12:07 AM, Jan Kesten wrote: > Hi, > > are your nodes at high load? Are there any dropped messages (nodetool > tpstats) on any node? > > Also have a look at your system clocks. C* needs them in thight sync - via > ntp for example. Side hint: if you use ntp use the same set of upstreams on > all of your nodes - ideal your own one. Using pool.ntp.org might lead to > minimal dirfts in time across your cluster. > > Another thing that could help you out is using client side timestamps: > https://docs.datastax.com/en/developer/java-driver/3.1/manua > l/query_timestamps/ (of course only when you are using a single client or > all clients are in sync via ntp). > > > > Am 24.02.2017 um 07:29 schrieb Charulata Sharma (charshar): > >> >> Hi All, >> >> In my application sometimes I cannot read data that just got inserted. >> This happens very intermittently. Both write and read use LOCAL QUOROM. >> >> We have a cluster of 12 nodes which spans across 2 Data Centers and a RF >> of 3. >> >> Has anyone encountered this problem and if yes what steps have you taken >> to solve it >> >> Thanks, >> Charu >> > > -- > Jan Kesten, mailto:j.kes...@enercast.de > Tel.: +49 561/4739664-0 FAX: -9 Mobil: +49 160 / 90 98 41 68 > enercast GmbH Universitätsplatz 12 D-34127 Kassel HRB15471 > http://www.enercast.de Online-Prognosen für erneuerbare Energien > Geschäftsführung: Thomas Landgraf (CEO), Bernd Kratz (CTO), Philipp Rinder > (CSO) > > Diese E-Mail und etwaige Anhänge können vertrauliche und/oder rechtlich > geschützte Informationen enthalten. Falls Sie nicht der angegebene > Empfänger sind oder falls diese E-Mail irrtümlich an Sie adressiert wurde, > benachrichtigen Sie uns bitte sofort durch Antwort-E-Mail und löschen Sie > diese E-Mail nebst etwaigen Anlagen von Ihrem System. Ebenso dürfen Sie > diese E-Mail oder ihre Anlagen nicht kopieren oder an Dritte weitergeben. > Vielen Dank. > > This e-mail and any attachment may contain confidential and/or privileged > information. If you are not the named addressee or if this transmission has > been addressed to you in error, please notify us immediately by reply > e-mail and then delete this e-mail and any attachment from your system. > Please understand that you must not copy this e-mail or any attachment or > disclose the contents to any other person. Thank you for your cooperation. > >
Re: How do you monitoring Cassandra Cluster?
I'm using JMX+Prometheus and Grafana. JMX = https://github.com/prometheus/jmx_exporter Prometheus + Grafana = https://prometheus.io/docs/visualization/grafana/ There are some dashboard examples like that: https://grafana.com/dashboards/371 Looks good. Thanks, Petrus Silva On Wed, Jun 28, 2017 at 5:55 AM, Peng Xiao <2535...@qq.com> wrote: > Dear All, > > we are currently using Cassandra 2.1.13,and it has grown to 5TB size with > 32 nodes in one DC. > For monitoring,opsCenter does not send alarm and not free in higher > version.so we have to use a simple JMX+Zabbix template.And we plan to use > Jolokia+JMX2Graphite to draw the metrics chart now. > > Could you please advise? > > Thanks, > Henry >
Re: Cassandra seems slow when having many read operations
How is your Percent Repaired when you run " nodetool info" ? Search for : "reduced num_token = improved performance ??" topic. The people were discussing that. How is your compaction is configured? Could you run the same process in command line to have a measurement? Thanks, Petrus Silva On Thu, Jul 13, 2017 at 7:49 AM, Felipe Esteves < felipe.este...@b2wdigital.com> wrote: > Hi, > > I have a Cassandra 2.1 cluster running on AWS that receives high read > loads, jumping from 100k requests to 400k requests, for example. Then it > normalizes and later cames another high throughput. > > To the application, it appears that Cassandra is slow. However, cpu and > disk use is ok in every instance, row cache is enabled and with almost 100% > hit rate. > > The logs from Cassandra instances doesn't have any errors, nor tombstone > messages or something liked that. It's mostly compactions and G1GC > operations. > > Any hints on where to investigate more? > > > Felipe Esteves > > > > >
Re: Cassandra seems slow when having many read operations
Hi Felipe, Yes, try it and let us know how it goes. Thanks, Petrus Silva. On Fri, Jul 14, 2017 at 11:37 AM, Felipe Esteves < felipe.este...@b2wdigital.com> wrote: > Hi Petrus, thanks for the feedback. > > I couldn't found the percent repaired in nodetool info, C* version is > 2.1.8, maybe it's something newer than that? > > I'm analyzing this thread about num_token. > > Compaction is "compaction_throughput_mb_per_sec: 16", I don't get pending > compactions in Opscenter. > > One point I've noticed, is that Opscenter show "OS: Disk Latency" max with > high values when the problem occurs, but it doesn't reflect in server > directly monitoring, in these tools the IO and latency of disks seems ok. > But seems to me that "read repair attempted" is a bit high, maybe it will > explain the latency in reads. I will try to run a repair on cluster to see > how it goes. > > Felipe Esteves > > Tecnologia > > felipe.este...@b2wdigital.com > > Tel.: (21) 3504-7162 ramal 57162 > > Skype: felipe2esteves > > 2017-07-13 15:02 GMT-03:00 Petrus Gomes : > >> How is your Percent Repaired when you run " nodetool info" ? >> >> Search for : >> "reduced num_token = improved performance ??" topic. >> The people were discussing that. >> >> How is your compaction is configured? >> >> Could you run the same process in command line to have a measurement? >> >> Thanks, >> Petrus Silva >> >> >> >> On Thu, Jul 13, 2017 at 7:49 AM, Felipe Esteves < >> felipe.este...@b2wdigital.com> wrote: >> >>> Hi, >>> >>> I have a Cassandra 2.1 cluster running on AWS that receives high read >>> loads, jumping from 100k requests to 400k requests, for example. Then it >>> normalizes and later cames another high throughput. >>> >>> To the application, it appears that Cassandra is slow. However, cpu and >>> disk use is ok in every instance, row cache is enabled and with almost 100% >>> hit rate. >>> >>> The logs from Cassandra instances doesn't have any errors, nor tombstone >>> messages or something liked that. It's mostly compactions and G1GC >>> operations. >>> >>> Any hints on where to investigate more? >>> >>> >>> Felipe Esteves >>> >>> >>> >>> >>> >> >> -- >> >> Esta mensagem pode conter informações confidenciais e somente o indivíduo >> ou entidade a quem foi destinada pode utilizá-la. A transmissão incorreta >> da mensagem não acarreta a perda de sua confidencialidade. Caso esta >> mensagem tenha sido recebida por engano, solicitamos que o fato seja >> comunicado ao remetente e que a mensagem seja eliminada de seu sistema >> imediatamente. É vedado a qualquer pessoa que não seja o destinatário usar, >> revelar, distribuir ou copiar qualquer parte desta mensagem. Ambiente de >> comunicação sujeito a monitoramento. >> >> This message may include confidential information and only the intended >> addresses have the right to use it as is, or any part of it. A wrong >> transmission does not break its confidentiality. If you've received it >> because of a mistake or erroneous transmission, please notify the sender >> and delete it from your system immediately. This communication environment >> is controlled and monitored. >> >> B2W Digital >> >> >> > > > >
Re: Quick question to config Prometheus to monitor Cassandra cluster
I use the same environment. Follow a few links: Use this link, is the best one to connect Cassandra and prometheus: https://www.robustperception.io/monitoring-cassandra-with-prometheus/ JMX agent: https://github.com/nabto/cassandra-prometheus https://community.grafana.com/t/how-to-connect-prometheus-to-cassandra/1153 Grafana dashboard : https://grafana.com/dashboards/371 Create a connection on Grafana connecting to Prometheus. On Thu, Jul 20, 2017 at 4:13 AM, Kiran mk wrote: > You have to download the Prometheus HTTP jmx dependencies jar and download > the Cassandra yaml and mention the jmx port in the config (7199). > > Run the agent on specific port" on all the Cassandra nodes. > > After this go to your Prometheus server and make the scrape config to > metrics from all clients. > > > > On 20-Jul-2017 3:27 PM, "wxn...@zjqunshuo.com" > wrote: > > Hi, > I'm going to set up Prometheus+Grafana to monitor Cassandra cluster. I > installed Prometheus and started it, but don't know how to config it to > support Cassandra. > Any ideas or related articles are appreciated. > > Cheers, > Simon > > >
Re: Cassandra seems slow when having many read operations
Thanks a lot to share the result. Boa Sorte. ;-) Take care. Petris Silva On Fri, Jul 21, 2017 at 12:19 PM, Felipe Esteves < felipe.este...@b2wdigital.com> wrote: > Hi, Petrus, > > Seems we've solved the problem, but it wasn't relationed to repair the > cluster or disk latency. > I've increased the memory available for Cassandra from 16GB to 24GB and > the performance was much improved! > The main symptom we've observed in Opscenter was a significantly decrease > in total compactions graph. > > Felipe Esteves > > Tecnologia > > felipe.este...@b2wdigital.com > > > > 2017-07-15 3:23 GMT-03:00 Petrus Gomes : > >> Hi Felipe, >> >> Yes, try it and let us know how it goes. >> >> Thanks, >> Petrus Silva. >> >> On Fri, Jul 14, 2017 at 11:37 AM, Felipe Esteves < >> felipe.este...@b2wdigital.com> wrote: >> >>> Hi Petrus, thanks for the feedback. >>> >>> I couldn't found the percent repaired in nodetool info, C* version is >>> 2.1.8, maybe it's something newer than that? >>> >>> I'm analyzing this thread about num_token. >>> >>> Compaction is "compaction_throughput_mb_per_sec: 16", I don't get >>> pending compactions in Opscenter. >>> >>> One point I've noticed, is that Opscenter show "OS: Disk Latency" max >>> with high values when the problem occurs, but it doesn't reflect in server >>> directly monitoring, in these tools the IO and latency of disks seems ok. >>> But seems to me that "read repair attempted" is a bit high, maybe it >>> will explain the latency in reads. I will try to run a repair on cluster to >>> see how it goes. >>> >>> Felipe Esteves >>> >>> Tecnologia >>> >>> felipe.este...@b2wdigital.com >>> >>> Tel.: (21) 3504-7162 ramal 57162 >>> >>> Skype: felipe2esteves >>> >>> 2017-07-13 15:02 GMT-03:00 Petrus Gomes : >>> >>>> How is your Percent Repaired when you run " nodetool info" ? >>>> >>>> Search for : >>>> "reduced num_token = improved performance ??" topic. >>>> The people were discussing that. >>>> >>>> How is your compaction is configured? >>>> >>>> Could you run the same process in command line to have a measurement? >>>> >>>> Thanks, >>>> Petrus Silva >>>> >>>> >>>> >>>> On Thu, Jul 13, 2017 at 7:49 AM, Felipe Esteves < >>>> felipe.este...@b2wdigital.com> wrote: >>>> >>>>> Hi, >>>>> >>>>> I have a Cassandra 2.1 cluster running on AWS that receives high read >>>>> loads, jumping from 100k requests to 400k requests, for example. Then it >>>>> normalizes and later cames another high throughput. >>>>> >>>>> To the application, it appears that Cassandra is slow. However, cpu >>>>> and disk use is ok in every instance, row cache is enabled and with almost >>>>> 100% hit rate. >>>>> >>>>> The logs from Cassandra instances doesn't have any errors, nor >>>>> tombstone messages or something liked that. It's mostly compactions and >>>>> G1GC operations. >>>>> >>>>> Any hints on where to investigate more? >>>>> >>>>> >>>>> Felipe Esteves >>>>> >>>>> >>>>> >>>>> >>>>> >>>> >>>> -- >>>> >>>> Esta mensagem pode conter informações confidenciais e somente o >>>> indivíduo ou entidade a quem foi destinada pode utilizá-la. A transmissão >>>> incorreta da mensagem não acarreta a perda de sua confidencialidade. Caso >>>> esta mensagem tenha sido recebida por engano, solicitamos que o fato seja >>>> comunicado ao remetente e que a mensagem seja eliminada de seu sistema >>>> imediatamente. É vedado a qualquer pessoa que não seja o destinatário usar, >>>> revelar, distribuir ou copiar qualquer parte desta mensagem. Ambiente de >>>> comunicação sujeito a monitoramento. >>>> >>>> This message may include confidential information and only the intended >>>> addresses have the right to use it as is, or any part of it. A wrong >>>> transmission does not break its confidentiality. If you've received it >>>> because of a mistake or erroneous transmission, please notify the sender >>>> and delete it from your system immediately. This communication environment >>>> is controlled and monitored. >>>> >>>> B2W Digital >>>> >>>> >>>> >>> >>> >>> >>> >> > > >
Re: Upgrading from Cassandra 3.9 to Cassandra 3.11
Hi Igor, Have a look into that document. ;-) https://github.com/apache/cassandra/blob/cassandra-3.11/NEWS.txt Thanks, Petrus Silva On Fri, Aug 4, 2017 at 10:15 AM, Igor Leão wrote: > Hi there, > > Is it possible to add a Cassandra 3.11 node into a Cassandra 3.9 cluster? > I wish I could upgrade this cluster without uninstalling and installing > Cassandra on existing nodes, just by adding and removing nodes. > > Thanks in advance! >
