[ https://issues.apache.org/jira/browse/CASSANDRA-7186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14233268#comment-14233268 ]
Alexander Bulaev commented on CASSANDRA-7186: --------------------------------------------- 1. No, test clusters are not under heavy load, they probably get 1-2 RPS. By the way, even the production cluster does not seem to be overloaded (LA is up to 3.5 in rush hour on 32-core systems, which seems perfectly OK). At peak we are having ~250 RPS writes and ~100 RPS reads in production environment. Config file is available here: https://www.dropbox.com/s/x5xhtkox7sj25u6/cassandra.yaml?dl=0 Unfortunately the logs were rotated, but when I was watching them when the issue had occurred, I could not find any suspicious messages. 2. Yes, we are using NTP and clocks are in perfect sync. > alter table add column not always propogating > --------------------------------------------- > > Key: CASSANDRA-7186 > URL: https://issues.apache.org/jira/browse/CASSANDRA-7186 > Project: Cassandra > Issue Type: Bug > Reporter: Martin Meyer > Assignee: Philip Thompson > Fix For: 2.0.12 > > > I've been many times in Cassandra 2.0.6 that adding columns to existing > tables seems to not fully propagate to our entire cluster. We add an extra > column to various tables maybe 0-2 times a week, and so far many of these > ALTERs have resulted in at least one node showing the old table description a > pretty long time (~30 mins) after the original ALTER command was issued. > We originally identified this issue when a connected clients would complain > that a column it issued a SELECT for wasn't a known column, at which point we > have to ask each node to describe the most recently altered table. One of > them will not know about the newly added field. Issuing the original ALTER > statement on that node makes everything work correctly. > We have seen this issue on multiple tables (we don't always alter the same > one). It has affected various nodes in the cluster (not always the same one > is not getting the mutation propagated). No new nodes have been added to the > cluster recently. All nodes are homogenous (hardware and software), running > 2.0.6. We don't see any particular errors or exceptions on the node that > didn't get the schema update, only the later error from a Java client about > asking for an unknown column in a SELECT. We have to check each node manually > to find the offender. The tables he have seen this on are under fairly heavy > read and write load, but we haven't altered any tables that are not, so that > might not be important. -- This message was sent by Atlassian JIRA (v6.3.4#6332)