[jira] [Commented] (CASSANDRA-8848) Protocol exceptions always use streamID 0
[ https://issues.apache.org/jira/browse/CASSANDRA-8848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14341456#comment-14341456 ] Chris Bannister commented on CASSANDRA-8848: thanks! > Protocol exceptions always use streamID 0 > - > > Key: CASSANDRA-8848 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8848 > Project: Cassandra > Issue Type: Bug > Components: Core >Reporter: Chris Bannister >Assignee: Chris Bannister >Priority: Minor > Fix For: 2.0.13, 2.1.4 > > Attachments: 8848-2.0.txt, 8848-2.1.txt, trunk-8848.patch > > > When decoding the binary protocol if a ProtocolException is thrown the > streamID the request came in on is lost. This makes it very hard for drivers > to correctly handle improper protocol implementations. > Included is a test case which sends a frame with version 0x82 and op 0x9 > which should return a ProtocolError indicating that PREPARE should be a > client Request not a response. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-8086) Cassandra should have ability to limit the number of native connections
[ https://issues.apache.org/jira/browse/CASSANDRA-8086?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Norman Maurer updated CASSANDRA-8086: - Attachment: 0001-CASSANDRA-8086-Allow-to-limit-the-number-of-native-c-final.patch This is the patch with comments addressed. It basically not add the handler to the pipeline if not explicit enabled. So it's kind of "dead-code" and so should expose no risk. > Cassandra should have ability to limit the number of native connections > --- > > Key: CASSANDRA-8086 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8086 > Project: Cassandra > Issue Type: Bug >Reporter: Vishy Kasar >Assignee: Norman Maurer > Fix For: 2.1.4 > > Attachments: > 0001-CASSANDRA-8086-Allow-to-limit-the-number-of-native-c-2.0.patch, > 0001-CASSANDRA-8086-Allow-to-limit-the-number-of-native-c-final.patch, > 0001-CASSANDRA-8086-Allow-to-limit-the-number-of-native-c.patch, > 0001-CASSANDRA-8086-Allow-to-limit-the-number-of-native-c.txt > > > We have a production cluster with 72 instances spread across 2 DCs. We have a > large number ( ~ 40,000 ) of clients hitting this cluster. Client normally > connects to 4 cassandra instances. Some event (we think it is a schema change > on server side) triggered the client to establish connections to all > cassandra instances of local DC. This brought the server to its knees. The > client connections failed and client attempted re-connections. > Cassandra should protect itself from such attack from client. Do we have any > knobs to control the number of max connections? If not, we need to add that > knob. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (CASSANDRA-8881) Add cassandra-coverage-deps - v2.0.12 jar to maven repo
Thilak created CASSANDRA-8881: - Summary: Add cassandra-coverage-deps - v2.0.12 jar to maven repo Key: CASSANDRA-8881 URL: https://issues.apache.org/jira/browse/CASSANDRA-8881 Project: Cassandra Issue Type: Bug Components: Config Environment: Ubuntu 14.04.1, Cassandra community edition 2.0.12, JDK 1.7.0_51, CCM (latest from git as on 28Feb2015) Reporter: Thilak When building Casandra v2.0.12 using CCM in Ubuntu, i get the below error : [artifact:dependencies] -- [artifact:dependencies] 1 required artifact is missing. [artifact:dependencies] [artifact:dependencies] for artifact: [artifact:dependencies] org.apache.cassandra:cassandra-coverage-deps:jar:2.0.12-SNAPSHOT [artifact:dependencies] [artifact:dependencies] from the specified remote repositories: [artifact:dependencies] central (http://repo1.maven.org/maven2) === I used the below command: ccm create demo_1node -v 2.0.12 -n 1 -s -d Could you add the jar to maven please? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8689) Assertion error in 2.1.2: ERROR [IndexSummaryManager:1]
[
https://issues.apache.org/jira/browse/CASSANDRA-8689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14341734#comment-14341734
]
Benedict commented on CASSANDRA-8689:
-
Since you're running 6 tests, I assume you're running with your test (as
stands) included? This breaks the other tests for the two reasons I gave above:
CASSANDRA-8805, and because the threads are still running mucking around with
the state even after the new test finishes successfully.
> Assertion error in 2.1.2: ERROR [IndexSummaryManager:1]
> ---
>
> Key: CASSANDRA-8689
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8689
> Project: Cassandra
> Issue Type: Bug
>Reporter: Jeff Liu
>Assignee: Benedict
> Fix For: 2.1.4
>
> Attachments: 8689-test.txt
>
>
> After upgrading a 6 nodes cassandra from 2.1.0 to 2.1.2, start getting the
> following assertion error.
> {noformat}
> ERROR [IndexSummaryManager:1] 2015-01-26 20:55:40,451
> CassandraDaemon.java:153 - Exception in thread
> Thread[IndexSummaryManager:1,1,main]
> java.lang.AssertionError: null
> at org.apache.cassandra.io.util.Memory.size(Memory.java:307)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.io.sstable.IndexSummary.getOffHeapSize(IndexSummary.java:192)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.io.sstable.SSTableReader.getIndexSummaryOffHeapSize(SSTableReader.java:1070)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.io.sstable.IndexSummaryManager.redistributeSummaries(IndexSummaryManager.java:292)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.io.sstable.IndexSummaryManager.redistributeSummaries(IndexSummaryManager.java:238)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.io.sstable.IndexSummaryManager$1.runMayThrow(IndexSummaryManager.java:139)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.concurrent.DebuggableScheduledThreadPoolExecutor$UncomplainingRunnable.run(DebuggableScheduledThreadPoolExecutor.java:77)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> [na:1.7.0_45]
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
> [na:1.7.0_45]
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
> [na:1.7.0_45]
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> [na:1.7.0_45]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> [na:1.7.0_45]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> [na:1.7.0_45]
> at java.lang.Thread.run(Thread.java:744) [na:1.7.0_45]
> {noformat}
> cassandra service is still running despite the issue. Node has total 8G
> memory with 2G allocated to heap. We are basically running read queries to
> retrieve data out of cassandra.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8860) Too many java.util.HashMap$Entry objects in heap
[
https://issues.apache.org/jira/browse/CASSANDRA-8860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14341781#comment-14341781
]
Benedict commented on CASSANDRA-8860:
-
bq. If createOverlapChain(a, overlapMap) return {a,b,c}, createOverlapChain(b,
overlapMap) must also return {a,b,c} because "overlapping" is a bidirectional
relationship. So if we have already get {a,b,c} by SStableReader a, we needn't
check b and c at all.
This is unfortunately not true. _a_ may overlap _b_ and _c_, but _b_ may only
overlap _a_, and not _c_, for instance. I knocked together a patch on the plane
with an average time complexity closer to O(N) but with worst case still
O(N^2), and achieved by filtering out sets of readers that are wholly contained
in other sets of overlapping readers. This may not be desirable, though, as it
may be that by considering a smaller group of readers we see a larger yield
from rewriting.
I think the best (medium term) solution to this complexity is to take ownership
of the hyperloglog implementation and to build a structure that can yield
overlaps efficiently from a multitude of hyperloglogs. If each hyperloglog
bucket maintained an ordered map from bucket cardinality to the set of objects
with that cardinality, an efficient search could be performed for overlap
candidates. Within each bucket we could find the largest overlapping
cardinalities, then take the intersection of these across all buckets, leaving
the top-ranked partition overlaps, and applying correction for the amount of
clustering column overlap between the candidates. We could then post-process to
include any files that are almost wholly overlapping with this set of best
overlaps. This is just a quick off-the-cuff sketch of a design, and no doubt
there are better and more developed approaches along these lines.
> Too many java.util.HashMap$Entry objects in heap
>
>
> Key: CASSANDRA-8860
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8860
> Project: Cassandra
> Issue Type: Bug
> Environment: Cassandra 2.1.3, jdk 1.7u51
>Reporter: Phil Yang
>Assignee: Phil Yang
> Fix For: 2.1.4
>
> Attachments: 8860-v2.txt, 8860.txt, cassandra-env.sh, cassandra.yaml,
> jmap.txt, jstack.txt, jstat-afterv1.txt, jstat-afterv2.txt, jstat-before.txt
>
>
> While I upgrading my cluster to 2.1.3, I find some nodes (not all) may have
> GC issue after the node restarting successfully. Old gen grows very fast and
> most of the space can not be recycled after setting its status to normal
> immediately. The qps of both reading and writing are very low and there is no
> heavy compaction.
> Jmap result seems strange that there are too many java.util.HashMap$Entry
> objects in heap, where in my experience the "[B" is usually the No1.
> If I downgrade it to 2.1.1, this issue will not appear.
> I uploaded conf files and jstack/jmap outputs. I'll upload heap dump if
> someone need it.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8881) Add cassandra-coverage-deps - v2.0.12 jar to maven repo
[ https://issues.apache.org/jira/browse/CASSANDRA-8881?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14341803#comment-14341803 ] Philip Thompson commented on CASSANDRA-8881: Are you having issues building 2.0.12 from source normally (not through ccm)? Could this have been a temporary problem accessing Maven Central? I am able to build 2.0.12 no problem. > Add cassandra-coverage-deps - v2.0.12 jar to maven repo > --- > > Key: CASSANDRA-8881 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8881 > Project: Cassandra > Issue Type: Bug > Components: Config > Environment: Ubuntu 14.04.1, Cassandra community edition 2.0.12, JDK > 1.7.0_51, CCM (latest from git as on 28Feb2015) >Reporter: Thilak > > When building Casandra v2.0.12 using CCM in Ubuntu, i get the below error : > > [artifact:dependencies] -- > [artifact:dependencies] 1 required artifact is missing. > [artifact:dependencies] > [artifact:dependencies] for artifact: > [artifact:dependencies] > org.apache.cassandra:cassandra-coverage-deps:jar:2.0.12-SNAPSHOT > [artifact:dependencies] > [artifact:dependencies] from the specified remote repositories: > [artifact:dependencies] central (http://repo1.maven.org/maven2) > === > I used the below command: > ccm create demo_1node -v 2.0.12 -n 1 -s -d > Could you add the jar to maven please? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (CASSANDRA-6541) New versions of Hotspot create new Class objects on every JMX connection causing the heap to fill up with them if CMSClassUnloadingEnabled isn't set.
[
https://issues.apache.org/jira/browse/CASSANDRA-6541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14341346#comment-14341346
]
Rekha Joshi edited comment on CASSANDRA-6541 at 2/28/15 11:08 PM:
--
On DSE 4.5.1, Java 1.7, linux ami on aws encountered long running GC before
dse was shutdown.This was a ring of 6 nodes, and no unexpected processes /load
were run.The error log attached.
I suspect to have hit that hotspot/java GC issue.But in this case the class
unloading is enabled.
JVM_OPTS="$JVM_OPTS -XX:+CMSClassUnloadingEnabled”
XX:CMSInitiatingOccupancyFraction=75
Now configured to have MAX_HEAP_SIZE=“8G”, HEAP_NEWSIZE=“2G” and GC logging
enabled and is stable.However if there is a heap leak, it needs a better fix.
Suggestion: If the CPU utilization is going up, or heap size leak/long running
GC is sensed, the nodetool -h localhost flush can auto kick in?
Thanks
Rekha
was (Author: rekhajoshm):
On DSE 4.5.1, Java 1.7, linux ami on aws encountered long running GC before
dse was shutdown.This was a ring of 6 nodes, and no unexpected processes /load
were run.The error log attached.
I suspect to have hit that hotspot/java GC issue.But in this case the class
unloading is enabled.
JVM_OPTS="$JVM_OPTS -XX:+CMSClassUnloadingEnabled”
XX:CMSInitiatingOccupancyFraction=75
Now configured to have MAX_HEAP_SIZE=“8G”, HEAP_NEWSIZE=“2G” and GC logging
enabled and is stable.However if there is a heap leak, it needs a better fix.
Thanks
Rekha
> New versions of Hotspot create new Class objects on every JMX connection
> causing the heap to fill up with them if CMSClassUnloadingEnabled isn't set.
> -
>
> Key: CASSANDRA-6541
> URL: https://issues.apache.org/jira/browse/CASSANDRA-6541
> Project: Cassandra
> Issue Type: Bug
> Components: Config
>Reporter: jonathan lacefield
>Assignee: Brandon Williams
>Priority: Minor
> Fix For: 1.2.16, 2.0.6, 2.1 beta2
>
> Attachments: dse_systemlog
>
>
> Newer versions of Oracle's Hotspot JVM , post 6u43 (maybe earlier) and 7u25
> (maybe earlier), are experiencing issues with GC and JMX where heap slowly
> fills up overtime until OOM or a full GC event occurs, specifically when CMS
> is leveraged. Adding:
> {noformat}
> JVM_OPTS="$JVM_OPTS -XX:+CMSClassUnloadingEnabled"
> {noformat}
> The the options in cassandra-env.sh alleviates the problem.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (CASSANDRA-6225) GCInspector should not wait after ConcurrentMarkSweep GC to flush memtables and reduce cache size
[
https://issues.apache.org/jira/browse/CASSANDRA-6225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14341350#comment-14341350
]
Rekha Joshi edited comment on CASSANDRA-6225 at 2/28/15 11:08 PM:
--
Hi,
On DSE 4.5.1, Java 1.7 encountered long running GC before dse was shutdown.
This was a ring of 6 nodes, and no unexpected load/processes were run.The error
log attached.
What are the flush/cache params in 4.5.1?I can check if they are configured.
My first suspect was that it has hit hotspot/java GC issue.But in this case the
class unloading is enabled.
JVM_OPTS="$JVM_OPTS -XX:+CMSClassUnloadingEnabled”
XX:CMSInitiatingOccupancyFraction=75
Now configured to have MAX_HEAP_SIZE=“8G”, HEAP_NEWSIZE=“2G” and GC logging
enabled and is stable.However if there is a heap leak, it needs a better fix.
Suggestion: If the CPU utilization is going up, or heap size leak/long running
GC is sensed, the nodetool -h localhost flush can auto kick in?
Thanks
Rekha
was (Author: rekhajoshm):
Hi,
On DSE 4.5.1, Java 1.7 encountered long running GC before dse was shutdown.
This was a ring of 6 nodes, and no unexpected load/processes were run.The error
log attached.
What are the flush/cache params in 4.5.1?I can check if they are configured.
My first suspect was that it has hit hotspot/java GC issue.But in this case the
class unloading is enabled.
JVM_OPTS="$JVM_OPTS -XX:+CMSClassUnloadingEnabled”
XX:CMSInitiatingOccupancyFraction=75
Now configured to have MAX_HEAP_SIZE=“8G”, HEAP_NEWSIZE=“2G” and GC logging
enabled and is stable.However if there is a heap leak, it needs a better fix.
Thanks
Rekha
> GCInspector should not wait after ConcurrentMarkSweep GC to flush memtables
> and reduce cache size
> -
>
> Key: CASSANDRA-6225
> URL: https://issues.apache.org/jira/browse/CASSANDRA-6225
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: Cassandra 1.2.9, SunOS, Java 7
>Reporter: Billow Gao
> Attachments: dse_systemlog
>
>
> In GCInspector.logGCResults, cassandra won't flush memtables and reduce Cache
> Sizes until there is a ConcurrentMarkSweep GC. It caused a long pause on the
> service. And other nodes could mark it as DEAD.
> In our stress test, we were using 64 concurrent threads to write data to
> cassandra. The heap usage grew up quickly and reach to maximum.
> We saw several ConcurrentMarkSweep GCs which only freed very few rams until a
> memtable flush was called. The other nodes marked the node as DOWN when GC
> took more than 20 seconds.
> {code}
> INFO [ScheduledTasks:1] 2013-10-18 15:42:36,176 GCInspector.java (line 119)
> GC for ConcurrentMarkSweep: 27481 ms for 1 collections, 5229917848 used; max
> is 6358564864
> INFO [ScheduledTasks:1] 2013-10-18 15:43:14,013 GCInspector.java (line 119)
> GC for ConcurrentMarkSweep: 27729 ms for 1 collections, 5381504752 used; max
> is 6358564864
> INFO [ScheduledTasks:1] 2013-10-18 15:43:50,565 GCInspector.java (line 119)
> GC for ConcurrentMarkSweep: 29867 ms for 1 collections, 5479631256 used; max
> is 6358564864
> INFO [ScheduledTasks:1] 2013-10-18 15:44:23,457 GCInspector.java (line 119)
> GC for ConcurrentMarkSweep: 28166 ms for 1 collections, 5545752344 used; max
> is 6358564864
> INFO [ScheduledTasks:1] 2013-10-18 15:44:58,290 GCInspector.java (line 119)
> GC for ConcurrentMarkSweep: 29377 ms for 2 collections, 5343255456 used; max
> is 6358564864
> {code}
> {code}
> INFO [GossipTasks:1] 2013-10-18 15:42:29,004 Gossiper.java (line 803)
> InetAddress /1.2.3.4 is now DOWN
> INFO [GossipTasks:1] 2013-10-18 15:43:06,901 Gossiper.java (line 803)
> InetAddress /1.2.3.4 is now DOWN
> INFO [GossipTasks:1] 2013-10-18 15:44:18,254 Gossiper.java (line 803)
> InetAddress /1.2.3.4 is now DOWN
> INFO [GossipTasks:1] 2013-10-18 15:44:48,507 Gossiper.java (line 803)
> InetAddress /1.2.3.4 is now DOWN
> INFO [GossipTasks:1] 2013-10-18 15:45:32,375 Gossiper.java (line 803)
> InetAddress /1.2.3.4 is now DOWN
> {code}
> We found two solutions to fix the long pause which result in a DOWN status.
> 1. We reduced the maximum ram to 3G. The behavior is the same, but gc was
> faster(under 20 seconds), so no nodes were marked as DOWN
> 2. Running a cronjob on the cassandra server which period call nodetool -h
> localhost flush.
> Flush after a full gc just make thing worse and waste time spent on GC. In a
> heavily load system, you would have several full GCs before a flush can
> finish. (a flush may take more than 30 seconds)
> Ideally, GCInspector should has a better logic on when to flush memtable.
> 1. Flush memtable/reduce cache size when it reached the threshold(smaller
> than full gc threshold).
> 2. prevent frequently flush by reme
[jira] [Commented] (CASSANDRA-8716) "java.util.concurrent.ExecutionException: java.lang.AssertionError: Memory was freed" when running cleanup
[
https://issues.apache.org/jira/browse/CASSANDRA-8716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14341848#comment-14341848
]
Gianluca Borello commented on CASSANDRA-8716:
-
+1 for a workaround, I could really use a cleanup without downgrading
production to 2.0.11
> "java.util.concurrent.ExecutionException: java.lang.AssertionError: Memory
> was freed" when running cleanup
> --
>
> Key: CASSANDRA-8716
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8716
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: Centos 6.6, Cassandra 2.0.12, Oracle JDK 1.7.0_67
>Reporter: Imri Zvik
>Assignee: Robert Stupp
>Priority: Minor
> Fix For: 2.0.13
>
> Attachments: 8716.txt, system.log.gz
>
>
> {code}Error occurred during cleanup
> java.util.concurrent.ExecutionException: java.lang.AssertionError: Memory was
> freed
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> at java.util.concurrent.FutureTask.get(FutureTask.java:188)
> at
> org.apache.cassandra.db.compaction.CompactionManager.performAllSSTableOperation(CompactionManager.java:234)
> at
> org.apache.cassandra.db.compaction.CompactionManager.performCleanup(CompactionManager.java:272)
> at
> org.apache.cassandra.db.ColumnFamilyStore.forceCleanup(ColumnFamilyStore.java:1115)
> at
> org.apache.cassandra.service.StorageService.forceKeyspaceCleanup(StorageService.java:2177)
> at sun.reflect.GeneratedMethodAccessor29.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:75)
> at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:279)
> at
> com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:112)
> at
> com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:46)
> at
> com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:237)
> at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)
> at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:252)
> at
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:819)
> at
> com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:801)
> at
> javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1487)
> at
> javax.management.remote.rmi.RMIConnectionImpl.access$300(RMIConnectionImpl.java:97)
> at
> javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1328)
> at
> javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1420)
> at
> javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:848)
> at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322)
> at sun.rmi.transport.Transport$1.run(Transport.java:177)
> at sun.rmi.transport.Transport$1.run(Transport.java:174)
> at java.security.AccessController.doPrivileged(Native Method)
> at sun.rmi.transport.Transport.serviceCall(Transport.java:173)
> at
> sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:556)
> at
> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:811)
> at
> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:670)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.AssertionError: Memory was freed
> at org.apache.cassandra.io.util.Memory.checkPosition(Memory.java:259)
> at org.apache.cassandra.io.util.Memory.getInt(Memory.java:211)
> at
> org.apache.cassandra.io.sstable.IndexSummary.getIndex(IndexSummary.java:79)
[jira] [Commented] (CASSANDRA-8860) Too many java.util.HashMap$Entry objects in heap
[ https://issues.apache.org/jira/browse/CASSANDRA-8860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14341913#comment-14341913 ] Jonathan Ellis commented on CASSANDRA-8860: --- That's a lot of complexity for something whose benefit I'm unclear on (I'm talking about the "ignore cold sstables feature"). To the best of my knowledge, CASSANDRA-8635 has never been observed as a problem in the wild. I'd be in favor of reverting it and documenting "don't enable cold sstable elision if you do a lot of overwrites." > Too many java.util.HashMap$Entry objects in heap > > > Key: CASSANDRA-8860 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8860 > Project: Cassandra > Issue Type: Bug > Environment: Cassandra 2.1.3, jdk 1.7u51 >Reporter: Phil Yang >Assignee: Phil Yang > Fix For: 2.1.4 > > Attachments: 8860-v2.txt, 8860.txt, cassandra-env.sh, cassandra.yaml, > jmap.txt, jstack.txt, jstat-afterv1.txt, jstat-afterv2.txt, jstat-before.txt > > > While I upgrading my cluster to 2.1.3, I find some nodes (not all) may have > GC issue after the node restarting successfully. Old gen grows very fast and > most of the space can not be recycled after setting its status to normal > immediately. The qps of both reading and writing are very low and there is no > heavy compaction. > Jmap result seems strange that there are too many java.util.HashMap$Entry > objects in heap, where in my experience the "[B" is usually the No1. > If I downgrade it to 2.1.1, this issue will not appear. > I uploaded conf files and jstack/jmap outputs. I'll upload heap dump if > someone need it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (CASSANDRA-8860) Too many java.util.HashMap$Entry objects in heap
[ https://issues.apache.org/jira/browse/CASSANDRA-8860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14341913#comment-14341913 ] Jonathan Ellis edited comment on CASSANDRA-8860 at 3/1/15 3:23 AM: --- That's a lot of complexity for something whose benefit I'm unclear on (I'm talking about the "ignore cold sstables" feature). To the best of my knowledge, CASSANDRA-8635 has never been observed as a problem in the wild. I'd be in favor of reverting it and documenting "don't enable cold sstable elision if you do a lot of overwrites." was (Author: jbellis): That's a lot of complexity for something whose benefit I'm unclear on (I'm talking about the "ignore cold sstables feature"). To the best of my knowledge, CASSANDRA-8635 has never been observed as a problem in the wild. I'd be in favor of reverting it and documenting "don't enable cold sstable elision if you do a lot of overwrites." > Too many java.util.HashMap$Entry objects in heap > > > Key: CASSANDRA-8860 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8860 > Project: Cassandra > Issue Type: Bug > Environment: Cassandra 2.1.3, jdk 1.7u51 >Reporter: Phil Yang >Assignee: Phil Yang > Fix For: 2.1.4 > > Attachments: 8860-v2.txt, 8860.txt, cassandra-env.sh, cassandra.yaml, > jmap.txt, jstack.txt, jstat-afterv1.txt, jstat-afterv2.txt, jstat-before.txt > > > While I upgrading my cluster to 2.1.3, I find some nodes (not all) may have > GC issue after the node restarting successfully. Old gen grows very fast and > most of the space can not be recycled after setting its status to normal > immediately. The qps of both reading and writing are very low and there is no > heavy compaction. > Jmap result seems strange that there are too many java.util.HashMap$Entry > objects in heap, where in my experience the "[B" is usually the No1. > If I downgrade it to 2.1.1, this issue will not appear. > I uploaded conf files and jstack/jmap outputs. I'll upload heap dump if > someone need it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8860) Too many java.util.HashMap$Entry objects in heap
[
https://issues.apache.org/jira/browse/CASSANDRA-8860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14342007#comment-14342007
]
Phil Yang commented on CASSANDRA-8860:
--
{quote}
This is unfortunately not true. a may overlap b and c, but b may only overlap
a, and not c, for instance.
{quote}
Your case is just like this, right?
{noformat}
b -
a-
c --
{noformat}
However, in the code of createOverlapChain, all the overlapping SSTables with b
will be pushed into the queue(for this case is a). When a is popped, all the
overlapping SSTables with a(for this case is c) will also be pushed, until
there is no more SSTables that overlap any of SSTables in overlapChain(\{a,b,c}
for this case). Just like the comment on this function says: "if we have 3
sstables, a, b, c where a overlaps with b, but not c and b overlaps with c, all
sstables would be returned."
So in graph theory, this function input "Map>
m" as an undirected graph and "SSTableReader s" as a node of this graph, will
return the connected component which contains this node. If this connected
component is \{a,b,c}, input a, b or c will return the same connected component.
> Too many java.util.HashMap$Entry objects in heap
>
>
> Key: CASSANDRA-8860
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8860
> Project: Cassandra
> Issue Type: Bug
> Environment: Cassandra 2.1.3, jdk 1.7u51
>Reporter: Phil Yang
>Assignee: Phil Yang
> Fix For: 2.1.4
>
> Attachments: 8860-v2.txt, 8860.txt, cassandra-env.sh, cassandra.yaml,
> jmap.txt, jstack.txt, jstat-afterv1.txt, jstat-afterv2.txt, jstat-before.txt
>
>
> While I upgrading my cluster to 2.1.3, I find some nodes (not all) may have
> GC issue after the node restarting successfully. Old gen grows very fast and
> most of the space can not be recycled after setting its status to normal
> immediately. The qps of both reading and writing are very low and there is no
> heavy compaction.
> Jmap result seems strange that there are too many java.util.HashMap$Entry
> objects in heap, where in my experience the "[B" is usually the No1.
> If I downgrade it to 2.1.1, this issue will not appear.
> I uploaded conf files and jstack/jmap outputs. I'll upload heap dump if
> someone need it.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8860) Too many java.util.HashMap$Entry objects in heap
[ https://issues.apache.org/jira/browse/CASSANDRA-8860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14342032#comment-14342032 ] Phil Yang commented on CASSANDRA-8860: -- Furthermore, if my algorithm is correct, my patch-v2 still need a 2-d loop to get the undirected graph. The space complexity is still O(N^2) if all the SSTables overlap each other. So I think there is a way to optimize it: Use a 2-d bitmap matrix with the boolean which means whether coldSSTables[i] overlaps coldSSTables[j] to replace the 2-d collection of SSTableReader. Each HashMap$Entry need 32*8=256 bits but each bitmap boolean needs only 1 bit. This method may need more memory in some case, but need much less in the worst case. If my algorithm is not wrong, we can discuss if this optimization is needed or not :) > Too many java.util.HashMap$Entry objects in heap > > > Key: CASSANDRA-8860 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8860 > Project: Cassandra > Issue Type: Bug > Environment: Cassandra 2.1.3, jdk 1.7u51 >Reporter: Phil Yang >Assignee: Phil Yang > Fix For: 2.1.4 > > Attachments: 8860-v2.txt, 8860.txt, cassandra-env.sh, cassandra.yaml, > jmap.txt, jstack.txt, jstat-afterv1.txt, jstat-afterv2.txt, jstat-before.txt > > > While I upgrading my cluster to 2.1.3, I find some nodes (not all) may have > GC issue after the node restarting successfully. Old gen grows very fast and > most of the space can not be recycled after setting its status to normal > immediately. The qps of both reading and writing are very low and there is no > heavy compaction. > Jmap result seems strange that there are too many java.util.HashMap$Entry > objects in heap, where in my experience the "[B" is usually the No1. > If I downgrade it to 2.1.1, this issue will not appear. > I uploaded conf files and jstack/jmap outputs. I'll upload heap dump if > someone need it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (CASSANDRA-8860) Too many java.util.HashMap$Entry objects in heap
[ https://issues.apache.org/jira/browse/CASSANDRA-8860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14342032#comment-14342032 ] Phil Yang edited comment on CASSANDRA-8860 at 3/1/15 7:42 AM: -- Furthermore, if my algorithm is correct, my patch-v2 still need a 2-d loop to get the undirected graph. The space complexity is still O(N^2) if all the SSTables overlap each other and the graph is a complete graph. So I think there is a way to optimize it: Use a 2-d bitmap matrix with the boolean which means whether coldSSTables[i] overlaps coldSSTables[j] to replace the 2-d collection of SSTableReader. Each HashMap$Entry need 32*8=256 bits but each bitmap boolean needs only 1 bit. This method may need more memory in some case, but need much less in the worst case. If my algorithm is not wrong, we can discuss if this optimization is needed or not :) was (Author: yangzhe1991): Furthermore, if my algorithm is correct, my patch-v2 still need a 2-d loop to get the undirected graph. The space complexity is still O(N^2) if all the SSTables overlap each other. So I think there is a way to optimize it: Use a 2-d bitmap matrix with the boolean which means whether coldSSTables[i] overlaps coldSSTables[j] to replace the 2-d collection of SSTableReader. Each HashMap$Entry need 32*8=256 bits but each bitmap boolean needs only 1 bit. This method may need more memory in some case, but need much less in the worst case. If my algorithm is not wrong, we can discuss if this optimization is needed or not :) > Too many java.util.HashMap$Entry objects in heap > > > Key: CASSANDRA-8860 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8860 > Project: Cassandra > Issue Type: Bug > Environment: Cassandra 2.1.3, jdk 1.7u51 >Reporter: Phil Yang >Assignee: Phil Yang > Fix For: 2.1.4 > > Attachments: 8860-v2.txt, 8860.txt, cassandra-env.sh, cassandra.yaml, > jmap.txt, jstack.txt, jstat-afterv1.txt, jstat-afterv2.txt, jstat-before.txt > > > While I upgrading my cluster to 2.1.3, I find some nodes (not all) may have > GC issue after the node restarting successfully. Old gen grows very fast and > most of the space can not be recycled after setting its status to normal > immediately. The qps of both reading and writing are very low and there is no > heavy compaction. > Jmap result seems strange that there are too many java.util.HashMap$Entry > objects in heap, where in my experience the "[B" is usually the No1. > If I downgrade it to 2.1.1, this issue will not appear. > I uploaded conf files and jstack/jmap outputs. I'll upload heap dump if > someone need it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
