[jira] [Issue Comment Deleted] (CASSANDRA-12888) Incremental repairs broken for MVs and CDC
[ https://issues.apache.org/jira/browse/CASSANDRA-12888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Jirsa updated CASSANDRA-12888: --- Comment: was deleted (was: {quote} What effect does the repairedAt flag have for future repairs except that a non-zero value means, that a table has been repaired at some time? {quote} It's used (heavily) for incremental repairs. CASSANDRA-9143 may be a starting point reading there (potentially).) > Incremental repairs broken for MVs and CDC > -- > > Key: CASSANDRA-12888 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12888 > Project: Cassandra > Issue Type: Bug > Components: Streaming and Messaging >Reporter: Stefan Podkowinski >Assignee: Benjamin Roth >Priority: Critical > Fix For: 3.0.x, 3.11.x > > > SSTables streamed during the repair process will first be written locally and > afterwards either simply added to the pool of existing sstables or, in case > of existing MVs or active CDC, replayed on mutation basis: > As described in {{StreamReceiveTask.OnCompletionRunnable}}: > {quote} > We have a special path for views and for CDC. > For views, since the view requires cleaning up any pre-existing state, we > must put all partitions through the same write path as normal mutations. This > also ensures any 2is are also updated. > For CDC-enabled tables, we want to ensure that the mutations are run through > the CommitLog so they can be archived by the CDC process on discard. > {quote} > Using the regular write path turns out to be an issue for incremental > repairs, as we loose the {{repaired_at}} state in the process. Eventually the > streamed rows will end up in the unrepaired set, in contrast to the rows on > the sender site moved to the repaired set. The next repair run will stream > the same data back again, causing rows to bounce on and on between nodes on > each repair. > See linked dtest on steps to reproduce. An example for reproducing this > manually using ccm can be found > [here|https://gist.github.com/spodkowinski/2d8e0408516609c7ae701f2bf1e515e8] -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (CASSANDRA-12888) Incremental repairs broken for MVs and CDC
[ https://issues.apache.org/jira/browse/CASSANDRA-12888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15895457#comment-15895457 ] Jeff Jirsa commented on CASSANDRA-12888: {quote} What effect does the repairedAt flag have for future repairs except that a non-zero value means, that a table has been repaired at some time? {quote} It's used (heavily) for incremental repairs. CASSANDRA-9143 may be a starting point reading there (potentially). > Incremental repairs broken for MVs and CDC > -- > > Key: CASSANDRA-12888 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12888 > Project: Cassandra > Issue Type: Bug > Components: Streaming and Messaging >Reporter: Stefan Podkowinski >Assignee: Benjamin Roth >Priority: Critical > Fix For: 3.0.x, 3.11.x > > > SSTables streamed during the repair process will first be written locally and > afterwards either simply added to the pool of existing sstables or, in case > of existing MVs or active CDC, replayed on mutation basis: > As described in {{StreamReceiveTask.OnCompletionRunnable}}: > {quote} > We have a special path for views and for CDC. > For views, since the view requires cleaning up any pre-existing state, we > must put all partitions through the same write path as normal mutations. This > also ensures any 2is are also updated. > For CDC-enabled tables, we want to ensure that the mutations are run through > the CommitLog so they can be archived by the CDC process on discard. > {quote} > Using the regular write path turns out to be an issue for incremental > repairs, as we loose the {{repaired_at}} state in the process. Eventually the > streamed rows will end up in the unrepaired set, in contrast to the rows on > the sender site moved to the repaired set. The next repair run will stream > the same data back again, causing rows to bounce on and on between nodes on > each repair. > See linked dtest on steps to reproduce. An example for reproducing this > manually using ccm can be found > [here|https://gist.github.com/spodkowinski/2d8e0408516609c7ae701f2bf1e515e8] -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (CASSANDRA-13294) Possible data loss on upgrade 2.1 - 3.0
[ https://issues.apache.org/jira/browse/CASSANDRA-13294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15895295#comment-15895295 ] Wei Deng commented on CASSANDRA-13294: -- If this problem is caused by the file name change, would we need to port to 2.2 as well, because the file name change was introduced in 2.2? > Possible data loss on upgrade 2.1 - 3.0 > --- > > Key: CASSANDRA-13294 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13294 > Project: Cassandra > Issue Type: Bug >Reporter: Marcus Eriksson >Assignee: Marcus Eriksson >Priority: Blocker > Fix For: 3.0.x > > > After finishing a compaction we delete the compacted away files. This is done > [here|https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/db/lifecycle/LogFile.java#L328-L337] > which uses > [this|https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/db/lifecycle/LogRecord.java#L265-L271] > to get the files - we get all files starting with {{absoluteFilePath}}. > Absolute file path is generated > [here|https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/io/sstable/Descriptor.java#L142-L153]. > For 3.0 version files the filename looks like this: > {{/blabla/keyspace1/standard1-bdb031c0ff7b11e6940fdd0479dd8912/mc-1332-big}} > but for 2.1 version files, they look like this: > {{/blabla/keyspace1/standard1-bdb031c0ff7b11e6940fdd0479dd8912/keyspace1-standard1-ka-2}}. > The problem is then that if we were to finish a compaction including the > legacy file, we would actually delete all legacy files having a generation > starting with '2' -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Comment Edited] (CASSANDRA-12588) Cannot find column durable_wrıtes
[ https://issues.apache.org/jira/browse/CASSANDRA-12588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15895222#comment-15895222 ] Jeff Jirsa edited comment on CASSANDRA-12588 at 3/3/17 11:17 PM: - When creating the system keyspace, it checks {{KeyspaceParams.Option.DURABLE_WRITES.toString()}} , - there's probably a wide class of bugs in cassandra where we assume everything is {{Locale.ENGLISH}}, but we break in awful ways in Turkish (and likely other locales). was (Author: jjirsa): When creating the system keyspace, it checks {{KeyspaceParams.Option.DURABLE_WRITES.toString()}}, but without a locale - there's probably a wide class of bugs in cassandra where we assume everything is {{Locale.ENGLISH}}, but we break in awful ways in Turkish (and likely other locales). > Cannot find column durable_wrıtes > - > > Key: CASSANDRA-12588 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12588 > Project: Cassandra > Issue Type: Bug >Reporter: LLc. > > help please > run : > cassandra -f > ERROR 17:00:16 Exception encountered during startup > java.lang.AssertionError: Cannot find column durable_wrıtes -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (CASSANDRA-12588) Cannot find column durable_wrıtes
[ https://issues.apache.org/jira/browse/CASSANDRA-12588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15895222#comment-15895222 ] Jeff Jirsa commented on CASSANDRA-12588: When creating the system keyspace, it checks {{KeyspaceParams.Option.DURABLE_WRITES.toString()}}, but without a locale - there's probably a wide class of bugs in cassandra where we assume everything is {{Locale.ENGLISH}}, but we break in awful ways in Turkish (and likely other locales). > Cannot find column durable_wrıtes > - > > Key: CASSANDRA-12588 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12588 > Project: Cassandra > Issue Type: Bug >Reporter: LLc. > > help please > run : > cassandra -f > ERROR 17:00:16 Exception encountered during startup > java.lang.AssertionError: Cannot find column durable_wrıtes -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (CASSANDRA-13297) Cleanup naming around native transport version parsing
Michael Kjellman created CASSANDRA-13297: Summary: Cleanup naming around native transport version parsing Key: CASSANDRA-13297 URL: https://issues.apache.org/jira/browse/CASSANDRA-13297 Project: Cassandra Issue Type: Bug Reporter: Michael Kjellman While [~jasobrown] and myself were chasing down how features and versions are parsed in the binary protocol, it took us a while to realize that the following check in StartupMessage that creates a {{CassandraVersion}} object is actually parsing the *CQL* Version (but again.. with a class named {{CassandraVersion}}...) {code} if (new CassandraVersion(cqlVersion).compareTo(new CassandraVersion("2.99.0")) < 0) {code} It then turns out there is already a {{CqlVersion}} class that has some of what we want but only appears to be being used by Stress. This isn't very clear so renaming the classes to make it obvious that we're parsing a CQL Language version vs. the Cassandra Server version vs. the Binary Native Protocol version will make this situation slightly better for people trying to contribute to the codebase. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (CASSANDRA-12888) Incremental repairs broken for MVs and CDC
[ https://issues.apache.org/jira/browse/CASSANDRA-12888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15895140#comment-15895140 ] Benjamin Roth commented on CASSANDRA-12888: --- Maybe my former question perished: What effect does the repairedAt flag have for future repairs except that a non-zero value means, that a table has been repaired at some time? I am happy about any code references. > Incremental repairs broken for MVs and CDC > -- > > Key: CASSANDRA-12888 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12888 > Project: Cassandra > Issue Type: Bug > Components: Streaming and Messaging >Reporter: Stefan Podkowinski >Assignee: Benjamin Roth >Priority: Critical > Fix For: 3.0.x, 3.11.x > > > SSTables streamed during the repair process will first be written locally and > afterwards either simply added to the pool of existing sstables or, in case > of existing MVs or active CDC, replayed on mutation basis: > As described in {{StreamReceiveTask.OnCompletionRunnable}}: > {quote} > We have a special path for views and for CDC. > For views, since the view requires cleaning up any pre-existing state, we > must put all partitions through the same write path as normal mutations. This > also ensures any 2is are also updated. > For CDC-enabled tables, we want to ensure that the mutations are run through > the CommitLog so they can be archived by the CDC process on discard. > {quote} > Using the regular write path turns out to be an issue for incremental > repairs, as we loose the {{repaired_at}} state in the process. Eventually the > streamed rows will end up in the unrepaired set, in contrast to the rows on > the sender site moved to the repaired set. The next repair run will stream > the same data back again, causing rows to bounce on and on between nodes on > each repair. > See linked dtest on steps to reproduce. An example for reproducing this > manually using ccm can be found > [here|https://gist.github.com/spodkowinski/2d8e0408516609c7ae701f2bf1e515e8] -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (CASSANDRA-13090) Coalescing strategy sleeps too much
[ https://issues.apache.org/jira/browse/CASSANDRA-13090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Shuler updated CASSANDRA-13090: --- Fix Version/s: (was: 3.0.11) 3.0.12 > Coalescing strategy sleeps too much > --- > > Key: CASSANDRA-13090 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13090 > Project: Cassandra > Issue Type: Bug > Components: Streaming and Messaging >Reporter: Corentin Chary >Assignee: Corentin Chary > Fix For: 2.2.10, 3.0.12, 3.11.0, 4.0 > > Attachments: 0001-Fix-wait-time-coalescing-CASSANDRA-13090-2.patch, > 0001-Fix-wait-time-coalescing-CASSANDRA-13090.patch > > > With the current code maybeSleep is called even if we managed to take > maxItems out of the backlog. In this case we should really avoid sleeping > because it means that backlog is building up. > I'll send a patch shortly. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
cassandra-builds git commit: Add cassandra-cqlsh-tests build script
Repository: cassandra-builds Updated Branches: refs/heads/master 0c4fc4713 -> 08f76054c Add cassandra-cqlsh-tests build script Project: http://git-wip-us.apache.org/repos/asf/cassandra-builds/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra-builds/commit/08f76054 Tree: http://git-wip-us.apache.org/repos/asf/cassandra-builds/tree/08f76054 Diff: http://git-wip-us.apache.org/repos/asf/cassandra-builds/diff/08f76054 Branch: refs/heads/master Commit: 08f76054c5adc4d5bae010e041f5b204b0880fba Parents: 0c4fc47 Author: Michael Shuler Authored: Fri Mar 3 15:59:46 2017 -0600 Committer: Michael Shuler Committed: Fri Mar 3 15:59:46 2017 -0600 -- build-scripts/cassandra-cqlsh-tests.sh | 99 + 1 file changed, 99 insertions(+) -- http://git-wip-us.apache.org/repos/asf/cassandra-builds/blob/08f76054/build-scripts/cassandra-cqlsh-tests.sh -- diff --git a/build-scripts/cassandra-cqlsh-tests.sh b/build-scripts/cassandra-cqlsh-tests.sh new file mode 100755 index 000..8cd867d --- /dev/null +++ b/build-scripts/cassandra-cqlsh-tests.sh @@ -0,0 +1,99 @@ +#!/bin/bash -x + + +# +# Prep +# + + +export PYTHONIOENCODING="utf-8" +export PYTHONUNBUFFERED=true +export CCM_MAX_HEAP_SIZE="2048M" +export CCM_HEAP_NEWSIZE="200M" +export NUM_TOKENS="32" +export CASSANDRA_DIR=${WORKSPACE} + +# Loop to prevent failure due to maven-ant-tasks not downloading a jar.. +for x in $(seq 1 3); do +ant clean jar +RETURN="$?" +if [ "${RETURN}" -eq "0" ]; then +break +fi +done +# Exit, if we didn't build successfully +if [ "${RETURN}" -ne "0" ]; then +echo "Build failed with exit code: ${RETURN}" +exit ${RETURN} +fi + + +# +# Main +# + + +if [ "$cython" = "yes" ]; then +virtualenv --python=python2 --no-site-packages venv +source venv/bin/activate +pip install "Cython>=0.20,<0.25" +pip freeze +cd pylib/; python setup.py build_ext --inplace +cd ${WORKSPACE} +fi + +ccm create test -n 1 +ccm updateconf "enable_user_defined_functions: true" + +version_from_build=$(ccm node1 versionfrombuild) +export pre_or_post_cdc=$(python -c """from distutils.version import LooseVersion +print \"postcdc\" if LooseVersion(\"${version_from_build}\") >= \"3.8\" else \"precdc\" +""") +case "${pre_or_post_cdc}" in +postcdc) +ccm updateconf "cdc_enabled: true" +;; +precdc) +: +;; +*) +echo "${pre_or_post_cdc}" is an invalid value. +exit 1 +;; +esac + +ccm start --wait-for-binary-proto + +cd pylib/cqlshlib/ +cat << 'EOF' > setup.cfg +[nosetests] +verbosity=3 +detailed-errors=1 +with-xunit=1 +EOF + +nosetests + +ccm remove +mv nosetests.xml ${WORKSPACE}/cqlshlib.xml +if [ "$cython" = "yes" ]; then +deactivate # venv +fi +cd ${WORKSPACE} + + +# run dtest cqlsh suite +cd cassandra-dtest/ +if [ "$cython" = "no" ]; then +export CASS_DRIVER_NO_EXTENSIONS=true +export CASS_DRIVER_NO_CYTHON=true +fi +virtualenv --python=python2 --no-site-packages venv +source venv/bin/activate +pip install -r requirements.txt +pip freeze + +nosetests --verbosity=3 --with-xunit --nocapture cqlsh_tests/ + +mv nosetests.xml ${WORKSPACE}/ +deactivate # venv
[jira] [Updated] (CASSANDRA-13090) Coalescing strategy sleeps too much
[ https://issues.apache.org/jira/browse/CASSANDRA-13090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brandon Williams updated CASSANDRA-13090: - Fix Version/s: (was: 2.2.9) 2.2.10 > Coalescing strategy sleeps too much > --- > > Key: CASSANDRA-13090 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13090 > Project: Cassandra > Issue Type: Bug > Components: Streaming and Messaging >Reporter: Corentin Chary >Assignee: Corentin Chary > Fix For: 2.2.10, 3.0.11, 3.11.0, 4.0 > > Attachments: 0001-Fix-wait-time-coalescing-CASSANDRA-13090-2.patch, > 0001-Fix-wait-time-coalescing-CASSANDRA-13090.patch > > > With the current code maybeSleep is called even if we managed to take > maxItems out of the backlog. In this case we should really avoid sleeping > because it means that backlog is building up. > I'll send a patch shortly. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (CASSANDRA-12588) Cannot find column durable_wrıtes
[ https://issues.apache.org/jira/browse/CASSANDRA-12588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15895030#comment-15895030 ] Gokhan Mansuroglu commented on CASSANDRA-12588: --- The same problem is troubling me. I came across a question on stackoverflow that emphasise that Turkish language could be the problem. In fact the "ı" character gives a clue. Here is the link : http://stackoverflow.com/questions/38113849/starting-cassandra-assertionerror > Cannot find column durable_wrıtes > - > > Key: CASSANDRA-12588 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12588 > Project: Cassandra > Issue Type: Bug >Reporter: LLc. > > help please > run : > cassandra -f > ERROR 17:00:16 Exception encountered during startup > java.lang.AssertionError: Cannot find column durable_wrıtes -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (CASSANDRA-13233) Improve testing on macOS by eliminating sigar logging
[ https://issues.apache.org/jira/browse/CASSANDRA-13233?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Brown updated CASSANDRA-13233: Resolution: Fixed Fix Version/s: 4.0 3.11.0 3.0.12 Status: Resolved (was: Patch Available) Committed as sha {{e3968cfd1e786f260029cf8cde4164719ae56c53}} [~mkjellman] feel free to change to subject of this ticket to something more apropos. It'll be the current name in the git history and CHANGES.txt, but the jira could have more more descriptive if you think it'll help. naming is hard ;) > Improve testing on macOS by eliminating sigar logging > - > > Key: CASSANDRA-13233 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13233 > Project: Cassandra > Issue Type: Bug >Reporter: Michael Kjellman >Assignee: Michael Kjellman > Fix For: 3.0.12, 3.11.0, 4.0 > > Attachments: 28827709.diff, CASSANDRA-13233-trunk-v2.diff > > > The changes introduced in CASSANDRA-7838 (Resolved; Fixed; 2.2.0 beta 1): > "Warn user when OS settings are poor / integrate sigar" are not Mac friendly. > {code} > INFO [main] 2016-10-18T11:20:10,330 SigarLibrary.java:44 - Initializing > SIGAR library > DEBUG [main] 2016-10-18T11:20:10,342 SigarLog.java:60 - no > libsigar-universal64-macosx.dylib in java.library.path > org.hyperic.sigar.SigarException: no libsigar-universal64-macosx.dylib in > java.library.path > at org.hyperic.sigar.Sigar.loadLibrary(Sigar.java:172) > ~[sigar-1.6.4.jar:?] > at org.hyperic.sigar.Sigar.(Sigar.java:100) > [sigar-1.6.4.jar:?] > at > org.apache.cassandra.utils.SigarLibrary.(SigarLibrary.java:47) [main/:?] > at > org.apache.cassandra.utils.SigarLibrary.(SigarLibrary.java:28) > [main/:?] > at org.apache.cassandra.utils.UUIDGen.hash(UUIDGen.java:363) [main/:?] > at org.apache.cassandra.utils.UUIDGen.makeNode(UUIDGen.java:342) > [main/:?] > at > org.apache.cassandra.utils.UUIDGen.makeClockSeqAndNode(UUIDGen.java:291) > [main/:?] > at org.apache.cassandra.utils.UUIDGen.(UUIDGen.java:42) > [main/:?] > at > org.apache.cassandra.config.CFMetaData$Builder.build(CFMetaData.java:1278) > [main/:?] > at > org.apache.cassandra.SchemaLoader.standardCFMD(SchemaLoader.java:369) > [classes/:?] > at > org.apache.cassandra.SchemaLoader.standardCFMD(SchemaLoader.java:356) > [classes/:?] > at > org.apache.cassandra.SchemaLoader.standardCFMD(SchemaLoader.java:351) > [classes/:?] > at > org.apache.cassandra.batchlog.BatchTest.defineSchema(BatchTest.java:59) > [classes/:?] > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > ~[?:1.8.0_66] > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > ~[?:1.8.0_66] > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > ~[?:1.8.0_66] > at java.lang.reflect.Method.invoke(Method.java:497) ~[?:1.8.0_66] > at > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44) > [junit-4.6.jar:?] > at > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15) > [junit-4.6.jar:?] > at > org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41) > [junit-4.6.jar:?] > at > org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:27) > [junit-4.6.jar:?] > at > org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:31) > [junit-4.6.jar:?] > at org.junit.runners.ParentRunner.run(ParentRunner.java:220) > [junit-4.6.jar:?] > at junit.framework.JUnit4TestAdapter.run(JUnit4TestAdapter.java:39) > [junit-4.6.jar:?] > at > org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:535) > [ant-junit.jar:?] > at > org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:1182) > [ant-junit.jar:?] > at > org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:1033) > [ant-junit.jar:?] > INFO [main] 2016-10-18T11:20:10,350 SigarLibrary.java:57 - Could not > initialize SIGAR library > org.hyperic.sigar.Sigar.getFileSystemListNative()[Lorg/hyperic/sigar/FileSystem; > {code} > There are 2 issues addressed by the attached patch: > # Create platform aware (windows, Darwin, linux) implementations of CLibrary > (for instance CLibrary today assumes all platforms have support for > posix_fadvise but this doesn't exist in the Darwin kernel). If methods are > defined with the "native" JNI keyword in java when the class is loaded it > will cause our jna check to fail incorrec
[5/6] cassandra git commit: Merge branch 'cassandra-3.0' into cassandra-3.11
Merge branch 'cassandra-3.0' into cassandra-3.11 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/92b7322b Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/92b7322b Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/92b7322b Branch: refs/heads/cassandra-3.11 Commit: 92b7322bbf0b53203adcfb6c4479022f975d9785 Parents: a830cf5 e3968cf Author: Jason Brown Authored: Fri Mar 3 11:17:17 2017 -0800 Committer: Jason Brown Committed: Fri Mar 3 11:19:30 2017 -0800 -- CHANGES.txt | 1 + .../org/apache/cassandra/utils/CLibrary.java| 135 +++ .../apache/cassandra/utils/CLibraryDarwin.java | 131 ++ .../apache/cassandra/utils/CLibraryLinux.java | 131 ++ .../apache/cassandra/utils/CLibraryWindows.java | 103 ++ .../apache/cassandra/utils/CLibraryWrapper.java | 40 ++ .../org/apache/cassandra/utils/HeapUtils.java | 6 +- .../org/apache/cassandra/utils/UUIDGen.java | 2 +- .../apache/cassandra/utils/CLibraryTest.java| 8 ++ 9 files changed, 495 insertions(+), 62 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/92b7322b/CHANGES.txt -- diff --cc CHANGES.txt index 5de9ece,076b337..b91edb4 --- a/CHANGES.txt +++ b/CHANGES.txt @@@ -1,17 -1,7 +1,18 @@@ -3.0.12 +3.11.0 + * Fix equality comparisons of columns using the duration type (CASSANDRA-13174) + * Obfuscate password in stress-graphs (CASSANDRA-12233) + * Move to FastThreadLocalThread and FastThreadLocal (CASSANDRA-13034) + * nodetool stopdaemon errors out (CASSANDRA-13030) + * Tables in system_distributed should not use gcgs of 0 (CASSANDRA-12954) + * Fix primary index calculation for SASI (CASSANDRA-12910) + * More fixes to the TokenAllocator (CASSANDRA-12990) + * NoReplicationTokenAllocator should work with zero replication factor (CASSANDRA-12983) + * Address message coalescing regression (CASSANDRA-12676) +Merged from 3.0: + * Improve testing on macOS by eliminating sigar logging (CASSANDRA-13233) * Cqlsh copy-from should error out when csv contains invalid data for collections (CASSANDRA-13071) - * Update c.yaml doc for offheap memtables (CASSANDRA-13179) + * Fix "multiple versions of ant detected..." when running ant test (CASSANDRA-13232) + * Coalescing strategy sleeps too much (CASSANDRA-13090) * Faster StreamingHistogram (CASSANDRA-13038) * Legacy deserializer can create unexpected boundary range tombstones (CASSANDRA-13237) * Remove unnecessary assertion from AntiCompactionTest (CASSANDRA-13070) http://git-wip-us.apache.org/repos/asf/cassandra/blob/92b7322b/src/java/org/apache/cassandra/utils/CLibrary.java -- diff --cc src/java/org/apache/cassandra/utils/CLibrary.java index 937d662,71f4148..ae3ce63 --- a/src/java/org/apache/cassandra/utils/CLibrary.java +++ b/src/java/org/apache/cassandra/utils/CLibrary.java @@@ -29,10 -29,12 +29,13 @@@ import org.slf4j.Logger import org.slf4j.LoggerFactory; import com.sun.jna.LastErrorException; - import com.sun.jna.Native; - import com.sun.jna.Pointer; +import sun.nio.ch.FileChannelImpl; + import static org.apache.cassandra.utils.CLibrary.OSType.LINUX; + import static org.apache.cassandra.utils.CLibrary.OSType.MAC; + import static org.apache.cassandra.utils.CLibrary.OSType.WINDOWS; + import static org.apache.cassandra.utils.CLibrary.OSType.AIX; + public final class CLibrary { private static final Logger logger = LoggerFactory.getLogger(CLibrary.class); @@@ -55,45 -68,22 +69,28 @@@ private static final int POSIX_FADV_DONTNEED = 4; /* fadvise.h */ private static final int POSIX_FADV_NOREUSE= 5; /* fadvise.h */ - static boolean jnaAvailable = true; - static boolean jnaLockable = false; - private static boolean fsyncUnavailable = false; - - private static final boolean OS_LINUX; - private static final boolean OS_AIX; - private static final boolean OS_MAC; + private static final CLibraryWrapper wrappedCLibrary; + private static boolean jnaLockable = false; +private static final Field FILE_DESCRIPTOR_FD_FIELD; +private static final Field FILE_CHANNEL_FD_FIELD; + static { - String os = System.getProperty("os.name").toLowerCase(); - OS_LINUX = os.contains("linux"); - OS_AIX = os.contains("aix"); - OS_MAC = os.contains("mac"); - +FILE_DESCRIPTOR_FD_FIELD = FBUtilities.getProtectedField(FileDescriptor.class, "fd"); +FILE_CHANNEL_FD_FIELD = FBUtilities.getProtectedField(FileChannelImpl.class, "fd"); + - try - { -
[1/6] cassandra git commit: Improve testing on macOS by eliminating sigar logging
Repository: cassandra Updated Branches: refs/heads/cassandra-3.0 244f887f4 -> e3968cfd1 refs/heads/cassandra-3.11 a830cf528 -> 92b7322bb refs/heads/trunk 643234664 -> 979b19b7c Improve testing on macOS by eliminating sigar logging patch by mkjellman, reviewed by jasobrown for CASSANDRA-13233 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/e3968cfd Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/e3968cfd Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/e3968cfd Branch: refs/heads/cassandra-3.0 Commit: e3968cfd1e786f260029cf8cde4164719ae56c53 Parents: 244f887 Author: Michael Kjellman Authored: Thu Mar 2 14:03:25 2017 -0800 Committer: Jason Brown Committed: Fri Mar 3 11:15:52 2017 -0800 -- CHANGES.txt | 1 + .../org/apache/cassandra/utils/CLibrary.java| 124 +++--- .../apache/cassandra/utils/CLibraryDarwin.java | 131 +++ .../apache/cassandra/utils/CLibraryLinux.java | 131 +++ .../apache/cassandra/utils/CLibraryWindows.java | 103 +++ .../apache/cassandra/utils/CLibraryWrapper.java | 40 ++ .../org/apache/cassandra/utils/HeapUtils.java | 6 +- .../org/apache/cassandra/utils/UUIDGen.java | 2 +- .../apache/cassandra/utils/CLibraryTest.java| 8 ++ 9 files changed, 494 insertions(+), 52 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/e3968cfd/CHANGES.txt -- diff --git a/CHANGES.txt b/CHANGES.txt index 1c3869f..076b337 100644 --- a/CHANGES.txt +++ b/CHANGES.txt @@ -1,4 +1,5 @@ 3.0.12 + * Improve testing on macOS by eliminating sigar logging (CASSANDRA-13233) * Cqlsh copy-from should error out when csv contains invalid data for collections (CASSANDRA-13071) * Update c.yaml doc for offheap memtables (CASSANDRA-13179) * Faster StreamingHistogram (CASSANDRA-13038) http://git-wip-us.apache.org/repos/asf/cassandra/blob/e3968cfd/src/java/org/apache/cassandra/utils/CLibrary.java -- diff --git a/src/java/org/apache/cassandra/utils/CLibrary.java b/src/java/org/apache/cassandra/utils/CLibrary.java index 347b0df..71f4148 100644 --- a/src/java/org/apache/cassandra/utils/CLibrary.java +++ b/src/java/org/apache/cassandra/utils/CLibrary.java @@ -23,21 +23,33 @@ import java.io.FileInputStream; import java.io.IOException; import java.lang.reflect.Field; import java.nio.channels.FileChannel; -import java.nio.file.Paths; -import java.nio.file.StandardOpenOption; import java.util.concurrent.TimeUnit; import org.slf4j.Logger; import org.slf4j.LoggerFactory; import com.sun.jna.LastErrorException; -import com.sun.jna.Native; -import com.sun.jna.Pointer; + +import static org.apache.cassandra.utils.CLibrary.OSType.LINUX; +import static org.apache.cassandra.utils.CLibrary.OSType.MAC; +import static org.apache.cassandra.utils.CLibrary.OSType.WINDOWS; +import static org.apache.cassandra.utils.CLibrary.OSType.AIX; public final class CLibrary { private static final Logger logger = LoggerFactory.getLogger(CLibrary.class); +public enum OSType +{ +LINUX, +MAC, +WINDOWS, +AIX, +OTHER; +} + +private static final OSType osType; + private static final int MCL_CURRENT; private static final int MCL_FUTURE; @@ -56,39 +68,32 @@ public final class CLibrary private static final int POSIX_FADV_DONTNEED = 4; /* fadvise.h */ private static final int POSIX_FADV_NOREUSE= 5; /* fadvise.h */ -static boolean jnaAvailable = true; -static boolean jnaLockable = false; +private static final CLibraryWrapper wrappedCLibrary; +private static boolean jnaLockable = false; static { -try -{ -Native.register("c"); -} -catch (NoClassDefFoundError e) -{ -logger.warn("JNA not found. Native methods will be disabled."); -jnaAvailable = false; -} -catch (UnsatisfiedLinkError e) +// detect the OS type the JVM is running on and then set the CLibraryWrapper +// instance to a compatable implementation of CLibraryWrapper for that OS type +osType = getOsType(); +switch (osType) { -logger.warn("JNA link failure, one or more native method will be unavailable."); -logger.trace("JNA link failure details: {}", e.getMessage()); -} -catch (NoSuchMethodError e) -{ -logger.warn("Obsolete version of JNA present; unable to register C library. Upgrade to JNA 3.2.7 or later"); -jnaAvailable = false; +case MAC: wr
[3/6] cassandra git commit: Improve testing on macOS by eliminating sigar logging
Improve testing on macOS by eliminating sigar logging patch by mkjellman, reviewed by jasobrown for CASSANDRA-13233 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/e3968cfd Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/e3968cfd Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/e3968cfd Branch: refs/heads/trunk Commit: e3968cfd1e786f260029cf8cde4164719ae56c53 Parents: 244f887 Author: Michael Kjellman Authored: Thu Mar 2 14:03:25 2017 -0800 Committer: Jason Brown Committed: Fri Mar 3 11:15:52 2017 -0800 -- CHANGES.txt | 1 + .../org/apache/cassandra/utils/CLibrary.java| 124 +++--- .../apache/cassandra/utils/CLibraryDarwin.java | 131 +++ .../apache/cassandra/utils/CLibraryLinux.java | 131 +++ .../apache/cassandra/utils/CLibraryWindows.java | 103 +++ .../apache/cassandra/utils/CLibraryWrapper.java | 40 ++ .../org/apache/cassandra/utils/HeapUtils.java | 6 +- .../org/apache/cassandra/utils/UUIDGen.java | 2 +- .../apache/cassandra/utils/CLibraryTest.java| 8 ++ 9 files changed, 494 insertions(+), 52 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/e3968cfd/CHANGES.txt -- diff --git a/CHANGES.txt b/CHANGES.txt index 1c3869f..076b337 100644 --- a/CHANGES.txt +++ b/CHANGES.txt @@ -1,4 +1,5 @@ 3.0.12 + * Improve testing on macOS by eliminating sigar logging (CASSANDRA-13233) * Cqlsh copy-from should error out when csv contains invalid data for collections (CASSANDRA-13071) * Update c.yaml doc for offheap memtables (CASSANDRA-13179) * Faster StreamingHistogram (CASSANDRA-13038) http://git-wip-us.apache.org/repos/asf/cassandra/blob/e3968cfd/src/java/org/apache/cassandra/utils/CLibrary.java -- diff --git a/src/java/org/apache/cassandra/utils/CLibrary.java b/src/java/org/apache/cassandra/utils/CLibrary.java index 347b0df..71f4148 100644 --- a/src/java/org/apache/cassandra/utils/CLibrary.java +++ b/src/java/org/apache/cassandra/utils/CLibrary.java @@ -23,21 +23,33 @@ import java.io.FileInputStream; import java.io.IOException; import java.lang.reflect.Field; import java.nio.channels.FileChannel; -import java.nio.file.Paths; -import java.nio.file.StandardOpenOption; import java.util.concurrent.TimeUnit; import org.slf4j.Logger; import org.slf4j.LoggerFactory; import com.sun.jna.LastErrorException; -import com.sun.jna.Native; -import com.sun.jna.Pointer; + +import static org.apache.cassandra.utils.CLibrary.OSType.LINUX; +import static org.apache.cassandra.utils.CLibrary.OSType.MAC; +import static org.apache.cassandra.utils.CLibrary.OSType.WINDOWS; +import static org.apache.cassandra.utils.CLibrary.OSType.AIX; public final class CLibrary { private static final Logger logger = LoggerFactory.getLogger(CLibrary.class); +public enum OSType +{ +LINUX, +MAC, +WINDOWS, +AIX, +OTHER; +} + +private static final OSType osType; + private static final int MCL_CURRENT; private static final int MCL_FUTURE; @@ -56,39 +68,32 @@ public final class CLibrary private static final int POSIX_FADV_DONTNEED = 4; /* fadvise.h */ private static final int POSIX_FADV_NOREUSE= 5; /* fadvise.h */ -static boolean jnaAvailable = true; -static boolean jnaLockable = false; +private static final CLibraryWrapper wrappedCLibrary; +private static boolean jnaLockable = false; static { -try -{ -Native.register("c"); -} -catch (NoClassDefFoundError e) -{ -logger.warn("JNA not found. Native methods will be disabled."); -jnaAvailable = false; -} -catch (UnsatisfiedLinkError e) +// detect the OS type the JVM is running on and then set the CLibraryWrapper +// instance to a compatable implementation of CLibraryWrapper for that OS type +osType = getOsType(); +switch (osType) { -logger.warn("JNA link failure, one or more native method will be unavailable."); -logger.trace("JNA link failure details: {}", e.getMessage()); -} -catch (NoSuchMethodError e) -{ -logger.warn("Obsolete version of JNA present; unable to register C library. Upgrade to JNA 3.2.7 or later"); -jnaAvailable = false; +case MAC: wrappedCLibrary = new CLibraryDarwin(); break; +case WINDOWS: wrappedCLibrary = new CLibraryWindows(); break; +case LINUX: +case AIX: +case OTHER:
[6/6] cassandra git commit: Merge branch 'cassandra-3.11' into trunk
Merge branch 'cassandra-3.11' into trunk Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/979b19b7 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/979b19b7 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/979b19b7 Branch: refs/heads/trunk Commit: 979b19b7c0ac64b14b51e8b92b00f50d0eed9be7 Parents: 6432346 92b7322 Author: Jason Brown Authored: Fri Mar 3 11:20:16 2017 -0800 Committer: Jason Brown Committed: Fri Mar 3 11:20:36 2017 -0800 -- CHANGES.txt | 1 + .../org/apache/cassandra/utils/CLibrary.java| 135 +++ .../apache/cassandra/utils/CLibraryDarwin.java | 131 ++ .../apache/cassandra/utils/CLibraryLinux.java | 131 ++ .../apache/cassandra/utils/CLibraryWindows.java | 103 ++ .../apache/cassandra/utils/CLibraryWrapper.java | 40 ++ .../org/apache/cassandra/utils/HeapUtils.java | 6 +- .../org/apache/cassandra/utils/UUIDGen.java | 2 +- .../apache/cassandra/utils/CLibraryTest.java| 8 ++ 9 files changed, 495 insertions(+), 62 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/979b19b7/CHANGES.txt --
[2/6] cassandra git commit: Improve testing on macOS by eliminating sigar logging
Improve testing on macOS by eliminating sigar logging patch by mkjellman, reviewed by jasobrown for CASSANDRA-13233 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/e3968cfd Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/e3968cfd Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/e3968cfd Branch: refs/heads/cassandra-3.11 Commit: e3968cfd1e786f260029cf8cde4164719ae56c53 Parents: 244f887 Author: Michael Kjellman Authored: Thu Mar 2 14:03:25 2017 -0800 Committer: Jason Brown Committed: Fri Mar 3 11:15:52 2017 -0800 -- CHANGES.txt | 1 + .../org/apache/cassandra/utils/CLibrary.java| 124 +++--- .../apache/cassandra/utils/CLibraryDarwin.java | 131 +++ .../apache/cassandra/utils/CLibraryLinux.java | 131 +++ .../apache/cassandra/utils/CLibraryWindows.java | 103 +++ .../apache/cassandra/utils/CLibraryWrapper.java | 40 ++ .../org/apache/cassandra/utils/HeapUtils.java | 6 +- .../org/apache/cassandra/utils/UUIDGen.java | 2 +- .../apache/cassandra/utils/CLibraryTest.java| 8 ++ 9 files changed, 494 insertions(+), 52 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/e3968cfd/CHANGES.txt -- diff --git a/CHANGES.txt b/CHANGES.txt index 1c3869f..076b337 100644 --- a/CHANGES.txt +++ b/CHANGES.txt @@ -1,4 +1,5 @@ 3.0.12 + * Improve testing on macOS by eliminating sigar logging (CASSANDRA-13233) * Cqlsh copy-from should error out when csv contains invalid data for collections (CASSANDRA-13071) * Update c.yaml doc for offheap memtables (CASSANDRA-13179) * Faster StreamingHistogram (CASSANDRA-13038) http://git-wip-us.apache.org/repos/asf/cassandra/blob/e3968cfd/src/java/org/apache/cassandra/utils/CLibrary.java -- diff --git a/src/java/org/apache/cassandra/utils/CLibrary.java b/src/java/org/apache/cassandra/utils/CLibrary.java index 347b0df..71f4148 100644 --- a/src/java/org/apache/cassandra/utils/CLibrary.java +++ b/src/java/org/apache/cassandra/utils/CLibrary.java @@ -23,21 +23,33 @@ import java.io.FileInputStream; import java.io.IOException; import java.lang.reflect.Field; import java.nio.channels.FileChannel; -import java.nio.file.Paths; -import java.nio.file.StandardOpenOption; import java.util.concurrent.TimeUnit; import org.slf4j.Logger; import org.slf4j.LoggerFactory; import com.sun.jna.LastErrorException; -import com.sun.jna.Native; -import com.sun.jna.Pointer; + +import static org.apache.cassandra.utils.CLibrary.OSType.LINUX; +import static org.apache.cassandra.utils.CLibrary.OSType.MAC; +import static org.apache.cassandra.utils.CLibrary.OSType.WINDOWS; +import static org.apache.cassandra.utils.CLibrary.OSType.AIX; public final class CLibrary { private static final Logger logger = LoggerFactory.getLogger(CLibrary.class); +public enum OSType +{ +LINUX, +MAC, +WINDOWS, +AIX, +OTHER; +} + +private static final OSType osType; + private static final int MCL_CURRENT; private static final int MCL_FUTURE; @@ -56,39 +68,32 @@ public final class CLibrary private static final int POSIX_FADV_DONTNEED = 4; /* fadvise.h */ private static final int POSIX_FADV_NOREUSE= 5; /* fadvise.h */ -static boolean jnaAvailable = true; -static boolean jnaLockable = false; +private static final CLibraryWrapper wrappedCLibrary; +private static boolean jnaLockable = false; static { -try -{ -Native.register("c"); -} -catch (NoClassDefFoundError e) -{ -logger.warn("JNA not found. Native methods will be disabled."); -jnaAvailable = false; -} -catch (UnsatisfiedLinkError e) +// detect the OS type the JVM is running on and then set the CLibraryWrapper +// instance to a compatable implementation of CLibraryWrapper for that OS type +osType = getOsType(); +switch (osType) { -logger.warn("JNA link failure, one or more native method will be unavailable."); -logger.trace("JNA link failure details: {}", e.getMessage()); -} -catch (NoSuchMethodError e) -{ -logger.warn("Obsolete version of JNA present; unable to register C library. Upgrade to JNA 3.2.7 or later"); -jnaAvailable = false; +case MAC: wrappedCLibrary = new CLibraryDarwin(); break; +case WINDOWS: wrappedCLibrary = new CLibraryWindows(); break; +case LINUX: +case AIX: +cas
[4/6] cassandra git commit: Merge branch 'cassandra-3.0' into cassandra-3.11
Merge branch 'cassandra-3.0' into cassandra-3.11 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/92b7322b Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/92b7322b Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/92b7322b Branch: refs/heads/trunk Commit: 92b7322bbf0b53203adcfb6c4479022f975d9785 Parents: a830cf5 e3968cf Author: Jason Brown Authored: Fri Mar 3 11:17:17 2017 -0800 Committer: Jason Brown Committed: Fri Mar 3 11:19:30 2017 -0800 -- CHANGES.txt | 1 + .../org/apache/cassandra/utils/CLibrary.java| 135 +++ .../apache/cassandra/utils/CLibraryDarwin.java | 131 ++ .../apache/cassandra/utils/CLibraryLinux.java | 131 ++ .../apache/cassandra/utils/CLibraryWindows.java | 103 ++ .../apache/cassandra/utils/CLibraryWrapper.java | 40 ++ .../org/apache/cassandra/utils/HeapUtils.java | 6 +- .../org/apache/cassandra/utils/UUIDGen.java | 2 +- .../apache/cassandra/utils/CLibraryTest.java| 8 ++ 9 files changed, 495 insertions(+), 62 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/92b7322b/CHANGES.txt -- diff --cc CHANGES.txt index 5de9ece,076b337..b91edb4 --- a/CHANGES.txt +++ b/CHANGES.txt @@@ -1,17 -1,7 +1,18 @@@ -3.0.12 +3.11.0 + * Fix equality comparisons of columns using the duration type (CASSANDRA-13174) + * Obfuscate password in stress-graphs (CASSANDRA-12233) + * Move to FastThreadLocalThread and FastThreadLocal (CASSANDRA-13034) + * nodetool stopdaemon errors out (CASSANDRA-13030) + * Tables in system_distributed should not use gcgs of 0 (CASSANDRA-12954) + * Fix primary index calculation for SASI (CASSANDRA-12910) + * More fixes to the TokenAllocator (CASSANDRA-12990) + * NoReplicationTokenAllocator should work with zero replication factor (CASSANDRA-12983) + * Address message coalescing regression (CASSANDRA-12676) +Merged from 3.0: + * Improve testing on macOS by eliminating sigar logging (CASSANDRA-13233) * Cqlsh copy-from should error out when csv contains invalid data for collections (CASSANDRA-13071) - * Update c.yaml doc for offheap memtables (CASSANDRA-13179) + * Fix "multiple versions of ant detected..." when running ant test (CASSANDRA-13232) + * Coalescing strategy sleeps too much (CASSANDRA-13090) * Faster StreamingHistogram (CASSANDRA-13038) * Legacy deserializer can create unexpected boundary range tombstones (CASSANDRA-13237) * Remove unnecessary assertion from AntiCompactionTest (CASSANDRA-13070) http://git-wip-us.apache.org/repos/asf/cassandra/blob/92b7322b/src/java/org/apache/cassandra/utils/CLibrary.java -- diff --cc src/java/org/apache/cassandra/utils/CLibrary.java index 937d662,71f4148..ae3ce63 --- a/src/java/org/apache/cassandra/utils/CLibrary.java +++ b/src/java/org/apache/cassandra/utils/CLibrary.java @@@ -29,10 -29,12 +29,13 @@@ import org.slf4j.Logger import org.slf4j.LoggerFactory; import com.sun.jna.LastErrorException; - import com.sun.jna.Native; - import com.sun.jna.Pointer; +import sun.nio.ch.FileChannelImpl; + import static org.apache.cassandra.utils.CLibrary.OSType.LINUX; + import static org.apache.cassandra.utils.CLibrary.OSType.MAC; + import static org.apache.cassandra.utils.CLibrary.OSType.WINDOWS; + import static org.apache.cassandra.utils.CLibrary.OSType.AIX; + public final class CLibrary { private static final Logger logger = LoggerFactory.getLogger(CLibrary.class); @@@ -55,45 -68,22 +69,28 @@@ private static final int POSIX_FADV_DONTNEED = 4; /* fadvise.h */ private static final int POSIX_FADV_NOREUSE= 5; /* fadvise.h */ - static boolean jnaAvailable = true; - static boolean jnaLockable = false; - private static boolean fsyncUnavailable = false; - - private static final boolean OS_LINUX; - private static final boolean OS_AIX; - private static final boolean OS_MAC; + private static final CLibraryWrapper wrappedCLibrary; + private static boolean jnaLockable = false; +private static final Field FILE_DESCRIPTOR_FD_FIELD; +private static final Field FILE_CHANNEL_FD_FIELD; + static { - String os = System.getProperty("os.name").toLowerCase(); - OS_LINUX = os.contains("linux"); - OS_AIX = os.contains("aix"); - OS_MAC = os.contains("mac"); - +FILE_DESCRIPTOR_FD_FIELD = FBUtilities.getProtectedField(FileDescriptor.class, "fd"); +FILE_CHANNEL_FD_FIELD = FBUtilities.getProtectedField(FileChannelImpl.class, "fd"); + - try - { - Native
[09/10] cassandra git commit: Merge branch 'cassandra-3.0' into cassandra-3.11
Merge branch 'cassandra-3.0' into cassandra-3.11 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/a830cf52 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/a830cf52 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/a830cf52 Branch: refs/heads/trunk Commit: a830cf528ea2ab85288bf7503b282507346c71db Parents: ff170af 244f887 Author: Michael Shuler Authored: Fri Mar 3 12:42:19 2017 -0600 Committer: Michael Shuler Committed: Fri Mar 3 12:42:19 2017 -0600 -- debian/control| 2 +- redhat/cassandra.spec | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/a830cf52/redhat/cassandra.spec --
[01/10] cassandra git commit: Update ant build dependency to >= 1.9 in deb/rpm
Repository: cassandra Updated Branches: refs/heads/cassandra-2.2 c0f99c4e6 -> 288d243d8 refs/heads/cassandra-3.0 e11f75081 -> 244f887f4 refs/heads/cassandra-3.11 ff170af2a -> a830cf528 refs/heads/trunk b9b63925f -> 643234664 Update ant build dependency to >= 1.9 in deb/rpm Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/288d243d Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/288d243d Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/288d243d Branch: refs/heads/cassandra-2.2 Commit: 288d243d806c4b5143c38875515776dae2fc99ae Parents: c0f99c4 Author: Michael Shuler Authored: Fri Mar 3 12:39:37 2017 -0600 Committer: Michael Shuler Committed: Fri Mar 3 12:39:37 2017 -0600 -- debian/control| 2 +- redhat/cassandra.spec | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/288d243d/debian/control -- diff --git a/debian/control b/debian/control index cc5928c..d45a875 100644 --- a/debian/control +++ b/debian/control @@ -3,7 +3,7 @@ Section: misc Priority: extra Maintainer: Eric Evans Uploaders: Sylvain Lebresne -Build-Depends: debhelper (>= 5), openjdk-7-jdk | java7-jdk, ant (>= 1.7), ant-optional (>= 1.7), dh-python, python-dev (>= 2.7), dpatch, bash-completion +Build-Depends: debhelper (>= 5), openjdk-7-jdk | java7-jdk, ant (>= 1.9), ant-optional (>= 1.9), dh-python, python-dev (>= 2.7), dpatch, bash-completion Homepage: http://cassandra.apache.org Vcs-Git: http://git-wip-us.apache.org/repos/asf/cassandra.git Vcs-Browser: https://git-wip-us.apache.org/repos/asf?p=cassandra.git http://git-wip-us.apache.org/repos/asf/cassandra/blob/288d243d/redhat/cassandra.spec -- diff --git a/redhat/cassandra.spec b/redhat/cassandra.spec index b40a706..c3c22d0 100644 --- a/redhat/cassandra.spec +++ b/redhat/cassandra.spec @@ -18,7 +18,7 @@ URL: http://cassandra.apache.org/ Source0: %{relname}-src.tar.gz BuildRoot: %{_tmppath}/%{relname}root-%(%{__id_u} -n) -BuildRequires: ant +BuildRequires: ant >= 1.9 Requires: jre >= 1.7.0 Requires: python(abi) >= 2.7
[08/10] cassandra git commit: Merge branch 'cassandra-3.0' into cassandra-3.11
Merge branch 'cassandra-3.0' into cassandra-3.11 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/a830cf52 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/a830cf52 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/a830cf52 Branch: refs/heads/cassandra-3.11 Commit: a830cf528ea2ab85288bf7503b282507346c71db Parents: ff170af 244f887 Author: Michael Shuler Authored: Fri Mar 3 12:42:19 2017 -0600 Committer: Michael Shuler Committed: Fri Mar 3 12:42:19 2017 -0600 -- debian/control| 2 +- redhat/cassandra.spec | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/a830cf52/redhat/cassandra.spec --
[03/10] cassandra git commit: Update ant build dependency to >= 1.9 in deb/rpm
Update ant build dependency to >= 1.9 in deb/rpm Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/288d243d Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/288d243d Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/288d243d Branch: refs/heads/cassandra-3.11 Commit: 288d243d806c4b5143c38875515776dae2fc99ae Parents: c0f99c4 Author: Michael Shuler Authored: Fri Mar 3 12:39:37 2017 -0600 Committer: Michael Shuler Committed: Fri Mar 3 12:39:37 2017 -0600 -- debian/control| 2 +- redhat/cassandra.spec | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/288d243d/debian/control -- diff --git a/debian/control b/debian/control index cc5928c..d45a875 100644 --- a/debian/control +++ b/debian/control @@ -3,7 +3,7 @@ Section: misc Priority: extra Maintainer: Eric Evans Uploaders: Sylvain Lebresne -Build-Depends: debhelper (>= 5), openjdk-7-jdk | java7-jdk, ant (>= 1.7), ant-optional (>= 1.7), dh-python, python-dev (>= 2.7), dpatch, bash-completion +Build-Depends: debhelper (>= 5), openjdk-7-jdk | java7-jdk, ant (>= 1.9), ant-optional (>= 1.9), dh-python, python-dev (>= 2.7), dpatch, bash-completion Homepage: http://cassandra.apache.org Vcs-Git: http://git-wip-us.apache.org/repos/asf/cassandra.git Vcs-Browser: https://git-wip-us.apache.org/repos/asf?p=cassandra.git http://git-wip-us.apache.org/repos/asf/cassandra/blob/288d243d/redhat/cassandra.spec -- diff --git a/redhat/cassandra.spec b/redhat/cassandra.spec index b40a706..c3c22d0 100644 --- a/redhat/cassandra.spec +++ b/redhat/cassandra.spec @@ -18,7 +18,7 @@ URL: http://cassandra.apache.org/ Source0: %{relname}-src.tar.gz BuildRoot: %{_tmppath}/%{relname}root-%(%{__id_u} -n) -BuildRequires: ant +BuildRequires: ant >= 1.9 Requires: jre >= 1.7.0 Requires: python(abi) >= 2.7
[04/10] cassandra git commit: Update ant build dependency to >= 1.9 in deb/rpm
Update ant build dependency to >= 1.9 in deb/rpm Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/288d243d Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/288d243d Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/288d243d Branch: refs/heads/trunk Commit: 288d243d806c4b5143c38875515776dae2fc99ae Parents: c0f99c4 Author: Michael Shuler Authored: Fri Mar 3 12:39:37 2017 -0600 Committer: Michael Shuler Committed: Fri Mar 3 12:39:37 2017 -0600 -- debian/control| 2 +- redhat/cassandra.spec | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/288d243d/debian/control -- diff --git a/debian/control b/debian/control index cc5928c..d45a875 100644 --- a/debian/control +++ b/debian/control @@ -3,7 +3,7 @@ Section: misc Priority: extra Maintainer: Eric Evans Uploaders: Sylvain Lebresne -Build-Depends: debhelper (>= 5), openjdk-7-jdk | java7-jdk, ant (>= 1.7), ant-optional (>= 1.7), dh-python, python-dev (>= 2.7), dpatch, bash-completion +Build-Depends: debhelper (>= 5), openjdk-7-jdk | java7-jdk, ant (>= 1.9), ant-optional (>= 1.9), dh-python, python-dev (>= 2.7), dpatch, bash-completion Homepage: http://cassandra.apache.org Vcs-Git: http://git-wip-us.apache.org/repos/asf/cassandra.git Vcs-Browser: https://git-wip-us.apache.org/repos/asf?p=cassandra.git http://git-wip-us.apache.org/repos/asf/cassandra/blob/288d243d/redhat/cassandra.spec -- diff --git a/redhat/cassandra.spec b/redhat/cassandra.spec index b40a706..c3c22d0 100644 --- a/redhat/cassandra.spec +++ b/redhat/cassandra.spec @@ -18,7 +18,7 @@ URL: http://cassandra.apache.org/ Source0: %{relname}-src.tar.gz BuildRoot: %{_tmppath}/%{relname}root-%(%{__id_u} -n) -BuildRequires: ant +BuildRequires: ant >= 1.9 Requires: jre >= 1.7.0 Requires: python(abi) >= 2.7
[10/10] cassandra git commit: Merge branch 'cassandra-3.11' into trunk
Merge branch 'cassandra-3.11' into trunk Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/64323466 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/64323466 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/64323466 Branch: refs/heads/trunk Commit: 64323466402f33536ba0318f02e7829eafcbd711 Parents: b9b6392 a830cf5 Author: Michael Shuler Authored: Fri Mar 3 12:42:40 2017 -0600 Committer: Michael Shuler Committed: Fri Mar 3 12:42:40 2017 -0600 -- debian/control| 2 +- redhat/cassandra.spec | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/64323466/redhat/cassandra.spec --
[06/10] cassandra git commit: Merge branch 'cassandra-2.2' into cassandra-3.0
Merge branch 'cassandra-2.2' into cassandra-3.0 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/244f887f Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/244f887f Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/244f887f Branch: refs/heads/cassandra-3.0 Commit: 244f887f4c61ed75cc7250386b4ae5371e79f48f Parents: e11f750 288d243 Author: Michael Shuler Authored: Fri Mar 3 12:42:00 2017 -0600 Committer: Michael Shuler Committed: Fri Mar 3 12:42:00 2017 -0600 -- debian/control| 2 +- redhat/cassandra.spec | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/244f887f/debian/control -- diff --cc debian/control index 2d30077,d45a875..6a0128e --- a/debian/control +++ b/debian/control @@@ -3,7 -3,7 +3,7 @@@ Section: mis Priority: extra Maintainer: Eric Evans Uploaders: Sylvain Lebresne - Build-Depends: debhelper (>= 5), openjdk-8-jdk | java8-jdk, ant (>= 1.7), ant-optional (>= 1.7), dh-python, python-dev (>= 2.7), dpatch, bash-completion -Build-Depends: debhelper (>= 5), openjdk-7-jdk | java7-jdk, ant (>= 1.9), ant-optional (>= 1.9), dh-python, python-dev (>= 2.7), dpatch, bash-completion ++Build-Depends: debhelper (>= 5), openjdk-8-jdk | java8-jdk, ant (>= 1.9), ant-optional (>= 1.9), dh-python, python-dev (>= 2.7), dpatch, bash-completion Homepage: http://cassandra.apache.org Vcs-Git: http://git-wip-us.apache.org/repos/asf/cassandra.git Vcs-Browser: https://git-wip-us.apache.org/repos/asf?p=cassandra.git http://git-wip-us.apache.org/repos/asf/cassandra/blob/244f887f/redhat/cassandra.spec -- diff --cc redhat/cassandra.spec index a1bfa5d,c3c22d0..d5e743d --- a/redhat/cassandra.spec +++ b/redhat/cassandra.spec @@@ -18,9 -18,9 +18,9 @@@ URL: http://cassandra.apache. Source0: %{relname}-src.tar.gz BuildRoot: %{_tmppath}/%{relname}root-%(%{__id_u} -n) - BuildRequires: ant + BuildRequires: ant >= 1.9 -Requires: jre >= 1.7.0 +Requires: jre >= 1.8.0 Requires: python(abi) >= 2.7 Requires(pre): user(cassandra) Requires(pre): group(cassandra)
[02/10] cassandra git commit: Update ant build dependency to >= 1.9 in deb/rpm
Update ant build dependency to >= 1.9 in deb/rpm Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/288d243d Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/288d243d Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/288d243d Branch: refs/heads/cassandra-3.0 Commit: 288d243d806c4b5143c38875515776dae2fc99ae Parents: c0f99c4 Author: Michael Shuler Authored: Fri Mar 3 12:39:37 2017 -0600 Committer: Michael Shuler Committed: Fri Mar 3 12:39:37 2017 -0600 -- debian/control| 2 +- redhat/cassandra.spec | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/288d243d/debian/control -- diff --git a/debian/control b/debian/control index cc5928c..d45a875 100644 --- a/debian/control +++ b/debian/control @@ -3,7 +3,7 @@ Section: misc Priority: extra Maintainer: Eric Evans Uploaders: Sylvain Lebresne -Build-Depends: debhelper (>= 5), openjdk-7-jdk | java7-jdk, ant (>= 1.7), ant-optional (>= 1.7), dh-python, python-dev (>= 2.7), dpatch, bash-completion +Build-Depends: debhelper (>= 5), openjdk-7-jdk | java7-jdk, ant (>= 1.9), ant-optional (>= 1.9), dh-python, python-dev (>= 2.7), dpatch, bash-completion Homepage: http://cassandra.apache.org Vcs-Git: http://git-wip-us.apache.org/repos/asf/cassandra.git Vcs-Browser: https://git-wip-us.apache.org/repos/asf?p=cassandra.git http://git-wip-us.apache.org/repos/asf/cassandra/blob/288d243d/redhat/cassandra.spec -- diff --git a/redhat/cassandra.spec b/redhat/cassandra.spec index b40a706..c3c22d0 100644 --- a/redhat/cassandra.spec +++ b/redhat/cassandra.spec @@ -18,7 +18,7 @@ URL: http://cassandra.apache.org/ Source0: %{relname}-src.tar.gz BuildRoot: %{_tmppath}/%{relname}root-%(%{__id_u} -n) -BuildRequires: ant +BuildRequires: ant >= 1.9 Requires: jre >= 1.7.0 Requires: python(abi) >= 2.7
[07/10] cassandra git commit: Merge branch 'cassandra-2.2' into cassandra-3.0
Merge branch 'cassandra-2.2' into cassandra-3.0 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/244f887f Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/244f887f Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/244f887f Branch: refs/heads/trunk Commit: 244f887f4c61ed75cc7250386b4ae5371e79f48f Parents: e11f750 288d243 Author: Michael Shuler Authored: Fri Mar 3 12:42:00 2017 -0600 Committer: Michael Shuler Committed: Fri Mar 3 12:42:00 2017 -0600 -- debian/control| 2 +- redhat/cassandra.spec | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/244f887f/debian/control -- diff --cc debian/control index 2d30077,d45a875..6a0128e --- a/debian/control +++ b/debian/control @@@ -3,7 -3,7 +3,7 @@@ Section: mis Priority: extra Maintainer: Eric Evans Uploaders: Sylvain Lebresne - Build-Depends: debhelper (>= 5), openjdk-8-jdk | java8-jdk, ant (>= 1.7), ant-optional (>= 1.7), dh-python, python-dev (>= 2.7), dpatch, bash-completion -Build-Depends: debhelper (>= 5), openjdk-7-jdk | java7-jdk, ant (>= 1.9), ant-optional (>= 1.9), dh-python, python-dev (>= 2.7), dpatch, bash-completion ++Build-Depends: debhelper (>= 5), openjdk-8-jdk | java8-jdk, ant (>= 1.9), ant-optional (>= 1.9), dh-python, python-dev (>= 2.7), dpatch, bash-completion Homepage: http://cassandra.apache.org Vcs-Git: http://git-wip-us.apache.org/repos/asf/cassandra.git Vcs-Browser: https://git-wip-us.apache.org/repos/asf?p=cassandra.git http://git-wip-us.apache.org/repos/asf/cassandra/blob/244f887f/redhat/cassandra.spec -- diff --cc redhat/cassandra.spec index a1bfa5d,c3c22d0..d5e743d --- a/redhat/cassandra.spec +++ b/redhat/cassandra.spec @@@ -18,9 -18,9 +18,9 @@@ URL: http://cassandra.apache. Source0: %{relname}-src.tar.gz BuildRoot: %{_tmppath}/%{relname}root-%(%{__id_u} -n) - BuildRequires: ant + BuildRequires: ant >= 1.9 -Requires: jre >= 1.7.0 +Requires: jre >= 1.8.0 Requires: python(abi) >= 2.7 Requires(pre): user(cassandra) Requires(pre): group(cassandra)
[05/10] cassandra git commit: Merge branch 'cassandra-2.2' into cassandra-3.0
Merge branch 'cassandra-2.2' into cassandra-3.0 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/244f887f Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/244f887f Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/244f887f Branch: refs/heads/cassandra-3.11 Commit: 244f887f4c61ed75cc7250386b4ae5371e79f48f Parents: e11f750 288d243 Author: Michael Shuler Authored: Fri Mar 3 12:42:00 2017 -0600 Committer: Michael Shuler Committed: Fri Mar 3 12:42:00 2017 -0600 -- debian/control| 2 +- redhat/cassandra.spec | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/244f887f/debian/control -- diff --cc debian/control index 2d30077,d45a875..6a0128e --- a/debian/control +++ b/debian/control @@@ -3,7 -3,7 +3,7 @@@ Section: mis Priority: extra Maintainer: Eric Evans Uploaders: Sylvain Lebresne - Build-Depends: debhelper (>= 5), openjdk-8-jdk | java8-jdk, ant (>= 1.7), ant-optional (>= 1.7), dh-python, python-dev (>= 2.7), dpatch, bash-completion -Build-Depends: debhelper (>= 5), openjdk-7-jdk | java7-jdk, ant (>= 1.9), ant-optional (>= 1.9), dh-python, python-dev (>= 2.7), dpatch, bash-completion ++Build-Depends: debhelper (>= 5), openjdk-8-jdk | java8-jdk, ant (>= 1.9), ant-optional (>= 1.9), dh-python, python-dev (>= 2.7), dpatch, bash-completion Homepage: http://cassandra.apache.org Vcs-Git: http://git-wip-us.apache.org/repos/asf/cassandra.git Vcs-Browser: https://git-wip-us.apache.org/repos/asf?p=cassandra.git http://git-wip-us.apache.org/repos/asf/cassandra/blob/244f887f/redhat/cassandra.spec -- diff --cc redhat/cassandra.spec index a1bfa5d,c3c22d0..d5e743d --- a/redhat/cassandra.spec +++ b/redhat/cassandra.spec @@@ -18,9 -18,9 +18,9 @@@ URL: http://cassandra.apache. Source0: %{relname}-src.tar.gz BuildRoot: %{_tmppath}/%{relname}root-%(%{__id_u} -n) - BuildRequires: ant + BuildRequires: ant >= 1.9 -Requires: jre >= 1.7.0 +Requires: jre >= 1.8.0 Requires: python(abi) >= 2.7 Requires(pre): user(cassandra) Requires(pre): group(cassandra)
[jira] [Commented] (CASSANDRA-13265) Expiration in OutboundTcpConnection can block the reader Thread
[ https://issues.apache.org/jira/browse/CASSANDRA-13265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15894783#comment-15894783 ] Jason Brown commented on CASSANDRA-13265: - I'm +1 on the ticket, but [~aweisberg] should also weigh in. As a note, we have {{ApproximateTime}} for an estimated time, but that works off of {{System.currentTimeMillis()}} rather than {{System.nanoTime()}}, so monotonicity. > Expiration in OutboundTcpConnection can block the reader Thread > --- > > Key: CASSANDRA-13265 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13265 > Project: Cassandra > Issue Type: Bug > Environment: Cassandra 3.0.9 > Java HotSpot(TM) 64-Bit Server VM version 25.112-b15 (Java version > 1.8.0_112-b15) > Linux 3.16 >Reporter: Christian Esken >Assignee: Christian Esken > Attachments: cassandra.pb-cache4-dus.2017-02-17-19-36-26.chist.xz, > cassandra.pb-cache4-dus.2017-02-17-19-36-26.td.xz > > > I observed that sometimes a single node in a Cassandra cluster fails to > communicate to the other nodes. This can happen at any time, during peak load > or low load. Restarting that single node from the cluster fixes the issue. > Before going in to details, I want to state that I have analyzed the > situation and am already developing a possible fix. Here is the analysis so > far: > - A Threaddump in this situation showed 324 Threads in the > OutboundTcpConnection class that want to lock the backlog queue for doing > expiration. > - A class histogram shows 262508 instances of > OutboundTcpConnection$QueuedMessage. > What is the effect of it? As soon as the Cassandra node has reached a certain > amount of queued messages, it starts thrashing itself to death. Each of the > Thread fully locks the Queue for reading and writing by calling > iterator.next(), making the situation worse and worse. > - Writing: Only after 262508 locking operation it can progress with actually > writing to the Queue. > - Reading: Is also blocked, as 324 Threads try to do iterator.next(), and > fully lock the Queue > This means: Writing blocks the Queue for reading, and readers might even be > starved which makes the situation even worse. > - > The setup is: > - 3-node cluster > - replication factor 2 > - Consistency LOCAL_ONE > - No remote DC's > - high write throughput (10 INSERT statements per second and more during > peak times). > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Resolved] (CASSANDRA-13271) Reduce lock contention for collection types and serializers
[ https://issues.apache.org/jira/browse/CASSANDRA-13271?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Stupp resolved CASSANDRA-13271. -- Resolution: Fixed Fix Version/s: (was: 4.x) 4.0 Thanks! Committed as [b9b63925f9b818637b986a300aee658df03a5500|https://github.com/apache/cassandra/commit/b9b63925f9b818637b986a300aee658df03a5500] to [trunk|https://github.com/apache/cassandra/tree/trunk] > Reduce lock contention for collection types and serializers > --- > > Key: CASSANDRA-13271 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13271 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: vincent royer >Priority: Minor > Labels: performance > Fix For: 4.0 > > Attachments: 0001-CASSANDRA-13271-computeIfAbsent.patch, > 0001-CASSANDRA-13271-singleton-factory-concurrency-opimiz.patch > > > By doing some performance tests, i noticed that getInstance() in > org.apache.cassandra.db.marshal.ListType and SetType could suffer from lock > contention on the singleton factory getInstance(). Here is a proposal to > reduce lock contention by using a ConcurrentMap and the putIfAbsent method > rather than a synchronized method. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
cassandra git commit: Reduce lock contention for collection types and serializers
Repository: cassandra Updated Branches: refs/heads/trunk a49cf2c6c -> b9b63925f Reduce lock contention for collection types and serializers patch by vincent royer; reviewed by Robert Stupp for CASSANDRA-13271 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/b9b63925 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/b9b63925 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/b9b63925 Branch: refs/heads/trunk Commit: b9b63925f9b818637b986a300aee658df03a5500 Parents: a49cf2c Author: vincent royer Authored: Fri Mar 3 19:02:25 2017 +0100 Committer: Robert Stupp Committed: Fri Mar 3 19:02:25 2017 +0100 -- CHANGES.txt | 1 + .../cassandra/db/marshal/DynamicCompositeType.java | 11 +-- .../org/apache/cassandra/db/marshal/ListType.java| 15 +++ .../org/apache/cassandra/db/marshal/MapType.java | 15 +++ .../org/apache/cassandra/db/marshal/SetType.java | 15 +++ .../apache/cassandra/serializers/ListSerializer.java | 11 +-- .../apache/cassandra/serializers/MapSerializer.java | 11 +-- .../apache/cassandra/serializers/SetSerializer.java | 11 +-- 8 files changed, 42 insertions(+), 48 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/b9b63925/CHANGES.txt -- diff --git a/CHANGES.txt b/CHANGES.txt index 9d07956..0ac9d50 100644 --- a/CHANGES.txt +++ b/CHANGES.txt @@ -1,4 +1,5 @@ 4.0 + * Reduce lock contention for collection types and serializers (CASSANDRA-13271) * Make it possible to override MessagingService.Verb ids (CASSANDRA-13283) * Avoid synchronized on prepareForRepair in ActiveRepairService (CASSANDRA-9292) * Adds the ability to use uncompressed chunks in compressed files (CASSANDRA-10520) http://git-wip-us.apache.org/repos/asf/cassandra/blob/b9b63925/src/java/org/apache/cassandra/db/marshal/DynamicCompositeType.java -- diff --git a/src/java/org/apache/cassandra/db/marshal/DynamicCompositeType.java b/src/java/org/apache/cassandra/db/marshal/DynamicCompositeType.java index d314bd9..cade725 100644 --- a/src/java/org/apache/cassandra/db/marshal/DynamicCompositeType.java +++ b/src/java/org/apache/cassandra/db/marshal/DynamicCompositeType.java @@ -21,6 +21,8 @@ import java.nio.charset.CharacterCodingException; import java.nio.ByteBuffer; import java.util.HashMap; import java.util.Map; +import java.util.concurrent.ConcurrentHashMap; +import java.util.concurrent.ConcurrentMap; import org.apache.cassandra.cql3.Term; import org.slf4j.Logger; @@ -59,21 +61,18 @@ public class DynamicCompositeType extends AbstractCompositeType private final Map> aliases; // interning instances -private static final Map>, DynamicCompositeType> instances = new HashMap>, DynamicCompositeType>(); +private static final ConcurrentMap>, DynamicCompositeType> instances = new ConcurrentHashMap>, DynamicCompositeType>(); public static synchronized DynamicCompositeType getInstance(TypeParser parser) throws ConfigurationException, SyntaxException { return getInstance(parser.getAliasParameters()); } -public static synchronized DynamicCompositeType getInstance(Map> aliases) +public static DynamicCompositeType getInstance(Map> aliases) { DynamicCompositeType dct = instances.get(aliases); if (dct == null) -{ -dct = new DynamicCompositeType(aliases); -instances.put(aliases, dct); -} +dct = instances.computeIfAbsent(aliases, k -> new DynamicCompositeType(k)); return dct; } http://git-wip-us.apache.org/repos/asf/cassandra/blob/b9b63925/src/java/org/apache/cassandra/db/marshal/ListType.java -- diff --git a/src/java/org/apache/cassandra/db/marshal/ListType.java b/src/java/org/apache/cassandra/db/marshal/ListType.java index e4c20e2..29ccaa5 100644 --- a/src/java/org/apache/cassandra/db/marshal/ListType.java +++ b/src/java/org/apache/cassandra/db/marshal/ListType.java @@ -19,6 +19,8 @@ package org.apache.cassandra.db.marshal; import java.nio.ByteBuffer; import java.util.*; +import java.util.concurrent.ConcurrentHashMap; +import java.util.concurrent.ConcurrentMap; import org.apache.cassandra.cql3.Json; import org.apache.cassandra.cql3.Lists; @@ -39,8 +41,8 @@ public class ListType extends CollectionType> private static final Logger logger = LoggerFactory.getLogger(ListType.class); // interning instances -private static final Map, ListType> instances = new HashMap<>(); -private s
[jira] [Updated] (CASSANDRA-13271) Reduce lock contention for collection types and serializers
[ https://issues.apache.org/jira/browse/CASSANDRA-13271?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Stupp updated CASSANDRA-13271: - Summary: Reduce lock contention for collection types and serializers (was: Reduce lock contention on instance factories of ListType and SetType) > Reduce lock contention for collection types and serializers > --- > > Key: CASSANDRA-13271 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13271 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: vincent royer >Priority: Minor > Labels: performance > Fix For: 4.x > > Attachments: 0001-CASSANDRA-13271-computeIfAbsent.patch, > 0001-CASSANDRA-13271-singleton-factory-concurrency-opimiz.patch > > > By doing some performance tests, i noticed that getInstance() in > org.apache.cassandra.db.marshal.ListType and SetType could suffer from lock > contention on the singleton factory getInstance(). Here is a proposal to > reduce lock contention by using a ConcurrentMap and the putIfAbsent method > rather than a synchronized method. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (CASSANDRA-13291) Replace usages of MessageDigest with Guava's Hasher
[ https://issues.apache.org/jira/browse/CASSANDRA-13291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15894759#comment-15894759 ] Michael Kjellman commented on CASSANDRA-13291: -- >From Hashing.java {code} private static class Md5Holder { static final HashFunction MD5 = new MessageDigestHashFunction("MD5", "Hashing.md5()"); private Md5Holder() { } } {code} And then from MessageDigestHashFunction.java {code} private static MessageDigest getMessageDigest(String algorithmName) { try { return MessageDigest.getInstance(algorithmName); } catch (NoSuchAlgorithmException var2) { throw new AssertionError(var2); } } {code} > Replace usages of MessageDigest with Guava's Hasher > --- > > Key: CASSANDRA-13291 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13291 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: Michael Kjellman >Assignee: Michael Kjellman > Attachments: CASSANDRA-13291-trunk.diff > > > During my profiling of C* I frequently see lots of aggregate time across > threads being spent inside the MD5 MessageDigest implementation. Given that > there are tons of modern alternative hashing functions better than MD5 > available -- both in terms of providing better collision resistance and > actual computational speed -- I wanted to switch out our usage of MD5 for > alternatives (like adler128 or murmur3_128) and test for performance > improvements. > Unfortunately, I found given the fact we use MessageDigest everywhere -- > switching out the hashing function to something like adler128 or murmur3_128 > (for example) -- which don't ship with the JDK -- wasn't straight forward. > The goal of this ticket is to propose switching out usages of MessageDigest > directly in favor of Hasher from Guava. This means going forward we can > change a single line of code to switch the hashing algorithm being used > (assuming there is an implementation in Guava). -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (CASSANDRA-13291) Replace usages of MessageDigest with Guava's Hasher
[ https://issues.apache.org/jira/browse/CASSANDRA-13291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15894754#comment-15894754 ] Robert Stupp commented on CASSANDRA-13291: -- Fair enough. Thought Guava has an alternative MD5 implementation. > Replace usages of MessageDigest with Guava's Hasher > --- > > Key: CASSANDRA-13291 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13291 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: Michael Kjellman >Assignee: Michael Kjellman > Attachments: CASSANDRA-13291-trunk.diff > > > During my profiling of C* I frequently see lots of aggregate time across > threads being spent inside the MD5 MessageDigest implementation. Given that > there are tons of modern alternative hashing functions better than MD5 > available -- both in terms of providing better collision resistance and > actual computational speed -- I wanted to switch out our usage of MD5 for > alternatives (like adler128 or murmur3_128) and test for performance > improvements. > Unfortunately, I found given the fact we use MessageDigest everywhere -- > switching out the hashing function to something like adler128 or murmur3_128 > (for example) -- which don't ship with the JDK -- wasn't straight forward. > The goal of this ticket is to propose switching out usages of MessageDigest > directly in favor of Hasher from Guava. This means going forward we can > change a single line of code to switch the hashing algorithm being used > (assuming there is an implementation in Guava). -- This message was sent by Atlassian JIRA (v6.3.15#6346)
cassandra-builds git commit: Fix ant target in dev branch job
Repository: cassandra-builds Updated Branches: refs/heads/master 481db1991 -> 0c4fc4713 Fix ant target in dev branch job Project: http://git-wip-us.apache.org/repos/asf/cassandra-builds/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra-builds/commit/0c4fc471 Tree: http://git-wip-us.apache.org/repos/asf/cassandra-builds/tree/0c4fc471 Diff: http://git-wip-us.apache.org/repos/asf/cassandra-builds/diff/0c4fc471 Branch: refs/heads/master Commit: 0c4fc4713582ac4d624198620373d89378fd365d Parents: 481db19 Author: Michael Shuler Authored: Fri Mar 3 11:36:32 2017 -0600 Committer: Michael Shuler Committed: Fri Mar 3 11:36:32 2017 -0600 -- jenkins-dsl/cassandra_job_dsl_seed.groovy | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra-builds/blob/0c4fc471/jenkins-dsl/cassandra_job_dsl_seed.groovy -- diff --git a/jenkins-dsl/cassandra_job_dsl_seed.groovy b/jenkins-dsl/cassandra_job_dsl_seed.groovy index a390ba4..857ea0c 100644 --- a/jenkins-dsl/cassandra_job_dsl_seed.groovy +++ b/jenkins-dsl/cassandra_job_dsl_seed.groovy @@ -260,7 +260,7 @@ cassandraBranches.each { /** - * Parameterized Dev Branch `ant testall` + * Parameterized Dev Branch `ant test-all` */ job('Cassandra-devbranch-testall') { description(jobDescription) @@ -295,7 +295,7 @@ job('Cassandra-devbranch-testall') { steps { buildDescription('', buildDescStr) shell("git clean -xdff ; git clone ${buildsRepo}") -shell('./cassandra-builds/build-scripts/cassandra-unittest.sh testall') +shell('./cassandra-builds/build-scripts/cassandra-unittest.sh test-all') } publishers { junit {
[jira] [Commented] (CASSANDRA-13291) Replace usages of MessageDigest with Guava's Hasher
[ https://issues.apache.org/jira/browse/CASSANDRA-13291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15894744#comment-15894744 ] Michael Kjellman commented on CASSANDRA-13291: -- [~snazy] I haven't benchmarked trunk with this patch -- however Guava internally for the md5 implementation calls the exact same thing we were calling -- MessageDigest to get an instance of md5 -- so it should be identical baring the additional overhead for the allocation of the Guava Hasher object wrapper. CASSANDRA-13292 is where we would need to do some benchmarking. > Replace usages of MessageDigest with Guava's Hasher > --- > > Key: CASSANDRA-13291 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13291 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: Michael Kjellman >Assignee: Michael Kjellman > Attachments: CASSANDRA-13291-trunk.diff > > > During my profiling of C* I frequently see lots of aggregate time across > threads being spent inside the MD5 MessageDigest implementation. Given that > there are tons of modern alternative hashing functions better than MD5 > available -- both in terms of providing better collision resistance and > actual computational speed -- I wanted to switch out our usage of MD5 for > alternatives (like adler128 or murmur3_128) and test for performance > improvements. > Unfortunately, I found given the fact we use MessageDigest everywhere -- > switching out the hashing function to something like adler128 or murmur3_128 > (for example) -- which don't ship with the JDK -- wasn't straight forward. > The goal of this ticket is to propose switching out usages of MessageDigest > directly in favor of Hasher from Guava. This means going forward we can > change a single line of code to switch the hashing algorithm being used > (assuming there is an implementation in Guava). -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (CASSANDRA-13294) Possible data loss on upgrade 2.1 - 3.0
[ https://issues.apache.org/jira/browse/CASSANDRA-13294?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcus Eriksson updated CASSANDRA-13294: Status: Patch Available (was: Open) patch that appends the separator when getting sstables: https://github.com/krummas/cassandra/commits/marcuse/13294 http://cassci.datastax.com/view/Dev/view/krummas/job/krummas-marcuse-13294-testall/ http://cassci.datastax.com/view/Dev/view/krummas/job/krummas-marcuse-13294-dtest/ Needs upgrade dtest as well, will work on that > Possible data loss on upgrade 2.1 - 3.0 > --- > > Key: CASSANDRA-13294 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13294 > Project: Cassandra > Issue Type: Bug >Reporter: Marcus Eriksson >Assignee: Marcus Eriksson >Priority: Blocker > Fix For: 3.0.x > > > After finishing a compaction we delete the compacted away files. This is done > [here|https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/db/lifecycle/LogFile.java#L328-L337] > which uses > [this|https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/db/lifecycle/LogRecord.java#L265-L271] > to get the files - we get all files starting with {{absoluteFilePath}}. > Absolute file path is generated > [here|https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/io/sstable/Descriptor.java#L142-L153]. > For 3.0 version files the filename looks like this: > {{/blabla/keyspace1/standard1-bdb031c0ff7b11e6940fdd0479dd8912/mc-1332-big}} > but for 2.1 version files, they look like this: > {{/blabla/keyspace1/standard1-bdb031c0ff7b11e6940fdd0479dd8912/keyspace1-standard1-ka-2}}. > The problem is then that if we were to finish a compaction including the > legacy file, we would actually delete all legacy files having a generation > starting with '2' -- This message was sent by Atlassian JIRA (v6.3.15#6346)
cassandra-builds git commit: Add Parameterized Dev Branch Jobs to DSL
Repository: cassandra-builds Updated Branches: refs/heads/master 2fff2e6b1 -> 481db1991 Add Parameterized Dev Branch Jobs to DSL Project: http://git-wip-us.apache.org/repos/asf/cassandra-builds/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra-builds/commit/481db199 Tree: http://git-wip-us.apache.org/repos/asf/cassandra-builds/tree/481db199 Diff: http://git-wip-us.apache.org/repos/asf/cassandra-builds/diff/481db199 Branch: refs/heads/master Commit: 481db1991136b05b5b6d3ec90da9183a93259df8 Parents: 2fff2e6 Author: Michael Shuler Authored: Fri Mar 3 11:18:37 2017 -0600 Committer: Michael Shuler Committed: Fri Mar 3 11:18:37 2017 -0600 -- jenkins-dsl/cassandra_job_dsl_seed.groovy | 110 - 1 file changed, 109 insertions(+), 1 deletion(-) -- http://git-wip-us.apache.org/repos/asf/cassandra-builds/blob/481db199/jenkins-dsl/cassandra_job_dsl_seed.groovy -- diff --git a/jenkins-dsl/cassandra_job_dsl_seed.groovy b/jenkins-dsl/cassandra_job_dsl_seed.groovy index 4b1dbdd..a390ba4 100644 --- a/jenkins-dsl/cassandra_job_dsl_seed.groovy +++ b/jenkins-dsl/cassandra_job_dsl_seed.groovy @@ -15,6 +15,7 @@ def buildDescStr = 'REF = ${GIT_BRANCH} COMMIT = ${GIT_COMMIT}' def cassandraBranches = ['cassandra-2.2', 'cassandra-3.0', 'cassandra-3.11', 'trunk'] // Ant test targets def testTargets = ['test', 'test-all', 'test-burn', 'test-cdc', 'test-compression'] +// Dtest test targets def dtestTargets = ['dtest', 'dtest-novnode', 'dtest-offheap'] // dtest-large target exists, but no large servers to run on.. @@ -250,6 +251,113 @@ cassandraBranches.each { } } } +} + + +// +// Parameterized Dev Branch Job Definitions +// + -// The End. +/** + * Parameterized Dev Branch `ant testall` + */ +job('Cassandra-devbranch-testall') { +description(jobDescription) +jdk(jdkLabel) +label(slaveLabel) +logRotator { +numToKeep(10) +} +wrappers { +timeout { +noActivity(1200) +} +} +throttleConcurrentBuilds { +categories(['Cassandra']) +} +parameters { +stringParam('REPO', 'apache', 'The github user/org to clone cassandra repo from') +stringParam('BRANCH', 'trunk', 'The branch of cassandra to checkout') +} +scm { +git { +remote { +url('https://github.com/${REPO}/cassandra.git') +} +branch('${BRANCH}') +extensions { +cleanAfterCheckout() +} +} +} +steps { +buildDescription('', buildDescStr) +shell("git clean -xdff ; git clone ${buildsRepo}") +shell('./cassandra-builds/build-scripts/cassandra-unittest.sh testall') +} +publishers { +junit { +testResults('**/TEST-*.xml') +testDataPublishers { +stabilityTestDataPublisher() +} +} +postBuildTask { +task('.', 'echo "Finding job process orphans.."; if pgrep -af ${JOB_BASE_NAME}; then pkill -9 -f ${JOB_BASE_NAME}; fi') +} +} +} + +/** + * Parameterized Dev Branch dtest + */ +job('Cassandra-devbranch-dtest') { +description(jobDescription) +jdk(jdkLabel) +label(slaveLabel) +logRotator { +numToKeep(10) +} +wrappers { +timeout { +noActivity(1200) +} +} +throttleConcurrentBuilds { +categories(['Cassandra']) +} +parameters { +stringParam('REPO', 'apache', 'The github user/org to clone cassandra repo from') +stringParam('BRANCH', 'trunk', 'The branch of cassandra to checkout') +} +scm { +git { +remote { +url('https://github.com/${REPO}/cassandra.git') +} +branch('${BRANCH}') +extensions { +cleanAfterCheckout() +} +} +} +steps { +buildDescription('', buildDescStr) +shell("git clean -xdff ; git clone ${buildsRepo} ; git clone ${dtestRepo}") +shell('./cassandra-builds/build-scripts/cassandra-dtest.sh') +} +publishers { +archiveArtifacts('test_stdout.txt') +junit { +testResults('cassandra-dtest/nosetests.xml') +testDataPublishers { +stabilityTestDataPublisher() +} +} +postBuildTask { +task('.', 'echo "Finding job process orphans.."; if pgrep -af ${JOB_BASE_NAME}; then pkill -9 -f ${JOB_BASE_NAME}; fi') +} +} }
[jira] [Updated] (CASSANDRA-13234) Add histogram for delay to deliver hints
[ https://issues.apache.org/jira/browse/CASSANDRA-13234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Jirsa updated CASSANDRA-13234: --- Fix Version/s: (was: 3.11.x) (was: 3.0.x) 4.0 > Add histogram for delay to deliver hints > > > Key: CASSANDRA-13234 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13234 > Project: Cassandra > Issue Type: Improvement > Components: Observability >Reporter: Jeff Jirsa >Assignee: Jeff Jirsa >Priority: Minor > Fix For: 4.0 > > > There is very little visibility into hint delivery in general - having > histograms available to understand how long it takes to deliver hints is > useful for operators to better identify problems. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (CASSANDRA-13271) Reduce lock contention on instance factories of ListType and SetType
[ https://issues.apache.org/jira/browse/CASSANDRA-13271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15894716#comment-15894716 ] vincent royer commented on CASSANDRA-13271: --- Sorry, this fixed in https://github.com/strapdata/cassandra/tree/13271-trunk > Reduce lock contention on instance factories of ListType and SetType > > > Key: CASSANDRA-13271 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13271 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: vincent royer >Priority: Minor > Labels: performance > Fix For: 4.x > > Attachments: 0001-CASSANDRA-13271-computeIfAbsent.patch, > 0001-CASSANDRA-13271-singleton-factory-concurrency-opimiz.patch > > > By doing some performance tests, i noticed that getInstance() in > org.apache.cassandra.db.marshal.ListType and SetType could suffer from lock > contention on the singleton factory getInstance(). Here is a proposal to > reduce lock contention by using a ConcurrentMap and the putIfAbsent method > rather than a synchronized method. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (CASSANDRA-12888) Incremental repairs broken for MVs and CDC
[ https://issues.apache.org/jira/browse/CASSANDRA-12888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15894645#comment-15894645 ] Benjamin Roth commented on CASSANDRA-12888: --- For detailed explanation an excerpt from that discussion: - ... there are still possible scenarios where it's possible to break consistency by repairing the base and the view separately even with QUORUM writes: Initial state: Base replica A: {k0=v0, ts=0} Base replica B: {k0=v0, ts=0} Base replica C: {k0=v0, ts=0} View paired replica A: {v1=k0, ts=0} View paired replica B: {v0=k0, ts=0} View paired replica C: {v0=k0, ts=0} Base replica A receives write {k1=v1, ts=1}, propagates to view paired replica A and dies. Current state is: Base replica A: {k1=v1, ts=1} Base replica B: {k0=v0, ts=0} Base replica C: {k0=v0, ts=0} View paired replica A: {v1=k1, ts=1} View paired replica B: {v0=k0, ts=0} View paired replica C: {v0=k0, ts=0} Base replica B and C receives write {k2=v2, ts=2}, write to their paired replica. Write is successful at QUORUM. Current state is: Base replica A: {k1=v1, ts=1} Base replica B: {k2=v2, ts=2} Base replica C: {k2=v2, ts=2} View paired replica A: {v1=k1, ts=1} View paired replica B: {v2=k2, ts=2} View paired replica C: {v2=k2, ts=2} A returns from the dead. Repair base table: Base replica A: {k2=v2, ts=2} Base replica B: {k2=v2, ts=2} Base replica C: {k2=v2, ts=2} Repair MV: View paired replica A: {v1=k1, ts=1} and {v2=k2, ts=2} View paired replica B: {v1=k1, ts=1} and {v2=k2, ts=2} View paired replica C: {v1=k1, ts=1} and {v2=k2, ts=2} So, this requires replica A to generate a tombstone for {v1=k1, ts=1} during repair of base table. > Incremental repairs broken for MVs and CDC > -- > > Key: CASSANDRA-12888 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12888 > Project: Cassandra > Issue Type: Bug > Components: Streaming and Messaging >Reporter: Stefan Podkowinski >Assignee: Benjamin Roth >Priority: Critical > Fix For: 3.0.x, 3.11.x > > > SSTables streamed during the repair process will first be written locally and > afterwards either simply added to the pool of existing sstables or, in case > of existing MVs or active CDC, replayed on mutation basis: > As described in {{StreamReceiveTask.OnCompletionRunnable}}: > {quote} > We have a special path for views and for CDC. > For views, since the view requires cleaning up any pre-existing state, we > must put all partitions through the same write path as normal mutations. This > also ensures any 2is are also updated. > For CDC-enabled tables, we want to ensure that the mutations are run through > the CommitLog so they can be archived by the CDC process on discard. > {quote} > Using the regular write path turns out to be an issue for incremental > repairs, as we loose the {{repaired_at}} state in the process. Eventually the > streamed rows will end up in the unrepaired set, in contrast to the rows on > the sender site moved to the repaired set. The next repair run will stream > the same data back again, causing rows to bounce on and on between nodes on > each repair. > See linked dtest on steps to reproduce. An example for reproducing this > manually using ccm can be found > [here|https://gist.github.com/spodkowinski/2d8e0408516609c7ae701f2bf1e515e8] -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (CASSANDRA-12888) Incremental repairs broken for MVs and CDC
[ https://issues.apache.org/jira/browse/CASSANDRA-12888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15894639#comment-15894639 ] Benjamin Roth commented on CASSANDRA-12888: --- A repair must go through the write path expect for some special cases. I also first had the idea to avoid it completely but in discussion with [~pauloricardomg] it turned out that this may introduce inconsistencies that these could only be fixed by a view rebuild because it leaves stale rows. I know that all this stuff is totally counter-intuitive but just streaming "blindly" all sstables (incl. MV tables) down is not correct. This is why I am trying to improve the mutation based approached. If the Sstables for MVs get corrupted or lost, the only way to fix it is to rebuild that view again. There is no way (at least none I see atm) that would consistenly repair a view from other nodes. The underlying principle is: - A view must always be consistent to its base-table - A view does not have to be consistent among nodes, thats handled by repairing the base table Thats also why you don't have to run a repair before building a view. Nevertheless it would not help anyway because you NEVER have a 100% guaranteed consistent state. A repair only guarantees consistency until the point of repair. The "know what you are doing" option is offered by CASSANDRA-13066 btw. In this ticket I also adopted the election of CFs (tables + mvs) when doing a keyspace repair depending if the MV is repaired by stream or by mutation. > Incremental repairs broken for MVs and CDC > -- > > Key: CASSANDRA-12888 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12888 > Project: Cassandra > Issue Type: Bug > Components: Streaming and Messaging >Reporter: Stefan Podkowinski >Assignee: Benjamin Roth >Priority: Critical > Fix For: 3.0.x, 3.11.x > > > SSTables streamed during the repair process will first be written locally and > afterwards either simply added to the pool of existing sstables or, in case > of existing MVs or active CDC, replayed on mutation basis: > As described in {{StreamReceiveTask.OnCompletionRunnable}}: > {quote} > We have a special path for views and for CDC. > For views, since the view requires cleaning up any pre-existing state, we > must put all partitions through the same write path as normal mutations. This > also ensures any 2is are also updated. > For CDC-enabled tables, we want to ensure that the mutations are run through > the CommitLog so they can be archived by the CDC process on discard. > {quote} > Using the regular write path turns out to be an issue for incremental > repairs, as we loose the {{repaired_at}} state in the process. Eventually the > streamed rows will end up in the unrepaired set, in contrast to the rows on > the sender site moved to the repaired set. The next repair run will stream > the same data back again, causing rows to bounce on and on between nodes on > each repair. > See linked dtest on steps to reproduce. An example for reproducing this > manually using ccm can be found > [here|https://gist.github.com/spodkowinski/2d8e0408516609c7ae701f2bf1e515e8] -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (CASSANDRA-12888) Incremental repairs broken for MVs and CDC
[ https://issues.apache.org/jira/browse/CASSANDRA-12888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15894617#comment-15894617 ] Stefan Podkowinski commented on CASSANDRA-12888: If you build from an inconsistent node and some missing data never makes it into a view, regular repairs alone won't be able to fix that. Is that your point? Although I see that this could happen, I'd instead still prefer to just tell the user to do a full repair before creating a view, if that would be a big issue. > Incremental repairs broken for MVs and CDC > -- > > Key: CASSANDRA-12888 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12888 > Project: Cassandra > Issue Type: Bug > Components: Streaming and Messaging >Reporter: Stefan Podkowinski >Assignee: Benjamin Roth >Priority: Critical > Fix For: 3.0.x, 3.11.x > > > SSTables streamed during the repair process will first be written locally and > afterwards either simply added to the pool of existing sstables or, in case > of existing MVs or active CDC, replayed on mutation basis: > As described in {{StreamReceiveTask.OnCompletionRunnable}}: > {quote} > We have a special path for views and for CDC. > For views, since the view requires cleaning up any pre-existing state, we > must put all partitions through the same write path as normal mutations. This > also ensures any 2is are also updated. > For CDC-enabled tables, we want to ensure that the mutations are run through > the CommitLog so they can be archived by the CDC process on discard. > {quote} > Using the regular write path turns out to be an issue for incremental > repairs, as we loose the {{repaired_at}} state in the process. Eventually the > streamed rows will end up in the unrepaired set, in contrast to the rows on > the sender site moved to the repaired set. The next repair run will stream > the same data back again, causing rows to bounce on and on between nodes on > each repair. > See linked dtest on steps to reproduce. An example for reproducing this > manually using ccm can be found > [here|https://gist.github.com/spodkowinski/2d8e0408516609c7ae701f2bf1e515e8] -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (CASSANDRA-12888) Incremental repairs broken for MVs and CDC
[ https://issues.apache.org/jira/browse/CASSANDRA-12888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15894542#comment-15894542 ] T Jake Luciani commented on CASSANDRA-12888: Just to restate the reason we need to put MV base table mutations through the regular write path (vs just streaming the MV and the base) from CASSANDRA-6477 bq. Let's say you put non-PK columns A and B into a Materialized view. Replica 1 has All of Column A, Replica 2 has All of column B. The build would end up with no data in the MV. You would need to subsequentally repair the data to build the MV. > Incremental repairs broken for MVs and CDC > -- > > Key: CASSANDRA-12888 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12888 > Project: Cassandra > Issue Type: Bug > Components: Streaming and Messaging >Reporter: Stefan Podkowinski >Assignee: Benjamin Roth >Priority: Critical > Fix For: 3.0.x, 3.11.x > > > SSTables streamed during the repair process will first be written locally and > afterwards either simply added to the pool of existing sstables or, in case > of existing MVs or active CDC, replayed on mutation basis: > As described in {{StreamReceiveTask.OnCompletionRunnable}}: > {quote} > We have a special path for views and for CDC. > For views, since the view requires cleaning up any pre-existing state, we > must put all partitions through the same write path as normal mutations. This > also ensures any 2is are also updated. > For CDC-enabled tables, we want to ensure that the mutations are run through > the CommitLog so they can be archived by the CDC process on discard. > {quote} > Using the regular write path turns out to be an issue for incremental > repairs, as we loose the {{repaired_at}} state in the process. Eventually the > streamed rows will end up in the unrepaired set, in contrast to the rows on > the sender site moved to the repaired set. The next repair run will stream > the same data back again, causing rows to bounce on and on between nodes on > each repair. > See linked dtest on steps to reproduce. An example for reproducing this > manually using ccm can be found > [here|https://gist.github.com/spodkowinski/2d8e0408516609c7ae701f2bf1e515e8] -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (CASSANDRA-13296) test failure in upgrade_tests.upgrade_through_versions_test.TestUpgrade_current_3_0_x_To_indev_3_x.rolling_upgrade_with_internode_ssl_test
Michael Shuler created CASSANDRA-13296: -- Summary: test failure in upgrade_tests.upgrade_through_versions_test.TestUpgrade_current_3_0_x_To_indev_3_x.rolling_upgrade_with_internode_ssl_test Key: CASSANDRA-13296 URL: https://issues.apache.org/jira/browse/CASSANDRA-13296 Project: Cassandra Issue Type: Bug Reporter: Michael Shuler Attachments: node1_debug.log, node1_gc.log, node1.log, node2_debug.log, node2_gc.log, node2.log, node3_debug.log, node3_gc.log, node3.log example failure: http://cassci.datastax.com/job/cassandra-3.11_large_dtest/22/testReport/upgrade_tests.upgrade_through_versions_test/TestUpgrade_current_3_0_x_To_indev_3_x/rolling_upgrade_with_internode_ssl_test {code} Error Message Ran out of time waiting for queue size (1) to be 'le' to 0. Aborting. Stacktrace File "/usr/lib/python2.7/unittest/case.py", line 329, in run testMethod() File "/home/automaton/cassandra-dtest/upgrade_tests/upgrade_through_versions_test.py", line 291, in rolling_upgrade_with_internode_ssl_test self.upgrade_scenario(rolling=True, internode_ssl=True) File "/home/automaton/cassandra-dtest/upgrade_tests/upgrade_through_versions_test.py", line 356, in upgrade_scenario self._wait_until_queue_condition('writes pending verification', verification_queue, operator.le, 0, max_wait_s=1200) File "/home/automaton/cassandra-dtest/upgrade_tests/upgrade_through_versions_test.py", line 541, in _wait_until_queue_condition raise RuntimeError("Ran out of time waiting for queue size ({}) to be '{}' to {}. Aborting.".format(qsize, opfunc.__name__, required_len)) {code} -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (CASSANDRA-13295) test failure in upgrade_tests.upgrade_through_versions_test.TestUpgrade_current_2_1_x_To_indev_3_x.bootstrap_test
Michael Shuler created CASSANDRA-13295: -- Summary: test failure in upgrade_tests.upgrade_through_versions_test.TestUpgrade_current_2_1_x_To_indev_3_x.bootstrap_test Key: CASSANDRA-13295 URL: https://issues.apache.org/jira/browse/CASSANDRA-13295 Project: Cassandra Issue Type: Bug Reporter: Michael Shuler Attachments: node1_debug.log, node1_gc.log, node1.log, node2_debug.log, node2_gc.log, node2.log, node3_debug.log, node3_gc.log, node3.log, node4_debug.log, node4_gc.log, node4.log example failure: http://cassci.datastax.com/job/cassandra-3.11_large_dtest/22/testReport/upgrade_tests.upgrade_through_versions_test/TestUpgrade_current_2_1_x_To_indev_3_x/bootstrap_test {code} Error Message 03 Mar 2017 12:58:17 [node2] Missing: ['127.0.0.4.* now UP']: INFO [HANDSHAKE-/127.0.0.4] 2017-03-03 12:54:12,1. See system.log for remainder Stacktrace File "/usr/lib/python2.7/unittest/case.py", line 329, in run testMethod() File "/home/automaton/cassandra-dtest/upgrade_tests/upgrade_through_versions_test.py", line 700, in bootstrap_test self.upgrade_scenario(after_upgrade_call=(self._bootstrap_new_node,)) File "/home/automaton/cassandra-dtest/upgrade_tests/upgrade_through_versions_test.py", line 376, in upgrade_scenario call() File "/home/automaton/cassandra-dtest/upgrade_tests/upgrade_through_versions_test.py", line 681, in _bootstrap_new_node nnode.start(use_jna=True, wait_other_notice=240, wait_for_binary_proto=True) File "/home/automaton/venv/local/lib/python2.7/site-packages/ccmlib/node.py", line 686, in start node.watch_log_for_alive(self, from_mark=mark, timeout=wait_other_notice) File "/home/automaton/venv/local/lib/python2.7/site-packages/ccmlib/node.py", line 503, in watch_log_for_alive self.watch_log_for(tofind, from_mark=from_mark, timeout=timeout, filename=filename) File "/home/automaton/venv/local/lib/python2.7/site-packages/ccmlib/node.py", line 471, in watch_log_for raise TimeoutError(time.strftime("%d %b %Y %H:%M:%S", time.gmtime()) + " [" + self.name + "] Missing: " + str([e.pattern for e in tofind]) + ":\n" + reads[:50] + ".\nSee {} for remainder".format(filename)) {code} -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (CASSANDRA-12888) Incremental repairs broken for MVs and CDC
[ https://issues.apache.org/jira/browse/CASSANDRA-12888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15894495#comment-15894495 ] Stefan Podkowinski commented on CASSANDRA-12888: Before moving on in the discussion, I think [~pauloricardomg]'s suggestion still stands and deserves some more thoughts: bq. As discussed on the mailing list and CASSANDRA-12905 we cannot ensure view consistency with sstable-based streaming of base/MVs in all scenarios, specially with repair, so I'm afraid this is not a viable solution just yet (needs to be further discussed). An alternative is to provide an option for repair to allow repairing base and MV separately when you know what you are doing©. Actually having to repair both, base and view table to get back into a consistent state, is something I would expect as a user and something that would happen anyway during my regular repairs, whereas the mutation based approach is probably unheard of by most users and totally unexpected, given how we handled anti-entropy in the past. Although we talk about "views", I think most users will understand that these are actually separate tables and should be handled as that from a operational perspective. So do we really want to introduce additional complexity that may bite us in the back at some point, instead of just relying on regular repairs? > Incremental repairs broken for MVs and CDC > -- > > Key: CASSANDRA-12888 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12888 > Project: Cassandra > Issue Type: Bug > Components: Streaming and Messaging >Reporter: Stefan Podkowinski >Assignee: Benjamin Roth >Priority: Critical > Fix For: 3.0.x, 3.11.x > > > SSTables streamed during the repair process will first be written locally and > afterwards either simply added to the pool of existing sstables or, in case > of existing MVs or active CDC, replayed on mutation basis: > As described in {{StreamReceiveTask.OnCompletionRunnable}}: > {quote} > We have a special path for views and for CDC. > For views, since the view requires cleaning up any pre-existing state, we > must put all partitions through the same write path as normal mutations. This > also ensures any 2is are also updated. > For CDC-enabled tables, we want to ensure that the mutations are run through > the CommitLog so they can be archived by the CDC process on discard. > {quote} > Using the regular write path turns out to be an issue for incremental > repairs, as we loose the {{repaired_at}} state in the process. Eventually the > streamed rows will end up in the unrepaired set, in contrast to the rows on > the sender site moved to the repaired set. The next repair run will stream > the same data back again, causing rows to bounce on and on between nodes on > each repair. > See linked dtest on steps to reproduce. An example for reproducing this > manually using ccm can be found > [here|https://gist.github.com/spodkowinski/2d8e0408516609c7ae701f2bf1e515e8] -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (CASSANDRA-13223) Unable to compute when histogram overflowed
[ https://issues.apache.org/jira/browse/CASSANDRA-13223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15894447#comment-15894447 ] Vladimir Bukhtoyarov commented on CASSANDRA-13223: -- Yes I threat overflown overflow bucket as normal value. {quote} This could potentially end up with a flooded log files. {quote} I want to speculate that logs should not be overflown because it is not really to execute big amount of very slow operations at rate which enough to flood. But I can implement simple non-blocking throttling, in order to log no often then *N* times per second(or other period). I suppose an implementation of rate limiter already exists in Cassandra code base, if not then implementation of throttling based on "Leaky Bucket" is very trivial. {quote} An option to consider would be to count the overflow bucket as Long.MAX_VALUE (or some other approximation between maximum register value and Long.MAX_VALUE). Just a thought. {quote} It is good idea. The anomaly big value on the graphs will be a very good alarm. > Unable to compute when histogram overflowed > --- > > Key: CASSANDRA-13223 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13223 > Project: Cassandra > Issue Type: Bug >Reporter: Vladimir Bukhtoyarov >Priority: Minor > > DecayingEstimatedHistogramReservoir throws exception when value upper max > recorded to reservoir. It is very undesired behavior, because functionality > like logging or monitoring should never fail with exception. Current behavior > of DecayingEstimatedHistogramReservoir violates contract for > [Reservoir|https://github.com/dropwizard/metrics/blob/3.2-development/metrics-core/src/main/java/com/codahale/metrics/Reservoir.java], > as you can see javadocs for Reservoir says nothing that implementation can > throw exception in getSnapshot method. As result all Dropwizzard/Metrics > reporters are broken, because nobody expect that metric will throw exception > on get, for example our monitoring pipeline is broken with exception: > {noformat} > com.fasterxml.jackson.databind.JsonMappingException: Unable to compute when > histogram overflowed (through reference chain: > java.util.UnmodifiableSortedMap["org.apache.cassandra.metrics.Table > .ColUpdateTimeDeltaHistogram.all"]) > at > com.fasterxml.jackson.databind.JsonMappingException.wrapWithPath(JsonMappingException.java:339) > at > com.fasterxml.jackson.databind.JsonMappingException.wrapWithPath(JsonMappingException.java:299) > at > com.fasterxml.jackson.databind.ser.std.StdSerializer.wrapAndThrow(StdSerializer.java:342) > at > com.fasterxml.jackson.databind.ser.std.MapSerializer.serializeFields(MapSerializer.java:620) > at > com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize(MapSerializer.java:519) > at > com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize(MapSerializer.java:31) > at > com.fasterxml.jackson.databind.ser.DefaultSerializerProvider.serializeValue(DefaultSerializerProvider.java:130) > at > com.fasterxml.jackson.databind.ObjectMapper.writeValue(ObjectMapper.java:2436) > at > com.fasterxml.jackson.core.base.GeneratorBase.writeObject(GeneratorBase.java:355) > at > com.fasterxml.jackson.core.JsonGenerator.writeObjectField(JsonGenerator.java:1442) > at > com.codahale.metrics.json.MetricsModule$MetricRegistrySerializer.serialize(MetricsModule.java:188) > at > com.codahale.metrics.json.MetricsModule$MetricRegistrySerializer.serialize(MetricsModule.java:171) > at > com.fasterxml.jackson.databind.ser.DefaultSerializerProvider.serializeValue(DefaultSerializerProvider.java:130) > at > com.fasterxml.jackson.databind.ObjectWriter$Prefetch.serialize(ObjectWriter.java:1428) > at > com.fasterxml.jackson.databind.ObjectWriter._configAndWriteValue(ObjectWriter.java:1129) > at > com.fasterxml.jackson.databind.ObjectWriter.writeValue(ObjectWriter.java:967) > at > com.codahale.metrics.servlets.MetricsServlet.doGet(MetricsServlet.java:176) > at javax.servlet.http.HttpServlet.service(HttpServlet.java:687) > at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) > at > org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:845) > at > org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1689) > at > com.ringcentral.slf4j.CleanMDCFilter.doFilter(CleanMDCFilter.java:18) > at > org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676) > at > org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581) > at > org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandl
[jira] [Commented] (CASSANDRA-13234) Add histogram for delay to deliver hints
[ https://issues.apache.org/jira/browse/CASSANDRA-13234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15894277#comment-15894277 ] Aleksey Yeschenko commented on CASSANDRA-13234: --- Going by the book, all of this should go to trunk only, not being a bug fix, by the way. > Add histogram for delay to deliver hints > > > Key: CASSANDRA-13234 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13234 > Project: Cassandra > Issue Type: Improvement > Components: Observability >Reporter: Jeff Jirsa >Assignee: Jeff Jirsa >Priority: Minor > Fix For: 3.0.x, 3.11.x > > > There is very little visibility into hint delivery in general - having > histograms available to understand how long it takes to deliver hints is > useful for operators to better identify problems. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (CASSANDRA-13289) Make it possible to monitor an ideal consistency level separate from actual consistency level
[ https://issues.apache.org/jira/browse/CASSANDRA-13289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15894255#comment-15894255 ] Romain Hardouin commented on CASSANDRA-13289: - Very interesting! Do you have an idea of the overhead that this monitoring layer would add? If this adds too much overhead we should be able to enable/disable it at runtime via nodetool. > Make it possible to monitor an ideal consistency level separate from actual > consistency level > - > > Key: CASSANDRA-13289 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13289 > Project: Cassandra > Issue Type: Bug > Components: Core >Reporter: Ariel Weisberg >Assignee: Ariel Weisberg > > As an operator there are several issues related to multi-datacenter > replication and consistency you may want to have more information on from > your production database. > For instance. If your application writes at LOCAL_QUORUM how often are those > writes failing to achieve EACH_QUORUM at other data centers. If you failed > your application over to one of those data centers roughly how inconsistent > might it be given the number of writes that didn't propagate since the last > incremental repair? > You might also want to know roughly what the latency of writes would be if > you switched to a different consistency level. For instance you are writing > at LOCAL_QUORUM and want to know what would happen if you switched to > EACH_QUORUM. > The proposed change is to allow an ideal_consistency_level to be specified in > cassandra.yaml as well as get/set via JMX. If no ideal consistency level is > specified no additional tracking is done. > if an ideal consistency level is specified then the > {{AbstractWriteResponesHandler}} will contain a delegate WriteResponseHandler > that tracks whether the ideal consistency level is met before a write times > out. It also tracks the latency for achieving the ideal CL of successful > writes. > These two metrics would be reported on a per keyspace basis. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (CASSANDRA-13223) Unable to compute when histogram overflowed
[ https://issues.apache.org/jira/browse/CASSANDRA-13223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15894242#comment-15894242 ] Per Otterström commented on CASSANDRA-13223: bq. I do not ignore big latency record, I write the maximum value of resolution Right, I missed that. Basically you are merging the count of the highest counting bucket with the overflow bucket? It would be nice with some logging on this. The downside I see with that is that this code is part of many critical code paths, including write-path and read-path. This could potentially end up with a flooded log files. An option to consider would be to count the overflow bucket as Long.MAX_VALUE (or some other approximation between maximum register value and Long.MAX_VALUE). Just a thought. Then values would really stand out in graphs, and you would know where to look. The existing solution with IllegalStateException is actually derived from [EstimatedHistogram|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/utils/EstimatedHistogram.java#L231]. I don't know the full background story on the original behavior. [~brandon.williams], do you know if there is a particular reason for throwing the IllegalStateException, or would you be OK with the changes proposed here? > Unable to compute when histogram overflowed > --- > > Key: CASSANDRA-13223 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13223 > Project: Cassandra > Issue Type: Bug >Reporter: Vladimir Bukhtoyarov >Priority: Minor > > DecayingEstimatedHistogramReservoir throws exception when value upper max > recorded to reservoir. It is very undesired behavior, because functionality > like logging or monitoring should never fail with exception. Current behavior > of DecayingEstimatedHistogramReservoir violates contract for > [Reservoir|https://github.com/dropwizard/metrics/blob/3.2-development/metrics-core/src/main/java/com/codahale/metrics/Reservoir.java], > as you can see javadocs for Reservoir says nothing that implementation can > throw exception in getSnapshot method. As result all Dropwizzard/Metrics > reporters are broken, because nobody expect that metric will throw exception > on get, for example our monitoring pipeline is broken with exception: > {noformat} > com.fasterxml.jackson.databind.JsonMappingException: Unable to compute when > histogram overflowed (through reference chain: > java.util.UnmodifiableSortedMap["org.apache.cassandra.metrics.Table > .ColUpdateTimeDeltaHistogram.all"]) > at > com.fasterxml.jackson.databind.JsonMappingException.wrapWithPath(JsonMappingException.java:339) > at > com.fasterxml.jackson.databind.JsonMappingException.wrapWithPath(JsonMappingException.java:299) > at > com.fasterxml.jackson.databind.ser.std.StdSerializer.wrapAndThrow(StdSerializer.java:342) > at > com.fasterxml.jackson.databind.ser.std.MapSerializer.serializeFields(MapSerializer.java:620) > at > com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize(MapSerializer.java:519) > at > com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize(MapSerializer.java:31) > at > com.fasterxml.jackson.databind.ser.DefaultSerializerProvider.serializeValue(DefaultSerializerProvider.java:130) > at > com.fasterxml.jackson.databind.ObjectMapper.writeValue(ObjectMapper.java:2436) > at > com.fasterxml.jackson.core.base.GeneratorBase.writeObject(GeneratorBase.java:355) > at > com.fasterxml.jackson.core.JsonGenerator.writeObjectField(JsonGenerator.java:1442) > at > com.codahale.metrics.json.MetricsModule$MetricRegistrySerializer.serialize(MetricsModule.java:188) > at > com.codahale.metrics.json.MetricsModule$MetricRegistrySerializer.serialize(MetricsModule.java:171) > at > com.fasterxml.jackson.databind.ser.DefaultSerializerProvider.serializeValue(DefaultSerializerProvider.java:130) > at > com.fasterxml.jackson.databind.ObjectWriter$Prefetch.serialize(ObjectWriter.java:1428) > at > com.fasterxml.jackson.databind.ObjectWriter._configAndWriteValue(ObjectWriter.java:1129) > at > com.fasterxml.jackson.databind.ObjectWriter.writeValue(ObjectWriter.java:967) > at > com.codahale.metrics.servlets.MetricsServlet.doGet(MetricsServlet.java:176) > at javax.servlet.http.HttpServlet.service(HttpServlet.java:687) > at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) > at > org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:845) > at > org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1689) > at > com.ringcentral.slf4j.CleanMDCFilter.doFilter(CleanMDCFilter.java:18) > at > org.eclipse.je
[jira] [Commented] (CASSANDRA-13271) Reduce lock contention on instance factories of ListType and SetType
[ https://issues.apache.org/jira/browse/CASSANDRA-13271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15894204#comment-15894204 ] Robert Stupp commented on CASSANDRA-13271: -- The change of the type of the parameter {{isMultiCell}} in {{ListType.getInstace()}} from primitive {{boolean}} to {{Boolean}} doesn't make sense. ||trunk|[branch|https://github.com/apache/cassandra/compare/trunk...snazy:13271-trunk]|[testall|http://cassci.datastax.com/view/Dev/view/snazy/job/snazy-13271-trunk-testall/lastSuccessfulBuild/]|[dtest|http://cassci.datastax.com/view/Dev/view/snazy/job/snazy-13271-trunk-dtest/lastSuccessfulBuild/] When the above issue is resolved, I think we can commit this one. > Reduce lock contention on instance factories of ListType and SetType > > > Key: CASSANDRA-13271 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13271 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: vincent royer >Priority: Minor > Labels: performance > Fix For: 4.x > > Attachments: 0001-CASSANDRA-13271-computeIfAbsent.patch, > 0001-CASSANDRA-13271-singleton-factory-concurrency-opimiz.patch > > > By doing some performance tests, i noticed that getInstance() in > org.apache.cassandra.db.marshal.ListType and SetType could suffer from lock > contention on the singleton factory getInstance(). Here is a proposal to > reduce lock contention by using a ConcurrentMap and the putIfAbsent method > rather than a synchronized method. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (CASSANDRA-11670) Rebuilding or streaming MV generates mutations larger than max_mutation_size_in_kb
[ https://issues.apache.org/jira/browse/CASSANDRA-11670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15894128#comment-15894128 ] Benjamin Roth commented on CASSANDRA-11670: --- Hmmm I guess this possibly breaks consistency in repair streams + MVs In StreamReceiveTasks mutations are applied without commitlog because cf is flushed at the end. But MVs are not flushed. Either: Also flush all MVs at the end of the stream task - it is not said that this is actually required for all MVs as we do not know where view replica updates eventually go. Or: Enable commitlog for view replica updates even if base table does not commit log writes. > Rebuilding or streaming MV generates mutations larger than > max_mutation_size_in_kb > -- > > Key: CASSANDRA-11670 > URL: https://issues.apache.org/jira/browse/CASSANDRA-11670 > Project: Cassandra > Issue Type: Bug > Components: Configuration, Streaming and Messaging >Reporter: Anastasia Osintseva >Assignee: Paulo Motta > Fix For: 3.0.10, 3.10 > > > I have in cluster 2 DC, in each DC - 2 Nodes. I wanted to add 1 node to each > DC. One node has been added successfully after I had made scrubing. > Now I'm trying to add node to another DC, but get error: > org.apache.cassandra.streaming.StreamException: Stream failed. > After scrubing and repair I get the same error. > {noformat} > ERROR [StreamReceiveTask:5] 2016-04-27 00:33:21,082 Keyspace.java:492 - > Unknown exception caught while attempting to update MaterializedView! > messages_dump.messages > java.lang.IllegalArgumentException: Mutation of 34974901 bytes is too large > for the maxiumum size of 33554432 > at org.apache.cassandra.db.commitlog.CommitLog.add(CommitLog.java:264) > ~[apache-cassandra-3.0.5.jar:3.0.5] > at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:469) > [apache-cassandra-3.0.5.jar:3.0.5] > at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:384) > [apache-cassandra-3.0.5.jar:3.0.5] > at org.apache.cassandra.db.Mutation.applyFuture(Mutation.java:205) > [apache-cassandra-3.0.5.jar:3.0.5] > at org.apache.cassandra.db.Mutation.apply(Mutation.java:217) > [apache-cassandra-3.0.5.jar:3.0.5] > at > org.apache.cassandra.batchlog.BatchlogManager.store(BatchlogManager.java:146) > ~[apache-cassandra-3.0.5.jar:3.0.5] > at > org.apache.cassandra.service.StorageProxy.mutateMV(StorageProxy.java:724) > ~[apache-cassandra-3.0.5.jar:3.0.5] > at > org.apache.cassandra.db.view.ViewManager.pushViewReplicaUpdates(ViewManager.java:149) > ~[apache-cassandra-3.0.5.jar:3.0.5] > at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:487) > [apache-cassandra-3.0.5.jar:3.0.5] > at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:384) > [apache-cassandra-3.0.5.jar:3.0.5] > at org.apache.cassandra.db.Mutation.applyFuture(Mutation.java:205) > [apache-cassandra-3.0.5.jar:3.0.5] > at org.apache.cassandra.db.Mutation.apply(Mutation.java:217) > [apache-cassandra-3.0.5.jar:3.0.5] > at org.apache.cassandra.db.Mutation.applyUnsafe(Mutation.java:236) > [apache-cassandra-3.0.5.jar:3.0.5] > at > org.apache.cassandra.streaming.StreamReceiveTask$OnCompletionRunnable.run(StreamReceiveTask.java:169) > [apache-cassandra-3.0.5.jar:3.0.5] > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > [na:1.8.0_11] > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > [na:1.8.0_11] > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > [na:1.8.0_11] > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > [na:1.8.0_11] > at java.lang.Thread.run(Thread.java:745) [na:1.8.0_11] > ERROR [StreamReceiveTask:5] 2016-04-27 00:33:21,082 > StreamReceiveTask.java:214 - Error applying streamed data: > java.lang.IllegalArgumentException: Mutation of 34974901 bytes is too large > for the maxiumum size of 33554432 > at org.apache.cassandra.db.commitlog.CommitLog.add(CommitLog.java:264) > ~[apache-cassandra-3.0.5.jar:3.0.5] > at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:469) > ~[apache-cassandra-3.0.5.jar:3.0.5] > at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:384) > ~[apache-cassandra-3.0.5.jar:3.0.5] > at org.apache.cassandra.db.Mutation.applyFuture(Mutation.java:205) > ~[apache-cassandra-3.0.5.jar:3.0.5] > at org.apache.cassandra.db.Mutation.apply(Mutation.java:217) > ~[apache-cassandra-3.0.5.jar:3.0.5] > at > org.apache.cassandra.batchlog.BatchlogManager.store(BatchlogManager.java:146) > ~[apache-cassandra-3.0.5.jar:3.0.5] > at > org.apache.cassandra.service.StorageProxy.mut
[jira] [Updated] (CASSANDRA-13294) Possible data loss on upgrade 2.1 - 3.0
[ https://issues.apache.org/jira/browse/CASSANDRA-13294?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcus Eriksson updated CASSANDRA-13294: Reviewer: Stefania > Possible data loss on upgrade 2.1 - 3.0 > --- > > Key: CASSANDRA-13294 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13294 > Project: Cassandra > Issue Type: Bug >Reporter: Marcus Eriksson >Assignee: Marcus Eriksson >Priority: Blocker > Fix For: 3.0.x > > > After finishing a compaction we delete the compacted away files. This is done > [here|https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/db/lifecycle/LogFile.java#L328-L337] > which uses > [this|https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/db/lifecycle/LogRecord.java#L265-L271] > to get the files - we get all files starting with {{absoluteFilePath}}. > Absolute file path is generated > [here|https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/io/sstable/Descriptor.java#L142-L153]. > For 3.0 version files the filename looks like this: > {{/blabla/keyspace1/standard1-bdb031c0ff7b11e6940fdd0479dd8912/mc-1332-big}} > but for 2.1 version files, they look like this: > {{/blabla/keyspace1/standard1-bdb031c0ff7b11e6940fdd0479dd8912/keyspace1-standard1-ka-2}}. > The problem is then that if we were to finish a compaction including the > legacy file, we would actually delete all legacy files having a generation > starting with '2' -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (CASSANDRA-13291) Replace usages of MessageDigest with Guava's Hasher
[ https://issues.apache.org/jira/browse/CASSANDRA-13291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15894050#comment-15894050 ] Robert Stupp commented on CASSANDRA-13291: -- [~mkjellman] can you provide some micro benchmarks for both implementations? > Replace usages of MessageDigest with Guava's Hasher > --- > > Key: CASSANDRA-13291 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13291 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: Michael Kjellman >Assignee: Michael Kjellman > Attachments: CASSANDRA-13291-trunk.diff > > > During my profiling of C* I frequently see lots of aggregate time across > threads being spent inside the MD5 MessageDigest implementation. Given that > there are tons of modern alternative hashing functions better than MD5 > available -- both in terms of providing better collision resistance and > actual computational speed -- I wanted to switch out our usage of MD5 for > alternatives (like adler128 or murmur3_128) and test for performance > improvements. > Unfortunately, I found given the fact we use MessageDigest everywhere -- > switching out the hashing function to something like adler128 or murmur3_128 > (for example) -- which don't ship with the JDK -- wasn't straight forward. > The goal of this ticket is to propose switching out usages of MessageDigest > directly in favor of Hasher from Guava. This means going forward we can > change a single line of code to switch the hashing algorithm being used > (assuming there is an implementation in Guava). -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (CASSANDRA-13294) Possible data loss on upgrade 2.1 - 3.0
Marcus Eriksson created CASSANDRA-13294: --- Summary: Possible data loss on upgrade 2.1 - 3.0 Key: CASSANDRA-13294 URL: https://issues.apache.org/jira/browse/CASSANDRA-13294 Project: Cassandra Issue Type: Bug Reporter: Marcus Eriksson Assignee: Marcus Eriksson Priority: Blocker Fix For: 3.0.x After finishing a compaction we delete the compacted away files. This is done [here|https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/db/lifecycle/LogFile.java#L328-L337] which uses [this|https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/db/lifecycle/LogRecord.java#L265-L271] to get the files - we get all files starting with {{absoluteFilePath}}. Absolute file path is generated [here|https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/io/sstable/Descriptor.java#L142-L153]. For 3.0 version files the filename looks like this: {{/blabla/keyspace1/standard1-bdb031c0ff7b11e6940fdd0479dd8912/mc-1332-big}} but for 2.1 version files, they look like this: {{/blabla/keyspace1/standard1-bdb031c0ff7b11e6940fdd0479dd8912/keyspace1-standard1-ka-2}}. The problem is then that if we were to finish a compaction including the legacy file, we would actually delete all legacy files having a generation starting with '2' -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Comment Edited] (CASSANDRA-12888) Incremental repairs broken for MVs and CDC
[ https://issues.apache.org/jira/browse/CASSANDRA-12888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15818955#comment-15818955 ] Benjamin Roth edited comment on CASSANDRA-12888 at 3/3/17 9:57 AM: --- Hi Victor, We use MVs in Production with billions of records without known data loss. Painful + slow refers to repairs and range movements (e.g. bootstrap + decommission). Also (as mentioned in this ticket) incremental repairs dont work, so full repair creates some overhead. Until 3.10 there are bugs leading to write timeouts, even to NPEs and completely blocked mutation stages. This could even bring your cluster down. In 3.10 some issues have been resolved - actually we use a patched trunk version which is 1-2 months old. Depending on your model, MVs can help a lot from a developer perspective. Some cases are very resource intensive to manage without MVs, requiring distributed locks and/or CAS. For append-only workloads, it may be simpler to NOT use MVs at the moment. They aren't very complex and MVs wont help that much compared to the problems that may raise with them. Painful scenarios: There is no recipe for that. You may or may not encounter performance issues, depending on your model and your workload. I'd recommend not to use MVs that use a different partition key on the MV than on the base table as this requires inter-node communication for EVERY write operation. So you can easily kill your cluster with bulk operations (like in streaming). At the moment our cluster runs stable but it took months to find all the bottlenecks, race conditions, resume from failures and so on. So my recommendation: You can get it work but you need time and you should not start with critical data, at least if it is not backed by another stable storage. And you should use 3.10 when it is finally released or build your own version from trunk. I would not recommend to use < 3.10 for MVs. Btw.: Our own patched version does some dirty tricks, that may lead to inconsistencies in some situations but we prefer some possible inconsistencies (we can deal with) over performance bottlenecks. I created several tickets to improve MV performance in some streaming situations but it will take some time to really improve that situation. Does this answer your question? was (Author: brstgt): Hi Victor, We use MVs in Production with billions of records without known data loss. Painful + slow refers to repairs and range movements (e.g. bootstrap + decommission). Also (as mentioned in this ticket) incremental repairs dont work, so full repair creates some overhead. Until 3.10 there are bugs leading to write timeouts, even to NPEs and completely blocked mutation stages. This could even bring your cluster down. In 3.10 some issues have been resolved - actually we use a patched trunk version which is 1-2 months old. Depending on your model, MVs can help a lot from a developer perspective. Some cases are very resource intensive to manage without MVs, requiring distributed locks and/or CAS. For append-only workloads, it may be simpler to NOT use MVs at the moment. They aren't very complex and MVs wont help that much compared to the problems that may raise with them. Painful scenarios: There is no recipe for that. You may or may not encounter performance issues, depending on your model and your workload. I'd recommend not to use MVs that use a different partition key on the MV than on the base table as this requires inter-node communication for EVERY write operation. So you can easily kill your cluster with bulk operations (like in streaming). At the moment our cluster runs stable but it took months to find all the bottlenecks, race conditions, resume from failures and so on. So my recommendation: You can get it work but you need time and you should not start with critical data, at least if it is not backed by another stable storage. And you should use 3.10 when it is finally released or build your own version from trunk. I would not recommend to use < 3.10 for MVs. Btw.: Our own patched version does some dirty tricks, that may lead to inconsistencies in some situations but we prefer some possible inconsistencies (we can deal with) over performance bottlenecks. I created several tickets to improve MV performance in some streaming situations but it will take some time to really improve that situation. Does this answer your question? -- Benjamin Roth Prokurist Jaumo GmbH · www.jaumo.com Wehrstraße 46 · 73035 Göppingen · Germany Phone +49 7161 304880-6 · Fax +49 7161 304880-1 AG Ulm · HRB 731058 · Managing Director: Jens Kammerer > Incremental repairs broken for MVs and CDC > -- > > Key: CASSANDRA-12888 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12888 > Project: Cassandra > Issue Type: Bug > Components: Str
[jira] [Comment Edited] (CASSANDRA-12888) Incremental repairs broken for MVs and CDC
[ https://issues.apache.org/jira/browse/CASSANDRA-12888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15894006#comment-15894006 ] Benjamin Roth edited comment on CASSANDRA-12888 at 3/3/17 9:55 AM: --- I am about to hack a proof of concept for that issue. Concept: Each mutation and each partition update have a "repairedAt" flag. This will be passed along through the whole write path like MV updates and serialization for remote MV updates. Then repair + non-repair mutations have to be separated in memtables and flushed to separate SSTables. From what I can see it should be easier to maintain a memtable each for repaired and non-repaired data than tracking the repair state within a memtable. Passing repair state to replicas isn't even necessary as replicas should not be repaired directly anyway, so no need for a repairedAt state. My question is: How important is the exact value of "repairedAt". Is it possible to merge updates with different repair timestamps into a single memtable and finally flush them to an SSTable with repairedAt set to the latest or earliest repairedAt timestamps of all mutations in the memtable? Or would that produce repair-inconsistencies or sth? Any feedback? was (Author: brstgt): I am about to hack a proof of concept for that issue. Concept: Each mutation and each partition update have a "repairedAt" flag. This will be passed along through the whole write path like MV updates and serialization for remote MV updates. Then repair + non-repair mutations have to be separated in memtables and flushed to separate SSTables. From what I can see it should be easier to maintain a memtable each for repaired and non-repaired data than tracking the repair state within a memtable. My question is: How important is the exact value of "repairedAt". Is it possible to merge updates with different repair timestamps into a single memtable and finally flush them to an SSTable with repairedAt set to the latest or earliest repairedAt timestamps of all mutations in the memtable? Or would that produce repair-inconsistencies or sth? Any feedback? > Incremental repairs broken for MVs and CDC > -- > > Key: CASSANDRA-12888 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12888 > Project: Cassandra > Issue Type: Bug > Components: Streaming and Messaging >Reporter: Stefan Podkowinski >Assignee: Benjamin Roth >Priority: Critical > Fix For: 3.0.x, 3.11.x > > > SSTables streamed during the repair process will first be written locally and > afterwards either simply added to the pool of existing sstables or, in case > of existing MVs or active CDC, replayed on mutation basis: > As described in {{StreamReceiveTask.OnCompletionRunnable}}: > {quote} > We have a special path for views and for CDC. > For views, since the view requires cleaning up any pre-existing state, we > must put all partitions through the same write path as normal mutations. This > also ensures any 2is are also updated. > For CDC-enabled tables, we want to ensure that the mutations are run through > the CommitLog so they can be archived by the CDC process on discard. > {quote} > Using the regular write path turns out to be an issue for incremental > repairs, as we loose the {{repaired_at}} state in the process. Eventually the > streamed rows will end up in the unrepaired set, in contrast to the rows on > the sender site moved to the repaired set. The next repair run will stream > the same data back again, causing rows to bounce on and on between nodes on > each repair. > See linked dtest on steps to reproduce. An example for reproducing this > manually using ccm can be found > [here|https://gist.github.com/spodkowinski/2d8e0408516609c7ae701f2bf1e515e8] -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (CASSANDRA-12888) Incremental repairs broken for MVs and CDC
[ https://issues.apache.org/jira/browse/CASSANDRA-12888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15894006#comment-15894006 ] Benjamin Roth commented on CASSANDRA-12888: --- I am about to hack a proof of concept for that issue. Concept: Each mutation and each partition update have a "repairedAt" flag. This will be passed along through the whole write path like MV updates and serialization for remote MV updates. Then repair + non-repair mutations have to be separated in memtables and flushed to separate SSTables. From what I can see it should be easier to maintain a memtable each for repaired and non-repaired data than tracking the repair state within a memtable. My question is: How important is the exact value of "repairedAt". Is it possible to merge updates with different repair timestamps into a single memtable and finally flush them to an SSTable with repairedAt set to the latest or earliest repairedAt timestamps of all mutations in the memtable? Or would that produce repair-inconsistencies or sth? Any feedback? > Incremental repairs broken for MVs and CDC > -- > > Key: CASSANDRA-12888 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12888 > Project: Cassandra > Issue Type: Bug > Components: Streaming and Messaging >Reporter: Stefan Podkowinski >Assignee: Benjamin Roth >Priority: Critical > Fix For: 3.0.x, 3.11.x > > > SSTables streamed during the repair process will first be written locally and > afterwards either simply added to the pool of existing sstables or, in case > of existing MVs or active CDC, replayed on mutation basis: > As described in {{StreamReceiveTask.OnCompletionRunnable}}: > {quote} > We have a special path for views and for CDC. > For views, since the view requires cleaning up any pre-existing state, we > must put all partitions through the same write path as normal mutations. This > also ensures any 2is are also updated. > For CDC-enabled tables, we want to ensure that the mutations are run through > the CommitLog so they can be archived by the CDC process on discard. > {quote} > Using the regular write path turns out to be an issue for incremental > repairs, as we loose the {{repaired_at}} state in the process. Eventually the > streamed rows will end up in the unrepaired set, in contrast to the rows on > the sender site moved to the repaired set. The next repair run will stream > the same data back again, causing rows to bounce on and on between nodes on > each repair. > See linked dtest on steps to reproduce. An example for reproducing this > manually using ccm can be found > [here|https://gist.github.com/spodkowinski/2d8e0408516609c7ae701f2bf1e515e8] -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (CASSANDRA-13265) Expiration in OutboundTcpConnection can block the reader Thread
[ https://issues.apache.org/jira/browse/CASSANDRA-13265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15894000#comment-15894000 ] Christian Esken commented on CASSANDRA-13265: - Your "nit" totally makes sense, as {{nanoTime()}} is expensive if called very often. Good catch, actually I should have seen that myself. I committed the change, including the required change to RetriedQueuedMessage. I also removed two warnings by adding the generic types. As a side note, one could even use an estimated time here, like EstimatorTimeSource which works with a regularly updated cached time. I was able to boost the performance of Triava Cache by factor 2 - 10 depending on the test. An example is in https://github.com/trivago/triava/blob/master/src/examples/java/com/trivago/examples/TimeSourceExample.java, estimatorTimeSource(). > Expiration in OutboundTcpConnection can block the reader Thread > --- > > Key: CASSANDRA-13265 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13265 > Project: Cassandra > Issue Type: Bug > Environment: Cassandra 3.0.9 > Java HotSpot(TM) 64-Bit Server VM version 25.112-b15 (Java version > 1.8.0_112-b15) > Linux 3.16 >Reporter: Christian Esken >Assignee: Christian Esken > Attachments: cassandra.pb-cache4-dus.2017-02-17-19-36-26.chist.xz, > cassandra.pb-cache4-dus.2017-02-17-19-36-26.td.xz > > > I observed that sometimes a single node in a Cassandra cluster fails to > communicate to the other nodes. This can happen at any time, during peak load > or low load. Restarting that single node from the cluster fixes the issue. > Before going in to details, I want to state that I have analyzed the > situation and am already developing a possible fix. Here is the analysis so > far: > - A Threaddump in this situation showed 324 Threads in the > OutboundTcpConnection class that want to lock the backlog queue for doing > expiration. > - A class histogram shows 262508 instances of > OutboundTcpConnection$QueuedMessage. > What is the effect of it? As soon as the Cassandra node has reached a certain > amount of queued messages, it starts thrashing itself to death. Each of the > Thread fully locks the Queue for reading and writing by calling > iterator.next(), making the situation worse and worse. > - Writing: Only after 262508 locking operation it can progress with actually > writing to the Queue. > - Reading: Is also blocked, as 324 Threads try to do iterator.next(), and > fully lock the Queue > This means: Writing blocks the Queue for reading, and readers might even be > starved which makes the situation even worse. > - > The setup is: > - 3-node cluster > - replication factor 2 > - Consistency LOCAL_ONE > - No remote DC's > - high write throughput (10 INSERT statements per second and more during > peak times). > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (CASSANDRA-13066) Fast streaming with materialized views
[ https://issues.apache.org/jira/browse/CASSANDRA-13066?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benjamin Roth updated CASSANDRA-13066: -- Fix Version/s: 4.0 Status: Patch Available (was: Open) Depends upon 13064+13065 https://github.com/Jaumo/cassandra/commits/CASSANDRA-13066 > Fast streaming with materialized views > -- > > Key: CASSANDRA-13066 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13066 > Project: Cassandra > Issue Type: Improvement >Reporter: Benjamin Roth >Assignee: Benjamin Roth > Fix For: 4.0 > > > I propose adding a configuration option to send streams of tables with MVs > not through the regular write path. > This may be either a global option or better a CF option. > Background: > A repair of a CF with an MV that is much out of sync creates many streams. > These streams all go through the regular write path to assert local > consistency of the MV. This again causes a read before write for every single > mutation which again puts a lot of pressure on the node - much more than > simply streaming the SSTable down. > In some cases this can be avoided. Instead of only repairing the base table, > all base + mv tables would have to be repaired. But this can break eventual > consistency between base table and MV. The proposed behaviour is always safe, > when having append-only MVs. It also works when using CL_QUORUM writes but it > cannot be absolutely guaranteed, that a quorum write is applied atomically, > so this can also lead to inconsistencies, if a quorum write is started but > one node dies in the middle of a request. > So, this proposal can help a lot in some situations but also can break > consistency in others. That's why it should be left upon the operator if that > behaviour is appropriate for individual use cases. > This issue came up here: > https://issues.apache.org/jira/browse/CASSANDRA-12888?focusedCommentId=15736599&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15736599 -- This message was sent by Atlassian JIRA (v6.3.15#6346)