[jira] [Updated] (CASSANDRA-13072) Cassandra failed to run on Linux-aarch64
[
https://issues.apache.org/jira/browse/CASSANDRA-13072?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alex Petrov updated CASSANDRA-13072:
Resolution: Fixed
Status: Resolved (was: Ready to Commit)
> Cassandra failed to run on Linux-aarch64
>
>
> Key: CASSANDRA-13072
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13072
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: Hardware: ARM aarch64
> OS: Ubuntu 16.04.1 LTS
>Reporter: Jun He
>Assignee: Benjamin Lerer
> Labels: incompatible
> Fix For: 4.0, 3.11.0, 3.0.14
>
> Attachments: compat_report.html
>
>
> Steps to reproduce:
> 1. Download cassandra latest source
> 2. Build it with "ant"
> 3. Run with "./bin/cassandra". Daemon is crashed with following error message:
> {quote}
> INFO 05:30:21 Initializing system.schema_functions
> INFO 05:30:21 Initializing system.schema_aggregates
> ERROR 05:30:22 Exception in thread Thread[MemtableFlushWriter:1,5,main]
> java.lang.NoClassDefFoundError: Could not initialize class com.sun.jna.Native
> at
> org.apache.cassandra.utils.memory.MemoryUtil.allocate(MemoryUtil.java:97)
> ~[main/:na]
> at org.apache.cassandra.io.util.Memory.(Memory.java:74)
> ~[main/:na]
> at org.apache.cassandra.io.util.SafeMemory.(SafeMemory.java:32)
> ~[main/:na]
> at
> org.apache.cassandra.io.compress.CompressionMetadata$Writer.(CompressionMetadata.java:316)
> ~[main/:na]
> at
> org.apache.cassandra.io.compress.CompressionMetadata$Writer.open(CompressionMetadata.java:330)
> ~[main/:na]
> at
> org.apache.cassandra.io.compress.CompressedSequentialWriter.(CompressedSequentialWriter.java:76)
> ~[main/:na]
> at
> org.apache.cassandra.io.util.SequentialWriter.open(SequentialWriter.java:163)
> ~[main/:na]
> at
> org.apache.cassandra.io.sstable.format.big.BigTableWriter.(BigTableWriter.java:73)
> ~[main/:na]
> at
> org.apache.cassandra.io.sstable.format.big.BigFormat$WriterFactory.open(BigFormat.java:93)
> ~[main/:na]
> at
> org.apache.cassandra.io.sstable.format.SSTableWriter.create(SSTableWriter.java:96)
> ~[main/:na]
> at
> org.apache.cassandra.io.sstable.SimpleSSTableMultiWriter.create(SimpleSSTableMultiWriter.java:114)
> ~[main/:na]
> at
> org.apache.cassandra.db.compaction.AbstractCompactionStrategy.createSSTableMultiWriter(AbstractCompactionStrategy.java:519)
> ~[main/:na]
> at
> org.apache.cassandra.db.compaction.CompactionStrategyManager.createSSTableMultiWriter(CompactionStrategyManager.java:497)
> ~[main/:na]
> at
> org.apache.cassandra.db.ColumnFamilyStore.createSSTableMultiWriter(ColumnFamilyStore.java:480)
> ~[main/:na]
> at
> org.apache.cassandra.db.Memtable.createFlushWriter(Memtable.java:439)
> ~[main/:na]
> at
> org.apache.cassandra.db.Memtable.writeSortedContents(Memtable.java:371)
> ~[main/:na]
> at org.apache.cassandra.db.Memtable.flush(Memtable.java:332)
> ~[main/:na]
> at
> org.apache.cassandra.db.ColumnFamilyStore$Flush.run(ColumnFamilyStore.java:1054)
> ~[main/:na]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> ~[na:1.8.0_111]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> ~[na:1.8.0_111]
> at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_111]
> {quote}
> Analyze:
> This issue is caused by bundled jna-4.0.0.jar which doesn't come with aarch64
> native support. Replace lib/jna-4.0.0.jar with jna-4.2.0.jar from
> http://central.maven.org/maven2/net/java/dev/jna/jna/4.2.0/ can fix this
> problem.
> Attached is the binary compatibility report of jna.jar between 4.0 and 4.2.
> The result is good (97.4%). So is there possibility to upgrade jna to 4.2.0
> in upstream? Should there be any kind of tests to execute, please kindly
> point me. Thanks a lot.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13072) Cassandra failed to run on Linux-aarch64
[
https://issues.apache.org/jira/browse/CASSANDRA-13072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16091196#comment-16091196
]
Alex Petrov commented on CASSANDRA-13072:
-
Committed to 3.0 as
[00a777ec8ab701b843172e23a6cbdc4d6cf48f8d|https://github.com/apache/cassandra/commit/00a777ec8ab701b843172e23a6cbdc4d6cf48f8d],
merged up to
[3.11|https://github.com/apache/cassandra/commit/478d54db4611278863a1736faf3c0b4c92f4412b]
and
[trunk|https://github.com/apache/cassandra/commit/fd065714003a8820cb4acf3521b8df5c726ef4b2]
> Cassandra failed to run on Linux-aarch64
>
>
> Key: CASSANDRA-13072
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13072
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: Hardware: ARM aarch64
> OS: Ubuntu 16.04.1 LTS
>Reporter: Jun He
>Assignee: Benjamin Lerer
> Labels: incompatible
> Fix For: 3.0.14, 3.11.0, 4.0
>
> Attachments: compat_report.html
>
>
> Steps to reproduce:
> 1. Download cassandra latest source
> 2. Build it with "ant"
> 3. Run with "./bin/cassandra". Daemon is crashed with following error message:
> {quote}
> INFO 05:30:21 Initializing system.schema_functions
> INFO 05:30:21 Initializing system.schema_aggregates
> ERROR 05:30:22 Exception in thread Thread[MemtableFlushWriter:1,5,main]
> java.lang.NoClassDefFoundError: Could not initialize class com.sun.jna.Native
> at
> org.apache.cassandra.utils.memory.MemoryUtil.allocate(MemoryUtil.java:97)
> ~[main/:na]
> at org.apache.cassandra.io.util.Memory.(Memory.java:74)
> ~[main/:na]
> at org.apache.cassandra.io.util.SafeMemory.(SafeMemory.java:32)
> ~[main/:na]
> at
> org.apache.cassandra.io.compress.CompressionMetadata$Writer.(CompressionMetadata.java:316)
> ~[main/:na]
> at
> org.apache.cassandra.io.compress.CompressionMetadata$Writer.open(CompressionMetadata.java:330)
> ~[main/:na]
> at
> org.apache.cassandra.io.compress.CompressedSequentialWriter.(CompressedSequentialWriter.java:76)
> ~[main/:na]
> at
> org.apache.cassandra.io.util.SequentialWriter.open(SequentialWriter.java:163)
> ~[main/:na]
> at
> org.apache.cassandra.io.sstable.format.big.BigTableWriter.(BigTableWriter.java:73)
> ~[main/:na]
> at
> org.apache.cassandra.io.sstable.format.big.BigFormat$WriterFactory.open(BigFormat.java:93)
> ~[main/:na]
> at
> org.apache.cassandra.io.sstable.format.SSTableWriter.create(SSTableWriter.java:96)
> ~[main/:na]
> at
> org.apache.cassandra.io.sstable.SimpleSSTableMultiWriter.create(SimpleSSTableMultiWriter.java:114)
> ~[main/:na]
> at
> org.apache.cassandra.db.compaction.AbstractCompactionStrategy.createSSTableMultiWriter(AbstractCompactionStrategy.java:519)
> ~[main/:na]
> at
> org.apache.cassandra.db.compaction.CompactionStrategyManager.createSSTableMultiWriter(CompactionStrategyManager.java:497)
> ~[main/:na]
> at
> org.apache.cassandra.db.ColumnFamilyStore.createSSTableMultiWriter(ColumnFamilyStore.java:480)
> ~[main/:na]
> at
> org.apache.cassandra.db.Memtable.createFlushWriter(Memtable.java:439)
> ~[main/:na]
> at
> org.apache.cassandra.db.Memtable.writeSortedContents(Memtable.java:371)
> ~[main/:na]
> at org.apache.cassandra.db.Memtable.flush(Memtable.java:332)
> ~[main/:na]
> at
> org.apache.cassandra.db.ColumnFamilyStore$Flush.run(ColumnFamilyStore.java:1054)
> ~[main/:na]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> ~[na:1.8.0_111]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> ~[na:1.8.0_111]
> at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_111]
> {quote}
> Analyze:
> This issue is caused by bundled jna-4.0.0.jar which doesn't come with aarch64
> native support. Replace lib/jna-4.0.0.jar with jna-4.2.0.jar from
> http://central.maven.org/maven2/net/java/dev/jna/jna/4.2.0/ can fix this
> problem.
> Attached is the binary compatibility report of jna.jar between 4.0 and 4.2.
> The result is good (97.4%). So is there possibility to upgrade jna to 4.2.0
> in upstream? Should there be any kind of tests to execute, please kindly
> point me. Thanks a lot.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[4/6] cassandra git commit: Merge branch 'cassandra-3.0' into cassandra-3.11
Merge branch 'cassandra-3.0' into cassandra-3.11 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/478d54db Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/478d54db Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/478d54db Branch: refs/heads/trunk Commit: 478d54db4611278863a1736faf3c0b4c92f4412b Parents: eb717f1 00a777e Author: Alex Petrov Authored: Tue Jul 18 08:36:07 2017 +0200 Committer: Alex Petrov Committed: Tue Jul 18 08:36:07 2017 +0200 -- build.xml | 2 +- lib/jna-4.2.2.jar | Bin 0 -> 1137286 bytes lib/jna-4.4.0.jar | Bin 1091208 -> 0 bytes lib/licenses/jna-4.2.2.txt | 202 lib/licenses/jna-4.4.0.txt | 202 5 files changed, 203 insertions(+), 203 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/478d54db/build.xml -- - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[6/6] cassandra git commit: Merge branch 'cassandra-3.11' into trunk
Merge branch 'cassandra-3.11' into trunk Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/fd065714 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/fd065714 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/fd065714 Branch: refs/heads/trunk Commit: fd065714003a8820cb4acf3521b8df5c726ef4b2 Parents: 6eb2758 478d54d Author: Alex Petrov Authored: Tue Jul 18 08:36:18 2017 +0200 Committer: Alex Petrov Committed: Tue Jul 18 08:36:18 2017 +0200 -- build.xml | 2 +- lib/jna-4.2.2.jar | Bin 0 -> 1137286 bytes lib/jna-4.4.0.jar | Bin 1091208 -> 0 bytes lib/licenses/jna-4.2.2.txt | 202 lib/licenses/jna-4.4.0.txt | 202 5 files changed, 203 insertions(+), 203 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/fd065714/build.xml -- - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[3/6] cassandra git commit: Downgrade JNA to 4.2.2
Downgrade JNA to 4.2.2 Patch by Benjamin Lerer; reviewed by Alex Petrov for CASSANDRA-13072 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/00a777ec Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/00a777ec Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/00a777ec Branch: refs/heads/trunk Commit: 00a777ec8ab701b843172e23a6cbdc4d6cf48f8d Parents: a033f51 Author: Benjamin Lerer Authored: Mon Jul 10 11:59:08 2017 +0200 Committer: Alex Petrov Committed: Tue Jul 18 08:35:59 2017 +0200 -- build.xml | 2 +- lib/jna-4.2.2.jar | Bin 0 -> 1137286 bytes lib/jna-4.4.0.jar | Bin 1091208 -> 0 bytes lib/licenses/jna-4.2.2.txt | 202 lib/licenses/jna-4.4.0.txt | 202 5 files changed, 203 insertions(+), 203 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/00a777ec/build.xml -- diff --git a/build.xml b/build.xml index 53c2cea..f07060d 100644 --- a/build.xml +++ b/build.xml @@ -381,7 +381,7 @@ - + http://git-wip-us.apache.org/repos/asf/cassandra/blob/00a777ec/lib/jna-4.2.2.jar -- diff --git a/lib/jna-4.2.2.jar b/lib/jna-4.2.2.jar new file mode 100644 index 000..a943670 Binary files /dev/null and b/lib/jna-4.2.2.jar differ http://git-wip-us.apache.org/repos/asf/cassandra/blob/00a777ec/lib/jna-4.4.0.jar -- diff --git a/lib/jna-4.4.0.jar b/lib/jna-4.4.0.jar deleted file mode 100644 index 521bd92..000 Binary files a/lib/jna-4.4.0.jar and /dev/null differ http://git-wip-us.apache.org/repos/asf/cassandra/blob/00a777ec/lib/licenses/jna-4.2.2.txt -- diff --git a/lib/licenses/jna-4.2.2.txt b/lib/licenses/jna-4.2.2.txt new file mode 100644 index 000..7a4a3ea --- /dev/null +++ b/lib/licenses/jna-4.2.2.txt @@ -0,0 +1,202 @@ + + Apache License + Version 2.0, January 2004 +http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally +
[5/6] cassandra git commit: Merge branch 'cassandra-3.0' into cassandra-3.11
Merge branch 'cassandra-3.0' into cassandra-3.11 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/478d54db Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/478d54db Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/478d54db Branch: refs/heads/cassandra-3.11 Commit: 478d54db4611278863a1736faf3c0b4c92f4412b Parents: eb717f1 00a777e Author: Alex Petrov Authored: Tue Jul 18 08:36:07 2017 +0200 Committer: Alex Petrov Committed: Tue Jul 18 08:36:07 2017 +0200 -- build.xml | 2 +- lib/jna-4.2.2.jar | Bin 0 -> 1137286 bytes lib/jna-4.4.0.jar | Bin 1091208 -> 0 bytes lib/licenses/jna-4.2.2.txt | 202 lib/licenses/jna-4.4.0.txt | 202 5 files changed, 203 insertions(+), 203 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/478d54db/build.xml -- - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[2/6] cassandra git commit: Downgrade JNA to 4.2.2
Downgrade JNA to 4.2.2 Patch by Benjamin Lerer; reviewed by Alex Petrov for CASSANDRA-13072 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/00a777ec Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/00a777ec Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/00a777ec Branch: refs/heads/cassandra-3.11 Commit: 00a777ec8ab701b843172e23a6cbdc4d6cf48f8d Parents: a033f51 Author: Benjamin Lerer Authored: Mon Jul 10 11:59:08 2017 +0200 Committer: Alex Petrov Committed: Tue Jul 18 08:35:59 2017 +0200 -- build.xml | 2 +- lib/jna-4.2.2.jar | Bin 0 -> 1137286 bytes lib/jna-4.4.0.jar | Bin 1091208 -> 0 bytes lib/licenses/jna-4.2.2.txt | 202 lib/licenses/jna-4.4.0.txt | 202 5 files changed, 203 insertions(+), 203 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/00a777ec/build.xml -- diff --git a/build.xml b/build.xml index 53c2cea..f07060d 100644 --- a/build.xml +++ b/build.xml @@ -381,7 +381,7 @@ - + http://git-wip-us.apache.org/repos/asf/cassandra/blob/00a777ec/lib/jna-4.2.2.jar -- diff --git a/lib/jna-4.2.2.jar b/lib/jna-4.2.2.jar new file mode 100644 index 000..a943670 Binary files /dev/null and b/lib/jna-4.2.2.jar differ http://git-wip-us.apache.org/repos/asf/cassandra/blob/00a777ec/lib/jna-4.4.0.jar -- diff --git a/lib/jna-4.4.0.jar b/lib/jna-4.4.0.jar deleted file mode 100644 index 521bd92..000 Binary files a/lib/jna-4.4.0.jar and /dev/null differ http://git-wip-us.apache.org/repos/asf/cassandra/blob/00a777ec/lib/licenses/jna-4.2.2.txt -- diff --git a/lib/licenses/jna-4.2.2.txt b/lib/licenses/jna-4.2.2.txt new file mode 100644 index 000..7a4a3ea --- /dev/null +++ b/lib/licenses/jna-4.2.2.txt @@ -0,0 +1,202 @@ + + Apache License + Version 2.0, January 2004 +http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionall
[1/6] cassandra git commit: Downgrade JNA to 4.2.2
Repository: cassandra Updated Branches: refs/heads/cassandra-3.0 a033f5165 -> 00a777ec8 refs/heads/cassandra-3.11 eb717f154 -> 478d54db4 refs/heads/trunk 6eb2758e4 -> fd0657140 Downgrade JNA to 4.2.2 Patch by Benjamin Lerer; reviewed by Alex Petrov for CASSANDRA-13072 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/00a777ec Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/00a777ec Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/00a777ec Branch: refs/heads/cassandra-3.0 Commit: 00a777ec8ab701b843172e23a6cbdc4d6cf48f8d Parents: a033f51 Author: Benjamin Lerer Authored: Mon Jul 10 11:59:08 2017 +0200 Committer: Alex Petrov Committed: Tue Jul 18 08:35:59 2017 +0200 -- build.xml | 2 +- lib/jna-4.2.2.jar | Bin 0 -> 1137286 bytes lib/jna-4.4.0.jar | Bin 1091208 -> 0 bytes lib/licenses/jna-4.2.2.txt | 202 lib/licenses/jna-4.4.0.txt | 202 5 files changed, 203 insertions(+), 203 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/00a777ec/build.xml -- diff --git a/build.xml b/build.xml index 53c2cea..f07060d 100644 --- a/build.xml +++ b/build.xml @@ -381,7 +381,7 @@ - + http://git-wip-us.apache.org/repos/asf/cassandra/blob/00a777ec/lib/jna-4.2.2.jar -- diff --git a/lib/jna-4.2.2.jar b/lib/jna-4.2.2.jar new file mode 100644 index 000..a943670 Binary files /dev/null and b/lib/jna-4.2.2.jar differ http://git-wip-us.apache.org/repos/asf/cassandra/blob/00a777ec/lib/jna-4.4.0.jar -- diff --git a/lib/jna-4.4.0.jar b/lib/jna-4.4.0.jar deleted file mode 100644 index 521bd92..000 Binary files a/lib/jna-4.4.0.jar and /dev/null differ http://git-wip-us.apache.org/repos/asf/cassandra/blob/00a777ec/lib/licenses/jna-4.2.2.txt -- diff --git a/lib/licenses/jna-4.2.2.txt b/lib/licenses/jna-4.2.2.txt new file mode 100644 index 000..7a4a3ea --- /dev/null +++ b/lib/licenses/jna-4.2.2.txt @@ -0,0 +1,202 @@ + + Apache License + Version 2.0, January 2004 +http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall
[jira] [Commented] (CASSANDRA-13561) Purge TTL on expiration
[ https://issues.apache.org/jira/browse/CASSANDRA-13561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16091165#comment-16091165 ] Kurt Greaves commented on CASSANDRA-13561: -- You are right. CASSANDRA-13418 is a different issue to what you are talking about. It is a specific option for allowing expiration of only completely expired SSTables where there are overlaps with another SSTable. My understanding is that you are saying we should purge TTL'd cells as soon as they expire, rather than waiting for GCGS to pass, but obviously we would still require a compaction to make that happen. Is that correct? In theory it would be nice but I think once you take into account that an update that sets a TTL may not make it to all nodes, you have a problem. For example: 3 nodes A B and C, RF=3 An INSERT without any TTL's set for PK "a" makes it to all 3 nodes B goes down An update setting a single column "b" to TTL after x seconds for PK "a" is sent. B comes back up (disregarding HH) A and C are now in a state with "b" having a TTL while B has no TTL for the same cell. "b" expires on A and C, a compaction is triggered and the cell gets removed. "b" returns from the dead from B. > Purge TTL on expiration > --- > > Key: CASSANDRA-13561 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13561 > Project: Cassandra > Issue Type: New Feature >Reporter: Andrew Whang >Assignee: Andrew Whang >Priority: Minor > Fix For: 4.0 > > > Tables with mostly TTL columns tend to suffer from high droppable tombstone > ratio, which results in higher read latency, cpu utilization, and disk usage. > Expired TTL data become tombstones, and the nature of purging tombstones > during compaction (due to checking for overlapping SSTables) make them > susceptible to surviving much longer than expected. A table option to purge > TTL on expiration would address this issue, by preventing them from becoming > tombstones. A boolean purge_ttl_on_expiration table setting would allow users > to easily turn the feature on or off. > Being more aggressive with gc_grace could also address the problem of long > lasting tombstones, but that would affect tombstones from deletes as well. > Even if a purged [expired] cell is revived via repair from a node that hasn't > yet compacted away the cell, it would be revived as an expiring cell with the > same localDeletionTime, so reads should properly handle them. As well, it > would be purged in the next compaction. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13696) Digest mismatch Exception if hints file has UnknownColumnFamily
[
https://issues.apache.org/jira/browse/CASSANDRA-13696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16091154#comment-16091154
]
Jay Zhuang commented on CASSANDRA-13696:
Sure, updated the test for trunk:
| Branch | uTest |
| [13696-3.0|https://github.com/cooldoger/cassandra/tree/13696-3.0] |
[circleci#21|https://circleci.com/gh/cooldoger/cassandra/21]|
| [13696-3.11|https://github.com/cooldoger/cassandra/tree/13696-3.11] |
[circleci#24|https://circleci.com/gh/cooldoger/cassandra/24]|
| [13696-trunk|https://github.com/cooldoger/cassandra/tree/13696-trunk] |
[circleci#25|https://circleci.com/gh/cooldoger/cassandra/25]|
Yes, I think it's a serious problem that all nodes go down at the same time.
And to recover, the user has to delete all hints file (Not sure if there's an
easy way to identify the "corrupted" hints files, system.log only shows the
first one. So for us, we just delete all the hints files.)
> Digest mismatch Exception if hints file has UnknownColumnFamily
> ---
>
> Key: CASSANDRA-13696
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13696
> Project: Cassandra
> Issue Type: Bug
> Components: Core
>Reporter: Jay Zhuang
>Assignee: Jay Zhuang
>Priority: Critical
>
> {noformat}
> WARN [HintsDispatcher:2] 2017-07-16 22:00:32,579 HintsReader.java:235 -
> Failed to read a hint for /127.0.0.2: a2b7daf1-a6a4-4dfc-89de-32d12d2d48b0 -
> table with id 3882bbb0-6a71-11e7-9bca-2759083e3964 is unknown in file
> a2b7daf1-a6a4-4dfc-89de-32d12d2d48b0-1500242103097-1.hints
> ERROR [HintsDispatcher:2] 2017-07-16 22:00:32,580
> HintsDispatchExecutor.java:234 - Failed to dispatch hints file
> a2b7daf1-a6a4-4dfc-89de-32d12d2d48b0-1500242103097-1.hints: file is corrupted
> ({})
> org.apache.cassandra.io.FSReadError: java.io.IOException: Digest mismatch
> exception
> at
> org.apache.cassandra.hints.HintsReader$HintsIterator.computeNext(HintsReader.java:199)
> ~[main/:na]
> at
> org.apache.cassandra.hints.HintsReader$HintsIterator.computeNext(HintsReader.java:164)
> ~[main/:na]
> at
> org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
> ~[main/:na]
> at
> org.apache.cassandra.hints.HintsDispatcher.sendHints(HintsDispatcher.java:157)
> ~[main/:na]
> at
> org.apache.cassandra.hints.HintsDispatcher.sendHintsAndAwait(HintsDispatcher.java:139)
> ~[main/:na]
> at
> org.apache.cassandra.hints.HintsDispatcher.dispatch(HintsDispatcher.java:123)
> ~[main/:na]
> at
> org.apache.cassandra.hints.HintsDispatcher.dispatch(HintsDispatcher.java:95)
> ~[main/:na]
> at
> org.apache.cassandra.hints.HintsDispatchExecutor$DispatchHintsTask.deliver(HintsDispatchExecutor.java:268)
> [main/:na]
> at
> org.apache.cassandra.hints.HintsDispatchExecutor$DispatchHintsTask.dispatch(HintsDispatchExecutor.java:251)
> [main/:na]
> at
> org.apache.cassandra.hints.HintsDispatchExecutor$DispatchHintsTask.dispatch(HintsDispatchExecutor.java:229)
> [main/:na]
> at
> org.apache.cassandra.hints.HintsDispatchExecutor$DispatchHintsTask.run(HintsDispatchExecutor.java:208)
> [main/:na]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> [na:1.8.0_111]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_111]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> [na:1.8.0_111]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [na:1.8.0_111]
> at
> org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:79)
> [main/:na]
> at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_111]
> Caused by: java.io.IOException: Digest mismatch exception
> at
> org.apache.cassandra.hints.HintsReader$HintsIterator.computeNextInternal(HintsReader.java:216)
> ~[main/:na]
> at
> org.apache.cassandra.hints.HintsReader$HintsIterator.computeNext(HintsReader.java:190)
> ~[main/:na]
> ... 16 common frames omitted
> {noformat}
> It causes multiple cassandra nodes stop [by
> default|https://github.com/apache/cassandra/blob/cassandra-3.0/conf/cassandra.yaml#L188].
> Here is the reproduce steps on a 3 nodes cluster, RF=3:
> 1. stop node1
> 2. send some data with quorum (or one), it will generate hints file on
> node2/node3
> 3. drop the table
> 4. start node1
> node2/node3 will report "corrupted hints file" and stop. The impact is very
> bad for a large cluster, when it happens, almost all the nodes are down at
> the same time and we have to remove all the hints files (which contain the
> dropped table) to bring the node back.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (CASSANDRA-12965) StreamReceiveTask causing high CPU utilization during repair
[
https://issues.apache.org/jira/browse/CASSANDRA-12965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16091147#comment-16091147
]
Jeff Jirsa commented on CASSANDRA-12965:
Relating to CASSANDRA-11303 , which is a "rethink inbound streaming throughput
throttle" ticket, which would let us better tune this sort of behavior.
> StreamReceiveTask causing high CPU utilization during repair
>
>
> Key: CASSANDRA-12965
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12965
> Project: Cassandra
> Issue Type: Bug
>Reporter: Randy Fradin
>
> During a full repair run, I observed one node in my cluster using 100% cpu
> (100% of all cores on a 48-core machine). When I took a stack trace I found
> exactly 48 running StreamReceiveTask threads. Each was in the same block of
> code in StreamReceiveTask.OnCompletionRunnable:
> {noformat}
> "StreamReceiveTask:8077" #1511134 daemon prio=5 os_prio=0
> tid=0x7f01520a8800 nid=0x6e77 runnable [0x7f020dfae000]
>java.lang.Thread.State: RUNNABLE
> at java.util.ComparableTimSort.binarySort(ComparableTimSort.java:258)
> at java.util.ComparableTimSort.sort(ComparableTimSort.java:203)
> at java.util.Arrays.sort(Arrays.java:1312)
> at java.util.Arrays.sort(Arrays.java:1506)
> at java.util.ArrayList.sort(ArrayList.java:1454)
> at java.util.Collections.sort(Collections.java:141)
> at
> org.apache.cassandra.utils.IntervalTree$IntervalNode.(IntervalTree.java:257)
> at
> org.apache.cassandra.utils.IntervalTree$IntervalNode.(IntervalTree.java:280)
> at
> org.apache.cassandra.utils.IntervalTree.(IntervalTree.java:72)
> at
> org.apache.cassandra.db.DataTracker$SSTableIntervalTree.(DataTracker.java:590)
> at
> org.apache.cassandra.db.DataTracker$SSTableIntervalTree.(DataTracker.java:584)
> at
> org.apache.cassandra.db.DataTracker.buildIntervalTree(DataTracker.java:565)
> at
> org.apache.cassandra.db.DataTracker$View.replace(DataTracker.java:761)
> at
> org.apache.cassandra.db.DataTracker.addSSTablesToTracker(DataTracker.java:428)
> at
> org.apache.cassandra.db.DataTracker.addSSTables(DataTracker.java:283)
> at
> org.apache.cassandra.db.ColumnFamilyStore.addSSTables(ColumnFamilyStore.java:1422)
> at
> org.apache.cassandra.streaming.StreamReceiveTask$OnCompletionRunnable.run(StreamReceiveTask.java:148)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
> All 48 threads were in ColumnFamilyStore.addSSTables(), and specifically in
> the IntervalNode constructor called from the IntervalTree constructor.
> It stayed this way for maybe an hour before we restarted the node. The repair
> was also generating thousands (20,000+) of tiny SSTables in a table that
> previously had just 20.
> I don't know enough about SSTables and ColumnFamilyStore to know if all this
> CPU work is necessary or a bug, but I did notice that these tasks are run on
> a thread pool constructed in StreamReceiveTask.java, so perhaps this pool
> should have a thread count max less than the number of processors on the
> machine, at least for machines with a lot of processors. Any reason not to do
> that? Any ideas for a reasonable # or formula to cap the thread count?
> Some additional info: We have never run incremental repair on this cluster,
> so that is not a factor. All our tables use LCS. Unfortunately I don't have
> the log files from the period saved.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13418) Allow TWCS to ignore overlaps when dropping fully expired sstables
[ https://issues.apache.org/jira/browse/CASSANDRA-13418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16091082#comment-16091082 ] mck commented on CASSANDRA-13418: - [~rgerard], are you looking to add any unit tests to [~krummas]'s patch? And have you also had any chance to do any manual testing? > Allow TWCS to ignore overlaps when dropping fully expired sstables > -- > > Key: CASSANDRA-13418 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13418 > Project: Cassandra > Issue Type: Improvement > Components: Compaction >Reporter: Corentin Chary > Labels: twcs > Attachments: twcs-cleanup.png > > > http://thelastpickle.com/blog/2016/12/08/TWCS-part1.html explains it well. If > you really want read-repairs you're going to have sstables blocking the > expiration of other fully expired SSTables because they overlap. > You can set unchecked_tombstone_compaction = true or tombstone_threshold to a > very low value and that will purge the blockers of old data that should > already have expired, thus removing the overlaps and allowing the other > SSTables to expire. > The thing is that this is rather CPU intensive and not optimal. If you have > time series, you might not care if all your data doesn't exactly expire at > the right time, or if data re-appears for some time, as long as it gets > deleted as soon as it can. And in this situation I believe it would be really > beneficial to allow users to simply ignore overlapping SSTables when looking > for fully expired ones. > To the question: why would you need read-repairs ? > - Full repairs basically take longer than the TTL of the data on my dataset, > so this isn't really effective. > - Even with a 10% chances of doing a repair, we found out that this would be > enough to greatly reduce entropy of the most used data (and if you have > timeseries, you're likely to have a dashboard doing the same important > queries over and over again). > - LOCAL_QUORUM is too expensive (need >3 replicas), QUORUM is too slow. > I'll try to come up with a patch demonstrating how this would work, try it on > our system and report the effects. > cc: [~adejanovski], [~rgerard] as I know you worked on similar issues already. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-12965) StreamReceiveTask causing high CPU utilization during repair
[
https://issues.apache.org/jira/browse/CASSANDRA-12965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16090973#comment-16090973
]
Randy Fradin commented on CASSANDRA-12965:
--
We set -Dcassandra.available_processors=(some number less than the # of cores
on the host) as suggested by lieangsibin. It limits the size of several thread
pools including this one. Not exactly a fix but at least prevents Cassandra
from monopolizing all of the CPU resources.
> StreamReceiveTask causing high CPU utilization during repair
>
>
> Key: CASSANDRA-12965
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12965
> Project: Cassandra
> Issue Type: Bug
>Reporter: Randy Fradin
>
> During a full repair run, I observed one node in my cluster using 100% cpu
> (100% of all cores on a 48-core machine). When I took a stack trace I found
> exactly 48 running StreamReceiveTask threads. Each was in the same block of
> code in StreamReceiveTask.OnCompletionRunnable:
> {noformat}
> "StreamReceiveTask:8077" #1511134 daemon prio=5 os_prio=0
> tid=0x7f01520a8800 nid=0x6e77 runnable [0x7f020dfae000]
>java.lang.Thread.State: RUNNABLE
> at java.util.ComparableTimSort.binarySort(ComparableTimSort.java:258)
> at java.util.ComparableTimSort.sort(ComparableTimSort.java:203)
> at java.util.Arrays.sort(Arrays.java:1312)
> at java.util.Arrays.sort(Arrays.java:1506)
> at java.util.ArrayList.sort(ArrayList.java:1454)
> at java.util.Collections.sort(Collections.java:141)
> at
> org.apache.cassandra.utils.IntervalTree$IntervalNode.(IntervalTree.java:257)
> at
> org.apache.cassandra.utils.IntervalTree$IntervalNode.(IntervalTree.java:280)
> at
> org.apache.cassandra.utils.IntervalTree.(IntervalTree.java:72)
> at
> org.apache.cassandra.db.DataTracker$SSTableIntervalTree.(DataTracker.java:590)
> at
> org.apache.cassandra.db.DataTracker$SSTableIntervalTree.(DataTracker.java:584)
> at
> org.apache.cassandra.db.DataTracker.buildIntervalTree(DataTracker.java:565)
> at
> org.apache.cassandra.db.DataTracker$View.replace(DataTracker.java:761)
> at
> org.apache.cassandra.db.DataTracker.addSSTablesToTracker(DataTracker.java:428)
> at
> org.apache.cassandra.db.DataTracker.addSSTables(DataTracker.java:283)
> at
> org.apache.cassandra.db.ColumnFamilyStore.addSSTables(ColumnFamilyStore.java:1422)
> at
> org.apache.cassandra.streaming.StreamReceiveTask$OnCompletionRunnable.run(StreamReceiveTask.java:148)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
> All 48 threads were in ColumnFamilyStore.addSSTables(), and specifically in
> the IntervalNode constructor called from the IntervalTree constructor.
> It stayed this way for maybe an hour before we restarted the node. The repair
> was also generating thousands (20,000+) of tiny SSTables in a table that
> previously had just 20.
> I don't know enough about SSTables and ColumnFamilyStore to know if all this
> CPU work is necessary or a bug, but I did notice that these tasks are run on
> a thread pool constructed in StreamReceiveTask.java, so perhaps this pool
> should have a thread count max less than the number of processors on the
> machine, at least for machines with a lot of processors. Any reason not to do
> that? Any ideas for a reasonable # or formula to cap the thread count?
> Some additional info: We have never run incremental repair on this cluster,
> so that is not a factor. All our tables use LCS. Unfortunately I don't have
> the log files from the period saved.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13409) Materialized Views: View cells are resurrected
[
https://issues.apache.org/jira/browse/CASSANDRA-13409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16090652#comment-16090652
]
Jeff Jirsa commented on CASSANDRA-13409:
[~slebresne] - since CASSANDRA-11475 seems to have caused this regression, and
since you probably understand current state of the engine better than anyone
else, any chance you have feedback here on the right way to solve this once and
for all?
> Materialized Views: View cells are resurrected
> --
>
> Key: CASSANDRA-13409
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13409
> Project: Cassandra
> Issue Type: Bug
> Components: Local Write-Read Paths, Materialized Views
>Reporter: Duarte Nunes
>Assignee: ZhaoYang
>
> Consider the following commands, ran against trunk@0f054fee5c:
> {code:xml}
> echo "create keyspace ks WITH replication = {'class': 'SimpleStrategy',
> 'replication_factor': 1};" | bin/cqlsh
> echo "create table ks.base (p int primary key, v1 int, v2 int) with
> gc_grace_seconds = 1;" | bin/cqlsh
> echo "create materialized view ks.my_view as select * from ks.base where p is
> not null and v1 is not null primary key (v1, p);" | bin/cqlsh
> echo "insert into ks.base (p, v1, v2) values (3, 1, 3) using timestamp 1;" |
> bin/cqlsh
> bin/nodetool flush ks my_view base
> echo "delete from ks.base using timestamp 2 where p = 3;" | bin/cqlsh
> bin/nodetool flush ks my_view base
> echo "insert into ks.base (p, v1) values (3, 1) using timestamp 3;" |
> bin/cqlsh
> bin/nodetool flush ks my_view base
> echo "select * from ks.my_view;" | bin/cqlsh
> v1 | p | v2
> +---+
> 1 | 3 | 3
> (1 rows)
> echo "select * from ks.base;" | bin/cqlsh
> p | v1 | v2
> ---++--
> 3 | 1 | null
> (1 rows)
> {code}
> As you can see, this incorrectly brings back cell v2=3.
> There is one definitive problem and a potential one:
> * Merging rows must be commutative. If a shadowable tombstone is applied
> after a row tombstone, it will replace that tombstone; if a row marker
> shadows the shadowable tombstone before the row containing the original data
> is applied, then any dead cells in said data will be resurrected;
> * Shadowable tombstones shouldn't compact away previous row tombstones or
> even deleted cells; if the relevant tombstones have been GCed from the base
> table, then a base table update won't carry them anymore (alongside a newer
> row marker).
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13688) Anticompaction race can leak sstables/txn
[
https://issues.apache.org/jira/browse/CASSANDRA-13688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16090549#comment-16090549
]
Ariel Weisberg commented on CASSANDRA-13688:
First dtest run timed out. dtests ran here
https://builds.apache.org/view/A-D/view/Cassandra/job/Cassandra-devbranch-dtest/127/
Does that materialized view test look like anything to you? I don't recall
seeing it fail recently. Don't see it failing in the last 30 builds on trunk. I
kicked off the dtests again.
> Anticompaction race can leak sstables/txn
> -
>
> Key: CASSANDRA-13688
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13688
> Project: Cassandra
> Issue Type: Bug
>Reporter: Blake Eggleston
>Assignee: Blake Eggleston
> Fix For: 4.0
>
>
> At the top of {{CompactionManager#performAntiCompaction}}, the parent repair
> session is loaded, if the session can't be found, a RuntimeException is
> thrown. This can happen if a participant is evicted after the IR prepare
> message is received, but before the anticompaction starts. This exception is
> thrown outside of the try/finally block that guards the sstable and lifecycle
> transaction, causing them to leak, and preventing the sstables from ever
> being removed from View.compacting.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-13652) Deadlock in AbstractCommitLogSegmentManager
[
https://issues.apache.org/jira/browse/CASSANDRA-13652?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ariel Weisberg updated CASSANDRA-13652:
---
Status: Ready to Commit (was: Patch Available)
> Deadlock in AbstractCommitLogSegmentManager
> ---
>
> Key: CASSANDRA-13652
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13652
> Project: Cassandra
> Issue Type: Bug
> Components: Core
>Reporter: Fuud
> Fix For: 3.11.1, 4.0
>
>
> AbstractCommitLogManager uses LockSupport.(un)park incorreclty. It invokes
> unpark without checking if manager thread was parked in approriate place.
> For example, logging frameworks uses queues and queues uses ReadWriteLock's
> that uses LockSupport. Therefore AbstractCommitLogManager.wakeManager can
> wake thread inside Lock and manager thread will sleep forever at park()
> method (because unpark permit was already consumed inside lock).
> For examle stack traces:
> {code}
> "MigrationStage:1" id=412 state=WAITING
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
> at
> org.apache.cassandra.utils.concurrent.WaitQueue$AbstractSignal.awaitUninterruptibly(WaitQueue.java:279)
> at
> org.apache.cassandra.db.commitlog.AbstractCommitLogSegmentManager.awaitAvailableSegment(AbstractCommitLogSegmentManager.java:263)
> at
> org.apache.cassandra.db.commitlog.AbstractCommitLogSegmentManager.advanceAllocatingFrom(AbstractCommitLogSegmentManager.java:237)
> at
> org.apache.cassandra.db.commitlog.AbstractCommitLogSegmentManager.forceRecycleAll(AbstractCommitLogSegmentManager.java:279)
> at
> org.apache.cassandra.db.commitlog.CommitLog.forceRecycleAllSegments(CommitLog.java:210)
> at org.apache.cassandra.config.Schema.dropView(Schema.java:708)
> at
> org.apache.cassandra.schema.SchemaKeyspace.lambda$updateKeyspace$23(SchemaKeyspace.java:1361)
> at
> org.apache.cassandra.schema.SchemaKeyspace$$Lambda$382/1123232162.accept(Unknown
> Source)
> at java.util.LinkedHashMap$LinkedValues.forEach(LinkedHashMap.java:608)
> at
> java.util.Collections$UnmodifiableCollection.forEach(Collections.java:1080)
> at
> org.apache.cassandra.schema.SchemaKeyspace.updateKeyspace(SchemaKeyspace.java:1361)
> at
> org.apache.cassandra.schema.SchemaKeyspace.mergeSchema(SchemaKeyspace.java:1332)
> at
> org.apache.cassandra.schema.SchemaKeyspace.mergeSchemaAndAnnounceVersion(SchemaKeyspace.java:1282)
> - locked java.lang.Class@cc38904
> at
> org.apache.cassandra.db.DefinitionsUpdateVerbHandler$1.runMayThrow(DefinitionsUpdateVerbHandler.java:51)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor$LocalSessionWrapper.run(DebuggableThreadPoolExecutor.java:322)
> at
> com.ringcentral.concurrent.executors.MonitoredRunnable.run(MonitoredRunnable.java:36)
> at MON_R_MigrationStage.run(NamedRunnableFactory.java:67)
> at
> com.ringcentral.concurrent.executors.MonitoredThreadPoolExecutor$MdcAwareRunnable.run(MonitoredThreadPoolExecutor.java:114)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at
> org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:79)
> at
> org.apache.cassandra.concurrent.NamedThreadFactory$$Lambda$61/179045.run(Unknown
> Source)
> at java.lang.Thread.run(Thread.java:745)
> "COMMIT-LOG-ALLOCATOR:1" id=80 state=WAITING
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
> at
> org.apache.cassandra.db.commitlog.AbstractCommitLogSegmentManager$1.runMayThrow(AbstractCommitLogSegmentManager.java:128)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> at
> org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:79)
> at
> org.apache.cassandra.concurrent.NamedThreadFactory$$Lambda$61/179045.run(Unknown
> Source)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> Solution is to use Semaphore instead of low-level LockSupport.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-13652) Deadlock in AbstractCommitLogSegmentManager
[
https://issues.apache.org/jira/browse/CASSANDRA-13652?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ariel Weisberg updated CASSANDRA-13652:
---
Fix Version/s: 4.0
3.11.1
> Deadlock in AbstractCommitLogSegmentManager
> ---
>
> Key: CASSANDRA-13652
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13652
> Project: Cassandra
> Issue Type: Bug
> Components: Core
>Reporter: Fuud
> Fix For: 3.11.1, 4.0
>
>
> AbstractCommitLogManager uses LockSupport.(un)park incorreclty. It invokes
> unpark without checking if manager thread was parked in approriate place.
> For example, logging frameworks uses queues and queues uses ReadWriteLock's
> that uses LockSupport. Therefore AbstractCommitLogManager.wakeManager can
> wake thread inside Lock and manager thread will sleep forever at park()
> method (because unpark permit was already consumed inside lock).
> For examle stack traces:
> {code}
> "MigrationStage:1" id=412 state=WAITING
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
> at
> org.apache.cassandra.utils.concurrent.WaitQueue$AbstractSignal.awaitUninterruptibly(WaitQueue.java:279)
> at
> org.apache.cassandra.db.commitlog.AbstractCommitLogSegmentManager.awaitAvailableSegment(AbstractCommitLogSegmentManager.java:263)
> at
> org.apache.cassandra.db.commitlog.AbstractCommitLogSegmentManager.advanceAllocatingFrom(AbstractCommitLogSegmentManager.java:237)
> at
> org.apache.cassandra.db.commitlog.AbstractCommitLogSegmentManager.forceRecycleAll(AbstractCommitLogSegmentManager.java:279)
> at
> org.apache.cassandra.db.commitlog.CommitLog.forceRecycleAllSegments(CommitLog.java:210)
> at org.apache.cassandra.config.Schema.dropView(Schema.java:708)
> at
> org.apache.cassandra.schema.SchemaKeyspace.lambda$updateKeyspace$23(SchemaKeyspace.java:1361)
> at
> org.apache.cassandra.schema.SchemaKeyspace$$Lambda$382/1123232162.accept(Unknown
> Source)
> at java.util.LinkedHashMap$LinkedValues.forEach(LinkedHashMap.java:608)
> at
> java.util.Collections$UnmodifiableCollection.forEach(Collections.java:1080)
> at
> org.apache.cassandra.schema.SchemaKeyspace.updateKeyspace(SchemaKeyspace.java:1361)
> at
> org.apache.cassandra.schema.SchemaKeyspace.mergeSchema(SchemaKeyspace.java:1332)
> at
> org.apache.cassandra.schema.SchemaKeyspace.mergeSchemaAndAnnounceVersion(SchemaKeyspace.java:1282)
> - locked java.lang.Class@cc38904
> at
> org.apache.cassandra.db.DefinitionsUpdateVerbHandler$1.runMayThrow(DefinitionsUpdateVerbHandler.java:51)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor$LocalSessionWrapper.run(DebuggableThreadPoolExecutor.java:322)
> at
> com.ringcentral.concurrent.executors.MonitoredRunnable.run(MonitoredRunnable.java:36)
> at MON_R_MigrationStage.run(NamedRunnableFactory.java:67)
> at
> com.ringcentral.concurrent.executors.MonitoredThreadPoolExecutor$MdcAwareRunnable.run(MonitoredThreadPoolExecutor.java:114)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at
> org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:79)
> at
> org.apache.cassandra.concurrent.NamedThreadFactory$$Lambda$61/179045.run(Unknown
> Source)
> at java.lang.Thread.run(Thread.java:745)
> "COMMIT-LOG-ALLOCATOR:1" id=80 state=WAITING
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
> at
> org.apache.cassandra.db.commitlog.AbstractCommitLogSegmentManager$1.runMayThrow(AbstractCommitLogSegmentManager.java:128)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> at
> org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:79)
> at
> org.apache.cassandra.concurrent.NamedThreadFactory$$Lambda$61/179045.run(Unknown
> Source)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> Solution is to use Semaphore instead of low-level LockSupport.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-13652) Deadlock in AbstractCommitLogSegmentManager
[
https://issues.apache.org/jira/browse/CASSANDRA-13652?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ariel Weisberg updated CASSANDRA-13652:
---
Resolution: Fixed
Status: Resolved (was: Ready to Commit)
Committed as
[eb717f154bd24453273d7175006fdef75e5724c2|https://github.com/apache/cassandra/commit/eb717f154bd24453273d7175006fdef75e5724c2].
Thanks.
> Deadlock in AbstractCommitLogSegmentManager
> ---
>
> Key: CASSANDRA-13652
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13652
> Project: Cassandra
> Issue Type: Bug
> Components: Core
>Reporter: Fuud
> Fix For: 3.11.1, 4.0
>
>
> AbstractCommitLogManager uses LockSupport.(un)park incorreclty. It invokes
> unpark without checking if manager thread was parked in approriate place.
> For example, logging frameworks uses queues and queues uses ReadWriteLock's
> that uses LockSupport. Therefore AbstractCommitLogManager.wakeManager can
> wake thread inside Lock and manager thread will sleep forever at park()
> method (because unpark permit was already consumed inside lock).
> For examle stack traces:
> {code}
> "MigrationStage:1" id=412 state=WAITING
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
> at
> org.apache.cassandra.utils.concurrent.WaitQueue$AbstractSignal.awaitUninterruptibly(WaitQueue.java:279)
> at
> org.apache.cassandra.db.commitlog.AbstractCommitLogSegmentManager.awaitAvailableSegment(AbstractCommitLogSegmentManager.java:263)
> at
> org.apache.cassandra.db.commitlog.AbstractCommitLogSegmentManager.advanceAllocatingFrom(AbstractCommitLogSegmentManager.java:237)
> at
> org.apache.cassandra.db.commitlog.AbstractCommitLogSegmentManager.forceRecycleAll(AbstractCommitLogSegmentManager.java:279)
> at
> org.apache.cassandra.db.commitlog.CommitLog.forceRecycleAllSegments(CommitLog.java:210)
> at org.apache.cassandra.config.Schema.dropView(Schema.java:708)
> at
> org.apache.cassandra.schema.SchemaKeyspace.lambda$updateKeyspace$23(SchemaKeyspace.java:1361)
> at
> org.apache.cassandra.schema.SchemaKeyspace$$Lambda$382/1123232162.accept(Unknown
> Source)
> at java.util.LinkedHashMap$LinkedValues.forEach(LinkedHashMap.java:608)
> at
> java.util.Collections$UnmodifiableCollection.forEach(Collections.java:1080)
> at
> org.apache.cassandra.schema.SchemaKeyspace.updateKeyspace(SchemaKeyspace.java:1361)
> at
> org.apache.cassandra.schema.SchemaKeyspace.mergeSchema(SchemaKeyspace.java:1332)
> at
> org.apache.cassandra.schema.SchemaKeyspace.mergeSchemaAndAnnounceVersion(SchemaKeyspace.java:1282)
> - locked java.lang.Class@cc38904
> at
> org.apache.cassandra.db.DefinitionsUpdateVerbHandler$1.runMayThrow(DefinitionsUpdateVerbHandler.java:51)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor$LocalSessionWrapper.run(DebuggableThreadPoolExecutor.java:322)
> at
> com.ringcentral.concurrent.executors.MonitoredRunnable.run(MonitoredRunnable.java:36)
> at MON_R_MigrationStage.run(NamedRunnableFactory.java:67)
> at
> com.ringcentral.concurrent.executors.MonitoredThreadPoolExecutor$MdcAwareRunnable.run(MonitoredThreadPoolExecutor.java:114)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at
> org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:79)
> at
> org.apache.cassandra.concurrent.NamedThreadFactory$$Lambda$61/179045.run(Unknown
> Source)
> at java.lang.Thread.run(Thread.java:745)
> "COMMIT-LOG-ALLOCATOR:1" id=80 state=WAITING
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
> at
> org.apache.cassandra.db.commitlog.AbstractCommitLogSegmentManager$1.runMayThrow(AbstractCommitLogSegmentManager.java:128)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> at
> org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:79)
> at
> org.apache.cassandra.concurrent.NamedThreadFactory$$Lambda$61/179045.run(Unknown
> Source)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> Solution is to use Semaphore instead of low-level LockSupport.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe,
[1/3] cassandra git commit: Deadlock in AbstractCommitLogSegmentManager
Repository: cassandra
Updated Branches:
refs/heads/cassandra-3.11 6d0e95af7 -> eb717f154
refs/heads/trunk 855dd0eee -> 6eb2758e4
Deadlock in AbstractCommitLogSegmentManager
Patch by Fedor Bobin; Reviewed by Ariel Weisberg for CASSANDRA-13652
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/eb717f15
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/eb717f15
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/eb717f15
Branch: refs/heads/cassandra-3.11
Commit: eb717f154bd24453273d7175006fdef75e5724c2
Parents: 6d0e95a
Author: Fedor Bobin
Authored: Tue Jul 11 15:10:28 2017 -0400
Committer: Ariel Weisberg
Committed: Mon Jul 17 13:50:09 2017 -0700
--
CHANGES.txt | 1 +
.../AbstractCommitLogSegmentManager.java| 11 +--
.../cassandra/utils/concurrent/WaitQueue.java | 20
3 files changed, 26 insertions(+), 6 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/eb717f15/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 8abf1f4..e7a10f5 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
3.11.1
+ * Deadlock in AbstractCommitLogSegmentManager (CASSANDRA-13652)
* Duplicate the buffer before passing it to analyser in SASI operation
(CASSANDRA-13512)
* Properly evict pstmts from prepared statements cache (CASSANDRA-13641)
Merged from 3.0:
http://git-wip-us.apache.org/repos/asf/cassandra/blob/eb717f15/src/java/org/apache/cassandra/db/commitlog/AbstractCommitLogSegmentManager.java
--
diff --git
a/src/java/org/apache/cassandra/db/commitlog/AbstractCommitLogSegmentManager.java
b/src/java/org/apache/cassandra/db/commitlog/AbstractCommitLogSegmentManager.java
old mode 100644
new mode 100755
index 0ab941b..7f3c9f8
---
a/src/java/org/apache/cassandra/db/commitlog/AbstractCommitLogSegmentManager.java
+++
b/src/java/org/apache/cassandra/db/commitlog/AbstractCommitLogSegmentManager.java
@@ -22,7 +22,7 @@ import java.io.IOException;
import java.util.*;
import java.util.concurrent.*;
import java.util.concurrent.atomic.AtomicLong;
-import java.util.concurrent.locks.LockSupport;
+import java.util.function.BooleanSupplier;
import com.google.common.annotations.VisibleForTesting;
import com.google.common.util.concurrent.*;
@@ -82,6 +82,8 @@ public abstract class AbstractCommitLogSegmentManager
private Thread managerThread;
protected final CommitLog commitLog;
private volatile boolean shutdown;
+private final BooleanSupplier managerThreadWaitCondition = () ->
(availableSegment == null && !atSegmentBufferLimit()) || shutdown;
+private final WaitQueue managerThreadWaitQueue = new WaitQueue();
private static final SimpleCachedBufferPool bufferPool =
new
SimpleCachedBufferPool(DatabaseDescriptor.getCommitLogMaxCompressionBuffersInPool(),
DatabaseDescriptor.getCommitLogSegmentSize());
@@ -124,8 +126,6 @@ public abstract class AbstractCommitLogSegmentManager
// Writing threads are not waiting for new segments,
we can spend time on other tasks.
// flush old Cfs if we're full
maybeFlushToReclaim();
-
-LockSupport.park();
}
catch (Throwable t)
{
@@ -140,8 +140,7 @@ public abstract class AbstractCommitLogSegmentManager
// shutting down-- nothing more can or needs to be
done in that case.
}
-while (availableSegment != null || atSegmentBufferLimit()
&& !shutdown)
-LockSupport.park();
+WaitQueue.waitOnCondition(managerThreadWaitCondition,
managerThreadWaitQueue);
}
}
};
@@ -530,7 +529,7 @@ public abstract class AbstractCommitLogSegmentManager
void wakeManager()
{
-LockSupport.unpark(managerThread);
+managerThreadWaitQueue.signalAll();
}
/**
http://git-wip-us.apache.org/repos/asf/cassandra/blob/eb717f15/src/java/org/apache/cassandra/utils/concurrent/WaitQueue.java
--
diff --git a/src/java/org/apache/cassandra/utils/concurrent/WaitQueue.java
b/src/java/org/apache/cassandra/utils/concurrent/WaitQueue.java
index be271b6..5b453b0 100644
--- a/src/java/org/apache/cassandra/utils/concurrent/WaitQueue.java
+++ b/src/java/org/apache/cassandra/utils/concurrent/WaitQueue.java
@@ -22,6 +22,7 @@ import java.util.Iterator;
import java.util.concurrent.ConcurrentLinkedQueue;
import java.util.concurrent.
[2/3] cassandra git commit: Deadlock in AbstractCommitLogSegmentManager
Deadlock in AbstractCommitLogSegmentManager
Patch by Fedor Bobin; Reviewed by Ariel Weisberg for CASSANDRA-13652
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/eb717f15
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/eb717f15
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/eb717f15
Branch: refs/heads/trunk
Commit: eb717f154bd24453273d7175006fdef75e5724c2
Parents: 6d0e95a
Author: Fedor Bobin
Authored: Tue Jul 11 15:10:28 2017 -0400
Committer: Ariel Weisberg
Committed: Mon Jul 17 13:50:09 2017 -0700
--
CHANGES.txt | 1 +
.../AbstractCommitLogSegmentManager.java| 11 +--
.../cassandra/utils/concurrent/WaitQueue.java | 20
3 files changed, 26 insertions(+), 6 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/eb717f15/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 8abf1f4..e7a10f5 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
3.11.1
+ * Deadlock in AbstractCommitLogSegmentManager (CASSANDRA-13652)
* Duplicate the buffer before passing it to analyser in SASI operation
(CASSANDRA-13512)
* Properly evict pstmts from prepared statements cache (CASSANDRA-13641)
Merged from 3.0:
http://git-wip-us.apache.org/repos/asf/cassandra/blob/eb717f15/src/java/org/apache/cassandra/db/commitlog/AbstractCommitLogSegmentManager.java
--
diff --git
a/src/java/org/apache/cassandra/db/commitlog/AbstractCommitLogSegmentManager.java
b/src/java/org/apache/cassandra/db/commitlog/AbstractCommitLogSegmentManager.java
old mode 100644
new mode 100755
index 0ab941b..7f3c9f8
---
a/src/java/org/apache/cassandra/db/commitlog/AbstractCommitLogSegmentManager.java
+++
b/src/java/org/apache/cassandra/db/commitlog/AbstractCommitLogSegmentManager.java
@@ -22,7 +22,7 @@ import java.io.IOException;
import java.util.*;
import java.util.concurrent.*;
import java.util.concurrent.atomic.AtomicLong;
-import java.util.concurrent.locks.LockSupport;
+import java.util.function.BooleanSupplier;
import com.google.common.annotations.VisibleForTesting;
import com.google.common.util.concurrent.*;
@@ -82,6 +82,8 @@ public abstract class AbstractCommitLogSegmentManager
private Thread managerThread;
protected final CommitLog commitLog;
private volatile boolean shutdown;
+private final BooleanSupplier managerThreadWaitCondition = () ->
(availableSegment == null && !atSegmentBufferLimit()) || shutdown;
+private final WaitQueue managerThreadWaitQueue = new WaitQueue();
private static final SimpleCachedBufferPool bufferPool =
new
SimpleCachedBufferPool(DatabaseDescriptor.getCommitLogMaxCompressionBuffersInPool(),
DatabaseDescriptor.getCommitLogSegmentSize());
@@ -124,8 +126,6 @@ public abstract class AbstractCommitLogSegmentManager
// Writing threads are not waiting for new segments,
we can spend time on other tasks.
// flush old Cfs if we're full
maybeFlushToReclaim();
-
-LockSupport.park();
}
catch (Throwable t)
{
@@ -140,8 +140,7 @@ public abstract class AbstractCommitLogSegmentManager
// shutting down-- nothing more can or needs to be
done in that case.
}
-while (availableSegment != null || atSegmentBufferLimit()
&& !shutdown)
-LockSupport.park();
+WaitQueue.waitOnCondition(managerThreadWaitCondition,
managerThreadWaitQueue);
}
}
};
@@ -530,7 +529,7 @@ public abstract class AbstractCommitLogSegmentManager
void wakeManager()
{
-LockSupport.unpark(managerThread);
+managerThreadWaitQueue.signalAll();
}
/**
http://git-wip-us.apache.org/repos/asf/cassandra/blob/eb717f15/src/java/org/apache/cassandra/utils/concurrent/WaitQueue.java
--
diff --git a/src/java/org/apache/cassandra/utils/concurrent/WaitQueue.java
b/src/java/org/apache/cassandra/utils/concurrent/WaitQueue.java
index be271b6..5b453b0 100644
--- a/src/java/org/apache/cassandra/utils/concurrent/WaitQueue.java
+++ b/src/java/org/apache/cassandra/utils/concurrent/WaitQueue.java
@@ -22,6 +22,7 @@ import java.util.Iterator;
import java.util.concurrent.ConcurrentLinkedQueue;
import java.util.concurrent.atomic.AtomicIntegerFieldUpdater;
import java.util.concurrent.locks.LockSupport;
+import java.util.function.BooleanSupplier;
import com.coda
[3/3] cassandra git commit: Merge branch 'cassandra-3.11' into trunk
Merge branch 'cassandra-3.11' into trunk Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/6eb2758e Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/6eb2758e Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/6eb2758e Branch: refs/heads/trunk Commit: 6eb2758e4038564b084f000ac411061d3b942d1f Parents: 855dd0e eb717f1 Author: Ariel Weisberg Authored: Mon Jul 17 13:51:57 2017 -0700 Committer: Ariel Weisberg Committed: Mon Jul 17 13:51:57 2017 -0700 -- CHANGES.txt | 1 + .../AbstractCommitLogSegmentManager.java| 11 +-- .../cassandra/utils/concurrent/WaitQueue.java | 20 3 files changed, 26 insertions(+), 6 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/6eb2758e/CHANGES.txt -- diff --cc CHANGES.txt index 0be0e39,e7a10f5..3fb2716 --- a/CHANGES.txt +++ b/CHANGES.txt @@@ -1,102 -1,5 +1,103 @@@ +4.0 + * Don't add localhost to the graph when calculating where to stream from (CASSANDRA-13583) + * Allow skipping equality-restricted clustering columns in ORDER BY clause (CASSANDRA-10271) + * Use common nowInSec for validation compactions (CASSANDRA-13671) + * Improve handling of IR prepare failures (CASSANDRA-13672) + * Send IR coordinator messages synchronously (CASSANDRA-13673) + * Flush system.repair table before IR finalize promise (CASSANDRA-13660) + * Fix column filter creation for wildcard queries (CASSANDRA-13650) + * Add 'nodetool getbatchlogreplaythrottle' and 'nodetool setbatchlogreplaythrottle' (CASSANDRA-13614) + * fix race condition in PendingRepairManager (CASSANDRA-13659) + * Allow noop incremental repair state transitions (CASSANDRA-13658) + * Run repair with down replicas (CASSANDRA-10446) + * Added started & completed repair metrics (CASSANDRA-13598) + * Added started & completed repair metrics (CASSANDRA-13598) + * Improve secondary index (re)build failure and concurrency handling (CASSANDRA-10130) + * Improve calculation of available disk space for compaction (CASSANDRA-13068) + * Change the accessibility of RowCacheSerializer for third party row cache plugins (CASSANDRA-13579) + * Allow sub-range repairs for a preview of repaired data (CASSANDRA-13570) + * NPE in IR cleanup when columnfamily has no sstables (CASSANDRA-13585) + * Fix Randomness of stress values (CASSANDRA-12744) + * Allow selecting Map values and Set elements (CASSANDRA-7396) + * Fast and garbage-free Streaming Histogram (CASSANDRA-13444) + * Update repairTime for keyspaces on completion (CASSANDRA-13539) + * Add configurable upper bound for validation executor threads (CASSANDRA-13521) + * Bring back maxHintTTL propery (CASSANDRA-12982) + * Add testing guidelines (CASSANDRA-13497) + * Add more repair metrics (CASSANDRA-13531) + * RangeStreamer should be smarter when picking endpoints for streaming (CASSANDRA-4650) + * Avoid rewrapping an exception thrown for cache load functions (CASSANDRA-13367) + * Log time elapsed for each incremental repair phase (CASSANDRA-13498) + * Add multiple table operation support to cassandra-stress (CASSANDRA-8780) + * Fix incorrect cqlsh results when selecting same columns multiple times (CASSANDRA-13262) + * Fix WriteResponseHandlerTest is sensitive to test execution order (CASSANDRA-13421) + * Improve incremental repair logging (CASSANDRA-13468) + * Start compaction when incremental repair finishes (CASSANDRA-13454) + * Add repair streaming preview (CASSANDRA-13257) + * Cleanup isIncremental/repairedAt usage (CASSANDRA-13430) + * Change protocol to allow sending key space independent of query string (CASSANDRA-10145) + * Make gc_log and gc_warn settable at runtime (CASSANDRA-12661) + * Take number of files in L0 in account when estimating remaining compaction tasks (CASSANDRA-13354) + * Skip building views during base table streams on range movements (CASSANDRA-13065) + * Improve error messages for +/- operations on maps and tuples (CASSANDRA-13197) + * Remove deprecated repair JMX APIs (CASSANDRA-11530) + * Fix version check to enable streaming keep-alive (CASSANDRA-12929) + * Make it possible to monitor an ideal consistency level separate from actual consistency level (CASSANDRA-13289) + * Outbound TCP connections ignore internode authenticator (CASSANDRA-13324) + * Upgrade junit from 4.6 to 4.12 (CASSANDRA-13360) + * Cleanup ParentRepairSession after repairs (CASSANDRA-13359) + * Upgrade snappy-java to 1.1.2.6 (CASSANDRA-13336) + * Incremental repair not streaming correct sstables (CASSANDRA-13328) + * Upgrade the jna version to 4.3.0 (CASSANDRA-13300) + * Add the currentTimestamp, currentDate, currentTime and currentTimeUUID
[jira] [Commented] (CASSANDRA-13696) Digest mismatch Exception if hints file has UnknownColumnFamily
[
https://issues.apache.org/jira/browse/CASSANDRA-13696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16090530#comment-16090530
]
Jeff Jirsa commented on CASSANDRA-13696:
Looks pretty serious, I'll try to get to it soon. Short term, your trunk patch
doesn't actually compile ( {{CFMetaData}} is gone in trunk), can you fix that
and push that branch?
> Digest mismatch Exception if hints file has UnknownColumnFamily
> ---
>
> Key: CASSANDRA-13696
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13696
> Project: Cassandra
> Issue Type: Bug
> Components: Core
>Reporter: Jay Zhuang
>Assignee: Jay Zhuang
>Priority: Critical
>
> {noformat}
> WARN [HintsDispatcher:2] 2017-07-16 22:00:32,579 HintsReader.java:235 -
> Failed to read a hint for /127.0.0.2: a2b7daf1-a6a4-4dfc-89de-32d12d2d48b0 -
> table with id 3882bbb0-6a71-11e7-9bca-2759083e3964 is unknown in file
> a2b7daf1-a6a4-4dfc-89de-32d12d2d48b0-1500242103097-1.hints
> ERROR [HintsDispatcher:2] 2017-07-16 22:00:32,580
> HintsDispatchExecutor.java:234 - Failed to dispatch hints file
> a2b7daf1-a6a4-4dfc-89de-32d12d2d48b0-1500242103097-1.hints: file is corrupted
> ({})
> org.apache.cassandra.io.FSReadError: java.io.IOException: Digest mismatch
> exception
> at
> org.apache.cassandra.hints.HintsReader$HintsIterator.computeNext(HintsReader.java:199)
> ~[main/:na]
> at
> org.apache.cassandra.hints.HintsReader$HintsIterator.computeNext(HintsReader.java:164)
> ~[main/:na]
> at
> org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
> ~[main/:na]
> at
> org.apache.cassandra.hints.HintsDispatcher.sendHints(HintsDispatcher.java:157)
> ~[main/:na]
> at
> org.apache.cassandra.hints.HintsDispatcher.sendHintsAndAwait(HintsDispatcher.java:139)
> ~[main/:na]
> at
> org.apache.cassandra.hints.HintsDispatcher.dispatch(HintsDispatcher.java:123)
> ~[main/:na]
> at
> org.apache.cassandra.hints.HintsDispatcher.dispatch(HintsDispatcher.java:95)
> ~[main/:na]
> at
> org.apache.cassandra.hints.HintsDispatchExecutor$DispatchHintsTask.deliver(HintsDispatchExecutor.java:268)
> [main/:na]
> at
> org.apache.cassandra.hints.HintsDispatchExecutor$DispatchHintsTask.dispatch(HintsDispatchExecutor.java:251)
> [main/:na]
> at
> org.apache.cassandra.hints.HintsDispatchExecutor$DispatchHintsTask.dispatch(HintsDispatchExecutor.java:229)
> [main/:na]
> at
> org.apache.cassandra.hints.HintsDispatchExecutor$DispatchHintsTask.run(HintsDispatchExecutor.java:208)
> [main/:na]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> [na:1.8.0_111]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_111]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> [na:1.8.0_111]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [na:1.8.0_111]
> at
> org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:79)
> [main/:na]
> at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_111]
> Caused by: java.io.IOException: Digest mismatch exception
> at
> org.apache.cassandra.hints.HintsReader$HintsIterator.computeNextInternal(HintsReader.java:216)
> ~[main/:na]
> at
> org.apache.cassandra.hints.HintsReader$HintsIterator.computeNext(HintsReader.java:190)
> ~[main/:na]
> ... 16 common frames omitted
> {noformat}
> It causes multiple cassandra nodes stop [by
> default|https://github.com/apache/cassandra/blob/cassandra-3.0/conf/cassandra.yaml#L188].
> Here is the reproduce steps on a 3 nodes cluster, RF=3:
> 1. stop node1
> 2. send some data with quorum (or one), it will generate hints file on
> node2/node3
> 3. drop the table
> 4. start node1
> node2/node3 will report "corrupted hints file" and stop. The impact is very
> bad for a large cluster, when it happens, almost all the nodes are down at
> the same time and we have to remove all the hints files (which contain the
> dropped table) to bring the node back.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-13696) Digest mismatch Exception if hints file has UnknownColumnFamily
[
https://issues.apache.org/jira/browse/CASSANDRA-13696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jeff Jirsa updated CASSANDRA-13696:
---
Reviewer: Jeff Jirsa
> Digest mismatch Exception if hints file has UnknownColumnFamily
> ---
>
> Key: CASSANDRA-13696
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13696
> Project: Cassandra
> Issue Type: Bug
> Components: Core
>Reporter: Jay Zhuang
>Assignee: Jay Zhuang
>Priority: Critical
>
> {noformat}
> WARN [HintsDispatcher:2] 2017-07-16 22:00:32,579 HintsReader.java:235 -
> Failed to read a hint for /127.0.0.2: a2b7daf1-a6a4-4dfc-89de-32d12d2d48b0 -
> table with id 3882bbb0-6a71-11e7-9bca-2759083e3964 is unknown in file
> a2b7daf1-a6a4-4dfc-89de-32d12d2d48b0-1500242103097-1.hints
> ERROR [HintsDispatcher:2] 2017-07-16 22:00:32,580
> HintsDispatchExecutor.java:234 - Failed to dispatch hints file
> a2b7daf1-a6a4-4dfc-89de-32d12d2d48b0-1500242103097-1.hints: file is corrupted
> ({})
> org.apache.cassandra.io.FSReadError: java.io.IOException: Digest mismatch
> exception
> at
> org.apache.cassandra.hints.HintsReader$HintsIterator.computeNext(HintsReader.java:199)
> ~[main/:na]
> at
> org.apache.cassandra.hints.HintsReader$HintsIterator.computeNext(HintsReader.java:164)
> ~[main/:na]
> at
> org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
> ~[main/:na]
> at
> org.apache.cassandra.hints.HintsDispatcher.sendHints(HintsDispatcher.java:157)
> ~[main/:na]
> at
> org.apache.cassandra.hints.HintsDispatcher.sendHintsAndAwait(HintsDispatcher.java:139)
> ~[main/:na]
> at
> org.apache.cassandra.hints.HintsDispatcher.dispatch(HintsDispatcher.java:123)
> ~[main/:na]
> at
> org.apache.cassandra.hints.HintsDispatcher.dispatch(HintsDispatcher.java:95)
> ~[main/:na]
> at
> org.apache.cassandra.hints.HintsDispatchExecutor$DispatchHintsTask.deliver(HintsDispatchExecutor.java:268)
> [main/:na]
> at
> org.apache.cassandra.hints.HintsDispatchExecutor$DispatchHintsTask.dispatch(HintsDispatchExecutor.java:251)
> [main/:na]
> at
> org.apache.cassandra.hints.HintsDispatchExecutor$DispatchHintsTask.dispatch(HintsDispatchExecutor.java:229)
> [main/:na]
> at
> org.apache.cassandra.hints.HintsDispatchExecutor$DispatchHintsTask.run(HintsDispatchExecutor.java:208)
> [main/:na]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> [na:1.8.0_111]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_111]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> [na:1.8.0_111]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [na:1.8.0_111]
> at
> org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:79)
> [main/:na]
> at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_111]
> Caused by: java.io.IOException: Digest mismatch exception
> at
> org.apache.cassandra.hints.HintsReader$HintsIterator.computeNextInternal(HintsReader.java:216)
> ~[main/:na]
> at
> org.apache.cassandra.hints.HintsReader$HintsIterator.computeNext(HintsReader.java:190)
> ~[main/:na]
> ... 16 common frames omitted
> {noformat}
> It causes multiple cassandra nodes stop [by
> default|https://github.com/apache/cassandra/blob/cassandra-3.0/conf/cassandra.yaml#L188].
> Here is the reproduce steps on a 3 nodes cluster, RF=3:
> 1. stop node1
> 2. send some data with quorum (or one), it will generate hints file on
> node2/node3
> 3. drop the table
> 4. start node1
> node2/node3 will report "corrupted hints file" and stop. The impact is very
> bad for a large cluster, when it happens, almost all the nodes are down at
> the same time and we have to remove all the hints files (which contain the
> dropped table) to bring the node back.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Assigned] (CASSANDRA-10346) Create tool to repair inconsistencies between base tables and materialized views when dataloss ocurrs
[ https://issues.apache.org/jira/browse/CASSANDRA-10346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nate McCall reassigned CASSANDRA-10346: --- Assignee: (was: Carl Yeksigian) > Create tool to repair inconsistencies between base tables and materialized > views when dataloss ocurrs > - > > Key: CASSANDRA-10346 > URL: https://issues.apache.org/jira/browse/CASSANDRA-10346 > Project: Cassandra > Issue Type: New Feature >Reporter: T Jake Luciani >Priority: Minor > Fix For: 4.x > > > As mentioned in CASSANDRA-10230, you can have inconsistencies between base > and view if you have data loss on all ack'd replicas in the base table. > We should add a way to identify and ideally fix/drop discrepancies between > the base and view replicas. > I think implementing this efficiently will not be so easy but it should work > like repair in theory. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Assigned] (CASSANDRA-10965) Shadowable tombstones can continue to shadow view results when timestamps match
[
https://issues.apache.org/jira/browse/CASSANDRA-10965?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nate McCall reassigned CASSANDRA-10965:
---
Assignee: (was: Carl Yeksigian)
> Shadowable tombstones can continue to shadow view results when timestamps
> match
> ---
>
> Key: CASSANDRA-10965
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10965
> Project: Cassandra
> Issue Type: Bug
> Components: Local Write-Read Paths, Materialized Views
>Reporter: Carl Yeksigian
> Fix For: 3.0.x
>
> Attachments: shadow-ts.cql
>
>
> I've attached a script which reproduces the issue. The first time we insert
> with {{TIMESTAMP 2}}, we are inserting a new row which has the same timestamp
> as the previous shadow tombstone, and it continues to be shadowed by that
> tombstone because we shadow values with the same timestamp.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-1361) Introduce ability to change seed addresses on a running node
[ https://issues.apache.org/jira/browse/CASSANDRA-1361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16090498#comment-16090498 ] Brandon Williams commented on CASSANDRA-1361: - Since seed providers were introduced, the seed list is reloaded anytime a seed is removed. So to avoid a restart, first promote an existing node to a seed in the yaml files, then remove the defunct seed with removenode and the promoted seed will become active. > Introduce ability to change seed addresses on a running node > > > Key: CASSANDRA-1361 > URL: https://issues.apache.org/jira/browse/CASSANDRA-1361 > Project: Cassandra > Issue Type: New Feature >Reporter: Valentino Volonghi > > Since seed nodes are not only used for bootstrap but also for gossiping it > would be really useful to be able to replace seed addresses on running nodes > without restarting them since at the moment you can only change seed nodes by > changing configuration file and then restarting cassandra. > My usecase is to be able to replace failed seed nodes in cassandra without > having to proceed to a rolling restart of the cluster. And also to be able to > start X nodes that initially might be independent but then would all join an > existing ring by changing their seeds. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-12373) 3.0 breaks CQL compatibility with super columns families
[ https://issues.apache.org/jira/browse/CASSANDRA-12373?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alex Petrov updated CASSANDRA-12373: Status: Open (was: Patch Available) I'll rebase on top of the latest trunk as it seems like the patch has gotten a bit out of date. > 3.0 breaks CQL compatibility with super columns families > > > Key: CASSANDRA-12373 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12373 > Project: Cassandra > Issue Type: Bug > Components: CQL >Reporter: Sylvain Lebresne >Assignee: Alex Petrov > Fix For: 3.0.x > > > This is a follow-up to CASSANDRA-12335 to fix the CQL side of super column > compatibility. > The details and a proposed solution can be found in the comments of > CASSANDRA-12335 but the crux of the issue is that super column famillies show > up differently in CQL in 3.0.x/3.x compared to 2.x, hence breaking backward > compatibilty. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-12373) 3.0 breaks CQL compatibility with super columns families
[ https://issues.apache.org/jira/browse/CASSANDRA-12373?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alex Petrov updated CASSANDRA-12373: Reviewer: Sylvain Lebresne > 3.0 breaks CQL compatibility with super columns families > > > Key: CASSANDRA-12373 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12373 > Project: Cassandra > Issue Type: Bug > Components: CQL >Reporter: Sylvain Lebresne >Assignee: Alex Petrov > Fix For: 3.0.x > > > This is a follow-up to CASSANDRA-12335 to fix the CQL side of super column > compatibility. > The details and a proposed solution can be found in the comments of > CASSANDRA-12335 but the crux of the issue is that super column famillies show > up differently in CQL in 3.0.x/3.x compared to 2.x, hence breaking backward > compatibilty. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-13583) test failure in rebuild_test.TestRebuild.disallow_rebuild_from_nonreplica_test
[
https://issues.apache.org/jira/browse/CASSANDRA-13583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Marcus Eriksson updated CASSANDRA-13583:
Resolution: Fixed
Fix Version/s: (was: 4.x)
4.0
Status: Resolved (was: Ready to Commit)
committed, thanks
> test failure in rebuild_test.TestRebuild.disallow_rebuild_from_nonreplica_test
> --
>
> Key: CASSANDRA-13583
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13583
> Project: Cassandra
> Issue Type: Bug
>Reporter: Michael Hamm
>Assignee: Marcus Eriksson
> Labels: dtest, test-failure
> Fix For: 4.0
>
> Attachments: node1_debug.log, node1_gc.log, node1.log,
> node2_debug.log, node2_gc.log, node2.log, node3_debug.log, node3_gc.log,
> node3.log
>
>
> example failure:
> http://cassci.datastax.com/job/trunk_novnode_dtest/524/testReport/rebuild_test/TestRebuild/disallow_rebuild_from_nonreplica_test
> {noformat}
> Error Message
> ToolError not raised
> >> begin captured logging <<
> dtest: DEBUG: Python driver version in use: 3.10
> dtest: DEBUG: cluster ccm directory: /tmp/dtest-0tUjhX
> dtest: DEBUG: Done setting configuration options:
> { 'num_tokens': None,
> 'phi_convict_threshold': 5,
> 'range_request_timeout_in_ms': 1,
> 'read_request_timeout_in_ms': 1,
> 'request_timeout_in_ms': 1,
> 'truncate_request_timeout_in_ms': 1,
> 'write_request_timeout_in_ms': 1}
> cassandra.cluster: INFO: New Cassandra host discovered
> cassandra.cluster: INFO: New Cassandra host discovered
> - >> end captured logging << -
> {noformat}
> {noformat}
> Stacktrace
> File "/usr/lib/python2.7/unittest/case.py", line 329, in run
> testMethod()
> File "/home/automaton/cassandra-dtest/tools/decorators.py", line 48, in
> wrappedtestrebuild
> f(obj)
> File "/home/automaton/cassandra-dtest/rebuild_test.py", line 357, in
> disallow_rebuild_from_nonreplica_test
> node1.nodetool('rebuild -ks ks1 -ts (%s,%s] -s %s' % (node3_token,
> node1_token, node3_address))
> File "/usr/lib/python2.7/unittest/case.py", line 116, in __exit__
> "{0} not raised".format(exc_name))
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
cassandra git commit: Don't add localhost to the graph when calculating where to stream from
Repository: cassandra
Updated Branches:
refs/heads/trunk c3373012d -> 855dd0eee
Don't add localhost to the graph when calculating where to stream from
Patch by marcuse; reviewed by Ariel Weisberg for CASSANDRA-13583
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/855dd0ee
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/855dd0ee
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/855dd0ee
Branch: refs/heads/trunk
Commit: 855dd0e4b8cd00bca48a40f1210efa8bd7f8
Parents: c337301
Author: Marcus Eriksson
Authored: Tue Jun 20 10:04:17 2017 +0200
Committer: Marcus Eriksson
Committed: Mon Jul 17 09:56:13 2017 -0700
--
CHANGES.txt | 1 +
.../org/apache/cassandra/dht/RangeFetchMapCalculator.java | 9 -
2 files changed, 5 insertions(+), 5 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/855dd0ee/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 087bacc..0be0e39 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
4.0
+ * Don't add localhost to the graph when calculating where to stream from
(CASSANDRA-13583)
* Allow skipping equality-restricted clustering columns in ORDER BY clause
(CASSANDRA-10271)
* Use common nowInSec for validation compactions (CASSANDRA-13671)
* Improve handling of IR prepare failures (CASSANDRA-13672)
http://git-wip-us.apache.org/repos/asf/cassandra/blob/855dd0ee/src/java/org/apache/cassandra/dht/RangeFetchMapCalculator.java
--
diff --git a/src/java/org/apache/cassandra/dht/RangeFetchMapCalculator.java
b/src/java/org/apache/cassandra/dht/RangeFetchMapCalculator.java
index 355d7a9..1186eab 100644
--- a/src/java/org/apache/cassandra/dht/RangeFetchMapCalculator.java
+++ b/src/java/org/apache/cassandra/dht/RangeFetchMapCalculator.java
@@ -251,8 +251,7 @@ public class RangeFetchMapCalculator
sourceFound = addEndpoints(capacityGraph, rangeVertex, false);
}
-//We could not find any source for this range which passed the
filters. Ignore if localhost is part of the endpoints for this range
-if (!sourceFound &&
!rangesWithSources.get(range).contains(FBUtilities.getBroadcastAddress()))
+if (!sourceFound)
throw new IllegalStateException("Unable to find sufficient
sources for streaming range " + range + " in keyspace " + keyspace);
}
@@ -275,6 +274,9 @@ public class RangeFetchMapCalculator
if (passFilters(endpoint, localDCCheck))
{
sourceFound = true;
+// if we pass filters, it means that we don't filter away
localhost and we can count it as a source:
+if (endpoint.equals(FBUtilities.getBroadcastAddress()))
+continue; // but don't add localhost to the graph to avoid
streaming locally
final Vertex endpointVertex = new EndpointVertex(endpoint);
capacityGraph.insertVertex(rangeVertex);
capacityGraph.insertVertex(endpointVertex);
@@ -303,9 +305,6 @@ public class RangeFetchMapCalculator
return false;
}
-if(endpoint.equals(FBUtilities.getBroadcastAddress()))
-return false;
-
return !localDCCheck || isInLocalDC(endpoint);
}
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Created] (CASSANDRA-13697) CDC and VIEW writeType missing from spec for write_timeout / write_failure
Andy Tolbert created CASSANDRA-13697:
Summary: CDC and VIEW writeType missing from spec for
write_timeout / write_failure
Key: CASSANDRA-13697
URL: https://issues.apache.org/jira/browse/CASSANDRA-13697
Project: Cassandra
Issue Type: Improvement
Reporter: Andy Tolbert
Priority: Minor
In cassandra 3.0 a new {{WriteType}} {{VIEW}} was added which appears to be
used when raising a {{WriteTimeoutException}} when the local view lock for a
key cannot be acquired within timeout.
In cassandra 3.8 {{CDC}} {{WriteType}} was added for when
{{cdc_total_space_in_mb}} is exceeded when doing a write to data tracked by cdc.
The [v4
spec|https://github.com/apache/cassandra/blob/cassandra-3.11.0/doc/native_protocol_v4.spec#L1051-L1066]
currently doesn't cover these two write types. While the protocol allows for
a free form string for write type, it would be nice to document that types are
available since some drivers (java, cpp, python) attempt to deserialize write
type into an enum and may not handle it well.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13425) nodetool refresh should try to insert new sstables in existing leveling
[
https://issues.apache.org/jira/browse/CASSANDRA-13425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16090082#comment-16090082
]
Marcus Eriksson commented on CASSANDRA-13425:
-
[~pauloricardomg] agree that it pollutes the ACS interface, will check if there
is something nicer we can do (looks like LCS.addSSTables is called from other
places though, so that might not work)
> nodetool refresh should try to insert new sstables in existing leveling
> ---
>
> Key: CASSANDRA-13425
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13425
> Project: Cassandra
> Issue Type: Bug
>Reporter: Marcus Eriksson
>Assignee: Marcus Eriksson
> Fix For: 4.x
>
>
> Currently {{nodetool refresh}} sets level to 0 on all new sstables, instead
> we could try to find gaps in the existing leveling and insert them there.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13127) Materialized Views: View row expires too soon
[
https://issues.apache.org/jira/browse/CASSANDRA-13127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16090043#comment-16090043
]
Fridtjof Sander commented on CASSANDRA-13127:
-
bq. More precisely only for base column/cell that has timestamp and not used in
MV. (In cases of insertion, no column timestamp)
You win.
bq. For now, base table with view doesn't support drop columns. but there is a
ticket open to support alter MV.
I just checked it, and that restriction is actually only checked via
{{view.includes(columnName)}} which is not the case here.
Anyway, it's probably a little to early in the discussion for that kind of
detail, we can just set a mental mark to remember this, I guess.
> Materialized Views: View row expires too soon
> -
>
> Key: CASSANDRA-13127
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13127
> Project: Cassandra
> Issue Type: Bug
> Components: Local Write-Read Paths, Materialized Views
>Reporter: Duarte Nunes
>Assignee: ZhaoYang
>
> Consider the following commands, ran against trunk:
> {code}
> echo "DROP MATERIALIZED VIEW ks.mv; DROP TABLE ks.base;" | bin/cqlsh
> echo "CREATE TABLE ks.base (p int, c int, v int, PRIMARY KEY (p, c));" |
> bin/cqlsh
> echo "CREATE MATERIALIZED VIEW ks.mv AS SELECT p, c FROM base WHERE p IS NOT
> NULL AND c IS NOT NULL PRIMARY KEY (c, p);" | bin/cqlsh
> echo "INSERT INTO ks.base (p, c) VALUES (0, 0) USING TTL 10;" | bin/cqlsh
> # wait for row liveness to get closer to expiration
> sleep 6;
> echo "UPDATE ks.base USING TTL 8 SET v = 0 WHERE p = 0 and c = 0;" | bin/cqlsh
> echo "SELECT p, c, ttl(v) FROM ks.base; SELECT * FROM ks.mv;" | bin/cqlsh
> p | c | ttl(v)
> ---+---+
> 0 | 0 | 7
> (1 rows)
> c | p
> ---+---
> 0 | 0
> (1 rows)
> # wait for row liveness to expire
> sleep 4;
> echo "SELECT p, c, ttl(v) FROM ks.base; SELECT * FROM ks.mv;" | bin/cqlsh
> p | c | ttl(v)
> ---+---+
> 0 | 0 | 3
> (1 rows)
> c | p
> ---+---
> (0 rows)
> {code}
> Notice how the view row is removed even though the base row is still live. I
> would say this is because in ViewUpdateGenerator#computeLivenessInfoForEntry
> the TTLs are compared instead of the expiration times, but I'm not sure I'm
> getting that far ahead in the code when updating a column that's not in the
> view.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-13127) Materialized Views: View row expires too soon
[
https://issues.apache.org/jira/browse/CASSANDRA-13127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16090024#comment-16090024
]
ZhaoYang edited comment on CASSANDRA-13127 at 7/17/17 4:33 PM:
---
For now, base table with view doesn't support drop columns. but there is a
ticket open to support alter MV.
More precisely only for base column/cell that has timestamp/ttl and not used in
MV. (In cases of insertion, no column timestamp/ttl)
Good to carry column->(timestamp/ttl) mapping.
was (Author: jasonstack):
For now, base table with view doesn't support drop columns. but there is a
ticket open to support alter MV.
More precisely only for base column/cell that has timestamp and not used in MV.
(In cases of insertion, no column timestamp)
Good to carry column->timestamp mapping.
> Materialized Views: View row expires too soon
> -
>
> Key: CASSANDRA-13127
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13127
> Project: Cassandra
> Issue Type: Bug
> Components: Local Write-Read Paths, Materialized Views
>Reporter: Duarte Nunes
>Assignee: ZhaoYang
>
> Consider the following commands, ran against trunk:
> {code}
> echo "DROP MATERIALIZED VIEW ks.mv; DROP TABLE ks.base;" | bin/cqlsh
> echo "CREATE TABLE ks.base (p int, c int, v int, PRIMARY KEY (p, c));" |
> bin/cqlsh
> echo "CREATE MATERIALIZED VIEW ks.mv AS SELECT p, c FROM base WHERE p IS NOT
> NULL AND c IS NOT NULL PRIMARY KEY (c, p);" | bin/cqlsh
> echo "INSERT INTO ks.base (p, c) VALUES (0, 0) USING TTL 10;" | bin/cqlsh
> # wait for row liveness to get closer to expiration
> sleep 6;
> echo "UPDATE ks.base USING TTL 8 SET v = 0 WHERE p = 0 and c = 0;" | bin/cqlsh
> echo "SELECT p, c, ttl(v) FROM ks.base; SELECT * FROM ks.mv;" | bin/cqlsh
> p | c | ttl(v)
> ---+---+
> 0 | 0 | 7
> (1 rows)
> c | p
> ---+---
> 0 | 0
> (1 rows)
> # wait for row liveness to expire
> sleep 4;
> echo "SELECT p, c, ttl(v) FROM ks.base; SELECT * FROM ks.mv;" | bin/cqlsh
> p | c | ttl(v)
> ---+---+
> 0 | 0 | 3
> (1 rows)
> c | p
> ---+---
> (0 rows)
> {code}
> Notice how the view row is removed even though the base row is still live. I
> would say this is because in ViewUpdateGenerator#computeLivenessInfoForEntry
> the TTLs are compared instead of the expiration times, but I'm not sure I'm
> getting that far ahead in the code when updating a column that's not in the
> view.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13127) Materialized Views: View row expires too soon
[
https://issues.apache.org/jira/browse/CASSANDRA-13127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16090024#comment-16090024
]
ZhaoYang commented on CASSANDRA-13127:
--
For now, base table with view doesn't support drop columns. but there is a
ticket open to support alter MV.
More precisely only for base column/cell that has timestamp and not used in MV.
(In cases of insertion, no column timestamp)
Good to carry column->timestamp mapping.
> Materialized Views: View row expires too soon
> -
>
> Key: CASSANDRA-13127
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13127
> Project: Cassandra
> Issue Type: Bug
> Components: Local Write-Read Paths, Materialized Views
>Reporter: Duarte Nunes
>Assignee: ZhaoYang
>
> Consider the following commands, ran against trunk:
> {code}
> echo "DROP MATERIALIZED VIEW ks.mv; DROP TABLE ks.base;" | bin/cqlsh
> echo "CREATE TABLE ks.base (p int, c int, v int, PRIMARY KEY (p, c));" |
> bin/cqlsh
> echo "CREATE MATERIALIZED VIEW ks.mv AS SELECT p, c FROM base WHERE p IS NOT
> NULL AND c IS NOT NULL PRIMARY KEY (c, p);" | bin/cqlsh
> echo "INSERT INTO ks.base (p, c) VALUES (0, 0) USING TTL 10;" | bin/cqlsh
> # wait for row liveness to get closer to expiration
> sleep 6;
> echo "UPDATE ks.base USING TTL 8 SET v = 0 WHERE p = 0 and c = 0;" | bin/cqlsh
> echo "SELECT p, c, ttl(v) FROM ks.base; SELECT * FROM ks.mv;" | bin/cqlsh
> p | c | ttl(v)
> ---+---+
> 0 | 0 | 7
> (1 rows)
> c | p
> ---+---
> 0 | 0
> (1 rows)
> # wait for row liveness to expire
> sleep 4;
> echo "SELECT p, c, ttl(v) FROM ks.base; SELECT * FROM ks.mv;" | bin/cqlsh
> p | c | ttl(v)
> ---+---+
> 0 | 0 | 3
> (1 rows)
> c | p
> ---+---
> (0 rows)
> {code}
> Notice how the view row is removed even though the base row is still live. I
> would say this is because in ViewUpdateGenerator#computeLivenessInfoForEntry
> the TTLs are compared instead of the expiration times, but I'm not sure I'm
> getting that far ahead in the code when updating a column that's not in the
> view.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13127) Materialized Views: View row expires too soon
[
https://issues.apache.org/jira/browse/CASSANDRA-13127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1609#comment-1609
]
Fridtjof Sander commented on CASSANDRA-13127:
-
Exactly, but only from those columns not projected into the view. The
timestamps of all projected columns are available from the cells as usual.
Would we also need to carry information to identify from which column a
timestamp comes from, in case a unprojected base-column is dropped and hence
becomes irrelevant?
> Materialized Views: View row expires too soon
> -
>
> Key: CASSANDRA-13127
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13127
> Project: Cassandra
> Issue Type: Bug
> Components: Local Write-Read Paths, Materialized Views
>Reporter: Duarte Nunes
>Assignee: ZhaoYang
>
> Consider the following commands, ran against trunk:
> {code}
> echo "DROP MATERIALIZED VIEW ks.mv; DROP TABLE ks.base;" | bin/cqlsh
> echo "CREATE TABLE ks.base (p int, c int, v int, PRIMARY KEY (p, c));" |
> bin/cqlsh
> echo "CREATE MATERIALIZED VIEW ks.mv AS SELECT p, c FROM base WHERE p IS NOT
> NULL AND c IS NOT NULL PRIMARY KEY (c, p);" | bin/cqlsh
> echo "INSERT INTO ks.base (p, c) VALUES (0, 0) USING TTL 10;" | bin/cqlsh
> # wait for row liveness to get closer to expiration
> sleep 6;
> echo "UPDATE ks.base USING TTL 8 SET v = 0 WHERE p = 0 and c = 0;" | bin/cqlsh
> echo "SELECT p, c, ttl(v) FROM ks.base; SELECT * FROM ks.mv;" | bin/cqlsh
> p | c | ttl(v)
> ---+---+
> 0 | 0 | 7
> (1 rows)
> c | p
> ---+---
> 0 | 0
> (1 rows)
> # wait for row liveness to expire
> sleep 4;
> echo "SELECT p, c, ttl(v) FROM ks.base; SELECT * FROM ks.mv;" | bin/cqlsh
> p | c | ttl(v)
> ---+---+
> 0 | 0 | 3
> (1 rows)
> c | p
> ---+---
> (0 rows)
> {code}
> Notice how the view row is removed even though the base row is still live. I
> would say this is because in ViewUpdateGenerator#computeLivenessInfoForEntry
> the TTLs are compared instead of the expiration times, but I'm not sure I'm
> getting that far ahead in the code when updating a column that's not in the
> view.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13127) Materialized Views: View row expires too soon
[
https://issues.apache.org/jira/browse/CASSANDRA-13127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16089988#comment-16089988
]
ZhaoYang commented on CASSANDRA-13127:
--
Then single base column timestamp in {{shadowable-liveness-info or
shadowable-tombstone}} won't be sufficient. All base columns' timestamp are
required in {{shadowable-liveness-info or shadowable tombstone}}.
> Materialized Views: View row expires too soon
> -
>
> Key: CASSANDRA-13127
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13127
> Project: Cassandra
> Issue Type: Bug
> Components: Local Write-Read Paths, Materialized Views
>Reporter: Duarte Nunes
>Assignee: ZhaoYang
>
> Consider the following commands, ran against trunk:
> {code}
> echo "DROP MATERIALIZED VIEW ks.mv; DROP TABLE ks.base;" | bin/cqlsh
> echo "CREATE TABLE ks.base (p int, c int, v int, PRIMARY KEY (p, c));" |
> bin/cqlsh
> echo "CREATE MATERIALIZED VIEW ks.mv AS SELECT p, c FROM base WHERE p IS NOT
> NULL AND c IS NOT NULL PRIMARY KEY (c, p);" | bin/cqlsh
> echo "INSERT INTO ks.base (p, c) VALUES (0, 0) USING TTL 10;" | bin/cqlsh
> # wait for row liveness to get closer to expiration
> sleep 6;
> echo "UPDATE ks.base USING TTL 8 SET v = 0 WHERE p = 0 and c = 0;" | bin/cqlsh
> echo "SELECT p, c, ttl(v) FROM ks.base; SELECT * FROM ks.mv;" | bin/cqlsh
> p | c | ttl(v)
> ---+---+
> 0 | 0 | 7
> (1 rows)
> c | p
> ---+---
> 0 | 0
> (1 rows)
> # wait for row liveness to expire
> sleep 4;
> echo "SELECT p, c, ttl(v) FROM ks.base; SELECT * FROM ks.mv;" | bin/cqlsh
> p | c | ttl(v)
> ---+---+
> 0 | 0 | 3
> (1 rows)
> c | p
> ---+---
> (0 rows)
> {code}
> Notice how the view row is removed even though the base row is still live. I
> would say this is because in ViewUpdateGenerator#computeLivenessInfoForEntry
> the TTLs are compared instead of the expiration times, but I'm not sure I'm
> getting that far ahead in the code when updating a column that's not in the
> view.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13127) Materialized Views: View row expires too soon
[
https://issues.apache.org/jira/browse/CASSANDRA-13127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16089940#comment-16089940
]
Fridtjof Sander commented on CASSANDRA-13127:
-
I found another bug, that I believe falls into the same ballpark as this issue:
{code:java}
@Test
public void testFilterViewTombstones() throws Throwable {
execute("USE " + keyspace());
executeNet(protocolVersion, "USE " + keyspace());
createTable("CREATE TABLE %s (k int, c int, a int, b int, PRIMARY KEY
(k, c))");
createView("mv", "CREATE MATERIALIZED VIEW %s AS SELECT k,c FROM %%s
WHERE k IS NOT NULL AND c IS NOT NULL PRIMARY KEY (k,c)");
updateView("UPDATE %s USING TIMESTAMP 10 SET b=1 WHERE k=1 AND c=1");
assertRows(execute("SELECT * from %s"), row(1, 1, null, 1));
assertRows(execute("SELECT * from mv"), row(1, 1));
updateView("DELETE b FROM %s USING TIMESTAMP 11 WHERE k=1 AND c=1");
assertEmpty(execute("SELECT * from %s"));
assertEmpty(execute("SELECT * from mv"));
updateView("UPDATE %s USING TIMESTAMP 1 SET a=1 WHERE k=1 AND c=1");
assertRows(execute("SELECT * from %s"), row(1, 1, 1, null));
assertRows(execute("SELECT * from mv"), row(1, 1));
}
{code}
A view entry is created based on {{b=1@ts=10}} and deleted with {{ts=11}}.
Later another cell is added with {{a=1@ts=1}}. The view's liveness-info with
{{ts=1}} does not supersede the previously created tombstone with {{ts=11}}.
If you disagree on the similarity, I'll open a new ticket for this. Both issues
are about handling the contribution of base-cells to the view-row's liveness
that are projected out of the view.
On trunk, this doesn't even create an mv-entry (because of the empty
liveness-info), but also fails with [~jasonstack]'s patch. I'm not sure, if
this can be fixed at all without keeping hidden timestamp/liveness-data of
discarded columns.
> Materialized Views: View row expires too soon
> -
>
> Key: CASSANDRA-13127
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13127
> Project: Cassandra
> Issue Type: Bug
> Components: Local Write-Read Paths, Materialized Views
>Reporter: Duarte Nunes
>Assignee: ZhaoYang
>
> Consider the following commands, ran against trunk:
> {code}
> echo "DROP MATERIALIZED VIEW ks.mv; DROP TABLE ks.base;" | bin/cqlsh
> echo "CREATE TABLE ks.base (p int, c int, v int, PRIMARY KEY (p, c));" |
> bin/cqlsh
> echo "CREATE MATERIALIZED VIEW ks.mv AS SELECT p, c FROM base WHERE p IS NOT
> NULL AND c IS NOT NULL PRIMARY KEY (c, p);" | bin/cqlsh
> echo "INSERT INTO ks.base (p, c) VALUES (0, 0) USING TTL 10;" | bin/cqlsh
> # wait for row liveness to get closer to expiration
> sleep 6;
> echo "UPDATE ks.base USING TTL 8 SET v = 0 WHERE p = 0 and c = 0;" | bin/cqlsh
> echo "SELECT p, c, ttl(v) FROM ks.base; SELECT * FROM ks.mv;" | bin/cqlsh
> p | c | ttl(v)
> ---+---+
> 0 | 0 | 7
> (1 rows)
> c | p
> ---+---
> 0 | 0
> (1 rows)
> # wait for row liveness to expire
> sleep 4;
> echo "SELECT p, c, ttl(v) FROM ks.base; SELECT * FROM ks.mv;" | bin/cqlsh
> p | c | ttl(v)
> ---+---+
> 0 | 0 | 3
> (1 rows)
> c | p
> ---+---
> (0 rows)
> {code}
> Notice how the view row is removed even though the base row is still live. I
> would say this is because in ViewUpdateGenerator#computeLivenessInfoForEntry
> the TTLs are compared instead of the expiration times, but I'm not sure I'm
> getting that far ahead in the code when updating a column that's not in the
> view.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-11500) Obsolete MV entry may not be properly deleted
[
https://issues.apache.org/jira/browse/CASSANDRA-11500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16082241#comment-16082241
]
ZhaoYang edited comment on CASSANDRA-11500 at 7/17/17 2:13 PM:
---
h3. *Idea*
{{ShadowableTombstone}} :
* deletion-time, isShadowable, and "viewKeyTs" aka. base column's ts which is
part of view pk(used to reconcile when timestamp tie), if there is no timestamp
associated with that column, use base pk timestamp instead.
* it's only generated when one base column is a pk in view and this base column
value is changed in base row, to mark previous view row as deleted. (original
definition of {{shadowable}} in CASSANDRA-10261). in other cases, {{standard
tombstone}} is generated for view rows.
* if {{ShadowableTombstone}} is superseded by {{LivenessInfo}}, columns
shadowed by {{ShadowableTombstone}} will come back alive. (original definition
of {{shadowable}} in CASSANDRA-10261)
* {{ShadowableTombstone}} should co-exist with {{Standard Tombstone}} if
{{shadowable}}'s deletion time supersedes {{standard tombstone}} to avoid
bringing columns older than {{standard tombstone}} coming back alive( as in
CASSANDRA-13409)
{{ShadowableLivenessInfo}}:
* timestamp, and "viewKeyTs"
* if shadowable and not live, all columns in this row are considered deleted as
in CASSANDRA-13657 and CASSANDRA-13127 to solve partial update issues
When reconcile {{ShadowableTombstone}} and {{ShadowableLivenessInfo}}:
{quote}
if deletion-time greater than timestamp, tombstone wins
if deletion-time smaller than timestamp, livenessInfo wins
when deletion-time ties with timestamp,
- if {{ShadowableTombstone}}'s {{viewKeyTs}} >= {{ShadowableLivenessInfo}}'s,
then tombstone wins
- else livesnessInfo wins.
{quote}
When inserting to view, always use the greatest timestamp of all base columns
in view similar to how view deletion timestamp is computed.
h3. *Example*
{quote}
CREATE TABLE t (k int PRIMARY KEY, a int, b int);
CREATE MATERIALIZED VIEW mv AS SELECT * FROM t WHERE k IS NOT NULL AND a IS NOT
NULL PRIMARY KEY (k, a);
{{q1}} INSERT INTO t(k, a, b) VALUES (1, 1, 1) USING TIMESTAMP 0;
{{q2}} UPDATE t USING TIMESTAMP 10 SET b = 2 WHERE k = 1;
{{q3}} UPDATE t USING TIMESTAMP 2 SET a = 2 WHERE k = 1;
{{q3}} UPDATE t USING TIMESTAMP 3 SET a = 1 WHERE k = 1;
{quote}
* After {{q1}}:
** in base: {{k=1@0, a=1, b=1}}// 'k' is having value '1' with timestamp '0'
** in view:
*** sstable1: {{(k=1&&a=1)@TS(0,0), b=1}} // 'k:a' is having value '1:1' with
timestamp '0' and viewKeyTs '0' from base's pk because column 'a' has no TS
* After {{q2}}
** in base(merged): {{k=1@0, a=1, b=2@10}}
** in view:
*** sstable1: {{(k=1&&a=1)@TS(0,0), b=1}}
*** sstable2: {{(k=1&&a=1)@TS(10,0), b=2@10}}
*** or merged: {{(k=1&&a=1)@TS(10,0), b=2@10}}
* After {{q3}}
** in base(merged): {{k=1@0, a=2@2, b=2@10}}
** in view:
*** sstable1: {{(k=1&&a=1)@TS(0,0), b=1}}
*** sstable2: {{(k=1&&a=1)@TS(10,0), b=2@10}}
*** sstable3: {{(k=1&&a=1)@Shadowable(10,0)}} & {{(k=1&&a=2)@TS(10,2),
b=2@10}} // '(k=1&&a=2)' is having biggest timestamp '10' and viewKeyTs '2'
from column 'a'
*** or merged: {{(k=1&&a=2)@TS(10,2), b=2@10}}
* After {{q4}}
** in base(merged): {{k=1@0, a=1@3, b=2@10}}
** in view:
*** sstable1: {{(k=1&&a=1)@TS(0,0), b=1}}
*** sstable2: {{(k=1&&a=1)@TS(10,0), b=2@10}}
*** sstable3: {{(k=1&&a=1)@Shadowable(10,0)}} & {{(k=1&&a=2)@TS(10,2),
b=2@10}}
*** sstable4: {{(k=1&&a=2)@Shadowable(10,2)}} & {{(k=1&&a=1)@TS(10,3),
b=2@10}} // '(k=1&&a=1)' is having biggest timestamp '10' and viewKeyTs '3'
from column 'a'
*** or merged: {{(k=1&&a=1)@TS(10,3), b=2@10}}
h3. *Changes*
* Extra flag in storage serialization format to facilitate {{viewKeyTs}} and
{{co-existed standard tombstones under shadowable}}
* Message serialization to store {{viewKeyTs}}
* Row.Merger Process
* BTreeRow Filter Process
was (Author: jasonstack):
h3. *Idea*
{{ShadowableTombstone}} :
* deletion-time, isShadowable, and "viewKeyTs" aka. base column's ts which is
part of view pk(used to reconcile when timestamp tie), if there is no timestamp
associated with that column, use base pk timestamp instead.
* it's only generated when one base column is a pk in view and this base column
value is changed in base row, to mark previous view row as deleted. (original
definition of {{shadowable}} in CASSANDRA-10261). in other cases, {{standard
tombstone}} is generated for view rows.
* if {{ShadowableTombstone}} is superseded by {{LivenessInfo}}, columns
shadowed by {{ShadowableTombstone}} will come back alive. (original definition
of {{shadowable}} in CASSANDRA-10261)
* {{ShadowableTombstone}} should co-exist with {{Standard Tombstone}} if
{{shadowable}}'s deletion time supersedes {{standard tombstone}} to avoid
bringing columns older than {{standard tombstone}} coming back alive( as in
CASSANDRA-13409)
[jira] [Comment Edited] (CASSANDRA-11500) Obsolete MV entry may not be properly deleted
[
https://issues.apache.org/jira/browse/CASSANDRA-11500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16089856#comment-16089856
]
ZhaoYang edited comment on CASSANDRA-11500 at 7/17/17 1:56 PM:
---
I plan to solve: {{partial update}},{{ttl}}, {{co-existed shadowable
tombstone}}, {{view timestamp tie}} all inside this ticket using extended
shadowable approach(mentioned
[here|https://issues.apache.org/jira/browse/CASSANDRA-11500?focusedCommentId=16082241&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16082241]).
Because all these issues require some storage format changes(extendedFlag),
it's better to fix them and refactor in one commit.
Drafted a
[patch|https://github.com/jasonstack/cassandra/commits/CASSANDRA-11500-update-time]..(refactoring
and adding more dtest)
Any suggestions would be appreciated.
was (Author: jasonstack):
I plan to solve: partial update, ttl, co-existed shadowable tombstone, view
timestamp tie all inside this ticket using extended shadowable
approach(mentioned
[here|https://issues.apache.org/jira/browse/CASSANDRA-11500?focusedCommentId=16082241&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16082241]).
Because all these issues require some storage format changes(extendedFlag),
it's better to fix them and refactor in one commit.
Drafted a
[patch|https://github.com/jasonstack/cassandra/commits/CASSANDRA-11500-update-time]..(refactoring
and adding more dtest)
Any suggestions would be appreciated.
> Obsolete MV entry may not be properly deleted
> -
>
> Key: CASSANDRA-11500
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11500
> Project: Cassandra
> Issue Type: Bug
> Components: Materialized Views
>Reporter: Sylvain Lebresne
>Assignee: ZhaoYang
>
> When a Materialized View uses a non-PK base table column in its PK, if an
> update changes that column value, we add the new view entry and remove the
> old one. When doing that removal, the current code uses the same timestamp
> than for the liveness info of the new entry, which is the max timestamp for
> any columns participating to the view PK. This is not correct for the
> deletion as the old view entry could have other columns with higher timestamp
> which won't be deleted as can easily shown by the failing of the following
> test:
> {noformat}
> CREATE TABLE t (k int PRIMARY KEY, a int, b int);
> CREATE MATERIALIZED VIEW mv AS SELECT * FROM t WHERE k IS NOT NULL AND a IS
> NOT NULL PRIMARY KEY (k, a);
> INSERT INTO t(k, a, b) VALUES (1, 1, 1) USING TIMESTAMP 0;
> UPDATE t USING TIMESTAMP 4 SET b = 2 WHERE k = 1;
> UPDATE t USING TIMESTAMP 2 SET a = 2 WHERE k = 1;
> SELECT * FROM mv WHERE k = 1; // This currently return 2 entries, the old
> (invalid) and the new one
> {noformat}
> So the correct timestamp to use for the deletion is the biggest timestamp in
> the old view entry (which we know since we read the pre-existing base row),
> and that is what CASSANDRA-11475 does (the test above thus doesn't fail on
> that branch).
> Unfortunately, even then we can still have problems if further updates
> requires us to overide the old entry. Consider the following case:
> {noformat}
> CREATE TABLE t (k int PRIMARY KEY, a int, b int);
> CREATE MATERIALIZED VIEW mv AS SELECT * FROM t WHERE k IS NOT NULL AND a IS
> NOT NULL PRIMARY KEY (k, a);
> INSERT INTO t(k, a, b) VALUES (1, 1, 1) USING TIMESTAMP 0;
> UPDATE t USING TIMESTAMP 10 SET b = 2 WHERE k = 1;
> UPDATE t USING TIMESTAMP 2 SET a = 2 WHERE k = 1; // This will delete the
> entry for a=1 with timestamp 10
> UPDATE t USING TIMESTAMP 3 SET a = 1 WHERE k = 1; // This needs to re-insert
> an entry for a=1 but shouldn't be deleted by the prior deletion
> UPDATE t USING TIMESTAMP 4 SET a = 2 WHERE k = 1; // ... and we can play this
> game more than once
> UPDATE t USING TIMESTAMP 5 SET a = 1 WHERE k = 1;
> ...
> {noformat}
> In a way, this is saying that the "shadowable" deletion mechanism is not
> general enough: we need to be able to re-insert an entry when a prior one had
> been deleted before, but we can't rely on timestamps being strictly bigger on
> the re-insert. In that sense, this can be though as a similar problem than
> CASSANDRA-10965, though the solution there of a single flag is not enough
> since we can have to replace more than once.
> I think the proper solution would be to ship enough information to always be
> able to decide when a view deletion is shadowed. Which means that both
> liveness info (for updates) and shadowable deletion would need to ship the
> timestamp of any base table column that is part the view PK (so {{a}} in the
> example below). It's doable (and not that hard really), but it does require
> a change
[jira] [Commented] (CASSANDRA-11500) Obsolete MV entry may not be properly deleted
[
https://issues.apache.org/jira/browse/CASSANDRA-11500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16089856#comment-16089856
]
ZhaoYang commented on CASSANDRA-11500:
--
I plan to solve: partial update, ttl, co-existed shadowable tombstone, view
timestamp tie all inside this ticket using extended shadowable
approach(mentioned
[here|https://issues.apache.org/jira/browse/CASSANDRA-11500?focusedCommentId=16082241&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16082241]).
Because all these issues require some storage format changes(extendedFlag),
it's better to fix them and refactor in one commit.
Drafted a
[patch|https://github.com/jasonstack/cassandra/commits/CASSANDRA-11500-update-time]..(refactoring
and adding more dtest)
Any suggestions would be appreciated.
> Obsolete MV entry may not be properly deleted
> -
>
> Key: CASSANDRA-11500
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11500
> Project: Cassandra
> Issue Type: Bug
> Components: Materialized Views
>Reporter: Sylvain Lebresne
>Assignee: ZhaoYang
>
> When a Materialized View uses a non-PK base table column in its PK, if an
> update changes that column value, we add the new view entry and remove the
> old one. When doing that removal, the current code uses the same timestamp
> than for the liveness info of the new entry, which is the max timestamp for
> any columns participating to the view PK. This is not correct for the
> deletion as the old view entry could have other columns with higher timestamp
> which won't be deleted as can easily shown by the failing of the following
> test:
> {noformat}
> CREATE TABLE t (k int PRIMARY KEY, a int, b int);
> CREATE MATERIALIZED VIEW mv AS SELECT * FROM t WHERE k IS NOT NULL AND a IS
> NOT NULL PRIMARY KEY (k, a);
> INSERT INTO t(k, a, b) VALUES (1, 1, 1) USING TIMESTAMP 0;
> UPDATE t USING TIMESTAMP 4 SET b = 2 WHERE k = 1;
> UPDATE t USING TIMESTAMP 2 SET a = 2 WHERE k = 1;
> SELECT * FROM mv WHERE k = 1; // This currently return 2 entries, the old
> (invalid) and the new one
> {noformat}
> So the correct timestamp to use for the deletion is the biggest timestamp in
> the old view entry (which we know since we read the pre-existing base row),
> and that is what CASSANDRA-11475 does (the test above thus doesn't fail on
> that branch).
> Unfortunately, even then we can still have problems if further updates
> requires us to overide the old entry. Consider the following case:
> {noformat}
> CREATE TABLE t (k int PRIMARY KEY, a int, b int);
> CREATE MATERIALIZED VIEW mv AS SELECT * FROM t WHERE k IS NOT NULL AND a IS
> NOT NULL PRIMARY KEY (k, a);
> INSERT INTO t(k, a, b) VALUES (1, 1, 1) USING TIMESTAMP 0;
> UPDATE t USING TIMESTAMP 10 SET b = 2 WHERE k = 1;
> UPDATE t USING TIMESTAMP 2 SET a = 2 WHERE k = 1; // This will delete the
> entry for a=1 with timestamp 10
> UPDATE t USING TIMESTAMP 3 SET a = 1 WHERE k = 1; // This needs to re-insert
> an entry for a=1 but shouldn't be deleted by the prior deletion
> UPDATE t USING TIMESTAMP 4 SET a = 2 WHERE k = 1; // ... and we can play this
> game more than once
> UPDATE t USING TIMESTAMP 5 SET a = 1 WHERE k = 1;
> ...
> {noformat}
> In a way, this is saying that the "shadowable" deletion mechanism is not
> general enough: we need to be able to re-insert an entry when a prior one had
> been deleted before, but we can't rely on timestamps being strictly bigger on
> the re-insert. In that sense, this can be though as a similar problem than
> CASSANDRA-10965, though the solution there of a single flag is not enough
> since we can have to replace more than once.
> I think the proper solution would be to ship enough information to always be
> able to decide when a view deletion is shadowed. Which means that both
> liveness info (for updates) and shadowable deletion would need to ship the
> timestamp of any base table column that is part the view PK (so {{a}} in the
> example below). It's doable (and not that hard really), but it does require
> a change to the sstable and intra-node protocol, which makes this a bit
> painful right now.
> But I'll also note that as CASSANDRA-1096 shows, the timestamp is not even
> enough since on equal timestamp the value can be the deciding factor. So in
> theory we'd have to ship the value of those columns (in the case of a
> deletion at least since we have it in the view PK for updates). That said, on
> that last problem, my preference would be that we start prioritizing
> CASSANDRA-6123 seriously so we don't have to care about conflicting timestamp
> anymore, which would make this problem go away.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To u
[jira] [Updated] (CASSANDRA-11500) Obsolete MV entry may not be properly deleted
[
https://issues.apache.org/jira/browse/CASSANDRA-11500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ZhaoYang updated CASSANDRA-11500:
-
Status: In Progress (was: Awaiting Feedback)
> Obsolete MV entry may not be properly deleted
> -
>
> Key: CASSANDRA-11500
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11500
> Project: Cassandra
> Issue Type: Bug
> Components: Materialized Views
>Reporter: Sylvain Lebresne
>Assignee: ZhaoYang
>
> When a Materialized View uses a non-PK base table column in its PK, if an
> update changes that column value, we add the new view entry and remove the
> old one. When doing that removal, the current code uses the same timestamp
> than for the liveness info of the new entry, which is the max timestamp for
> any columns participating to the view PK. This is not correct for the
> deletion as the old view entry could have other columns with higher timestamp
> which won't be deleted as can easily shown by the failing of the following
> test:
> {noformat}
> CREATE TABLE t (k int PRIMARY KEY, a int, b int);
> CREATE MATERIALIZED VIEW mv AS SELECT * FROM t WHERE k IS NOT NULL AND a IS
> NOT NULL PRIMARY KEY (k, a);
> INSERT INTO t(k, a, b) VALUES (1, 1, 1) USING TIMESTAMP 0;
> UPDATE t USING TIMESTAMP 4 SET b = 2 WHERE k = 1;
> UPDATE t USING TIMESTAMP 2 SET a = 2 WHERE k = 1;
> SELECT * FROM mv WHERE k = 1; // This currently return 2 entries, the old
> (invalid) and the new one
> {noformat}
> So the correct timestamp to use for the deletion is the biggest timestamp in
> the old view entry (which we know since we read the pre-existing base row),
> and that is what CASSANDRA-11475 does (the test above thus doesn't fail on
> that branch).
> Unfortunately, even then we can still have problems if further updates
> requires us to overide the old entry. Consider the following case:
> {noformat}
> CREATE TABLE t (k int PRIMARY KEY, a int, b int);
> CREATE MATERIALIZED VIEW mv AS SELECT * FROM t WHERE k IS NOT NULL AND a IS
> NOT NULL PRIMARY KEY (k, a);
> INSERT INTO t(k, a, b) VALUES (1, 1, 1) USING TIMESTAMP 0;
> UPDATE t USING TIMESTAMP 10 SET b = 2 WHERE k = 1;
> UPDATE t USING TIMESTAMP 2 SET a = 2 WHERE k = 1; // This will delete the
> entry for a=1 with timestamp 10
> UPDATE t USING TIMESTAMP 3 SET a = 1 WHERE k = 1; // This needs to re-insert
> an entry for a=1 but shouldn't be deleted by the prior deletion
> UPDATE t USING TIMESTAMP 4 SET a = 2 WHERE k = 1; // ... and we can play this
> game more than once
> UPDATE t USING TIMESTAMP 5 SET a = 1 WHERE k = 1;
> ...
> {noformat}
> In a way, this is saying that the "shadowable" deletion mechanism is not
> general enough: we need to be able to re-insert an entry when a prior one had
> been deleted before, but we can't rely on timestamps being strictly bigger on
> the re-insert. In that sense, this can be though as a similar problem than
> CASSANDRA-10965, though the solution there of a single flag is not enough
> since we can have to replace more than once.
> I think the proper solution would be to ship enough information to always be
> able to decide when a view deletion is shadowed. Which means that both
> liveness info (for updates) and shadowable deletion would need to ship the
> timestamp of any base table column that is part the view PK (so {{a}} in the
> example below). It's doable (and not that hard really), but it does require
> a change to the sstable and intra-node protocol, which makes this a bit
> painful right now.
> But I'll also note that as CASSANDRA-1096 shows, the timestamp is not even
> enough since on equal timestamp the value can be the deciding factor. So in
> theory we'd have to ship the value of those columns (in the case of a
> deletion at least since we have it in the view PK for updates). That said, on
> that last problem, my preference would be that we start prioritizing
> CASSANDRA-6123 seriously so we don't have to care about conflicting timestamp
> anymore, which would make this problem go away.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-11500) Obsolete MV entry may not be properly deleted
[
https://issues.apache.org/jira/browse/CASSANDRA-11500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16066749#comment-16066749
]
ZhaoYang edited comment on CASSANDRA-11500 at 7/17/17 1:46 PM:
---
{quote}
we need to be able to re-insert an entry when a prior one had been deleted
before, but we can't rely on timestamps being strictly bigger on the re-insert.
In that sense, this can be though as a similar problem than CASSANDRA-10965,
though the solution there of a single flag is not enough since we can have to
replace more than once.
{quote}
Agreed.
-How about shipping an extra "view-update-time" (`nowInSecond` that
view-operation is triggered) per view row. it will be used to check who's new
when TS ties.-
-The `nowInSeconds` could still be the same in certain cases, but rare, similar
to CASSANDRA-1096-
was (Author: jasonstack):
{quote}
we need to be able to re-insert an entry when a prior one had been deleted
before, but we can't rely on timestamps being strictly bigger on the re-insert.
In that sense, this can be though as a similar problem than CASSANDRA-10965,
though the solution there of a single flag is not enough since we can have to
replace more than once.
{quote}
Agreed.
How about shipping an extra "view-update-time" (`nowInSecond` that
view-operation is triggered) per view row. it will be used to check who's new
when TS ties.
The `nowInSeconds` could still be the same in certain cases, but rare, similar
to CASSANDRA-1096
> Obsolete MV entry may not be properly deleted
> -
>
> Key: CASSANDRA-11500
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11500
> Project: Cassandra
> Issue Type: Bug
> Components: Materialized Views
>Reporter: Sylvain Lebresne
>Assignee: ZhaoYang
>
> When a Materialized View uses a non-PK base table column in its PK, if an
> update changes that column value, we add the new view entry and remove the
> old one. When doing that removal, the current code uses the same timestamp
> than for the liveness info of the new entry, which is the max timestamp for
> any columns participating to the view PK. This is not correct for the
> deletion as the old view entry could have other columns with higher timestamp
> which won't be deleted as can easily shown by the failing of the following
> test:
> {noformat}
> CREATE TABLE t (k int PRIMARY KEY, a int, b int);
> CREATE MATERIALIZED VIEW mv AS SELECT * FROM t WHERE k IS NOT NULL AND a IS
> NOT NULL PRIMARY KEY (k, a);
> INSERT INTO t(k, a, b) VALUES (1, 1, 1) USING TIMESTAMP 0;
> UPDATE t USING TIMESTAMP 4 SET b = 2 WHERE k = 1;
> UPDATE t USING TIMESTAMP 2 SET a = 2 WHERE k = 1;
> SELECT * FROM mv WHERE k = 1; // This currently return 2 entries, the old
> (invalid) and the new one
> {noformat}
> So the correct timestamp to use for the deletion is the biggest timestamp in
> the old view entry (which we know since we read the pre-existing base row),
> and that is what CASSANDRA-11475 does (the test above thus doesn't fail on
> that branch).
> Unfortunately, even then we can still have problems if further updates
> requires us to overide the old entry. Consider the following case:
> {noformat}
> CREATE TABLE t (k int PRIMARY KEY, a int, b int);
> CREATE MATERIALIZED VIEW mv AS SELECT * FROM t WHERE k IS NOT NULL AND a IS
> NOT NULL PRIMARY KEY (k, a);
> INSERT INTO t(k, a, b) VALUES (1, 1, 1) USING TIMESTAMP 0;
> UPDATE t USING TIMESTAMP 10 SET b = 2 WHERE k = 1;
> UPDATE t USING TIMESTAMP 2 SET a = 2 WHERE k = 1; // This will delete the
> entry for a=1 with timestamp 10
> UPDATE t USING TIMESTAMP 3 SET a = 1 WHERE k = 1; // This needs to re-insert
> an entry for a=1 but shouldn't be deleted by the prior deletion
> UPDATE t USING TIMESTAMP 4 SET a = 2 WHERE k = 1; // ... and we can play this
> game more than once
> UPDATE t USING TIMESTAMP 5 SET a = 1 WHERE k = 1;
> ...
> {noformat}
> In a way, this is saying that the "shadowable" deletion mechanism is not
> general enough: we need to be able to re-insert an entry when a prior one had
> been deleted before, but we can't rely on timestamps being strictly bigger on
> the re-insert. In that sense, this can be though as a similar problem than
> CASSANDRA-10965, though the solution there of a single flag is not enough
> since we can have to replace more than once.
> I think the proper solution would be to ship enough information to always be
> able to decide when a view deletion is shadowed. Which means that both
> liveness info (for updates) and shadowable deletion would need to ship the
> timestamp of any base table column that is part the view PK (so {{a}} in the
> example below). It's doable (and not that hard really), but it does require
> a change to the sstable and intra-node protocol, which m
[jira] [Commented] (CASSANDRA-13564) Mismatch Documentation on MATERIALIZE VIEW
[ https://issues.apache.org/jira/browse/CASSANDRA-13564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16089816#comment-16089816 ] Nick Hryhoriev commented on CASSANDRA-13564: I have try create Table with two columns in PK, and add Materialize View for it with 3 Pk, with out clustering key. > Mismatch Documentation on MATERIALIZE VIEW > -- > > Key: CASSANDRA-13564 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13564 > Project: Cassandra > Issue Type: Improvement > Components: Documentation and Website, Materialized Views > Environment: 3.10 >Reporter: Nick Hryhoriev > Labels: doc-impacting > > During create MATERIALIZED VIEW with out cluster key. > I've got exception "No columns are defined for Materialized View other than > primary key" > Does it expected behave? Because i can't find nothing about this in > Documentation. But this check exist in code. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-12965) StreamReceiveTask causing high CPU utilization during repair
[
https://issues.apache.org/jira/browse/CASSANDRA-12965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16089687#comment-16089687
]
Guillaume Drot commented on CASSANDRA-12965:
[~rbfblk] Did you find a solution to your problem ? We have the same problem
since we have upgraded to DSE OpsCenter 6.
[~pauloricardomg] Did you had time to investigate the issue ?
Thanks.
> StreamReceiveTask causing high CPU utilization during repair
>
>
> Key: CASSANDRA-12965
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12965
> Project: Cassandra
> Issue Type: Bug
>Reporter: Randy Fradin
>
> During a full repair run, I observed one node in my cluster using 100% cpu
> (100% of all cores on a 48-core machine). When I took a stack trace I found
> exactly 48 running StreamReceiveTask threads. Each was in the same block of
> code in StreamReceiveTask.OnCompletionRunnable:
> {noformat}
> "StreamReceiveTask:8077" #1511134 daemon prio=5 os_prio=0
> tid=0x7f01520a8800 nid=0x6e77 runnable [0x7f020dfae000]
>java.lang.Thread.State: RUNNABLE
> at java.util.ComparableTimSort.binarySort(ComparableTimSort.java:258)
> at java.util.ComparableTimSort.sort(ComparableTimSort.java:203)
> at java.util.Arrays.sort(Arrays.java:1312)
> at java.util.Arrays.sort(Arrays.java:1506)
> at java.util.ArrayList.sort(ArrayList.java:1454)
> at java.util.Collections.sort(Collections.java:141)
> at
> org.apache.cassandra.utils.IntervalTree$IntervalNode.(IntervalTree.java:257)
> at
> org.apache.cassandra.utils.IntervalTree$IntervalNode.(IntervalTree.java:280)
> at
> org.apache.cassandra.utils.IntervalTree.(IntervalTree.java:72)
> at
> org.apache.cassandra.db.DataTracker$SSTableIntervalTree.(DataTracker.java:590)
> at
> org.apache.cassandra.db.DataTracker$SSTableIntervalTree.(DataTracker.java:584)
> at
> org.apache.cassandra.db.DataTracker.buildIntervalTree(DataTracker.java:565)
> at
> org.apache.cassandra.db.DataTracker$View.replace(DataTracker.java:761)
> at
> org.apache.cassandra.db.DataTracker.addSSTablesToTracker(DataTracker.java:428)
> at
> org.apache.cassandra.db.DataTracker.addSSTables(DataTracker.java:283)
> at
> org.apache.cassandra.db.ColumnFamilyStore.addSSTables(ColumnFamilyStore.java:1422)
> at
> org.apache.cassandra.streaming.StreamReceiveTask$OnCompletionRunnable.run(StreamReceiveTask.java:148)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
> All 48 threads were in ColumnFamilyStore.addSSTables(), and specifically in
> the IntervalNode constructor called from the IntervalTree constructor.
> It stayed this way for maybe an hour before we restarted the node. The repair
> was also generating thousands (20,000+) of tiny SSTables in a table that
> previously had just 20.
> I don't know enough about SSTables and ColumnFamilyStore to know if all this
> CPU work is necessary or a bug, but I did notice that these tasks are run on
> a thread pool constructed in StreamReceiveTask.java, so perhaps this pool
> should have a thread count max less than the number of processors on the
> machine, at least for machines with a lot of processors. Any reason not to do
> that? Any ideas for a reasonable # or formula to cap the thread count?
> Some additional info: We have never run incremental repair on this cluster,
> so that is not a factor. All our tables use LCS. Unfortunately I don't have
> the log files from the period saved.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13639) SSTableLoader always uses hostname to stream files from
[ https://issues.apache.org/jira/browse/CASSANDRA-13639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16089554#comment-16089554 ] Stefan Podkowinski commented on CASSANDRA-13639: This should better be addressed by adding a new command line parameter that would work for both ssl/non-ssl. I'm also wondering if this really qualifies as a bug, since nothing really indicates listenAddress or listenInterface is used from cassandra.yaml. > SSTableLoader always uses hostname to stream files from > --- > > Key: CASSANDRA-13639 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13639 > Project: Cassandra > Issue Type: Bug > Components: Tools >Reporter: Jan Karlsson >Assignee: Jan Karlsson > Fix For: 4.x > > Attachments: 13639-trunk > > > I stumbled upon an issue where SSTableLoader was ignoring our routing by > using the wrong interface to send the SSTables to the other nodes. Looking at > the code, it seems that we are using FBUtilities.getLocalAddress() to fetch > out the hostname, even if the yaml file specifies a different host. I am not > sure why we call this function instead of using the routing by leaving it > blank, perhaps someone could enlighten me. > This behaviour comes from the fact that we use a default created > DatabaseDescriptor which does not set the values for listenAddress and > listenInterface. This causes the aforementioned function to retrieve the > hostname at all times, even if it is not the interface used in the yaml file. > I propose we break out the function that handles listenAddress and > listenInterface and call it so that listenAddress or listenInterface is > getting populated in the DatabaseDescriptor. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13639) SSTableLoader always uses hostname to stream files from
[
https://issues.apache.org/jira/browse/CASSANDRA-13639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16089522#comment-16089522
]
Jan Karlsson commented on CASSANDRA-13639:
--
The problem stems from the fact that the SSTableLoader has his own way of
reading the yaml file but still uses the a default created DatabaseDescriptor
to connect by using {{FBUtilities.getLocalAddress()}}. Perhaps another solution
maybe to add this as a parameter to SSTableLoader.
In BulkLoadConnectionFactory, after a rather strange if clause that is always
false, {{SSLFactory.getSocket(encryptionOptions, peer, secureStoragePort,
FBUtilities.getLocalAddress(), 0);}} fetches the IP address from the
DatabaseDescriptor which will return null because the listenAddress is not set
by default on the DatabaseDescriptor object. My patch applies the listen
address from the yaml file to the DatabaseDescriptor which in turn fixes the
issue.
> SSTableLoader always uses hostname to stream files from
> ---
>
> Key: CASSANDRA-13639
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13639
> Project: Cassandra
> Issue Type: Bug
> Components: Tools
>Reporter: Jan Karlsson
>Assignee: Jan Karlsson
> Fix For: 4.x
>
> Attachments: 13639-trunk
>
>
> I stumbled upon an issue where SSTableLoader was ignoring our routing by
> using the wrong interface to send the SSTables to the other nodes. Looking at
> the code, it seems that we are using FBUtilities.getLocalAddress() to fetch
> out the hostname, even if the yaml file specifies a different host. I am not
> sure why we call this function instead of using the routing by leaving it
> blank, perhaps someone could enlighten me.
> This behaviour comes from the fact that we use a default created
> DatabaseDescriptor which does not set the values for listenAddress and
> listenInterface. This causes the aforementioned function to retrieve the
> hostname at all times, even if it is not the interface used in the yaml file.
> I propose we break out the function that handles listenAddress and
> listenInterface and call it so that listenAddress or listenInterface is
> getting populated in the DatabaseDescriptor.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13127) Materialized Views: View row expires too soon
[
https://issues.apache.org/jira/browse/CASSANDRA-13127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16089502#comment-16089502
]
ZhaoYang commented on CASSANDRA-13127:
--
Updated the patch. Now MV should handle partial update properly, UPDATE
semantic is now same as INSERT which means as long as data exists in base, the
data should exist in view.
It introduces {{Shadowable-LivenessInfo}}. if {{livenessInfo}} is
{{shadowable}} and {{not live}}, all columns in the row are considered deleted.
And this {{Shadowable-LivenessInfo}} can be further extended for
CASSANDRA-11500 to solve view timestamp tie.
> Materialized Views: View row expires too soon
> -
>
> Key: CASSANDRA-13127
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13127
> Project: Cassandra
> Issue Type: Bug
> Components: Local Write-Read Paths, Materialized Views
>Reporter: Duarte Nunes
>Assignee: ZhaoYang
>
> Consider the following commands, ran against trunk:
> {code}
> echo "DROP MATERIALIZED VIEW ks.mv; DROP TABLE ks.base;" | bin/cqlsh
> echo "CREATE TABLE ks.base (p int, c int, v int, PRIMARY KEY (p, c));" |
> bin/cqlsh
> echo "CREATE MATERIALIZED VIEW ks.mv AS SELECT p, c FROM base WHERE p IS NOT
> NULL AND c IS NOT NULL PRIMARY KEY (c, p);" | bin/cqlsh
> echo "INSERT INTO ks.base (p, c) VALUES (0, 0) USING TTL 10;" | bin/cqlsh
> # wait for row liveness to get closer to expiration
> sleep 6;
> echo "UPDATE ks.base USING TTL 8 SET v = 0 WHERE p = 0 and c = 0;" | bin/cqlsh
> echo "SELECT p, c, ttl(v) FROM ks.base; SELECT * FROM ks.mv;" | bin/cqlsh
> p | c | ttl(v)
> ---+---+
> 0 | 0 | 7
> (1 rows)
> c | p
> ---+---
> 0 | 0
> (1 rows)
> # wait for row liveness to expire
> sleep 4;
> echo "SELECT p, c, ttl(v) FROM ks.base; SELECT * FROM ks.mv;" | bin/cqlsh
> p | c | ttl(v)
> ---+---+
> 0 | 0 | 3
> (1 rows)
> c | p
> ---+---
> (0 rows)
> {code}
> Notice how the view row is removed even though the base row is still live. I
> would say this is because in ViewUpdateGenerator#computeLivenessInfoForEntry
> the TTLs are compared instead of the expiration times, but I'm not sure I'm
> getting that far ahead in the code when updating a column that's not in the
> view.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-11500) Obsolete MV entry may not be properly deleted
[
https://issues.apache.org/jira/browse/CASSANDRA-11500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16082241#comment-16082241
]
ZhaoYang edited comment on CASSANDRA-11500 at 7/17/17 8:35 AM:
---
h3. *Idea*
{{ShadowableTombstone}} :
* deletion-time, isShadowable, and "viewKeyTs" aka. base column's ts which is
part of view pk(used to reconcile when timestamp tie), if there is no timestamp
associated with that column, use base pk timestamp instead.
* it's only generated when one base column is a pk in view and this base column
value is changed in base row, to mark previous view row as deleted. (original
definition of {{shadowable}} in CASSANDRA-10261). in other cases, {{standard
tombstone}} is generated for view rows.
* if {{ShadowableTombstone}} is superseded by {{LivenessInfo}}, columns
shadowed by {{ShadowableTombstone}} will come back alive. (original definition
of {{shadowable}} in CASSANDRA-10261)
* {{ShadowableTombstone}} should co-exist with {{Standard Tombstone}} if
{{shadowable}}'s deletion time supersedes {{standard tombstone}} to avoid
bringing columns older than {{standard tombstone}} coming back alive( as in
CASSANDRA-13409)
{{ShadowableLivenessInfo}}:
* timestamp, and "viewKeyTs"
* if shadowable and not live, all columns in this row are considered deleted as
in CASSANDRA-13657 and CASSANDRA-13127 to solve partial update issues
When reconcile {{ShadowableTombstone}} and {{ShadowableLivenessInfo}}:
{quote}
if deletion-time greater than timestamp, tombstone wins
if deletion-time smaller than timestamp, livenessInfo wins
when deletion-time ties with timestamp,
- if {{ShadowableTombstone}}'s {{viewKeyTs}} >= {{ShadowableLivenessInfo}}'s,
then tombstone wins
- else livesnessInfo wins.
{quote}
When inserting to view, always use the greatest timestamp of all base columns
in view similar to how view deletion timestamp is computed.
h3. *Example*
{quote}
CREATE TABLE t (k int PRIMARY KEY, a int, b int);
CREATE MATERIALIZED VIEW mv AS SELECT * FROM t WHERE k IS NOT NULL AND a IS NOT
NULL PRIMARY KEY (k, a);
{{q1}} INSERT INTO t(k, a, b) VALUES (1, 1, 1) USING TIMESTAMP 0;
{{q2}} UPDATE t USING TIMESTAMP 10 SET b = 2 WHERE k = 1;
{{q3}} UPDATE t USING TIMESTAMP 2 SET a = 2 WHERE k = 1;
{{q3}} UPDATE t USING TIMESTAMP 3 SET a = 1 WHERE k = 1;
{quote}
* After {{q1}}:
** in base: {{k=1@0, a=1, b=1}}// 'k' is having value '1' with timestamp '0'
** in view:
*** sstable1: {{(k=1&&a=1)@TS(0,0), b=1}} // 'k:a' is having value '1:1' with
timestamp '0' and viewKeyTs '0' from base's pk because column 'a' has no TS
* After {{q2}}
** in base(merged): {{k=1@0, a=1, b=2@10}}
** in view:
*** sstable1: {{(k=1&&a=1)@TS(0,0), b=1}}
*** sstable2: {{(k=1&&a=1)@TS(10,0), b=2@10}}
*** or merged: {{(k=1&&a=1)@TS(10,0), b=2@10}}
* After {{q3}}
** in base(merged): {{k=1@0, a=2@2, b=2@10}}
** in view:
*** sstable1: {{(k=1&&a=1)@TS(0,0), b=1}}
*** sstable2: {{(k=1&&a=1)@TS(10,0), b=2@10}}
*** sstable3: {{(k=1&&a=1)@Shadowable(10,0)}} & {{(k=1&&a=2)@TS(10,2),
b=2@10}} // '(k=1&&a=2)' is having biggest timestamp '10' and viewKeyTs '2'
from column 'a'
*** or merged: {{(k=1&&a=2)@TS(10,2), b=2@10}}
* After {{q4}}
** in base(merged): {{k=1@0, a=1@3, b=2@10}}
** in view:
*** sstable1: {{(k=1&&a=1)@TS(0,0), b=1}}
*** sstable2: {{(k=1&&a=1)@TS(10,0), b=2@10}}
*** sstable3: {{(k=1&&a=1)@Shadowable(10,0)}} & {{(k=1&&a=2)@TS(10,2),
b=2@10}}
*** sstable4: {{(k=1&&a=2)@Shadowable(10,2)}} & {{(k=1&&a=1)@TS(10,3),
b=2@10}} // '(k=1&&a=1)' is having biggest timestamp '10' and viewKeyTs '3'
from column 'a'
*** or merged: {{(k=1&&a=1)@TS(10,3), b=2@10}}
h3. *Changes*
* Extra flag in storage serialization format to facilitate {{viewKeyTs}} and
{{co-existed standard tombstones under shadowable}}
* Message serialization to store {{viewKeyTs}}
* Row.Merger Process
was (Author: jasonstack):
h3. *Idea*
{{ShadowableTombstone}} :
* deletion-time, isShadowable, and "viewKeyTs" aka. base column's ts which is
part of view pk(used to reconcile when timestamp tie), if there is no timestamp
associated with that column, use base pk timestamp instead.
* it's only generated when one base column is a pk in view and this base column
value is changed in base row, to mark previous view row as deleted. (original
definition of {{shadowable}} in CASSANDRA-10261). in other cases, {{standard
tombstone}} is generated for view rows.
* if {{ShadowableTombstone}} is superseded by {{LivenessInfo}}, columns
shadowed by {{ShadowableTombstone}} will come back alive. (original definition
of {{shadowable}} in CASSANDRA-10261)
* {{ShadowableTombstone}} should co-exist with {{Standard Tombstone}} if
{{shadowable}}'s deletion time supersedes {{standard tombstone}} to avoid
bringing columns older than {{standard tombstone}} coming back alive( as in
CASSANDRA-13409)
{{ShadowableLivenessInfo
[jira] [Commented] (CASSANDRA-13657) Materialized Views: Index MV on TTL'ed column produces orphanized view entry if another column keeps entry live
[
https://issues.apache.org/jira/browse/CASSANDRA-13657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16089494#comment-16089494
]
ZhaoYang commented on CASSANDRA-13657:
--
I updated the patch for CASSANDRA-13127 which now handles partial updates using
similar idea here and CASSANDRA-11500.
It's to put one more marker in {{LivessnessInfo}} called {{shadowable}} ( we
have already have shadowable-deletion..consistent naming).
If {{LivessnessInfo}} is {{shadowable}} and {{not live()}}, all columns in that
row are considered deleted.
> Materialized Views: Index MV on TTL'ed column produces orphanized view entry
> if another column keeps entry live
> ---
>
> Key: CASSANDRA-13657
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13657
> Project: Cassandra
> Issue Type: Bug
> Components: Materialized Views
>Reporter: Fridtjof Sander
>Assignee: Krishna Dattu Koneru
> Labels: materializedviews, ttl
>
> {noformat}
> CREATE TABLE t (k int, a int, b int, PRIMARY KEY (k));
> CREATE MATERIALIZED VIEW mv AS SELECT * FROM t WHERE k IS NOT NULL AND a IS
> NOT NULL PRIMARY KEY (a, k);
> INSERT INTO t (k) VALUES (1);
> UPDATE t USING TTL 5 SET a = 10 WHERE k = 1;
> UPDATE t SET b = 100 WHERE k = 1;
> SELECT * from t; SELECT * from mv;
> k | a | b
> ---++-
> 1 | 10 | 100
> (1 rows)
> a | k | b
> +---+-
> 10 | 1 | 100
> (1 rows)
> -- 5 seconds later
> SELECT * from t; SELECT * from mv;
> k | a| b
> ---+--+-
> 1 | null | 100
> (1 rows)
> a | k | b
> +---+-
> 10 | 1 | 100
> (1 rows)
> -- that view entry's liveness-info is (probably) dead, but the entry is kept
> alive by b=100
> DELETE b FROM t WHERE k=1;
> SELECT * from t; SELECT * from mv;
> k | a| b
> ---+--+--
> 1 | null | null
> (1 rows)
> a | k | b
> +---+-
> 10 | 1 | 100
> (1 rows)
> DELETE FROM t WHERE k=1;
> cqlsh:test> SELECT * from t; SELECT * from mv;
> k | a | b
> ---+---+---
> (0 rows)
> a | k | b
> +---+-
> 10 | 1 | 100
> (1 rows)
> -- deleting the base-entry doesn't help, because the view-key can not be
> constructed anymore (a=10 already expired)
> {noformat}
> The problem here is that although the view-entry's liveness-info (probably)
> expired correctly a regular column (`b`) keeps the view-entry live. It should
> have disappeared since it's indexed column (`a`) expired in the corresponding
> base-row. This is pretty severe, since that view-entry is now orphanized.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-13127) Materialized Views: View row expires too soon
[
https://issues.apache.org/jira/browse/CASSANDRA-13127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16083764#comment-16083764
]
ZhaoYang edited comment on CASSANDRA-13127 at 7/17/17 8:28 AM:
---
{{when updating base's columns which are not used in view.}} is about whether
we should consider `Update semantic the same as Insert`. (for now it's not the
same. update-statement has no primary key liveness info. pk's liveness of
update-statement is depending on liveness of normal columns)
eg. current semantic/behavior:
{quote}
base table: create table ks.base (p int, c int, v int, primary key (p, c))
view table: select p,c from ks.base ... primary key (c, p)
an update query on base table normal column {{v}} will not generate any rows..
a delete query on base table normal column {{v}} will remove the base row.
{quote}
imo, to avoid user confusion, it's better to keep view matched with base
regardless semantic of update..
[~tjake][~slebresne] what do you think?
was (Author: jasonstack):
{{when updating base's columns which are not used in view.}} is about whether
we should consider `Update semantic the same as Insert`. (for now it's not the
same. update-statement has no primary key liveness info. pk's liveness of
update-statement is depending on liveness of normal columns)
eg. current semantic/behavior:
{quote}
base table: create table ks.base (p int, c int, v int, primary key (p, c))
view table: select p,c from ks.base ... primary key (c, p)
an update query on base table normal column {{v}} will not generate any rows..
a delete query on base table normal column {{v}} will remove the base row.
{quote}
imo, to avoid user confusion, it's better to keep view matched with base
regardless semantic of update..(the current patch is not general enough to
cover all such cases yet)
[~tjake][~slebresne] what do you think?
> Materialized Views: View row expires too soon
> -
>
> Key: CASSANDRA-13127
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13127
> Project: Cassandra
> Issue Type: Bug
> Components: Local Write-Read Paths, Materialized Views
>Reporter: Duarte Nunes
>Assignee: ZhaoYang
>
> Consider the following commands, ran against trunk:
> {code}
> echo "DROP MATERIALIZED VIEW ks.mv; DROP TABLE ks.base;" | bin/cqlsh
> echo "CREATE TABLE ks.base (p int, c int, v int, PRIMARY KEY (p, c));" |
> bin/cqlsh
> echo "CREATE MATERIALIZED VIEW ks.mv AS SELECT p, c FROM base WHERE p IS NOT
> NULL AND c IS NOT NULL PRIMARY KEY (c, p);" | bin/cqlsh
> echo "INSERT INTO ks.base (p, c) VALUES (0, 0) USING TTL 10;" | bin/cqlsh
> # wait for row liveness to get closer to expiration
> sleep 6;
> echo "UPDATE ks.base USING TTL 8 SET v = 0 WHERE p = 0 and c = 0;" | bin/cqlsh
> echo "SELECT p, c, ttl(v) FROM ks.base; SELECT * FROM ks.mv;" | bin/cqlsh
> p | c | ttl(v)
> ---+---+
> 0 | 0 | 7
> (1 rows)
> c | p
> ---+---
> 0 | 0
> (1 rows)
> # wait for row liveness to expire
> sleep 4;
> echo "SELECT p, c, ttl(v) FROM ks.base; SELECT * FROM ks.mv;" | bin/cqlsh
> p | c | ttl(v)
> ---+---+
> 0 | 0 | 3
> (1 rows)
> c | p
> ---+---
> (0 rows)
> {code}
> Notice how the view row is removed even though the base row is still live. I
> would say this is because in ViewUpdateGenerator#computeLivenessInfoForEntry
> the TTLs are compared instead of the expiration times, but I'm not sure I'm
> getting that far ahead in the code when updating a column that's not in the
> view.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13652) Deadlock in AbstractCommitLogSegmentManager
[
https://issues.apache.org/jira/browse/CASSANDRA-13652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16089481#comment-16089481
]
Branimir Lambov commented on CASSANDRA-13652:
-
This looks good to me.
> Deadlock in AbstractCommitLogSegmentManager
> ---
>
> Key: CASSANDRA-13652
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13652
> Project: Cassandra
> Issue Type: Bug
> Components: Core
>Reporter: Fuud
>
> AbstractCommitLogManager uses LockSupport.(un)park incorreclty. It invokes
> unpark without checking if manager thread was parked in approriate place.
> For example, logging frameworks uses queues and queues uses ReadWriteLock's
> that uses LockSupport. Therefore AbstractCommitLogManager.wakeManager can
> wake thread inside Lock and manager thread will sleep forever at park()
> method (because unpark permit was already consumed inside lock).
> For examle stack traces:
> {code}
> "MigrationStage:1" id=412 state=WAITING
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
> at
> org.apache.cassandra.utils.concurrent.WaitQueue$AbstractSignal.awaitUninterruptibly(WaitQueue.java:279)
> at
> org.apache.cassandra.db.commitlog.AbstractCommitLogSegmentManager.awaitAvailableSegment(AbstractCommitLogSegmentManager.java:263)
> at
> org.apache.cassandra.db.commitlog.AbstractCommitLogSegmentManager.advanceAllocatingFrom(AbstractCommitLogSegmentManager.java:237)
> at
> org.apache.cassandra.db.commitlog.AbstractCommitLogSegmentManager.forceRecycleAll(AbstractCommitLogSegmentManager.java:279)
> at
> org.apache.cassandra.db.commitlog.CommitLog.forceRecycleAllSegments(CommitLog.java:210)
> at org.apache.cassandra.config.Schema.dropView(Schema.java:708)
> at
> org.apache.cassandra.schema.SchemaKeyspace.lambda$updateKeyspace$23(SchemaKeyspace.java:1361)
> at
> org.apache.cassandra.schema.SchemaKeyspace$$Lambda$382/1123232162.accept(Unknown
> Source)
> at java.util.LinkedHashMap$LinkedValues.forEach(LinkedHashMap.java:608)
> at
> java.util.Collections$UnmodifiableCollection.forEach(Collections.java:1080)
> at
> org.apache.cassandra.schema.SchemaKeyspace.updateKeyspace(SchemaKeyspace.java:1361)
> at
> org.apache.cassandra.schema.SchemaKeyspace.mergeSchema(SchemaKeyspace.java:1332)
> at
> org.apache.cassandra.schema.SchemaKeyspace.mergeSchemaAndAnnounceVersion(SchemaKeyspace.java:1282)
> - locked java.lang.Class@cc38904
> at
> org.apache.cassandra.db.DefinitionsUpdateVerbHandler$1.runMayThrow(DefinitionsUpdateVerbHandler.java:51)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor$LocalSessionWrapper.run(DebuggableThreadPoolExecutor.java:322)
> at
> com.ringcentral.concurrent.executors.MonitoredRunnable.run(MonitoredRunnable.java:36)
> at MON_R_MigrationStage.run(NamedRunnableFactory.java:67)
> at
> com.ringcentral.concurrent.executors.MonitoredThreadPoolExecutor$MdcAwareRunnable.run(MonitoredThreadPoolExecutor.java:114)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at
> org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:79)
> at
> org.apache.cassandra.concurrent.NamedThreadFactory$$Lambda$61/179045.run(Unknown
> Source)
> at java.lang.Thread.run(Thread.java:745)
> "COMMIT-LOG-ALLOCATOR:1" id=80 state=WAITING
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
> at
> org.apache.cassandra.db.commitlog.AbstractCommitLogSegmentManager$1.runMayThrow(AbstractCommitLogSegmentManager.java:128)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> at
> org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:79)
> at
> org.apache.cassandra.concurrent.NamedThreadFactory$$Lambda$61/179045.run(Unknown
> Source)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> Solution is to use Semaphore instead of low-level LockSupport.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13639) SSTableLoader always uses hostname to stream files from
[ https://issues.apache.org/jira/browse/CASSANDRA-13639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16089445#comment-16089445 ] Stefan Podkowinski commented on CASSANDRA-13639: Just had a quick look at the patch and noticed that the intention behind providing a separate config file option is to initialize streaming and ssl settings and not to actually apply the config. Where exactly is the problematic getLocalAddress() call being made? Maybe we can find a cleaner solution for that. > SSTableLoader always uses hostname to stream files from > --- > > Key: CASSANDRA-13639 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13639 > Project: Cassandra > Issue Type: Bug > Components: Tools >Reporter: Jan Karlsson >Assignee: Jan Karlsson > Fix For: 4.x > > Attachments: 13639-trunk > > > I stumbled upon an issue where SSTableLoader was ignoring our routing by > using the wrong interface to send the SSTables to the other nodes. Looking at > the code, it seems that we are using FBUtilities.getLocalAddress() to fetch > out the hostname, even if the yaml file specifies a different host. I am not > sure why we call this function instead of using the routing by leaving it > blank, perhaps someone could enlighten me. > This behaviour comes from the fact that we use a default created > DatabaseDescriptor which does not set the values for listenAddress and > listenInterface. This causes the aforementioned function to retrieve the > hostname at all times, even if it is not the interface used in the yaml file. > I propose we break out the function that handles listenAddress and > listenInterface and call it so that listenAddress or listenInterface is > getting populated in the DatabaseDescriptor. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13682) Include cassandra-lucene-index plugin description in doc.
[ https://issues.apache.org/jira/browse/CASSANDRA-13682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16089417#comment-16089417 ] Eduardo Alonso de Blas commented on CASSANDRA-13682: Thank you [~spo...@gmail.com] > Include cassandra-lucene-index plugin description in doc. > - > > Key: CASSANDRA-13682 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13682 > Project: Cassandra > Issue Type: Improvement > Components: Documentation and Website >Reporter: Eduardo Alonso de Blas >Assignee: Eduardo Alonso de Blas >Priority: Trivial > Labels: easy-fix > Fix For: 4.0 > > > In cassandra user-list, in thread "UDf for sorting", Jeff Jirsa asks if > anyone could add this. > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
