[jira] [Commented] (CASSANDRA-10198) 3.0 hints should be streamed on decomission
[
https://issues.apache.org/jira/browse/CASSANDRA-10198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14744981#comment-14744981
]
Marcus Eriksson commented on CASSANDRA-10198:
-
bq. We should probably fail the decom?
yup, pushed a new commit which throws exception if it fails to stream hints
> 3.0 hints should be streamed on decomission
> ---
>
> Key: CASSANDRA-10198
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10198
> Project: Cassandra
> Issue Type: Bug
>Reporter: Aleksey Yeschenko
>Assignee: Marcus Eriksson
> Fix For: 3.0.0 rc1
>
>
> CASSANDRA-6230 added all the necessary pieces in the initial release, but
> streaming itself didn't make it in time.
> Now that hints are stored in flat files, we cannot just stream hints
> sstables. Instead we need to handoff hints files.
> Essentially we need to rewrite {{StorageService::streamHints}} to be
> CASSANDRA-6230 aware.
> {{HintMessage}} and {{HintVerbHandler}} can already handle hints targeted for
> other nodes (see javadoc for both, it's documented reasonably).
> {{HintsDispatcher}} also takes hostId as an argument, and can stream any
> hints to any nodes.
> The building blocks are all there - we just need
> {{StorageService::streamHints}} to pick the optimal candidate for each file
> and use {{HintsDispatcher}} to stream the files.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9961) cqlsh should have DESCRIBE MATERIALIZED VIEW
[
https://issues.apache.org/jira/browse/CASSANDRA-9961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14744993#comment-14744993
]
Stefania commented on CASSANDRA-9961:
-
Thanks [~aholmber]. I've attached a patch based on the CASSANDRA-9921 branch
and your driver branch but I still haven't zipped the driver, so {{cqlsh}} must
be run with the environment variable {{CQLSH_NO_BUNDLED}} defined (in case
anyone wants to try it out).
There is one more thing I need: either {{MaterializedViewMetadata}} implements
{{export_as_string}} by returning {{as_cql_query(formatted=True)}} or all
metadata objects should accept the {{formatted}} parameter in
{{as_cql_query()}}. Basically I need to be able to call one unique method on
any metadata object that can be described (currently ks, table, index and mv).
> cqlsh should have DESCRIBE MATERIALIZED VIEW
>
>
> Key: CASSANDRA-9961
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9961
> Project: Cassandra
> Issue Type: Improvement
>Reporter: Carl Yeksigian
>Assignee: Stefania
> Labels: client-impacting, materializedviews
> Fix For: 3.0.0 rc1
>
>
> cqlsh doesn't currently produce describe output that can be used to recreate
> a MV. Needs to add a new {{DESCRIBE MATERIALIZED VIEW}} command, and also add
> to {{DESCRIBE KEYSPACE}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (CASSANDRA-9961) cqlsh should have DESCRIBE MATERIALIZED VIEW
[
https://issues.apache.org/jira/browse/CASSANDRA-9961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14744993#comment-14744993
]
Stefania edited comment on CASSANDRA-9961 at 9/15/15 7:33 AM:
--
Thanks [~aholmber]. I've attached a patch based on the CASSANDRA-9921 branch
and your driver branch but I still haven't zipped the driver, so {{cqlsh}} must
be run with the environment variable {{CQLSH_NO_BUNDLED}} defined (in case
anyone wants to try it out).
There is one more thing I need: can {{MaterializedViewMetadata}} implement
{{export_as_string}} by returning {{as_cql_query(formatted=True)}} so we can
call a unique method on any metadata object that can be described after a
simple {{DESCRIBE}} with no type specified (currently ks, table, index and mv)?
was (Author: stefania):
Thanks [~aholmber]. I've attached a patch based on the CASSANDRA-9921 branch
and your driver branch but I still haven't zipped the driver, so {{cqlsh}} must
be run with the environment variable {{CQLSH_NO_BUNDLED}} defined (in case
anyone wants to try it out).
There is one more thing I need: either {{MaterializedViewMetadata}} implements
{{export_as_string}} by returning {{as_cql_query(formatted=True)}} or all
metadata objects should accept the {{formatted}} parameter in
{{as_cql_query()}}. Basically I need to be able to call one unique method on
any metadata object that can be described (currently ks, table, index and mv).
> cqlsh should have DESCRIBE MATERIALIZED VIEW
>
>
> Key: CASSANDRA-9961
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9961
> Project: Cassandra
> Issue Type: Improvement
>Reporter: Carl Yeksigian
>Assignee: Stefania
> Labels: client-impacting, materializedviews
> Fix For: 3.0.0 rc1
>
>
> cqlsh doesn't currently produce describe output that can be used to recreate
> a MV. Needs to add a new {{DESCRIBE MATERIALIZED VIEW}} command, and also add
> to {{DESCRIBE KEYSPACE}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (CASSANDRA-9798) Cassandra seems to have deadlocks during flush operations
[
https://issues.apache.org/jira/browse/CASSANDRA-9798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Łukasz Mrożkiewicz resolved CASSANDRA-9798.
---
Resolution: Fixed
in 2.1.9 seems to be fixed
> Cassandra seems to have deadlocks during flush operations
> -
>
> Key: CASSANDRA-9798
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9798
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: 4x HP Gen9 dl 360 servers
> 2x8 cpu each (Intel(R) Xeon E5-2667 v3 @ 3.20GHz)
> 6x900GB 10kRPM disk for data
> 1x900GB 10kRPM disk for commitlog
> 64GB ram
> ETH: 10Gb/s
> Red Hat Enterprise Linux Server release 6.6 (Santiago) 2.6.32-504.el6.x86_64
> java build 1.8.0_45-b14 (openjdk) (tested on oracle java 8 too)
>Reporter: Łukasz Mrożkiewicz
> Fix For: 2.1.x
>
> Attachments: cassandra.2.1.8.log, cassandra.log, cassandra.yaml,
> cassandra.yaml, gc.log.0.current, stack.txt, topHbn1.txt
>
>
> Hi,
> We noticed some problem with dropped mutationstages. Usually on one random
> node there is a situation that:
> MutationStage "active" is full, "pending" is increasing "completed" is
> stalled.
> MemtableFlushWriter "active" 6, pending: 25 completed: stalled
> MemtablePostFlush "active" is 1, pending 29 completed: stalled
> after a some time (30s-10min) pending mutations are dropped and everything is
> working.
> When it happened:
> 1. Cpu idle is ~95%
> 2. no gc long pauses or more activity.
> 3. memory usage 3.5GB form 8GB
> 4. only writes is processed by cassandra
> 5. when LOAD > 400GB/node problems appeared
> 6. cassandra 2.1.6
> There is gap in logs:
> {code}
> INFO 08:47:01 Timed out replaying hints to /192.168.100.83; aborting (0
> delivered)
> INFO 08:47:01 Enqueuing flush of hints: 7870567 (0%) on-heap, 0 (0%) off-heap
> INFO 08:47:30 Enqueuing flush of table1: 95301807 (4%) on-heap, 0 (0%)
> off-heap
> INFO 08:47:31 Enqueuing flush of table1: 60462632 (3%) on-heap, 0 (0%)
> off-heap
> INFO 08:47:31 Enqueuing flush of table2: 76973746 (4%) on-heap, 0 (0%)
> off-heap
> INFO 08:47:31 Enqueuing flush of table1: 84290135 (4%) on-heap, 0 (0%)
> off-heap
> INFO 08:47:32 Enqueuing flush of table3: 56926652 (3%) on-heap, 0 (0%)
> off-heap
> INFO 08:47:32 Enqueuing flush of table1: 85124218 (4%) on-heap, 0 (0%)
> off-heap
> INFO 08:47:33 Enqueuing flush of table2: 95663415 (4%) on-heap, 0 (0%)
> off-heap
> INFO 08:47:58 CompactionManager 239
> INFO 08:47:58 Writing Memtable-table2@1767938721(13843064 serialized bytes,
> 162359 ops, 4%/0% of on/off-heap l
> imit)
> INFO 08:47:58 Writing Memtable-hints@1433125911(478703 serialized bytes, 424
> ops, 0%/0% of on/off-heap limit)
> INFO 08:47:58 Writing Memtable-table2@1318583275(11783615 serialized bytes,
> 137378 ops, 4%/0% of on/off-heap l
> imit)
> INFO 08:47:58 Enqueuing flush of compactions_in_progress: 969 (0%) on-heap,
> 0 (0%) off-heap
> INFO 08:47:58 Writing Memtable-table1@541175113(17221327 serialized bytes,
> 180792 ops, 4%/0% of on/off-heap
> limit)
> INFO 08:47:58 Writing Memtable-table1@1361154669(27138519 serialized bytes,
> 273472 ops, 6%/0% of on/off-hea
> p limit)
> INFO 08:48:03 2176 MUTATION messages dropped in last 5000ms
> {code}
> use case:
> 100% write - 100Mb/s, couples of CF ~10column each. max cell size 100B
> CMS and G1GC tested - no difference
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (CASSANDRA-10322) skipBytes is used extensively, but is slow
[
https://issues.apache.org/jira/browse/CASSANDRA-10322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Benedict updated CASSANDRA-10322:
-
Priority: Trivial (was: Minor)
> skipBytes is used extensively, but is slow
> --
>
> Key: CASSANDRA-10322
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10322
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
>Reporter: Benedict
>Assignee: Benedict
>Priority: Trivial
> Fix For: 3.0.x
>
>
> We skip a great deal to avoid materializing data. Ironically, however,
> skipping is just as (perhaps more) expensive, as it allocates a temporary
> array of the size of the number of bytes we want to skip.
> This trivial patch implements {{skipBytes}} more efficiently, and simplifies
> {{FileUtils.skipBytesFully}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9446) Failure detector should ignore local pauses per endpoint
[ https://issues.apache.org/jira/browse/CASSANDRA-9446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14745030#comment-14745030 ] Benedict commented on CASSANDRA-9446: - [~kohlisankalp]: was that a change of position on the existing patch, in order to get something in sooner? If so, which versions are we targeting with this? Given it's a very small patch, it's arguably quite acceptable to sneak it into the last maintenance release of 2.0. > Failure detector should ignore local pauses per endpoint > > > Key: CASSANDRA-9446 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9446 > Project: Cassandra > Issue Type: Improvement >Reporter: sankalp kohli >Assignee: Stefania >Priority: Minor > Attachments: 9446.txt, 9644-v2.txt > > > In CASSANDRA-9183, we added a feature to ignore local pauses. But it will > only not mark 2 endpoints as down. > We should do this per endpoint as suggested by Brandon in CASSANDRA-9183. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9798) Cassandra seems to have deadlocks during flush operations
[
https://issues.apache.org/jira/browse/CASSANDRA-9798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14745035#comment-14745035
]
Benedict commented on CASSANDRA-9798:
-
Thanks for reporting back. Sorry I couldn't be more help tracking this down at
the time, but glad to hear it is no longer an issue for you.
> Cassandra seems to have deadlocks during flush operations
> -
>
> Key: CASSANDRA-9798
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9798
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: 4x HP Gen9 dl 360 servers
> 2x8 cpu each (Intel(R) Xeon E5-2667 v3 @ 3.20GHz)
> 6x900GB 10kRPM disk for data
> 1x900GB 10kRPM disk for commitlog
> 64GB ram
> ETH: 10Gb/s
> Red Hat Enterprise Linux Server release 6.6 (Santiago) 2.6.32-504.el6.x86_64
> java build 1.8.0_45-b14 (openjdk) (tested on oracle java 8 too)
>Reporter: Łukasz Mrożkiewicz
> Fix For: 2.1.x
>
> Attachments: cassandra.2.1.8.log, cassandra.log, cassandra.yaml,
> cassandra.yaml, gc.log.0.current, stack.txt, topHbn1.txt
>
>
> Hi,
> We noticed some problem with dropped mutationstages. Usually on one random
> node there is a situation that:
> MutationStage "active" is full, "pending" is increasing "completed" is
> stalled.
> MemtableFlushWriter "active" 6, pending: 25 completed: stalled
> MemtablePostFlush "active" is 1, pending 29 completed: stalled
> after a some time (30s-10min) pending mutations are dropped and everything is
> working.
> When it happened:
> 1. Cpu idle is ~95%
> 2. no gc long pauses or more activity.
> 3. memory usage 3.5GB form 8GB
> 4. only writes is processed by cassandra
> 5. when LOAD > 400GB/node problems appeared
> 6. cassandra 2.1.6
> There is gap in logs:
> {code}
> INFO 08:47:01 Timed out replaying hints to /192.168.100.83; aborting (0
> delivered)
> INFO 08:47:01 Enqueuing flush of hints: 7870567 (0%) on-heap, 0 (0%) off-heap
> INFO 08:47:30 Enqueuing flush of table1: 95301807 (4%) on-heap, 0 (0%)
> off-heap
> INFO 08:47:31 Enqueuing flush of table1: 60462632 (3%) on-heap, 0 (0%)
> off-heap
> INFO 08:47:31 Enqueuing flush of table2: 76973746 (4%) on-heap, 0 (0%)
> off-heap
> INFO 08:47:31 Enqueuing flush of table1: 84290135 (4%) on-heap, 0 (0%)
> off-heap
> INFO 08:47:32 Enqueuing flush of table3: 56926652 (3%) on-heap, 0 (0%)
> off-heap
> INFO 08:47:32 Enqueuing flush of table1: 85124218 (4%) on-heap, 0 (0%)
> off-heap
> INFO 08:47:33 Enqueuing flush of table2: 95663415 (4%) on-heap, 0 (0%)
> off-heap
> INFO 08:47:58 CompactionManager 239
> INFO 08:47:58 Writing Memtable-table2@1767938721(13843064 serialized bytes,
> 162359 ops, 4%/0% of on/off-heap l
> imit)
> INFO 08:47:58 Writing Memtable-hints@1433125911(478703 serialized bytes, 424
> ops, 0%/0% of on/off-heap limit)
> INFO 08:47:58 Writing Memtable-table2@1318583275(11783615 serialized bytes,
> 137378 ops, 4%/0% of on/off-heap l
> imit)
> INFO 08:47:58 Enqueuing flush of compactions_in_progress: 969 (0%) on-heap,
> 0 (0%) off-heap
> INFO 08:47:58 Writing Memtable-table1@541175113(17221327 serialized bytes,
> 180792 ops, 4%/0% of on/off-heap
> limit)
> INFO 08:47:58 Writing Memtable-table1@1361154669(27138519 serialized bytes,
> 273472 ops, 6%/0% of on/off-hea
> p limit)
> INFO 08:48:03 2176 MUTATION messages dropped in last 5000ms
> {code}
> use case:
> 100% write - 100Mb/s, couples of CF ~10column each. max cell size 100B
> CMS and G1GC tested - no difference
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-10336) SinglePartitionSliceCommandTest.staticColumnsAreReturned flappy (3.0)
[
https://issues.apache.org/jira/browse/CASSANDRA-10336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14745051#comment-14745051
]
Sylvain Lebresne commented on CASSANDRA-10336:
--
I think I've seen that test flap since the beginning (but wasn't entirely sure
so my attitude was wait and see) so I'm not sure this isn't just a test issue.
Not sure why it's flapping in any case but assigning to [~bdeggleston] since he
is the original test author.
> SinglePartitionSliceCommandTest.staticColumnsAreReturned flappy (3.0)
> -
>
> Key: CASSANDRA-10336
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10336
> Project: Cassandra
> Issue Type: Bug
>Reporter: Robert Stupp
>
> UTest {{SinglePartitionSliceCommandTest.staticColumnsAreReturned}} seems to
> flap.
> The test failed during build
> [#110|http://cassci.datastax.com/view/cassandra-3.0/job/cassandra-3.0_testall/110],
> 112, 115, 116, 118, 123, 125 and 128 (didn't check before build 95)
> It _might_ be related to the changes of CASSANDRA-10232 - but I don't get why
> it flaps.
> /cc [~slebresne]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (CASSANDRA-10336) SinglePartitionSliceCommandTest.staticColumnsAreReturned flappy (3.0)
[
https://issues.apache.org/jira/browse/CASSANDRA-10336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sylvain Lebresne updated CASSANDRA-10336:
-
Assignee: Blake Eggleston
> SinglePartitionSliceCommandTest.staticColumnsAreReturned flappy (3.0)
> -
>
> Key: CASSANDRA-10336
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10336
> Project: Cassandra
> Issue Type: Bug
>Reporter: Robert Stupp
>Assignee: Blake Eggleston
>
> UTest {{SinglePartitionSliceCommandTest.staticColumnsAreReturned}} seems to
> flap.
> The test failed during build
> [#110|http://cassci.datastax.com/view/cassandra-3.0/job/cassandra-3.0_testall/110],
> 112, 115, 116, 118, 123, 125 and 128 (didn't check before build 95)
> It _might_ be related to the changes of CASSANDRA-10232 - but I don't get why
> it flaps.
> /cc [~slebresne]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9961) cqlsh should have DESCRIBE MATERIALIZED VIEW
[
https://issues.apache.org/jira/browse/CASSANDRA-9961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14745060#comment-14745060
]
Stefania commented on CASSANDRA-9961:
-
The dtests are [here|https://github.com/stef1927/cassandra-dtest/commits/9961]
but before I create a pull request I need to remove {{CQLSH_NO_BUNDLED}} after
the driver has been bundled.
CI is here:
http://cassci.datastax.com/view/Dev/view/stef1927/job/stef1927-9961-3.0-dtest/
> cqlsh should have DESCRIBE MATERIALIZED VIEW
>
>
> Key: CASSANDRA-9961
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9961
> Project: Cassandra
> Issue Type: Improvement
>Reporter: Carl Yeksigian
>Assignee: Stefania
> Labels: client-impacting, materializedviews
> Fix For: 3.0.0 rc1
>
>
> cqlsh doesn't currently produce describe output that can be used to recreate
> a MV. Needs to add a new {{DESCRIBE MATERIALIZED VIEW}} command, and also add
> to {{DESCRIBE KEYSPACE}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (CASSANDRA-10337) Make SSTableReader and Partition access paths uniform
Benedict created CASSANDRA-10337:
Summary: Make SSTableReader and Partition access paths uniform
Key: CASSANDRA-10337
URL: https://issues.apache.org/jira/browse/CASSANDRA-10337
Project: Cassandra
Issue Type: Improvement
Components: Core
Reporter: Benedict
Assignee: Benedict
Priority: Minor
Fix For: 3.x
CASSANDRA-9986 attempts to do this in a quick manner to help simplify the
codebase a little, but it's still a long way from being optimal. We can follow
up with some changes to make it cleaner still:
We can modify {{Partition}} a little so that {{SSTableReader}} could return a
lightweight instance (or a common base class) with the set of accessors
necessary for answering queries. Filters can then operate on this object
uniformly. We can _perhaps_ simultaneously unify the access methods within
{{Partition}} by removing {{getRow}} and {{unfilteredIterator(Slices)}}, and
introduce an {{UnfilteredRowSearchIterator}} that permits indexed access to the
underlying data.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-10286) Windows utest 3.0: org.apache.cassandra.io.sstable.SSTableLoaderTest.testLoadingIncompleteSSTable
[
https://issues.apache.org/jira/browse/CASSANDRA-10286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14745097#comment-14745097
]
Benedict commented on CASSANDRA-10286:
--
Well, utests seem OK. dtests don't seem to be succeeding though
> Windows utest 3.0:
> org.apache.cassandra.io.sstable.SSTableLoaderTest.testLoadingIncompleteSSTable
> -
>
> Key: CASSANDRA-10286
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10286
> Project: Cassandra
> Issue Type: Sub-task
>Reporter: Philip Thompson
>Assignee: Stefania
> Fix For: 3.x
>
>
> Distinct error message from CASSANDRA-10210.
> {code}
> junit.framework.AssertionFailedError:
> at
> org.apache.cassandra.io.sstable.SSTableLoaderTest.testLoadingIncompleteSSTable(SSTableLoaderTest.java:186)
> {code}
> which I believe is from
> {code}
> ERROR 22:05:57 Unable to delete
> D:\temp\1441663555277-0\SSTableLoaderTest\Standard2\ma-1-big-Data.db
> java.nio.file.AccessDeniedException:
> D:\temp\1441663555277-0\SSTableLoaderTest\Standard2\ma-1-big-Data.db
> at
> sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:83)
> ~[na:1.8.0_51]
> at
> sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97)
> ~[na:1.8.0_51]
> at
> sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:102)
> ~[na:1.8.0_51]
> at
> sun.nio.fs.WindowsFileSystemProvider.implDelete(WindowsFileSystemProvider.java:269)
> ~[na:1.8.0_51]
> at
> sun.nio.fs.AbstractFileSystemProvider.delete(AbstractFileSystemProvider.java:103)
> ~[na:1.8.0_51]
> at java.nio.file.Files.delete(Files.java:1126) ~[na:1.8.0_51]
> at
> org.apache.cassandra.db.lifecycle.TransactionLog.delete(TransactionLog.java:792)
> [main/:na]
> at
> org.apache.cassandra.db.lifecycle.TransactionLog.access$100(TransactionLog.java:97)
> [main/:na]
> at
> org.apache.cassandra.db.lifecycle.TransactionLog$TransactionFile.lambda$deleteRecord$108(TransactionLog.java:504)
> [main/:na]
> at
> org.apache.cassandra.db.lifecycle.TransactionLog$TransactionFile$$Lambda$108/1799732569.accept(Unknown
> Source) [main/:na]
> at java.util.Arrays$ArrayList.forEach(Arrays.java:3880) [na:1.8.0_51]
> at
> org.apache.cassandra.db.lifecycle.TransactionLog$TransactionFile.deleteRecord(TransactionLog.java:504)
> [main/:na]
> at
> org.apache.cassandra.db.lifecycle.TransactionLog$TransactionFile$$Lambda$88/744169296.accept(Unknown
> Source) [main/:na]
> at
> java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184)
> [na:1.8.0_51]
> at
> java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)
> [na:1.8.0_51]
> at java.util.HashMap$KeySpliterator.forEachRemaining(HashMap.java:1540)
> [na:1.8.0_51]
> at
> java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:512)
> [na:1.8.0_51]
> at
> java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:502)
> [na:1.8.0_51]
> at
> java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151)
> [na:1.8.0_51]
> at
> java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174)
> [na:1.8.0_51]
> at
> java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
> [na:1.8.0_51]
> at
> java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418)
> [na:1.8.0_51]
> at
> org.apache.cassandra.db.lifecycle.TransactionLog$TransactionFile.deleteRecords(TransactionLog.java:490)
> [main/:na]
> at
> org.apache.cassandra.db.lifecycle.TransactionLog$TransactionData.removeUnfinishedLeftovers(TransactionLog.java:622)
> [main/:na]
> at
> org.apache.cassandra.db.lifecycle.TransactionLog$TransactionTidier.run(TransactionLog.java:837)
> [main/:na]
> at
> org.apache.cassandra.db.lifecycle.TransactionLog$TransactionTidier.tidy(TransactionLog.java:822)
> [main/:na]
> at
> org.apache.cassandra.utils.concurrent.Ref$GlobalState.release(Ref.java:294)
> [main/:na]
> at
> org.apache.cassandra.utils.concurrent.Ref$State.ensureReleased(Ref.java:172)
> [main/:na]
> at
> org.apache.cassandra.utils.concurrent.Ref.ensureReleased(Ref.java:92)
> [main/:na]
> at
> org.apache.cassandra.db.lifecycle.TransactionLog.complete(TransactionLog.java:950)
> [main/:na]
> at
> org.apache.cassandra.db.lifecycle.TransactionLog.doAbort(TransactionLog.java:969)
> [main/:na]
> at
> org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.abort(Transactional.java:144)
> [main/:na]
> at
> org.apache.cassandra.db.lifecycle.LifecycleTransaction.doAbort(LifecycleTransaction.java:256)
>
[jira] [Commented] (CASSANDRA-10286) Windows utest 3.0: org.apache.cassandra.io.sstable.SSTableLoaderTest.testLoadingIncompleteSSTable
[
https://issues.apache.org/jira/browse/CASSANDRA-10286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14745126#comment-14745126
]
Stefania commented on CASSANDRA-10286:
--
I restarted the dtests. The previous build failed to report the results:
{code}
ERROR: Publisher 'Publish JUnit test result report' failed: None of the test
reports contained any result
{code}
Not sure why.
> Windows utest 3.0:
> org.apache.cassandra.io.sstable.SSTableLoaderTest.testLoadingIncompleteSSTable
> -
>
> Key: CASSANDRA-10286
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10286
> Project: Cassandra
> Issue Type: Sub-task
>Reporter: Philip Thompson
>Assignee: Stefania
> Fix For: 3.x
>
>
> Distinct error message from CASSANDRA-10210.
> {code}
> junit.framework.AssertionFailedError:
> at
> org.apache.cassandra.io.sstable.SSTableLoaderTest.testLoadingIncompleteSSTable(SSTableLoaderTest.java:186)
> {code}
> which I believe is from
> {code}
> ERROR 22:05:57 Unable to delete
> D:\temp\1441663555277-0\SSTableLoaderTest\Standard2\ma-1-big-Data.db
> java.nio.file.AccessDeniedException:
> D:\temp\1441663555277-0\SSTableLoaderTest\Standard2\ma-1-big-Data.db
> at
> sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:83)
> ~[na:1.8.0_51]
> at
> sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97)
> ~[na:1.8.0_51]
> at
> sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:102)
> ~[na:1.8.0_51]
> at
> sun.nio.fs.WindowsFileSystemProvider.implDelete(WindowsFileSystemProvider.java:269)
> ~[na:1.8.0_51]
> at
> sun.nio.fs.AbstractFileSystemProvider.delete(AbstractFileSystemProvider.java:103)
> ~[na:1.8.0_51]
> at java.nio.file.Files.delete(Files.java:1126) ~[na:1.8.0_51]
> at
> org.apache.cassandra.db.lifecycle.TransactionLog.delete(TransactionLog.java:792)
> [main/:na]
> at
> org.apache.cassandra.db.lifecycle.TransactionLog.access$100(TransactionLog.java:97)
> [main/:na]
> at
> org.apache.cassandra.db.lifecycle.TransactionLog$TransactionFile.lambda$deleteRecord$108(TransactionLog.java:504)
> [main/:na]
> at
> org.apache.cassandra.db.lifecycle.TransactionLog$TransactionFile$$Lambda$108/1799732569.accept(Unknown
> Source) [main/:na]
> at java.util.Arrays$ArrayList.forEach(Arrays.java:3880) [na:1.8.0_51]
> at
> org.apache.cassandra.db.lifecycle.TransactionLog$TransactionFile.deleteRecord(TransactionLog.java:504)
> [main/:na]
> at
> org.apache.cassandra.db.lifecycle.TransactionLog$TransactionFile$$Lambda$88/744169296.accept(Unknown
> Source) [main/:na]
> at
> java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184)
> [na:1.8.0_51]
> at
> java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)
> [na:1.8.0_51]
> at java.util.HashMap$KeySpliterator.forEachRemaining(HashMap.java:1540)
> [na:1.8.0_51]
> at
> java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:512)
> [na:1.8.0_51]
> at
> java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:502)
> [na:1.8.0_51]
> at
> java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151)
> [na:1.8.0_51]
> at
> java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174)
> [na:1.8.0_51]
> at
> java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
> [na:1.8.0_51]
> at
> java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418)
> [na:1.8.0_51]
> at
> org.apache.cassandra.db.lifecycle.TransactionLog$TransactionFile.deleteRecords(TransactionLog.java:490)
> [main/:na]
> at
> org.apache.cassandra.db.lifecycle.TransactionLog$TransactionData.removeUnfinishedLeftovers(TransactionLog.java:622)
> [main/:na]
> at
> org.apache.cassandra.db.lifecycle.TransactionLog$TransactionTidier.run(TransactionLog.java:837)
> [main/:na]
> at
> org.apache.cassandra.db.lifecycle.TransactionLog$TransactionTidier.tidy(TransactionLog.java:822)
> [main/:na]
> at
> org.apache.cassandra.utils.concurrent.Ref$GlobalState.release(Ref.java:294)
> [main/:na]
> at
> org.apache.cassandra.utils.concurrent.Ref$State.ensureReleased(Ref.java:172)
> [main/:na]
> at
> org.apache.cassandra.utils.concurrent.Ref.ensureReleased(Ref.java:92)
> [main/:na]
> at
> org.apache.cassandra.db.lifecycle.TransactionLog.complete(TransactionLog.java:950)
> [main/:na]
> at
> org.apache.cassandra.db.lifecycle.TransactionLog.doAbort(TransactionLog.java:969)
> [main/:na]
> at
> org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.abort(Tr
[jira] [Commented] (CASSANDRA-7392) Abort in-progress queries that time out
[ https://issues.apache.org/jira/browse/CASSANDRA-7392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14745135#comment-14745135 ] Stefania commented on CASSANDRA-7392: - Thanks for the comments, I hope to resume this tomorrow. [~aweisberg]: aside from logging at DEBUG level is there any other blocker? Do we still retain logging of the top N entries or shall we log all timed out queries? bq. For writes, we are going to be retaining the queries past the timeout for the logger. If someone has a memory utilization issue they can't fix it by setting the timeout lower since the logger will still retain it and it runs on it's own period. bq. Even in a separate debug log file rolling is a concern. One bad log statement can wipe away all the other information in a failure scenario. I'm not sure if there is anything we can do about this? I think we also have one failing unit test. > Abort in-progress queries that time out > --- > > Key: CASSANDRA-7392 > URL: https://issues.apache.org/jira/browse/CASSANDRA-7392 > Project: Cassandra > Issue Type: New Feature > Components: Core >Reporter: Jonathan Ellis >Assignee: Stefania >Priority: Critical > Fix For: 3.x > > > Currently we drop queries that time out before we get to them (because node > is overloaded) but not queries that time out while being processed. > (Particularly common for index queries on data that shouldn't be indexed.) > Adding the latter and logging when we have to interrupt one gets us a poor > man's "slow query log" for free. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (CASSANDRA-10338) Secondary index large values dtest failing on 3.0
Sam Tunnicliffe created CASSANDRA-10338:
---
Summary: Secondary index large values dtest failing on 3.0
Key: CASSANDRA-10338
URL: https://issues.apache.org/jira/browse/CASSANDRA-10338
Project: Cassandra
Issue Type: Sub-task
Reporter: Sam Tunnicliffe
Assignee: Sam Tunnicliffe
Fix For: 3.0.0 rc1
{{secondary_index_test:TestSecondaryIndex.test_8280_validate_indexed_values}}
has been failing since build
[#147|http://cassci.datastax.com/view/cassandra-3.0/job/cassandra-3.0_dtest/147/].
The most likely cause is CASSANDRA-6237 (though I haven't confirmed that).
The problem is that if an oversized value is provided, the validation for LWT
statements (both regular and in batches) is not performed up front, but when
the {{Cql3CasRequest}} is executed via {{StorageProxy}}. This causes a timeout,
rather than an immediate validation error & hence the test fails.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-10264) Unable to use conditions on static columns for DELETE
[
https://issues.apache.org/jira/browse/CASSANDRA-10264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14745197#comment-14745197
]
Benjamin Lerer commented on CASSANDRA-10264:
I checked with 2.1 and the problem is also there as I got also {{Invalid
restriction on clustering column ord since the DELETE statement modifies only
static columns}}.
This seems wrong to me as the delete statement modify also non-static columns.
> Unable to use conditions on static columns for DELETE
> -
>
> Key: CASSANDRA-10264
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10264
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: Cassandra 2.2.0
>Reporter: DOAN DuyHai
>Assignee: Benjamin Lerer
>
> {noformat}
> cqlsh:test> create table static_table(id int, stat int static, ord int, val
> text, primary key(id,ord));
> cqlsh:test> insert into static_table (id,stat,ord,val) VALUES ( 1, 1, 1, '1');
> cqlsh:test> delete from static_table where id=1 and ord=1 if stat != 1;
> Invalid syntax at line 1, char 55
> delete from static_table where id=1 and ord=1 if stat != 1;
> ^
> {noformat}
> Same error if using =, <, <=, >= or > condition
> According to [~thobbs] the syntax should work. Plus, the error message is
> wrong
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (CASSANDRA-10264) Unable to use conditions on static columns for DELETE
[
https://issues.apache.org/jira/browse/CASSANDRA-10264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Benjamin Lerer updated CASSANDRA-10264:
---
Reproduced In: 3.0 beta 1, 2.2.0 (was: 2.2.0, 3.0 beta 1)
Since Version: 2.1.x (was: 2.2.0)
> Unable to use conditions on static columns for DELETE
> -
>
> Key: CASSANDRA-10264
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10264
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: Cassandra 2.2.0
>Reporter: DOAN DuyHai
>Assignee: Benjamin Lerer
>
> {noformat}
> cqlsh:test> create table static_table(id int, stat int static, ord int, val
> text, primary key(id,ord));
> cqlsh:test> insert into static_table (id,stat,ord,val) VALUES ( 1, 1, 1, '1');
> cqlsh:test> delete from static_table where id=1 and ord=1 if stat != 1;
> Invalid syntax at line 1, char 55
> delete from static_table where id=1 and ord=1 if stat != 1;
> ^
> {noformat}
> Same error if using =, <, <=, >= or > condition
> According to [~thobbs] the syntax should work. Plus, the error message is
> wrong
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (CASSANDRA-10264) Unable to use conditions on static columns for DELETE
[
https://issues.apache.org/jira/browse/CASSANDRA-10264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Benjamin Lerer updated CASSANDRA-10264:
---
Reproduced In: 3.0 beta 1, 2.2.0, 2.1.9 (was: 2.2.0, 3.0 beta 1)
> Unable to use conditions on static columns for DELETE
> -
>
> Key: CASSANDRA-10264
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10264
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: Cassandra 2.2.0
>Reporter: DOAN DuyHai
>Assignee: Benjamin Lerer
>
> {noformat}
> cqlsh:test> create table static_table(id int, stat int static, ord int, val
> text, primary key(id,ord));
> cqlsh:test> insert into static_table (id,stat,ord,val) VALUES ( 1, 1, 1, '1');
> cqlsh:test> delete from static_table where id=1 and ord=1 if stat != 1;
> Invalid syntax at line 1, char 55
> delete from static_table where id=1 and ord=1 if stat != 1;
> ^
> {noformat}
> Same error if using =, <, <=, >= or > condition
> According to [~thobbs] the syntax should work. Plus, the error message is
> wrong
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (CASSANDRA-10339) Prevent ALTER TYPE from creating circular references
Olivier Michallat created CASSANDRA-10339:
-
Summary: Prevent ALTER TYPE from creating circular references
Key: CASSANDRA-10339
URL: https://issues.apache.org/jira/browse/CASSANDRA-10339
Project: Cassandra
Issue Type: Bug
Reporter: Olivier Michallat
Priority: Minor
It's possible to define circular/recursive types using {{ALTER TYPE}}. They
won't work in practice when you try to insert data, but we should detect this
earlier and prevent the type modification.
Recursive type example (from
[JAVA-908|https://datastax-oss.atlassian.net/browse/JAVA-908]):
{code}
CREATE TYPE node (name text,);
ALTER TYPE node ADD children frozen>;
{code}
Circular example (from [Stack
overflow|http://stackoverflow.com/questions/29037733/cassandra-2-1-recursion-by-nesting-udts]):
{code}
create type ping(pingid int);
create type pong(pongid int, ping frozen);
alter type ping ADD pong frozen;
{code}
Note that, in the circular example, references are properly checked when
dropping the types, so neither type can be dropped.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[3/3] cassandra git commit: Merge branch 'cassandra-3.0' into trunk
Merge branch 'cassandra-3.0' into trunk Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/ef54c6c7 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/ef54c6c7 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/ef54c6c7 Branch: refs/heads/trunk Commit: ef54c6c72f863bcf7211925efc38773307fc56a7 Parents: acb8fbb 7f5580b Author: Sam Tunnicliffe Authored: Tue Sep 15 11:22:24 2015 +0100 Committer: Sam Tunnicliffe Committed: Tue Sep 15 11:22:24 2015 +0100 -- CHANGES.txt | 1 + src/java/org/apache/cassandra/index/Index.java | 13 .../cassandra/index/SecondaryIndexManager.java | 19 -- .../index/internal/CassandraIndex.java | 6 ++ .../org/apache/cassandra/index/StubIndex.java | 5 ++ .../index/internal/CustomCassandraIndex.java| 5 ++ .../index/internal/CustomIndexTest.java | 67 +++- 7 files changed, 109 insertions(+), 7 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/ef54c6c7/CHANGES.txt -- diff --cc CHANGES.txt index 6e04690,cf982dd..ee1aa69 --- a/CHANGES.txt +++ b/CHANGES.txt @@@ -1,8 -1,5 +1,9 @@@ +3.2 + * Add transparent data encryption core classes (CASSANDRA-9945) + + 3.0.0-rc1 + * Give index implementations more control over rebuild operations (CASSANDRA-10312) * Update index file format (CASSANDRA-10314) * Add "shadowable" row tombstones to deal with mv timestamp issues (CASSANDRA-10261) * CFS.loadNewSSTables() broken for pre-3.0 sstables
[1/3] cassandra git commit: Give more control over building to 2i impls
Repository: cassandra
Updated Branches:
refs/heads/cassandra-3.0 51b1a1c6d -> 7f5580b22
refs/heads/trunk acb8fbb8f -> ef54c6c72
Give more control over building to 2i impls
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/7f5580b2
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/7f5580b2
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/7f5580b2
Branch: refs/heads/cassandra-3.0
Commit: 7f5580b2241425fd524336d60374f1d759ad93cc
Parents: 51b1a1c
Author: Sam Tunnicliffe
Authored: Mon Sep 14 11:12:02 2015 +0100
Committer: Sam Tunnicliffe
Committed: Tue Sep 15 11:19:43 2015 +0100
--
CHANGES.txt | 1 +
src/java/org/apache/cassandra/index/Index.java | 13
.../cassandra/index/SecondaryIndexManager.java | 19 --
.../index/internal/CassandraIndex.java | 6 ++
.../org/apache/cassandra/index/StubIndex.java | 5 ++
.../index/internal/CustomCassandraIndex.java| 5 ++
.../index/internal/CustomIndexTest.java | 67 +++-
7 files changed, 109 insertions(+), 7 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/7f5580b2/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 1a1ddeb..cf982dd 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
3.0.0-rc1
+ * Give index implementations more control over rebuild operations
(CASSANDRA-10312)
* Update index file format (CASSANDRA-10314)
* Add "shadowable" row tombstones to deal with mv timestamp issues
(CASSANDRA-10261)
* CFS.loadNewSSTables() broken for pre-3.0 sstables
http://git-wip-us.apache.org/repos/asf/cassandra/blob/7f5580b2/src/java/org/apache/cassandra/index/Index.java
--
diff --git a/src/java/org/apache/cassandra/index/Index.java
b/src/java/org/apache/cassandra/index/Index.java
index 8f126fe..5dc44a4 100644
--- a/src/java/org/apache/cassandra/index/Index.java
+++ b/src/java/org/apache/cassandra/index/Index.java
@@ -178,6 +178,19 @@ public interface Index
*/
public Callable getTruncateTask(long truncatedAt);
+/**
+ * Return true if this index can be built or rebuilt when the index
manager determines it is necessary. Returning
+ * false enables the index implementation (or some other component) to
control if and when SSTable data is
+ * incorporated into the index.
+ *
+ * This is called by SecondaryIndexManager in buildIndexBlocking,
buildAllIndexesBlocking & rebuildIndexesBlocking
+ * where a return value of false causes the index to be exluded from the
set of those which will process the
+ * SSTable data.
+ * @return if the index should be included in the set which processes
SSTable data, false otherwise.
+ */
+public boolean shouldBuildBlocking();
+
+
/*
* Index selection
*/
http://git-wip-us.apache.org/repos/asf/cassandra/blob/7f5580b2/src/java/org/apache/cassandra/index/SecondaryIndexManager.java
--
diff --git a/src/java/org/apache/cassandra/index/SecondaryIndexManager.java
b/src/java/org/apache/cassandra/index/SecondaryIndexManager.java
index a916bd2..ff4567b 100644
--- a/src/java/org/apache/cassandra/index/SecondaryIndexManager.java
+++ b/src/java/org/apache/cassandra/index/SecondaryIndexManager.java
@@ -209,7 +209,8 @@ public class SecondaryIndexManager implements IndexRegistry
public void rebuildIndexesBlocking(Collection sstables,
Set indexNames)
{
Set toRebuild = indexes.values().stream()
- .filter(indexer ->
indexNames.contains(indexer.getIndexName()))
+ .filter(index ->
indexNames.contains(index.getIndexName()))
+
.filter(Index::shouldBuildBlocking)
.collect(Collectors.toSet());
if (toRebuild.isEmpty())
{
@@ -226,17 +227,23 @@ public class SecondaryIndexManager implements
IndexRegistry
public void buildAllIndexesBlocking(Collection sstables)
{
-buildIndexesBlocking(sstables, ImmutableSet.copyOf(indexes.values()));
+buildIndexesBlocking(sstables, indexes.values()
+ .stream()
+
.filter(Index::shouldBuildBlocking)
+ .collect(Collectors.toSet()));
}
// For convenience, may be called directly from Index impls
public void buildIndexBlocking(Index index)
{
-try (ColumnFamilyStore.RefViewFragment viewFr
[2/3] cassandra git commit: Give more control over building to 2i impls
Give more control over building to 2i impls
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/7f5580b2
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/7f5580b2
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/7f5580b2
Branch: refs/heads/trunk
Commit: 7f5580b2241425fd524336d60374f1d759ad93cc
Parents: 51b1a1c
Author: Sam Tunnicliffe
Authored: Mon Sep 14 11:12:02 2015 +0100
Committer: Sam Tunnicliffe
Committed: Tue Sep 15 11:19:43 2015 +0100
--
CHANGES.txt | 1 +
src/java/org/apache/cassandra/index/Index.java | 13
.../cassandra/index/SecondaryIndexManager.java | 19 --
.../index/internal/CassandraIndex.java | 6 ++
.../org/apache/cassandra/index/StubIndex.java | 5 ++
.../index/internal/CustomCassandraIndex.java| 5 ++
.../index/internal/CustomIndexTest.java | 67 +++-
7 files changed, 109 insertions(+), 7 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/7f5580b2/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 1a1ddeb..cf982dd 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
3.0.0-rc1
+ * Give index implementations more control over rebuild operations
(CASSANDRA-10312)
* Update index file format (CASSANDRA-10314)
* Add "shadowable" row tombstones to deal with mv timestamp issues
(CASSANDRA-10261)
* CFS.loadNewSSTables() broken for pre-3.0 sstables
http://git-wip-us.apache.org/repos/asf/cassandra/blob/7f5580b2/src/java/org/apache/cassandra/index/Index.java
--
diff --git a/src/java/org/apache/cassandra/index/Index.java
b/src/java/org/apache/cassandra/index/Index.java
index 8f126fe..5dc44a4 100644
--- a/src/java/org/apache/cassandra/index/Index.java
+++ b/src/java/org/apache/cassandra/index/Index.java
@@ -178,6 +178,19 @@ public interface Index
*/
public Callable getTruncateTask(long truncatedAt);
+/**
+ * Return true if this index can be built or rebuilt when the index
manager determines it is necessary. Returning
+ * false enables the index implementation (or some other component) to
control if and when SSTable data is
+ * incorporated into the index.
+ *
+ * This is called by SecondaryIndexManager in buildIndexBlocking,
buildAllIndexesBlocking & rebuildIndexesBlocking
+ * where a return value of false causes the index to be exluded from the
set of those which will process the
+ * SSTable data.
+ * @return if the index should be included in the set which processes
SSTable data, false otherwise.
+ */
+public boolean shouldBuildBlocking();
+
+
/*
* Index selection
*/
http://git-wip-us.apache.org/repos/asf/cassandra/blob/7f5580b2/src/java/org/apache/cassandra/index/SecondaryIndexManager.java
--
diff --git a/src/java/org/apache/cassandra/index/SecondaryIndexManager.java
b/src/java/org/apache/cassandra/index/SecondaryIndexManager.java
index a916bd2..ff4567b 100644
--- a/src/java/org/apache/cassandra/index/SecondaryIndexManager.java
+++ b/src/java/org/apache/cassandra/index/SecondaryIndexManager.java
@@ -209,7 +209,8 @@ public class SecondaryIndexManager implements IndexRegistry
public void rebuildIndexesBlocking(Collection sstables,
Set indexNames)
{
Set toRebuild = indexes.values().stream()
- .filter(indexer ->
indexNames.contains(indexer.getIndexName()))
+ .filter(index ->
indexNames.contains(index.getIndexName()))
+
.filter(Index::shouldBuildBlocking)
.collect(Collectors.toSet());
if (toRebuild.isEmpty())
{
@@ -226,17 +227,23 @@ public class SecondaryIndexManager implements
IndexRegistry
public void buildAllIndexesBlocking(Collection sstables)
{
-buildIndexesBlocking(sstables, ImmutableSet.copyOf(indexes.values()));
+buildIndexesBlocking(sstables, indexes.values()
+ .stream()
+
.filter(Index::shouldBuildBlocking)
+ .collect(Collectors.toSet()));
}
// For convenience, may be called directly from Index impls
public void buildIndexBlocking(Index index)
{
-try (ColumnFamilyStore.RefViewFragment viewFragment =
baseCfs.selectAndReference(View.select(SSTableSet.CANONICAL));
- Refs sstables = viewFragment.refs)
+if (index.s
[jira] [Commented] (CASSANDRA-10216) Remove target type from internal index metadata
[
https://issues.apache.org/jira/browse/CASSANDRA-10216?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14745261#comment-14745261
]
Alexandre Dutra commented on CASSANDRA-10216:
-
[~beobal] A compatible java driver version can be found
[here|https://github.com/datastax/java-driver/pull/452].
We have a small problem with case-sensitive columns: suppose one creates an
index on {{values("MixedCaseColumn")}}; the target column would then contain
{{values(MixedCaseColumn)}} - note that the column identifier is unquoted;
since the driver is not parsing this string, it is unable to recreate the
original CREATE INDEX statement.
> Remove target type from internal index metadata
> ---
>
> Key: CASSANDRA-10216
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10216
> Project: Cassandra
> Issue Type: Improvement
>Reporter: Sam Tunnicliffe
>Assignee: Sam Tunnicliffe
> Labels: client-impacting
> Fix For: 3.0.0 rc1
>
>
> As part of CASSANDRA-6716 & in anticipation of CASSANDRA-10124, a distinction
> was introduced between secondary indexes which target a fixed set of 1 or
> more columns in the base data, and those which are agnostic to the structure
> of the underlying rows. This distinction is manifested in
> {{IndexMetadata.targetType}} and {{system_schema.indexes}}, in the
> {{target_type}} column. It could be argued that this distinction complicates
> the codebase without providing any tangible benefit, given that the target
> type is not actually used anywhere.
> It's only the impact on {{system_schema.indexes}} that makes puts this on the
> critical path for 3.0, any code changes are just implementation details.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9748) Can't see other nodes when using multiple network interfaces
[ https://issues.apache.org/jira/browse/CASSANDRA-9748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14745299#comment-14745299 ] Roman Bielik commented on CASSANDRA-9748: - I have retested on Cassandra 2.2.1 and the results are pretty much the same. If my premise is correct, I need to use public_IP as seeds as this is the interface/network where different datacenters can connect to each other. I have tested all possible permutations of listen_address, broadcast_address and broadcast_rpc_address. The seeds were set to public IP and rpc_address to 0.0.0.0 in all tests. Results: listen_address - if set to private_IP, the ring showed only one node broadcast_address - if set to private_IP, there was always problem with gossip broadcast_rpc_address - it did not matter in terms of connection or ring, so I could use private IP here. To sum up: I still needed to use public IP in seeds, listen_address and broadcast_address in order to start a Cassandra cluster. > Can't see other nodes when using multiple network interfaces > > > Key: CASSANDRA-9748 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9748 > Project: Cassandra > Issue Type: Bug > Environment: Cassandra 2.0.16; multi-DC configuration >Reporter: Roman Bielik > > The idea is to setup a multi-DC environment across 2 different networks based > on the following configuration recommendations: > http://docs.datastax.com/en/cassandra/2.0/cassandra/configuration/configMultiNetworks.html > Each node has 2 network interfaces. One used as a private network (DC1: > 10.0.1.x and DC2: 10.0.2.x). The second one a "public" network where all > nodes can see each other (this one has a higher latency). > Using the following settings in cassandra.yaml: > *seeds:* public IP (same as used in broadcast_address) > *listen_address:* private IP > *broadcast_address:* public IP > *rpc_address:* 0.0.0.0 > *endpoint_snitch:* GossipingPropertyFileSnitch > _(tried different combinations with no luck)_ > No firewall and no SSL/encryption used. > The problem is that nodes do not see each other (a gossip problem I guess). > The nodetool ring/status shows only the local node but not the other ones > (even from the same DC). > When I set listen_address to public IP, then everything works fine, but that > is not the required configuration. > _Note: Not using EC2 cloud!_ > netstat -anp | grep -E "(7199|9160|9042|7000)" > tcp0 0 0.0.0.0:71990.0.0.0:* > LISTEN 3587/java > tcp0 0 10.0.1.1:9160 0.0.0.0:* > LISTEN 3587/java > tcp0 0 10.0.1.1:9042 0.0.0.0:* > LISTEN 3587/java > tcp0 0 10.0.1.1:7000 0.0.0.0:* > LISTEN 3587/java > tcp0 0 127.0.0.1:7199 127.0.0.1:52874 > ESTABLISHED 3587/java > tcp0 0 10.0.1.1:7199 10.0.1.1:39650 > ESTABLISHED 3587/java -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9669) If sstable flushes complete out of order, on restart we can fail to replay necessary commit log records
[
https://issues.apache.org/jira/browse/CASSANDRA-9669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14745367#comment-14745367
]
Branimir Lambov commented on CASSANDRA-9669:
[Memtable.isCleanAfter|https://github.com/apache/cassandra/compare/trunk...belliottsmith:9669-2.1#diff-f0a15c3588b56c5ce53ece7c48e325b5R171]
doesn't look right. This guarantees that it does not contain data _before_ the
given position, and thus {{CFS.forceFlush(ReplayPosition)}} does not appear to
use it correctly.
[Descriptor.Version.isCompatible|https://github.com/apache/cassandra/compare/trunk...belliottsmith:9669-2.1#diff-d1b9592895e36631679f1d5cab0f61f1R97]
is suspect in the context of this ticket, "ka" reader says compatible with
"kb" data but read will fail. Later readers (e.g. "la") will also fail to read
these tables but say they are compatible. Could this cause trouble?
The new format needs to be added to {{LegacySSTableTest}} and
{{test/data/legacy-sstables}} and included in 2.2 and 3.0 versions of the patch.
\\
\\
Nits:
[Memtable.approximateCommitLogLowerBound|https://github.com/belliottsmith/cassandra/blob/d190c29d5e24e0b2a93a844d250695107e8bb942/src/java/org/apache/cassandra/db/Memtable.java#L64]:
This must satisfy {{approximateCommitLogLowerBound <= commitLogLowerBound}}
(after discarding predecessor), mustn't it? Could you add a comment to say so?
[Memtable.forceFlush(ReplayPosition)|https://github.com/apache/cassandra/compare/trunk...belliottsmith:9669-2.1#diff-98f5acb96aa6d684781936c141132e2aR972]
no longer loops through index CFSes, can you add a comment to state why this
is the right thing to do?
[writeSortedContents|https://github.com/apache/cassandra/compare/trunk...belliottsmith:9669-2.1#diff-f0a15c3588b56c5ce53ece7c48e325b5R379]:
duplicate log?
[Descriptor.Version version
text|https://github.com/apache/cassandra/compare/trunk...belliottsmith:9669-2.1#diff-d1b9592895e36631679f1d5cab0f61f1R56]:
2._1_.7; j versions are compatible according to {{isCompatible}}, shouldn't
comment be retained?
[LegacyMetadataSerializer.serialize|https://github.com/apache/cassandra/compare/trunk...belliottsmith:9669-2.1#diff-0f1c5b6135b34548f033e5168f6761a5R96]:
No need to declare {{commitLogUpperBound}} here. Declare with assignment below
to ease reading.
[SSTableMetadataViewer|https://github.com/apache/cassandra/compare/trunk...belliottsmith:9669-2.1#diff-63785fd0e03996f22c0c2c38eb137f7fR71]:
Could use some text describing what's being printed.
> If sstable flushes complete out of order, on restart we can fail to replay
> necessary commit log records
> ---
>
> Key: CASSANDRA-9669
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9669
> Project: Cassandra
> Issue Type: Bug
> Components: Core
>Reporter: Benedict
>Assignee: Benedict
>Priority: Critical
> Labels: correctness
> Fix For: 3.x, 2.1.x, 2.2.x, 3.0.x
>
>
> While {{postFlushExecutor}} ensures it never expires CL entries out-of-order,
> on restart we simply take the maximum replay position of any sstable on disk,
> and ignore anything prior.
> It is quite possible for there to be two flushes triggered for a given table,
> and for the second to finish first by virtue of containing a much smaller
> quantity of live data (or perhaps the disk is just under less pressure). If
> we crash before the first sstable has been written, then on restart the data
> it would have represented will disappear, since we will not replay the CL
> records.
> This looks to be a bug present since time immemorial, and also seems pretty
> serious.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-10316) Improve ColumnDefinition comparison performance
[ https://issues.apache.org/jira/browse/CASSANDRA-10316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14745377#comment-14745377 ] Sylvain Lebresne commented on CASSANDRA-10316: -- Assuming CI passes, +1. > Improve ColumnDefinition comparison performance > --- > > Key: CASSANDRA-10316 > URL: https://issues.apache.org/jira/browse/CASSANDRA-10316 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: Benedict >Assignee: Benedict > Fix For: 3.0.x > > > We have already improved the performance here for the basic comparison, > however since these happen exceedingly frequently we may as well go a little > (easy step) further. This is a really tiny patch, and we should aim to > include before GA, but not RC. > Right now we use all of an int for the metadata presorting, but in fact we > only need 2-3 bytes for this. We can upcast the int to a long, and use the > remaining bits to store the prefix of the _name_. This way, a majority of > comparisons become a single long comparison. Which will be considerably > cheaper. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-10216) Remove target type from internal index metadata
[
https://issues.apache.org/jira/browse/CASSANDRA-10216?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14745380#comment-14745380
]
Sam Tunnicliffe commented on CASSANDRA-10216:
-
bq.We have a small problem with case-sensitive columns
Good point, I've pushed a further commit to deal with that by quoting such
column names in the {{target}} string.
> Remove target type from internal index metadata
> ---
>
> Key: CASSANDRA-10216
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10216
> Project: Cassandra
> Issue Type: Improvement
>Reporter: Sam Tunnicliffe
>Assignee: Sam Tunnicliffe
> Labels: client-impacting
> Fix For: 3.0.0 rc1
>
>
> As part of CASSANDRA-6716 & in anticipation of CASSANDRA-10124, a distinction
> was introduced between secondary indexes which target a fixed set of 1 or
> more columns in the base data, and those which are agnostic to the structure
> of the underlying rows. This distinction is manifested in
> {{IndexMetadata.targetType}} and {{system_schema.indexes}}, in the
> {{target_type}} column. It could be argued that this distinction complicates
> the codebase without providing any tangible benefit, given that the target
> type is not actually used anywhere.
> It's only the impact on {{system_schema.indexes}} that makes puts this on the
> critical path for 3.0, any code changes are just implementation details.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9669) If sstable flushes complete out of order, on restart we can fail to replay necessary commit log records
[
https://issues.apache.org/jira/browse/CASSANDRA-9669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14745432#comment-14745432
]
Benedict commented on CASSANDRA-9669:
-
bq. Memtable.isCleanAfter doesn't look right.
Good catch, I have inverted the implementation and meaning, and renamed it to
{{mayContainDataSince}}
bq. "ka" reader says compatible with "kb" data but read will fail
Hmm. This is a bit of a problem. I must admit I didn't look too closely at
compatibility, since I assumed the whole point of the major/minor chars was to
permit intra- and extra-version sstable version increments. Without that, this
seems to be a bit of a mess. I guess we will need to increment all of the
sstable versions past the max of the current. We should perhaps rethink
accepting all versions <= first char of current, as it doesn't permit much
flexibility.
Thanks. I'll address this, your nits, and what looks like a relatively
innocuous problem with DTCS shortly.
> If sstable flushes complete out of order, on restart we can fail to replay
> necessary commit log records
> ---
>
> Key: CASSANDRA-9669
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9669
> Project: Cassandra
> Issue Type: Bug
> Components: Core
>Reporter: Benedict
>Assignee: Benedict
>Priority: Critical
> Labels: correctness
> Fix For: 3.x, 2.1.x, 2.2.x, 3.0.x
>
>
> While {{postFlushExecutor}} ensures it never expires CL entries out-of-order,
> on restart we simply take the maximum replay position of any sstable on disk,
> and ignore anything prior.
> It is quite possible for there to be two flushes triggered for a given table,
> and for the second to finish first by virtue of containing a much smaller
> quantity of live data (or perhaps the disk is just under less pressure). If
> we crash before the first sstable has been written, then on restart the data
> it would have represented will disappear, since we will not replay the CL
> records.
> This looks to be a bug present since time immemorial, and also seems pretty
> serious.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (CASSANDRA-10340) Stress should exit with non-zero status after failure
Paulo Motta created CASSANDRA-10340: --- Summary: Stress should exit with non-zero status after failure Key: CASSANDRA-10340 URL: https://issues.apache.org/jira/browse/CASSANDRA-10340 Project: Cassandra Issue Type: Bug Reporter: Paulo Motta Priority: Minor Currently, stress always exits with sucess status, even if after a failure. In order to be able to rely on stress exit status during dtests it would be nice if it exited with a non-zero status after failures. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-10340) Stress should exit with non-zero status after failure
[ https://issues.apache.org/jira/browse/CASSANDRA-10340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Paulo Motta updated CASSANDRA-10340: Labels: stress (was: ) > Stress should exit with non-zero status after failure > - > > Key: CASSANDRA-10340 > URL: https://issues.apache.org/jira/browse/CASSANDRA-10340 > Project: Cassandra > Issue Type: Bug >Reporter: Paulo Motta >Priority: Minor > Labels: stress > > Currently, stress always exits with sucess status, even if after a failure. > In order to be able to rely on stress exit status during dtests it would be > nice if it exited with a non-zero status after failures. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (CASSANDRA-10341) Streaming does not guarantee cache invalidation
Benedict created CASSANDRA-10341: Summary: Streaming does not guarantee cache invalidation Key: CASSANDRA-10341 URL: https://issues.apache.org/jira/browse/CASSANDRA-10341 Project: Cassandra Issue Type: Bug Components: Core Reporter: Benedict Looking at the code, we attempt to invalidate the row cache for any rows we receive via streaming, however we invalidate them immediately, before the new data is available. So, if it is requested (which is likely if it is "hot") in the interval, it will be re-cached and not invalidated. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-10216) Remove target type from internal index metadata
[
https://issues.apache.org/jira/browse/CASSANDRA-10216?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14745462#comment-14745462
]
Sylvain Lebresne commented on CASSANDRA-10216:
--
I don't think
{{!column.toString().equals(column.toString().toLowerCase(Locale.US));}} works
all the time as a test to know if the name should be quoted, because we
actually allow identifiers with spaces in them if they are quoted (so a name
could be all lowercase, but still require quoting because it has spaces or some
other weird character). The proper test is probably to check if the name
matches the grammar for non-quoted identifer, i.e. {{IDENT : LETTER (LETTER |
DIGIT | '_')*}}.
Similarly, {{columnName.replaceAll("\\\"", "");}} is a tad dangerous because
you can have quotes inside a column identifier if you escape it by doubling it
(so {{"foo""bar"}} is the identifier {{foo"bar}}). I guess just checking if the
first character is a quote, and remove that and the last character would be
good enough however.
> Remove target type from internal index metadata
> ---
>
> Key: CASSANDRA-10216
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10216
> Project: Cassandra
> Issue Type: Improvement
>Reporter: Sam Tunnicliffe
>Assignee: Sam Tunnicliffe
> Labels: client-impacting
> Fix For: 3.0.0 rc1
>
>
> As part of CASSANDRA-6716 & in anticipation of CASSANDRA-10124, a distinction
> was introduced between secondary indexes which target a fixed set of 1 or
> more columns in the base data, and those which are agnostic to the structure
> of the underlying rows. This distinction is manifested in
> {{IndexMetadata.targetType}} and {{system_schema.indexes}}, in the
> {{target_type}} column. It could be argued that this distinction complicates
> the codebase without providing any tangible benefit, given that the target
> type is not actually used anywhere.
> It's only the impact on {{system_schema.indexes}} that makes puts this on the
> critical path for 3.0, any code changes are just implementation details.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9669) If sstable flushes complete out of order, on restart we can fail to replay necessary commit log records
[
https://issues.apache.org/jira/browse/CASSANDRA-9669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14745464#comment-14745464
]
Benedict commented on CASSANDRA-9669:
-
bq. Hmm. This is a bit of a problem.
I remember now why it is not. We don't generally permit downgrades, and when
upgrading you must upgrade via the latest of any intervening minor version.
Whether or not this is a problem, I think it's probably better left for another
ticket, but I think we would be best off fixing it so a given c* version knows
the maximum sstable version it can read for each prior (and equal) minor
cassandra version. So long as the main data format is the same (which is the
case here) there shouldn't be a problem, as we only stream the data contents
between nodes, so there's no reason for an old version to see a new file.
However we should probably make that more robust to sstable version changes
also.
> If sstable flushes complete out of order, on restart we can fail to replay
> necessary commit log records
> ---
>
> Key: CASSANDRA-9669
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9669
> Project: Cassandra
> Issue Type: Bug
> Components: Core
>Reporter: Benedict
>Assignee: Benedict
>Priority: Critical
> Labels: correctness
> Fix For: 3.x, 2.1.x, 2.2.x, 3.0.x
>
>
> While {{postFlushExecutor}} ensures it never expires CL entries out-of-order,
> on restart we simply take the maximum replay position of any sstable on disk,
> and ignore anything prior.
> It is quite possible for there to be two flushes triggered for a given table,
> and for the second to finish first by virtue of containing a much smaller
> quantity of live data (or perhaps the disk is just under less pressure). If
> we crash before the first sstable has been written, then on restart the data
> it would have represented will disappear, since we will not replay the CL
> records.
> This looks to be a bug present since time immemorial, and also seems pretty
> serious.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (CASSANDRA-10342) Read defragmentation can cause unnecessary repairs
Marcus Olsson created CASSANDRA-10342: - Summary: Read defragmentation can cause unnecessary repairs Key: CASSANDRA-10342 URL: https://issues.apache.org/jira/browse/CASSANDRA-10342 Project: Cassandra Issue Type: Bug Reporter: Marcus Olsson Priority: Minor After applying the fix from CASSANDRA-10299 to the cluster we started having a problem of ~20k small sstables appearing for the table with static data when running incremental repair. In the logs there were several messages about flushes for that table, one for each repaired range. The flushed sstables were 0.000kb in size with < 100 ops in each. When checking cfstats there were several writes to that table, even though we were only reading from it and read repair did not repair anything. After digging around in the codebase I noticed that defragmentation of data can occur while reading, depending on the query and some other conditions. This causes the read data to be inserted again to have it in a more recent sstable, which can be a problem if that data was repaired using incremental repair. The defragmentation is done in [CollationController.java|https://github.com/apache/cassandra/blob/cassandra-2.1/src/java/org/apache/cassandra/db/CollationController.java#L151]. I guess this wasn't a problem with full repairs since I assume that the digest should be the same even if you have two copies of the same data. But with incremental repair this will most probably cause a mismatch between nodes if that data already was repaired, since the other nodes probably won't have that data in their unrepaired set. -- I can add that the problems on our cluster was probably due to the fact that CASSANDRA-10299 caused the same data to be streamed multiple times and ending up in several sstables. One of the conditions for the defragmentation is that the number of sstables read during a read request have to be more than the minimum number of sstables needed for a compaction(> 4 in our case). So normally I don't think this would cause ~20k sstables to appear, we probably hit an extreme. One workaround for this is to use another compaction strategy than STCS(it seems to be the only affected strategy, atleast in 2.1), but the solution might be to either make defragmentation configurable per table or avoid reinserting the data if any of the sstables involved in the read are repaired. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9669) If sstable flushes complete out of order, on restart we can fail to replay necessary commit log records
[
https://issues.apache.org/jira/browse/CASSANDRA-9669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14745503#comment-14745503
]
Aleksey Yeschenko commented on CASSANDRA-9669:
--
FWIW we do have startup checks that do this -
https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/service/StartupChecks.java#L209
> If sstable flushes complete out of order, on restart we can fail to replay
> necessary commit log records
> ---
>
> Key: CASSANDRA-9669
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9669
> Project: Cassandra
> Issue Type: Bug
> Components: Core
>Reporter: Benedict
>Assignee: Benedict
>Priority: Critical
> Labels: correctness
> Fix For: 3.x, 2.1.x, 2.2.x, 3.0.x
>
>
> While {{postFlushExecutor}} ensures it never expires CL entries out-of-order,
> on restart we simply take the maximum replay position of any sstable on disk,
> and ignore anything prior.
> It is quite possible for there to be two flushes triggered for a given table,
> and for the second to finish first by virtue of containing a much smaller
> quantity of live data (or perhaps the disk is just under less pressure). If
> we crash before the first sstable has been written, then on restart the data
> it would have represented will disappear, since we will not replay the CL
> records.
> This looks to be a bug present since time immemorial, and also seems pretty
> serious.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9669) If sstable flushes complete out of order, on restart we can fail to replay necessary commit log records
[
https://issues.apache.org/jira/browse/CASSANDRA-9669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14745510#comment-14745510
]
Benedict commented on CASSANDRA-9669:
-
Except that this won't fail the startup checks if you downgrade patch versions.
Or upgrade to a non-latest patch version when upgrading your major/minor.
> If sstable flushes complete out of order, on restart we can fail to replay
> necessary commit log records
> ---
>
> Key: CASSANDRA-9669
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9669
> Project: Cassandra
> Issue Type: Bug
> Components: Core
>Reporter: Benedict
>Assignee: Benedict
>Priority: Critical
> Labels: correctness
> Fix For: 3.x, 2.1.x, 2.2.x, 3.0.x
>
>
> While {{postFlushExecutor}} ensures it never expires CL entries out-of-order,
> on restart we simply take the maximum replay position of any sstable on disk,
> and ignore anything prior.
> It is quite possible for there to be two flushes triggered for a given table,
> and for the second to finish first by virtue of containing a much smaller
> quantity of live data (or perhaps the disk is just under less pressure). If
> we crash before the first sstable has been written, then on restart the data
> it would have represented will disappear, since we will not replay the CL
> records.
> This looks to be a bug present since time immemorial, and also seems pretty
> serious.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-10318) Update cqlsh COPY for new internal driver serialization interface
[
https://issues.apache.org/jira/browse/CASSANDRA-10318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14745517#comment-14745517
]
Adam Holmberg commented on CASSANDRA-10318:
---
That looks right. Sorry for the misdirection. I was most focused on the binary
serialization and didn't really do a proper analysis.
> Update cqlsh COPY for new internal driver serialization interface
> -
>
> Key: CASSANDRA-10318
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10318
> Project: Cassandra
> Issue Type: Bug
>Reporter: Adam Holmberg
>Assignee: Stefania
> Fix For: 3.0.0 rc1
>
>
> A recent driver update changed some of the internal serialization interface.
> cqlsh relies on this for the copy command and will need to be updated.
> Previously used
> {code}
> cassandra.protocol.QueryMesage.to_binary
> {code}
> now should use
> {code}
> cassandra.protocol.ProtocolHandler.encode_message(...)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (CASSANDRA-9870) Improve cassandra-stress graphing
[ https://issues.apache.org/jira/browse/CASSANDRA-9870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jim Witschey updated CASSANDRA-9870: Assignee: Ryan McGuire (was: Shawn Kumar) > Improve cassandra-stress graphing > - > > Key: CASSANDRA-9870 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9870 > Project: Cassandra > Issue Type: Improvement > Components: Tools >Reporter: Benedict >Assignee: Ryan McGuire > Attachments: reads.svg > > > CASSANDRA-7918 introduces graph output from a stress run, but these graphs > are a little limited. Attached to the ticket is an example of some improved > graphs which can serve as the *basis* for some improvements, which I will > briefly describe. They should not be taken as the exact end goal, but we > should aim for at least their functionality. Preferably with some Javascript > advantages thrown in, such as the hiding of datasets/graphs for clarity. Any > ideas for improvements are *definitely* encouraged. > Some overarching design principles: > * Display _on *one* screen_ all of the information necessary to get a good > idea of how two or more branches compare to each other. Ideally we will > reintroduce this, painting multiple graphs onto one screen, stretched to fit. > * Axes must be truncated to only the interesting dimensions, to ensure there > is no wasted space. > * Each graph displaying multiple kinds of data should use colour _and shape_ > to help easily distinguish the different datasets. > * Each graph should be tailored to the data it is representing, and we should > have multiple views of each data. > The data can roughly be partitioned into three kinds: > * throughput > * latency > * gc > These can each be viewed in different ways: > * as a continuous plot of: > ** raw data > ** scaled/compared to a "base" branch, or other metric > ** cumulatively > * as box plots > ** ideally, these will plot median, outer quartiles, outer deciles and > absolute limits of the distribution, so the shape of the data can be best > understood > Each compresses the information differently, losing different information, so > that collectively they help to understand the data. > Some basic rules for presentation that work well: > * Latency information should be plotted to a logarithmic scale, to avoid high > latencies drowning out low ones > * GC information should be plotted cumulatively, to avoid differing > throughputs giving the impression of worse GC. It should also have a line > that is rescaled by the amount of work (number of operations) completed > * Throughput should be plotted as the actual numbers > To walk the graphs top-left to bottom-right, we have: > * Spot throughput comparison of branches to the baseline branch, as an > improvement ratio (which can of course be negative, but is not in this > example) > * Raw throughput of all branches (no baseline) > * Raw throughput as a box plot > * Latency percentiles, compared to baseline. The percentage improvement at > any point in time vs baseline is calculated, and then multiplied by the > overall median for the entire run. This simply permits the non-baseline > branches to scatter their wins/loss around a relatively clustered line for > each percentile. It's probably the most "dishonest" graph but comparing > something like latency where each data point can have very high variance is > difficult, and this gives you an idea of clustering of improvements/losses. > * Latency percentiles, raw, each with a different shape; lowest percentiles > plotted as a solid line as they vary least, with higher percentiles each > getting their own subtly different shape to scatter. > * Latency box plots > * GC time, plotted cumulatively and also scaled by work done > * GC Mb, plotted cumulatively and also scaled by work done > * GC time, raw > * GC time as a box plot > These do mostly introduce the concept of a "baseline" branch. It may be that, > ideally, this baseline be selected by a dropdown so the javascript can > transform the output dynamically. This would permit more interesting > comparisons to be made on the fly. > There are also some complexities, such as deciding which datapoints to > compare against baseline when times get out-of-whack (due to GC, etc, causing > a lack of output for a period). The version I uploaded does a merge of the > times, permitting a small degree of variance, and ignoring those datapoints > we cannot pair. One option here might be to change stress' behaviour to > always print to a strict schedule, instead of trying to get absolutely > accurate apportionment of timings. If this makes things much simpler, it can > be done. > As previously stated, but may be lost in the wall-of-text, these should be > taken as a starting point / sign post, rather than a golden rule
[jira] [Commented] (CASSANDRA-9961) cqlsh should have DESCRIBE MATERIALIZED VIEW
[
https://issues.apache.org/jira/browse/CASSANDRA-9961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14745529#comment-14745529
]
Adam Holmberg commented on CASSANDRA-9961:
--
https://github.com/datastax/python-driver/commit/96883eb8b61753bd704a087717f2d556980bf4d2
> cqlsh should have DESCRIBE MATERIALIZED VIEW
>
>
> Key: CASSANDRA-9961
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9961
> Project: Cassandra
> Issue Type: Improvement
>Reporter: Carl Yeksigian
>Assignee: Stefania
> Labels: client-impacting, materializedviews
> Fix For: 3.0.0 rc1
>
>
> cqlsh doesn't currently produce describe output that can be used to recreate
> a MV. Needs to add a new {{DESCRIBE MATERIALIZED VIEW}} command, and also add
> to {{DESCRIBE KEYSPACE}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-10264) Unable to use conditions on static columns for DELETE
[
https://issues.apache.org/jira/browse/CASSANDRA-10264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14745538#comment-14745538
]
Jim Witschey commented on CASSANDRA-10264:
--
Thank you both for having a look and working on this. I just realized I wasn't
clear -- I get the {{invalid syntax}} message when executing via cqlsh, so
there may be another error there.
> Unable to use conditions on static columns for DELETE
> -
>
> Key: CASSANDRA-10264
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10264
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: Cassandra 2.2.0
>Reporter: DOAN DuyHai
>Assignee: Benjamin Lerer
>
> {noformat}
> cqlsh:test> create table static_table(id int, stat int static, ord int, val
> text, primary key(id,ord));
> cqlsh:test> insert into static_table (id,stat,ord,val) VALUES ( 1, 1, 1, '1');
> cqlsh:test> delete from static_table where id=1 and ord=1 if stat != 1;
> Invalid syntax at line 1, char 55
> delete from static_table where id=1 and ord=1 if stat != 1;
> ^
> {noformat}
> Same error if using =, <, <=, >= or > condition
> According to [~thobbs] the syntax should work. Plus, the error message is
> wrong
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-10323) Add more MaterializedView metrics
[ https://issues.apache.org/jira/browse/CASSANDRA-10323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14745548#comment-14745548 ] Joel Knighton commented on CASSANDRA-10323: --- I'd be interested to see counters for batchlog entries created/removed by a coordinator. Effectively, how often does batchlog replay kick in as a mechanism to work toward quorum writes? > Add more MaterializedView metrics > - > > Key: CASSANDRA-10323 > URL: https://issues.apache.org/jira/browse/CASSANDRA-10323 > Project: Cassandra > Issue Type: Improvement >Reporter: T Jake Luciani >Assignee: T Jake Luciani > Fix For: 3.0.0 rc1 > > > We need to add more metrics to help understand where time is spent in > materialized view writes. We currently track the ratio of async base -> view > mutations that fail. > We should also add > * The amount of time spent waiting for the partition lock (contention) > * The amount of time spent reading data > Any others? > [~carlyeks] [~jkni] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8530) Query on a secondary index creates huge CPU spike + unable to trace
[
https://issues.apache.org/jira/browse/CASSANDRA-8530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14745552#comment-14745552
]
Tom van den Berge commented on CASSANDRA-8530:
--
Pavel,
I'm having similar problems since I'm using vnodes, but I'm not sure if that's
causing the problems. Are you using vnodes? Did you manage to find a solution
or workaround for this problem?
Tom
> Query on a secondary index creates huge CPU spike + unable to trace
> ---
>
> Key: CASSANDRA-8530
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8530
> Project: Cassandra
> Issue Type: Bug
> Components: API, Core
> Environment: CentOs 6.5 / Cassandra 2.1.2
>Reporter: Pavel Baranov
>
> After upgrading cassandra from 2.0.10 to 2.1.2 we are having all kinds of
> issues, especially with performance.
> java version "1.7.0_65"
> Table creation:
> {noformat}
> tweets> desc table tweets;
> CREATE TABLE tweets.tweets (
> uname text,
> tweet_id bigint,
> tweet text,
> tweet_date timestamp,
> tweet_date_only text,
> uid bigint,
> PRIMARY KEY (uname, tweet_id)
> ) WITH CLUSTERING ORDER BY (tweet_id ASC)
> AND bloom_filter_fp_chance = 0.01
> AND caching = '{"keys":"ALL", "rows_per_partition":"NONE"}'
> AND comment = ''
> AND compaction = {'min_threshold': '10', 'class':
> 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy',
> 'max_threshold': '32'}
> AND compression = {'sstable_compression':
> 'org.apache.cassandra.io.compress.LZ4Compressor'}
> AND dclocal_read_repair_chance = 0.0
> AND default_time_to_live = 0
> AND gc_grace_seconds = 864000
> AND max_index_interval = 2048
> AND memtable_flush_period_in_ms = 0
> AND min_index_interval = 128
> AND read_repair_chance = 0.1
> AND speculative_retry = '99.0PERCENTILE';
> CREATE INDEX tweets_tweet_date_only_idx ON tweets.tweets (tweet_date_only);
> CREATE INDEX tweets_uid ON tweets.tweets (uid);
> {noformat}
> With Cassandra 2.0.10 this query:
> {noformat}
> select uname from tweets where uid = 636732672 limit 1;
> {noformat}
> did not have any issues. After upgrade, I can see the cpu spikes and load avg
> goes from ~1 to ~13, especially if I execute the query over and over again.
> Doing "tracing on" does not work and just returns:
> "Statement trace did not complete within 10 seconds"
> I've done:
> nodetool upgradesstables
> recreated indexes
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-10323) Add more MaterializedView metrics
[ https://issues.apache.org/jira/browse/CASSANDRA-10323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14745554#comment-14745554 ] Aleksey Yeschenko commented on CASSANDRA-10323: --- It's possible to do that quite efficiently now that CASSANDRA-9673 is in. Commented about it [here|https://issues.apache.org/jira/browse/CASSANDRA-9673?focusedCommentId=14712205&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14712205]. That's metrics. Batchlog however is not a mechanism for achieving quorum writes and shouldn't be seen as such. > Add more MaterializedView metrics > - > > Key: CASSANDRA-10323 > URL: https://issues.apache.org/jira/browse/CASSANDRA-10323 > Project: Cassandra > Issue Type: Improvement >Reporter: T Jake Luciani >Assignee: T Jake Luciani > Fix For: 3.0.0 rc1 > > > We need to add more metrics to help understand where time is spent in > materialized view writes. We currently track the ratio of async base -> view > mutations that fail. > We should also add > * The amount of time spent waiting for the partition lock (contention) > * The amount of time spent reading data > Any others? > [~carlyeks] [~jkni] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-10323) Add more MaterializedView metrics
[ https://issues.apache.org/jira/browse/CASSANDRA-10323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14745557#comment-14745557 ] Joel Knighton commented on CASSANDRA-10323: --- Agreed re: paragraph 3. I'll watch that closer in the future. > Add more MaterializedView metrics > - > > Key: CASSANDRA-10323 > URL: https://issues.apache.org/jira/browse/CASSANDRA-10323 > Project: Cassandra > Issue Type: Improvement >Reporter: T Jake Luciani >Assignee: T Jake Luciani > Fix For: 3.0.0 rc1 > > > We need to add more metrics to help understand where time is spent in > materialized view writes. We currently track the ratio of async base -> view > mutations that fail. > We should also add > * The amount of time spent waiting for the partition lock (contention) > * The amount of time spent reading data > Any others? > [~carlyeks] [~jkni] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8530) Query on a secondary index creates huge CPU spike + unable to trace
[
https://issues.apache.org/jira/browse/CASSANDRA-8530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14745573#comment-14745573
]
Pavel Baranov commented on CASSANDRA-8530:
--
Tom, we decided to restructure our tables and not use secondary indexes at all
(feedback from Datastax). It's been working great so far.
- Pavel
> Query on a secondary index creates huge CPU spike + unable to trace
> ---
>
> Key: CASSANDRA-8530
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8530
> Project: Cassandra
> Issue Type: Bug
> Components: API, Core
> Environment: CentOs 6.5 / Cassandra 2.1.2
>Reporter: Pavel Baranov
>
> After upgrading cassandra from 2.0.10 to 2.1.2 we are having all kinds of
> issues, especially with performance.
> java version "1.7.0_65"
> Table creation:
> {noformat}
> tweets> desc table tweets;
> CREATE TABLE tweets.tweets (
> uname text,
> tweet_id bigint,
> tweet text,
> tweet_date timestamp,
> tweet_date_only text,
> uid bigint,
> PRIMARY KEY (uname, tweet_id)
> ) WITH CLUSTERING ORDER BY (tweet_id ASC)
> AND bloom_filter_fp_chance = 0.01
> AND caching = '{"keys":"ALL", "rows_per_partition":"NONE"}'
> AND comment = ''
> AND compaction = {'min_threshold': '10', 'class':
> 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy',
> 'max_threshold': '32'}
> AND compression = {'sstable_compression':
> 'org.apache.cassandra.io.compress.LZ4Compressor'}
> AND dclocal_read_repair_chance = 0.0
> AND default_time_to_live = 0
> AND gc_grace_seconds = 864000
> AND max_index_interval = 2048
> AND memtable_flush_period_in_ms = 0
> AND min_index_interval = 128
> AND read_repair_chance = 0.1
> AND speculative_retry = '99.0PERCENTILE';
> CREATE INDEX tweets_tweet_date_only_idx ON tweets.tweets (tweet_date_only);
> CREATE INDEX tweets_uid ON tweets.tweets (uid);
> {noformat}
> With Cassandra 2.0.10 this query:
> {noformat}
> select uname from tweets where uid = 636732672 limit 1;
> {noformat}
> did not have any issues. After upgrade, I can see the cpu spikes and load avg
> goes from ~1 to ~13, especially if I execute the query over and over again.
> Doing "tracing on" does not work and just returns:
> "Statement trace did not complete within 10 seconds"
> I've done:
> nodetool upgradesstables
> recreated indexes
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (CASSANDRA-10343) Erroneous partition deletion events are delivered to indexers
Sam Tunnicliffe created CASSANDRA-10343:
---
Summary: Erroneous partition deletion events are delivered to
indexers
Key: CASSANDRA-10343
URL: https://issues.apache.org/jira/browse/CASSANDRA-10343
Project: Cassandra
Issue Type: Bug
Reporter: Sam Tunnicliffe
Assignee: Sam Tunnicliffe
Fix For: 3.0.0 rc1
In {{AtomicBTreePartition#addAllWithSizeDelta}} we check
{{update.deletionInfo().isLive()}} to determine whether the
{{PartitionDeletion}} should be passed to the supplied {{IndexTransaction}}.
This is incorrect and it results in the {{PartitionDeletion}} being passed on
if *either* a partition delete or any range tombstones are present in the
update.
We need to change:
{code}
if (!update.deletionInfo().isLive())
{code}
to
{code}
if (!update.deletionInfo().getPartitionDeletion().isLive())
{code}
Note that the built-in indexes don't currently do anything with the
{{deletionTime}} and so aren't affected, but any custom implementation which
didn't examine it properly might remove data from its index where it shouldn't
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
cassandra git commit: (Pig) support BulkOutputFormat as a URL parameter
Repository: cassandra
Updated Branches:
refs/heads/cassandra-2.1 ae5108645 -> c7b407357
(Pig) support BulkOutputFormat as a URL parameter
patch by Alex Liu; reviewed by Piotr KoÅaczkowski for CASSANDRA-7410
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/c7b40735
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/c7b40735
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/c7b40735
Branch: refs/heads/cassandra-2.1
Commit: c7b40735789c840529002eb3c11d8731f460d61c
Parents: ae51086
Author: Alex Liu
Authored: Tue Sep 15 16:06:18 2015 +0100
Committer: Aleksey Yeschenko
Committed: Tue Sep 15 16:08:54 2015 +0100
--
CHANGES.txt | 1 +
.../hadoop/cql3/CqlBulkOutputFormat.java| 93 +++-
.../hadoop/cql3/CqlBulkRecordWriter.java| 87
.../cassandra/hadoop/pig/CqlNativeStorage.java | 213 +--
.../apache/cassandra/hadoop/pig/CqlStorage.java | 1 -
.../org/apache/cassandra/tools/BulkLoader.java | 2 +-
test/conf/cassandra.yaml| 1 +
.../org/apache/cassandra/pig/CqlTableTest.java | 36
.../org/apache/cassandra/pig/PigTestBase.java | 3 +-
9 files changed, 336 insertions(+), 101 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/c7b40735/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index dff47fc..5f11049 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
2.1.10
+ * (Pig) support BulkOutputFormat as a URL parameter (CASSANDRA-7410)
* BATCH statement is broken in cqlsh (CASSANDRA-10272)
* Added configurable warning threshold for GC duration (CASSANDRA-8907)
* (cqlsh) Make cqlsh PEP8 compliant (CASSANDRA-10066)
http://git-wip-us.apache.org/repos/asf/cassandra/blob/c7b40735/src/java/org/apache/cassandra/hadoop/cql3/CqlBulkOutputFormat.java
--
diff --git a/src/java/org/apache/cassandra/hadoop/cql3/CqlBulkOutputFormat.java
b/src/java/org/apache/cassandra/hadoop/cql3/CqlBulkOutputFormat.java
index 887fe8e..7fedb41 100644
--- a/src/java/org/apache/cassandra/hadoop/cql3/CqlBulkOutputFormat.java
+++ b/src/java/org/apache/cassandra/hadoop/cql3/CqlBulkOutputFormat.java
@@ -22,6 +22,7 @@ import java.io.IOException;
import java.nio.ByteBuffer;
import java.util.List;
+import org.apache.cassandra.config.EncryptionOptions;
import org.apache.cassandra.hadoop.AbstractBulkOutputFormat;
import org.apache.cassandra.hadoop.ConfigHelper;
import org.apache.hadoop.conf.Configuration;
@@ -54,6 +55,16 @@ public class CqlBulkOutputFormat extends
AbstractBulkOutputFormathttp://git-wip-us.apache.org/repos/asf/cassandra/blob/c7b40735/src/java/org/apache/cassandra/hadoop/cql3/CqlBulkRecordWriter.java
--
diff --git a/src/java/org/apache/cassandra/hadoop/cql3/CqlBulkRecordWriter.java
b/src/java/org/apache/cassandra/hadoop/cql3/CqlBulkRecordWriter.java
index e60a240..ced8aa9 100644
--- a/src/java/org/apache/cassandra/hadoop/cql3/CqlBulkRecordWriter.java
+++ b/src/java/org/apache/cassandra/hadoop/cql3/CqlBulkRecordWriter.java
@@ -19,13 +19,16 @@ package org.apache.cassandra.hadoop.cql3;
import java.io.File;
import java.io.IOException;
+import java.net.InetAddress;
+import java.net.UnknownHostException;
import java.nio.ByteBuffer;
-import java.util.HashMap;
+import java.util.HashSet;
import java.util.List;
-import java.util.Map;
+import java.util.Set;
import java.util.UUID;
-import org.apache.cassandra.config.CFMetaData;
+import org.apache.cassandra.config.EncryptionOptions;
+import org.apache.cassandra.config.EncryptionOptions.ServerEncryptionOptions;
import org.apache.cassandra.exceptions.InvalidRequestException;
import org.apache.cassandra.hadoop.AbstractBulkRecordWriter;
import org.apache.cassandra.hadoop.BulkRecordWriter;
@@ -35,6 +38,9 @@ import org.apache.cassandra.io.sstable.CQLSSTableWriter;
import org.apache.cassandra.io.sstable.SSTableLoader;
import org.apache.cassandra.io.util.FileUtils;
import org.apache.cassandra.streaming.StreamState;
+import org.apache.cassandra.thrift.ITransportFactory;
+import org.apache.cassandra.tools.BulkLoader;
+import org.apache.commons.lang.StringUtils;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.mapreduce.TaskAttemptContext;
import org.apache.hadoop.util.Progressable;
@@ -108,10 +114,7 @@ public class CqlBulkRecordWriter extends
AbstractBulkRecordWriter> knownCqlCfs = new
HashMap<>();
-
-public ExternalClient(Configuration conf)
-{
-super(conf);
-}
-public void addKnownCfs(String keyspace, String cql)
[1/2] cassandra git commit: (Pig) support BulkOutputFormat as a URL parameter
Repository: cassandra
Updated Branches:
refs/heads/cassandra-2.2 979179609 -> cd4a1e6ac
(Pig) support BulkOutputFormat as a URL parameter
patch by Alex Liu; reviewed by Piotr KoÅaczkowski for CASSANDRA-7410
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/c7b40735
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/c7b40735
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/c7b40735
Branch: refs/heads/cassandra-2.2
Commit: c7b40735789c840529002eb3c11d8731f460d61c
Parents: ae51086
Author: Alex Liu
Authored: Tue Sep 15 16:06:18 2015 +0100
Committer: Aleksey Yeschenko
Committed: Tue Sep 15 16:08:54 2015 +0100
--
CHANGES.txt | 1 +
.../hadoop/cql3/CqlBulkOutputFormat.java| 93 +++-
.../hadoop/cql3/CqlBulkRecordWriter.java| 87
.../cassandra/hadoop/pig/CqlNativeStorage.java | 213 +--
.../apache/cassandra/hadoop/pig/CqlStorage.java | 1 -
.../org/apache/cassandra/tools/BulkLoader.java | 2 +-
test/conf/cassandra.yaml| 1 +
.../org/apache/cassandra/pig/CqlTableTest.java | 36
.../org/apache/cassandra/pig/PigTestBase.java | 3 +-
9 files changed, 336 insertions(+), 101 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/c7b40735/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index dff47fc..5f11049 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
2.1.10
+ * (Pig) support BulkOutputFormat as a URL parameter (CASSANDRA-7410)
* BATCH statement is broken in cqlsh (CASSANDRA-10272)
* Added configurable warning threshold for GC duration (CASSANDRA-8907)
* (cqlsh) Make cqlsh PEP8 compliant (CASSANDRA-10066)
http://git-wip-us.apache.org/repos/asf/cassandra/blob/c7b40735/src/java/org/apache/cassandra/hadoop/cql3/CqlBulkOutputFormat.java
--
diff --git a/src/java/org/apache/cassandra/hadoop/cql3/CqlBulkOutputFormat.java
b/src/java/org/apache/cassandra/hadoop/cql3/CqlBulkOutputFormat.java
index 887fe8e..7fedb41 100644
--- a/src/java/org/apache/cassandra/hadoop/cql3/CqlBulkOutputFormat.java
+++ b/src/java/org/apache/cassandra/hadoop/cql3/CqlBulkOutputFormat.java
@@ -22,6 +22,7 @@ import java.io.IOException;
import java.nio.ByteBuffer;
import java.util.List;
+import org.apache.cassandra.config.EncryptionOptions;
import org.apache.cassandra.hadoop.AbstractBulkOutputFormat;
import org.apache.cassandra.hadoop.ConfigHelper;
import org.apache.hadoop.conf.Configuration;
@@ -54,6 +55,16 @@ public class CqlBulkOutputFormat extends
AbstractBulkOutputFormathttp://git-wip-us.apache.org/repos/asf/cassandra/blob/c7b40735/src/java/org/apache/cassandra/hadoop/cql3/CqlBulkRecordWriter.java
--
diff --git a/src/java/org/apache/cassandra/hadoop/cql3/CqlBulkRecordWriter.java
b/src/java/org/apache/cassandra/hadoop/cql3/CqlBulkRecordWriter.java
index e60a240..ced8aa9 100644
--- a/src/java/org/apache/cassandra/hadoop/cql3/CqlBulkRecordWriter.java
+++ b/src/java/org/apache/cassandra/hadoop/cql3/CqlBulkRecordWriter.java
@@ -19,13 +19,16 @@ package org.apache.cassandra.hadoop.cql3;
import java.io.File;
import java.io.IOException;
+import java.net.InetAddress;
+import java.net.UnknownHostException;
import java.nio.ByteBuffer;
-import java.util.HashMap;
+import java.util.HashSet;
import java.util.List;
-import java.util.Map;
+import java.util.Set;
import java.util.UUID;
-import org.apache.cassandra.config.CFMetaData;
+import org.apache.cassandra.config.EncryptionOptions;
+import org.apache.cassandra.config.EncryptionOptions.ServerEncryptionOptions;
import org.apache.cassandra.exceptions.InvalidRequestException;
import org.apache.cassandra.hadoop.AbstractBulkRecordWriter;
import org.apache.cassandra.hadoop.BulkRecordWriter;
@@ -35,6 +38,9 @@ import org.apache.cassandra.io.sstable.CQLSSTableWriter;
import org.apache.cassandra.io.sstable.SSTableLoader;
import org.apache.cassandra.io.util.FileUtils;
import org.apache.cassandra.streaming.StreamState;
+import org.apache.cassandra.thrift.ITransportFactory;
+import org.apache.cassandra.tools.BulkLoader;
+import org.apache.commons.lang.StringUtils;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.mapreduce.TaskAttemptContext;
import org.apache.hadoop.util.Progressable;
@@ -108,10 +114,7 @@ public class CqlBulkRecordWriter extends
AbstractBulkRecordWriter> knownCqlCfs = new
HashMap<>();
-
-public ExternalClient(Configuration conf)
-{
-super(conf);
-}
-public void addKnownCfs(String keyspace, String cql)
[2/2] cassandra git commit: Merge branch 'cassandra-2.1' into cassandra-2.2
Merge branch 'cassandra-2.1' into cassandra-2.2 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/cd4a1e6a Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/cd4a1e6a Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/cd4a1e6a Branch: refs/heads/cassandra-2.2 Commit: cd4a1e6acc51b0f708127a50d37dd7832bbe8dfa Parents: 9791796 c7b4073 Author: Aleksey Yeschenko Authored: Tue Sep 15 16:12:45 2015 +0100 Committer: Aleksey Yeschenko Committed: Tue Sep 15 16:12:45 2015 +0100 -- CHANGES.txt | 1 + .../hadoop/AbstractColumnFamilyInputFormat.java | 2 + .../hadoop/cql3/CqlBulkRecordWriter.java| 13 +- .../cassandra/hadoop/pig/CqlNativeStorage.java | 171 --- test/conf/cassandra_pig.yaml| 41 + .../org/apache/cassandra/pig/CqlTableTest.java | 35 .../org/apache/cassandra/pig/PigTestBase.java | 3 +- 7 files changed, 205 insertions(+), 61 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/cd4a1e6a/CHANGES.txt -- diff --cc CHANGES.txt index bd24781,5f11049..b0ade42 --- a/CHANGES.txt +++ b/CHANGES.txt @@@ -1,17 -1,8 +1,18 @@@ -2.1.10 +2.2.2 + * Defer default role manager setup until all nodes are on 2.2+ (CASSANDRA-9761) + * Cancel transaction for sstables we wont redistribute index summary + for (CASSANDRA-10270) + * Handle missing RoleManager in config after upgrade to 2.2 (CASSANDRA-10209) + * Retry snapshot deletion after compaction and gc on Windows (CASSANDRA-10222) + * Fix failure to start with space in directory path on Windows (CASSANDRA-10239) + * Fix repair hang when snapshot failed (CASSANDRA-10057) + * Fall back to 1/4 commitlog volume for commitlog_total_space on small disks + (CASSANDRA-10199) +Merged from 2.1: + * (Pig) support BulkOutputFormat as a URL parameter (CASSANDRA-7410) * BATCH statement is broken in cqlsh (CASSANDRA-10272) * Added configurable warning threshold for GC duration (CASSANDRA-8907) - * (cqlsh) Make cqlsh PEP8 compliant (CASSANDRA-10066) + * (cqlsh) Make cqlsh PEP8 Compliant (CASSANDRA-10066) * (cqlsh) Fix error when starting cqlsh with --debug (CASSANDRA-10282) * Scrub, Cleanup and Upgrade do not unmark compacting until all operations have completed, regardless of the occurence of exceptions (CASSANDRA-10274) http://git-wip-us.apache.org/repos/asf/cassandra/blob/cd4a1e6a/src/java/org/apache/cassandra/hadoop/AbstractColumnFamilyInputFormat.java -- diff --cc src/java/org/apache/cassandra/hadoop/AbstractColumnFamilyInputFormat.java index 4dd53ff,e8de0f2..9c45bfe --- a/src/java/org/apache/cassandra/hadoop/AbstractColumnFamilyInputFormat.java +++ b/src/java/org/apache/cassandra/hadoop/AbstractColumnFamilyInputFormat.java @@@ -24,23 -32,31 +24,24 @@@ import java.util.concurrent.* import org.slf4j.Logger; import org.slf4j.LoggerFactory; -import org.apache.cassandra.auth.IAuthenticator; +import com.datastax.driver.core.Host; +import com.datastax.driver.core.Metadata; +import com.datastax.driver.core.ResultSet; +import com.datastax.driver.core.Row; +import com.datastax.driver.core.Session; +import com.datastax.driver.core.TokenRange; ++ +import org.apache.cassandra.db.SystemKeyspace; +import org.apache.cassandra.dht.ByteOrderedPartitioner; import org.apache.cassandra.dht.IPartitioner; +import org.apache.cassandra.dht.OrderPreservingPartitioner; import org.apache.cassandra.dht.Range; import org.apache.cassandra.dht.Token; -import org.apache.cassandra.thrift.AuthenticationRequest; -import org.apache.cassandra.thrift.Cassandra; -import org.apache.cassandra.thrift.CfSplit; -import org.apache.cassandra.thrift.InvalidRequestException; +import org.apache.cassandra.hadoop.cql3.*; import org.apache.cassandra.thrift.KeyRange; -import org.apache.cassandra.thrift.TokenRange; -import org.apache.commons.lang3.StringUtils; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.mapred.JobConf; -import org.apache.hadoop.mapreduce.InputFormat; -import org.apache.hadoop.mapreduce.InputSplit; -import org.apache.hadoop.mapreduce.JobContext; -import org.apache.hadoop.mapreduce.TaskAttemptContext; -import org.apache.hadoop.mapreduce.TaskAttemptID; -import org.apache.thrift.TApplicationException; -import org.apache.thrift.TException; -import org.apache.thrift.protocol.TBinaryProtocol; -import org.apache.thrift.protocol.TProtocol; -import org.apache.thrift.transport.TTransport; -import org.apache.thrift.transport.TTransportException; - +import org.apache.hadoop.mapreduce.*; public abstract class AbstractColumnFamilyInp
[1/3] cassandra git commit: (Pig) support BulkOutputFormat as a URL parameter
Repository: cassandra
Updated Branches:
refs/heads/cassandra-3.0 7f5580b22 -> 7ca4db0a1
(Pig) support BulkOutputFormat as a URL parameter
patch by Alex Liu; reviewed by Piotr KoÅaczkowski for CASSANDRA-7410
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/c7b40735
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/c7b40735
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/c7b40735
Branch: refs/heads/cassandra-3.0
Commit: c7b40735789c840529002eb3c11d8731f460d61c
Parents: ae51086
Author: Alex Liu
Authored: Tue Sep 15 16:06:18 2015 +0100
Committer: Aleksey Yeschenko
Committed: Tue Sep 15 16:08:54 2015 +0100
--
CHANGES.txt | 1 +
.../hadoop/cql3/CqlBulkOutputFormat.java| 93 +++-
.../hadoop/cql3/CqlBulkRecordWriter.java| 87
.../cassandra/hadoop/pig/CqlNativeStorage.java | 213 +--
.../apache/cassandra/hadoop/pig/CqlStorage.java | 1 -
.../org/apache/cassandra/tools/BulkLoader.java | 2 +-
test/conf/cassandra.yaml| 1 +
.../org/apache/cassandra/pig/CqlTableTest.java | 36
.../org/apache/cassandra/pig/PigTestBase.java | 3 +-
9 files changed, 336 insertions(+), 101 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/c7b40735/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index dff47fc..5f11049 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
2.1.10
+ * (Pig) support BulkOutputFormat as a URL parameter (CASSANDRA-7410)
* BATCH statement is broken in cqlsh (CASSANDRA-10272)
* Added configurable warning threshold for GC duration (CASSANDRA-8907)
* (cqlsh) Make cqlsh PEP8 compliant (CASSANDRA-10066)
http://git-wip-us.apache.org/repos/asf/cassandra/blob/c7b40735/src/java/org/apache/cassandra/hadoop/cql3/CqlBulkOutputFormat.java
--
diff --git a/src/java/org/apache/cassandra/hadoop/cql3/CqlBulkOutputFormat.java
b/src/java/org/apache/cassandra/hadoop/cql3/CqlBulkOutputFormat.java
index 887fe8e..7fedb41 100644
--- a/src/java/org/apache/cassandra/hadoop/cql3/CqlBulkOutputFormat.java
+++ b/src/java/org/apache/cassandra/hadoop/cql3/CqlBulkOutputFormat.java
@@ -22,6 +22,7 @@ import java.io.IOException;
import java.nio.ByteBuffer;
import java.util.List;
+import org.apache.cassandra.config.EncryptionOptions;
import org.apache.cassandra.hadoop.AbstractBulkOutputFormat;
import org.apache.cassandra.hadoop.ConfigHelper;
import org.apache.hadoop.conf.Configuration;
@@ -54,6 +55,16 @@ public class CqlBulkOutputFormat extends
AbstractBulkOutputFormathttp://git-wip-us.apache.org/repos/asf/cassandra/blob/c7b40735/src/java/org/apache/cassandra/hadoop/cql3/CqlBulkRecordWriter.java
--
diff --git a/src/java/org/apache/cassandra/hadoop/cql3/CqlBulkRecordWriter.java
b/src/java/org/apache/cassandra/hadoop/cql3/CqlBulkRecordWriter.java
index e60a240..ced8aa9 100644
--- a/src/java/org/apache/cassandra/hadoop/cql3/CqlBulkRecordWriter.java
+++ b/src/java/org/apache/cassandra/hadoop/cql3/CqlBulkRecordWriter.java
@@ -19,13 +19,16 @@ package org.apache.cassandra.hadoop.cql3;
import java.io.File;
import java.io.IOException;
+import java.net.InetAddress;
+import java.net.UnknownHostException;
import java.nio.ByteBuffer;
-import java.util.HashMap;
+import java.util.HashSet;
import java.util.List;
-import java.util.Map;
+import java.util.Set;
import java.util.UUID;
-import org.apache.cassandra.config.CFMetaData;
+import org.apache.cassandra.config.EncryptionOptions;
+import org.apache.cassandra.config.EncryptionOptions.ServerEncryptionOptions;
import org.apache.cassandra.exceptions.InvalidRequestException;
import org.apache.cassandra.hadoop.AbstractBulkRecordWriter;
import org.apache.cassandra.hadoop.BulkRecordWriter;
@@ -35,6 +38,9 @@ import org.apache.cassandra.io.sstable.CQLSSTableWriter;
import org.apache.cassandra.io.sstable.SSTableLoader;
import org.apache.cassandra.io.util.FileUtils;
import org.apache.cassandra.streaming.StreamState;
+import org.apache.cassandra.thrift.ITransportFactory;
+import org.apache.cassandra.tools.BulkLoader;
+import org.apache.commons.lang.StringUtils;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.mapreduce.TaskAttemptContext;
import org.apache.hadoop.util.Progressable;
@@ -108,10 +114,7 @@ public class CqlBulkRecordWriter extends
AbstractBulkRecordWriter> knownCqlCfs = new
HashMap<>();
-
-public ExternalClient(Configuration conf)
-{
-super(conf);
-}
-public void addKnownCfs(String keyspace, String cql)
[3/3] cassandra git commit: Merge branch 'cassandra-2.2' into cassandra-3.0
Merge branch 'cassandra-2.2' into cassandra-3.0 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/7ca4db0a Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/7ca4db0a Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/7ca4db0a Branch: refs/heads/cassandra-3.0 Commit: 7ca4db0a117b02457f73ebdbb10e6919f1535245 Parents: 7f5580b cd4a1e6 Author: Aleksey Yeschenko Authored: Tue Sep 15 16:16:33 2015 +0100 Committer: Aleksey Yeschenko Committed: Tue Sep 15 16:16:33 2015 +0100 -- CHANGES.txt | 1 + .../hadoop/cql3/CqlBulkRecordWriter.java| 13 +- .../cassandra/hadoop/pig/CqlNativeStorage.java | 171 --- test/conf/cassandra_pig.yaml| 41 + .../org/apache/cassandra/pig/CqlTableTest.java | 35 .../org/apache/cassandra/pig/PigTestBase.java | 3 +- 6 files changed, 203 insertions(+), 61 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/7ca4db0a/CHANGES.txt -- diff --cc CHANGES.txt index cf982dd,b0ade42..54d4ddf --- a/CHANGES.txt +++ b/CHANGES.txt @@@ -1,45 -1,8 +1,46 @@@ -2.2.2 +3.0.0-rc1 + * Give index implementations more control over rebuild operations (CASSANDRA-10312) + * Update index file format (CASSANDRA-10314) + * Add "shadowable" row tombstones to deal with mv timestamp issues (CASSANDRA-10261) + * CFS.loadNewSSTables() broken for pre-3.0 sstables + * Cache selected index in read command to reduce lookups (CASSANDRA-10215) + * Small optimizations of sstable index serialization (CASSANDRA-10232) + * Support for both encrypted and unencrypted native transport connections (CASSANDRA-9590) +Merged from 2.2: * Defer default role manager setup until all nodes are on 2.2+ (CASSANDRA-9761) + * Handle missing RoleManager in config after upgrade to 2.2 (CASSANDRA-10209) +Merged from 2.1: ++ * (Pig) support BulkOutputFormat as a URL parameter (CASSANDRA-7410) + * BATCH statement is broken in cqlsh (CASSANDRA-10272) + * (cqlsh) Make cqlsh PEP8 Compliant (CASSANDRA-10066) + * (cqlsh) Fix error when starting cqlsh with --debug (CASSANDRA-10282) + * Scrub, Cleanup and Upgrade do not unmark compacting until all operations + have completed, regardless of the occurence of exceptions (CASSANDRA-10274) + + +3.0.0-beta2 + * Fix columns returned by AbstractBtreePartitions (CASSANDRA-10220) + * Fix backward compatibility issue due to AbstractBounds serialization bug (CASSANDRA-9857) + * Fix startup error when upgrading nodes (CASSANDRA-10136) + * Base table PRIMARY KEY can be assumed to be NOT NULL in MV creation (CASSANDRA-10147) + * Improve batchlog write patch (CASSANDRA-9673) + * Re-apply MaterializedView updates on commitlog replay (CASSANDRA-10164) + * Require AbstractType.isByteOrderComparable declaration in constructor (CASSANDRA-9901) + * Avoid digest mismatch on upgrade to 3.0 (CASSANDRA-9554) + * Fix Materialized View builder when adding multiple MVs (CASSANDRA-10156) + * Choose better poolingOptions for protocol v4 in cassandra-stress (CASSANDRA-10182) + * Fix LWW bug affecting Materialized Views (CASSANDRA-10197) + * Ensures frozen sets and maps are always sorted (CASSANDRA-10162) + * Don't deadlock when flushing CFS backed custom indexes (CASSANDRA-10181) + * Fix double flushing of secondary index tables (CASSANDRA-10180) + * Fix incorrect handling of range tombstones in thrift (CASSANDRA-10046) + * Only use batchlog when paired materialized view replica is remote (CASSANDRA-10061) + * Reuse TemporalRow when updating multiple MaterializedViews (CASSANDRA-10060) + * Validate gc_grace_seconds for batchlog writes and MVs (CASSANDRA-9917) + * Fix sstablerepairedset (CASSANDRA-10132) +Merged from 2.2: * Cancel transaction for sstables we wont redistribute index summary for (CASSANDRA-10270) - * Handle missing RoleManager in config after upgrade to 2.2 (CASSANDRA-10209) * Retry snapshot deletion after compaction and gc on Windows (CASSANDRA-10222) * Fix failure to start with space in directory path on Windows (CASSANDRA-10239) * Fix repair hang when snapshot failed (CASSANDRA-10057) http://git-wip-us.apache.org/repos/asf/cassandra/blob/7ca4db0a/src/java/org/apache/cassandra/hadoop/cql3/CqlBulkRecordWriter.java -- diff --cc src/java/org/apache/cassandra/hadoop/cql3/CqlBulkRecordWriter.java index 3e69c2d,9e6e23b..4552d87 --- a/src/java/org/apache/cassandra/hadoop/cql3/CqlBulkRecordWriter.java +++ b/src/java/org/apache/cassandra/hadoop/cql3/CqlBulkRecordWriter.java @@@ -31,8 -31,10 +31,9 @@@ import org.slf4j.LoggerFactory import org.apache.cassandra.config.CFMetaD
[1/4] cassandra git commit: (Pig) support BulkOutputFormat as a URL parameter
Repository: cassandra
Updated Branches:
refs/heads/trunk ef54c6c72 -> 1944e402c
(Pig) support BulkOutputFormat as a URL parameter
patch by Alex Liu; reviewed by Piotr KoÅaczkowski for CASSANDRA-7410
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/c7b40735
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/c7b40735
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/c7b40735
Branch: refs/heads/trunk
Commit: c7b40735789c840529002eb3c11d8731f460d61c
Parents: ae51086
Author: Alex Liu
Authored: Tue Sep 15 16:06:18 2015 +0100
Committer: Aleksey Yeschenko
Committed: Tue Sep 15 16:08:54 2015 +0100
--
CHANGES.txt | 1 +
.../hadoop/cql3/CqlBulkOutputFormat.java| 93 +++-
.../hadoop/cql3/CqlBulkRecordWriter.java| 87
.../cassandra/hadoop/pig/CqlNativeStorage.java | 213 +--
.../apache/cassandra/hadoop/pig/CqlStorage.java | 1 -
.../org/apache/cassandra/tools/BulkLoader.java | 2 +-
test/conf/cassandra.yaml| 1 +
.../org/apache/cassandra/pig/CqlTableTest.java | 36
.../org/apache/cassandra/pig/PigTestBase.java | 3 +-
9 files changed, 336 insertions(+), 101 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/c7b40735/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index dff47fc..5f11049 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
2.1.10
+ * (Pig) support BulkOutputFormat as a URL parameter (CASSANDRA-7410)
* BATCH statement is broken in cqlsh (CASSANDRA-10272)
* Added configurable warning threshold for GC duration (CASSANDRA-8907)
* (cqlsh) Make cqlsh PEP8 compliant (CASSANDRA-10066)
http://git-wip-us.apache.org/repos/asf/cassandra/blob/c7b40735/src/java/org/apache/cassandra/hadoop/cql3/CqlBulkOutputFormat.java
--
diff --git a/src/java/org/apache/cassandra/hadoop/cql3/CqlBulkOutputFormat.java
b/src/java/org/apache/cassandra/hadoop/cql3/CqlBulkOutputFormat.java
index 887fe8e..7fedb41 100644
--- a/src/java/org/apache/cassandra/hadoop/cql3/CqlBulkOutputFormat.java
+++ b/src/java/org/apache/cassandra/hadoop/cql3/CqlBulkOutputFormat.java
@@ -22,6 +22,7 @@ import java.io.IOException;
import java.nio.ByteBuffer;
import java.util.List;
+import org.apache.cassandra.config.EncryptionOptions;
import org.apache.cassandra.hadoop.AbstractBulkOutputFormat;
import org.apache.cassandra.hadoop.ConfigHelper;
import org.apache.hadoop.conf.Configuration;
@@ -54,6 +55,16 @@ public class CqlBulkOutputFormat extends
AbstractBulkOutputFormathttp://git-wip-us.apache.org/repos/asf/cassandra/blob/c7b40735/src/java/org/apache/cassandra/hadoop/cql3/CqlBulkRecordWriter.java
--
diff --git a/src/java/org/apache/cassandra/hadoop/cql3/CqlBulkRecordWriter.java
b/src/java/org/apache/cassandra/hadoop/cql3/CqlBulkRecordWriter.java
index e60a240..ced8aa9 100644
--- a/src/java/org/apache/cassandra/hadoop/cql3/CqlBulkRecordWriter.java
+++ b/src/java/org/apache/cassandra/hadoop/cql3/CqlBulkRecordWriter.java
@@ -19,13 +19,16 @@ package org.apache.cassandra.hadoop.cql3;
import java.io.File;
import java.io.IOException;
+import java.net.InetAddress;
+import java.net.UnknownHostException;
import java.nio.ByteBuffer;
-import java.util.HashMap;
+import java.util.HashSet;
import java.util.List;
-import java.util.Map;
+import java.util.Set;
import java.util.UUID;
-import org.apache.cassandra.config.CFMetaData;
+import org.apache.cassandra.config.EncryptionOptions;
+import org.apache.cassandra.config.EncryptionOptions.ServerEncryptionOptions;
import org.apache.cassandra.exceptions.InvalidRequestException;
import org.apache.cassandra.hadoop.AbstractBulkRecordWriter;
import org.apache.cassandra.hadoop.BulkRecordWriter;
@@ -35,6 +38,9 @@ import org.apache.cassandra.io.sstable.CQLSSTableWriter;
import org.apache.cassandra.io.sstable.SSTableLoader;
import org.apache.cassandra.io.util.FileUtils;
import org.apache.cassandra.streaming.StreamState;
+import org.apache.cassandra.thrift.ITransportFactory;
+import org.apache.cassandra.tools.BulkLoader;
+import org.apache.commons.lang.StringUtils;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.mapreduce.TaskAttemptContext;
import org.apache.hadoop.util.Progressable;
@@ -108,10 +114,7 @@ public class CqlBulkRecordWriter extends
AbstractBulkRecordWriter> knownCqlCfs = new
HashMap<>();
-
-public ExternalClient(Configuration conf)
-{
-super(conf);
-}
-public void addKnownCfs(String keyspace, String cql)
+private Bu
[3/4] cassandra git commit: Merge branch 'cassandra-2.2' into cassandra-3.0
Merge branch 'cassandra-2.2' into cassandra-3.0 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/7ca4db0a Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/7ca4db0a Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/7ca4db0a Branch: refs/heads/trunk Commit: 7ca4db0a117b02457f73ebdbb10e6919f1535245 Parents: 7f5580b cd4a1e6 Author: Aleksey Yeschenko Authored: Tue Sep 15 16:16:33 2015 +0100 Committer: Aleksey Yeschenko Committed: Tue Sep 15 16:16:33 2015 +0100 -- CHANGES.txt | 1 + .../hadoop/cql3/CqlBulkRecordWriter.java| 13 +- .../cassandra/hadoop/pig/CqlNativeStorage.java | 171 --- test/conf/cassandra_pig.yaml| 41 + .../org/apache/cassandra/pig/CqlTableTest.java | 35 .../org/apache/cassandra/pig/PigTestBase.java | 3 +- 6 files changed, 203 insertions(+), 61 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/7ca4db0a/CHANGES.txt -- diff --cc CHANGES.txt index cf982dd,b0ade42..54d4ddf --- a/CHANGES.txt +++ b/CHANGES.txt @@@ -1,45 -1,8 +1,46 @@@ -2.2.2 +3.0.0-rc1 + * Give index implementations more control over rebuild operations (CASSANDRA-10312) + * Update index file format (CASSANDRA-10314) + * Add "shadowable" row tombstones to deal with mv timestamp issues (CASSANDRA-10261) + * CFS.loadNewSSTables() broken for pre-3.0 sstables + * Cache selected index in read command to reduce lookups (CASSANDRA-10215) + * Small optimizations of sstable index serialization (CASSANDRA-10232) + * Support for both encrypted and unencrypted native transport connections (CASSANDRA-9590) +Merged from 2.2: * Defer default role manager setup until all nodes are on 2.2+ (CASSANDRA-9761) + * Handle missing RoleManager in config after upgrade to 2.2 (CASSANDRA-10209) +Merged from 2.1: ++ * (Pig) support BulkOutputFormat as a URL parameter (CASSANDRA-7410) + * BATCH statement is broken in cqlsh (CASSANDRA-10272) + * (cqlsh) Make cqlsh PEP8 Compliant (CASSANDRA-10066) + * (cqlsh) Fix error when starting cqlsh with --debug (CASSANDRA-10282) + * Scrub, Cleanup and Upgrade do not unmark compacting until all operations + have completed, regardless of the occurence of exceptions (CASSANDRA-10274) + + +3.0.0-beta2 + * Fix columns returned by AbstractBtreePartitions (CASSANDRA-10220) + * Fix backward compatibility issue due to AbstractBounds serialization bug (CASSANDRA-9857) + * Fix startup error when upgrading nodes (CASSANDRA-10136) + * Base table PRIMARY KEY can be assumed to be NOT NULL in MV creation (CASSANDRA-10147) + * Improve batchlog write patch (CASSANDRA-9673) + * Re-apply MaterializedView updates on commitlog replay (CASSANDRA-10164) + * Require AbstractType.isByteOrderComparable declaration in constructor (CASSANDRA-9901) + * Avoid digest mismatch on upgrade to 3.0 (CASSANDRA-9554) + * Fix Materialized View builder when adding multiple MVs (CASSANDRA-10156) + * Choose better poolingOptions for protocol v4 in cassandra-stress (CASSANDRA-10182) + * Fix LWW bug affecting Materialized Views (CASSANDRA-10197) + * Ensures frozen sets and maps are always sorted (CASSANDRA-10162) + * Don't deadlock when flushing CFS backed custom indexes (CASSANDRA-10181) + * Fix double flushing of secondary index tables (CASSANDRA-10180) + * Fix incorrect handling of range tombstones in thrift (CASSANDRA-10046) + * Only use batchlog when paired materialized view replica is remote (CASSANDRA-10061) + * Reuse TemporalRow when updating multiple MaterializedViews (CASSANDRA-10060) + * Validate gc_grace_seconds for batchlog writes and MVs (CASSANDRA-9917) + * Fix sstablerepairedset (CASSANDRA-10132) +Merged from 2.2: * Cancel transaction for sstables we wont redistribute index summary for (CASSANDRA-10270) - * Handle missing RoleManager in config after upgrade to 2.2 (CASSANDRA-10209) * Retry snapshot deletion after compaction and gc on Windows (CASSANDRA-10222) * Fix failure to start with space in directory path on Windows (CASSANDRA-10239) * Fix repair hang when snapshot failed (CASSANDRA-10057) http://git-wip-us.apache.org/repos/asf/cassandra/blob/7ca4db0a/src/java/org/apache/cassandra/hadoop/cql3/CqlBulkRecordWriter.java -- diff --cc src/java/org/apache/cassandra/hadoop/cql3/CqlBulkRecordWriter.java index 3e69c2d,9e6e23b..4552d87 --- a/src/java/org/apache/cassandra/hadoop/cql3/CqlBulkRecordWriter.java +++ b/src/java/org/apache/cassandra/hadoop/cql3/CqlBulkRecordWriter.java @@@ -31,8 -31,10 +31,9 @@@ import org.slf4j.LoggerFactory import org.apache.cassandra.config.CFMetaData; i
[2/4] cassandra git commit: Merge branch 'cassandra-2.1' into cassandra-2.2
Merge branch 'cassandra-2.1' into cassandra-2.2 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/cd4a1e6a Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/cd4a1e6a Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/cd4a1e6a Branch: refs/heads/trunk Commit: cd4a1e6acc51b0f708127a50d37dd7832bbe8dfa Parents: 9791796 c7b4073 Author: Aleksey Yeschenko Authored: Tue Sep 15 16:12:45 2015 +0100 Committer: Aleksey Yeschenko Committed: Tue Sep 15 16:12:45 2015 +0100 -- CHANGES.txt | 1 + .../hadoop/AbstractColumnFamilyInputFormat.java | 2 + .../hadoop/cql3/CqlBulkRecordWriter.java| 13 +- .../cassandra/hadoop/pig/CqlNativeStorage.java | 171 --- test/conf/cassandra_pig.yaml| 41 + .../org/apache/cassandra/pig/CqlTableTest.java | 35 .../org/apache/cassandra/pig/PigTestBase.java | 3 +- 7 files changed, 205 insertions(+), 61 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/cd4a1e6a/CHANGES.txt -- diff --cc CHANGES.txt index bd24781,5f11049..b0ade42 --- a/CHANGES.txt +++ b/CHANGES.txt @@@ -1,17 -1,8 +1,18 @@@ -2.1.10 +2.2.2 + * Defer default role manager setup until all nodes are on 2.2+ (CASSANDRA-9761) + * Cancel transaction for sstables we wont redistribute index summary + for (CASSANDRA-10270) + * Handle missing RoleManager in config after upgrade to 2.2 (CASSANDRA-10209) + * Retry snapshot deletion after compaction and gc on Windows (CASSANDRA-10222) + * Fix failure to start with space in directory path on Windows (CASSANDRA-10239) + * Fix repair hang when snapshot failed (CASSANDRA-10057) + * Fall back to 1/4 commitlog volume for commitlog_total_space on small disks + (CASSANDRA-10199) +Merged from 2.1: + * (Pig) support BulkOutputFormat as a URL parameter (CASSANDRA-7410) * BATCH statement is broken in cqlsh (CASSANDRA-10272) * Added configurable warning threshold for GC duration (CASSANDRA-8907) - * (cqlsh) Make cqlsh PEP8 compliant (CASSANDRA-10066) + * (cqlsh) Make cqlsh PEP8 Compliant (CASSANDRA-10066) * (cqlsh) Fix error when starting cqlsh with --debug (CASSANDRA-10282) * Scrub, Cleanup and Upgrade do not unmark compacting until all operations have completed, regardless of the occurence of exceptions (CASSANDRA-10274) http://git-wip-us.apache.org/repos/asf/cassandra/blob/cd4a1e6a/src/java/org/apache/cassandra/hadoop/AbstractColumnFamilyInputFormat.java -- diff --cc src/java/org/apache/cassandra/hadoop/AbstractColumnFamilyInputFormat.java index 4dd53ff,e8de0f2..9c45bfe --- a/src/java/org/apache/cassandra/hadoop/AbstractColumnFamilyInputFormat.java +++ b/src/java/org/apache/cassandra/hadoop/AbstractColumnFamilyInputFormat.java @@@ -24,23 -32,31 +24,24 @@@ import java.util.concurrent.* import org.slf4j.Logger; import org.slf4j.LoggerFactory; -import org.apache.cassandra.auth.IAuthenticator; +import com.datastax.driver.core.Host; +import com.datastax.driver.core.Metadata; +import com.datastax.driver.core.ResultSet; +import com.datastax.driver.core.Row; +import com.datastax.driver.core.Session; +import com.datastax.driver.core.TokenRange; ++ +import org.apache.cassandra.db.SystemKeyspace; +import org.apache.cassandra.dht.ByteOrderedPartitioner; import org.apache.cassandra.dht.IPartitioner; +import org.apache.cassandra.dht.OrderPreservingPartitioner; import org.apache.cassandra.dht.Range; import org.apache.cassandra.dht.Token; -import org.apache.cassandra.thrift.AuthenticationRequest; -import org.apache.cassandra.thrift.Cassandra; -import org.apache.cassandra.thrift.CfSplit; -import org.apache.cassandra.thrift.InvalidRequestException; +import org.apache.cassandra.hadoop.cql3.*; import org.apache.cassandra.thrift.KeyRange; -import org.apache.cassandra.thrift.TokenRange; -import org.apache.commons.lang3.StringUtils; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.mapred.JobConf; -import org.apache.hadoop.mapreduce.InputFormat; -import org.apache.hadoop.mapreduce.InputSplit; -import org.apache.hadoop.mapreduce.JobContext; -import org.apache.hadoop.mapreduce.TaskAttemptContext; -import org.apache.hadoop.mapreduce.TaskAttemptID; -import org.apache.thrift.TApplicationException; -import org.apache.thrift.TException; -import org.apache.thrift.protocol.TBinaryProtocol; -import org.apache.thrift.protocol.TProtocol; -import org.apache.thrift.transport.TTransport; -import org.apache.thrift.transport.TTransportException; - +import org.apache.hadoop.mapreduce.*; public abstract class AbstractColumnFamilyInputFormat
[2/3] cassandra git commit: Merge branch 'cassandra-2.1' into cassandra-2.2
Merge branch 'cassandra-2.1' into cassandra-2.2 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/cd4a1e6a Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/cd4a1e6a Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/cd4a1e6a Branch: refs/heads/cassandra-3.0 Commit: cd4a1e6acc51b0f708127a50d37dd7832bbe8dfa Parents: 9791796 c7b4073 Author: Aleksey Yeschenko Authored: Tue Sep 15 16:12:45 2015 +0100 Committer: Aleksey Yeschenko Committed: Tue Sep 15 16:12:45 2015 +0100 -- CHANGES.txt | 1 + .../hadoop/AbstractColumnFamilyInputFormat.java | 2 + .../hadoop/cql3/CqlBulkRecordWriter.java| 13 +- .../cassandra/hadoop/pig/CqlNativeStorage.java | 171 --- test/conf/cassandra_pig.yaml| 41 + .../org/apache/cassandra/pig/CqlTableTest.java | 35 .../org/apache/cassandra/pig/PigTestBase.java | 3 +- 7 files changed, 205 insertions(+), 61 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/cd4a1e6a/CHANGES.txt -- diff --cc CHANGES.txt index bd24781,5f11049..b0ade42 --- a/CHANGES.txt +++ b/CHANGES.txt @@@ -1,17 -1,8 +1,18 @@@ -2.1.10 +2.2.2 + * Defer default role manager setup until all nodes are on 2.2+ (CASSANDRA-9761) + * Cancel transaction for sstables we wont redistribute index summary + for (CASSANDRA-10270) + * Handle missing RoleManager in config after upgrade to 2.2 (CASSANDRA-10209) + * Retry snapshot deletion after compaction and gc on Windows (CASSANDRA-10222) + * Fix failure to start with space in directory path on Windows (CASSANDRA-10239) + * Fix repair hang when snapshot failed (CASSANDRA-10057) + * Fall back to 1/4 commitlog volume for commitlog_total_space on small disks + (CASSANDRA-10199) +Merged from 2.1: + * (Pig) support BulkOutputFormat as a URL parameter (CASSANDRA-7410) * BATCH statement is broken in cqlsh (CASSANDRA-10272) * Added configurable warning threshold for GC duration (CASSANDRA-8907) - * (cqlsh) Make cqlsh PEP8 compliant (CASSANDRA-10066) + * (cqlsh) Make cqlsh PEP8 Compliant (CASSANDRA-10066) * (cqlsh) Fix error when starting cqlsh with --debug (CASSANDRA-10282) * Scrub, Cleanup and Upgrade do not unmark compacting until all operations have completed, regardless of the occurence of exceptions (CASSANDRA-10274) http://git-wip-us.apache.org/repos/asf/cassandra/blob/cd4a1e6a/src/java/org/apache/cassandra/hadoop/AbstractColumnFamilyInputFormat.java -- diff --cc src/java/org/apache/cassandra/hadoop/AbstractColumnFamilyInputFormat.java index 4dd53ff,e8de0f2..9c45bfe --- a/src/java/org/apache/cassandra/hadoop/AbstractColumnFamilyInputFormat.java +++ b/src/java/org/apache/cassandra/hadoop/AbstractColumnFamilyInputFormat.java @@@ -24,23 -32,31 +24,24 @@@ import java.util.concurrent.* import org.slf4j.Logger; import org.slf4j.LoggerFactory; -import org.apache.cassandra.auth.IAuthenticator; +import com.datastax.driver.core.Host; +import com.datastax.driver.core.Metadata; +import com.datastax.driver.core.ResultSet; +import com.datastax.driver.core.Row; +import com.datastax.driver.core.Session; +import com.datastax.driver.core.TokenRange; ++ +import org.apache.cassandra.db.SystemKeyspace; +import org.apache.cassandra.dht.ByteOrderedPartitioner; import org.apache.cassandra.dht.IPartitioner; +import org.apache.cassandra.dht.OrderPreservingPartitioner; import org.apache.cassandra.dht.Range; import org.apache.cassandra.dht.Token; -import org.apache.cassandra.thrift.AuthenticationRequest; -import org.apache.cassandra.thrift.Cassandra; -import org.apache.cassandra.thrift.CfSplit; -import org.apache.cassandra.thrift.InvalidRequestException; +import org.apache.cassandra.hadoop.cql3.*; import org.apache.cassandra.thrift.KeyRange; -import org.apache.cassandra.thrift.TokenRange; -import org.apache.commons.lang3.StringUtils; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.mapred.JobConf; -import org.apache.hadoop.mapreduce.InputFormat; -import org.apache.hadoop.mapreduce.InputSplit; -import org.apache.hadoop.mapreduce.JobContext; -import org.apache.hadoop.mapreduce.TaskAttemptContext; -import org.apache.hadoop.mapreduce.TaskAttemptID; -import org.apache.thrift.TApplicationException; -import org.apache.thrift.TException; -import org.apache.thrift.protocol.TBinaryProtocol; -import org.apache.thrift.protocol.TProtocol; -import org.apache.thrift.transport.TTransport; -import org.apache.thrift.transport.TTransportException; - +import org.apache.hadoop.mapreduce.*; public abstract class AbstractColumnFamilyInp
[4/4] cassandra git commit: Merge branch 'cassandra-3.0' into trunk
Merge branch 'cassandra-3.0' into trunk Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/1944e402 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/1944e402 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/1944e402 Branch: refs/heads/trunk Commit: 1944e402cf4b4f64e8e7ea82ee3df0075916c164 Parents: ef54c6c 7ca4db0 Author: Aleksey Yeschenko Authored: Tue Sep 15 16:16:56 2015 +0100 Committer: Aleksey Yeschenko Committed: Tue Sep 15 16:16:56 2015 +0100 -- CHANGES.txt | 1 + .../hadoop/cql3/CqlBulkRecordWriter.java| 13 +- .../cassandra/hadoop/pig/CqlNativeStorage.java | 171 --- test/conf/cassandra_pig.yaml| 41 + .../org/apache/cassandra/pig/CqlTableTest.java | 35 .../org/apache/cassandra/pig/PigTestBase.java | 3 +- 6 files changed, 203 insertions(+), 61 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/1944e402/CHANGES.txt --
[jira] [Commented] (CASSANDRA-7410) Pig support for BulkOutputFormat as a parameter in url
[ https://issues.apache.org/jira/browse/CASSANDRA-7410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14745593#comment-14745593 ] Aleksey Yeschenko commented on CASSANDRA-7410: -- Committed to cassandra-2.1 as [c7b40735789c840529002eb3c11d8731f460d61c|https://github.com/apache/cassandra/commit/c7b40735789c840529002eb3c11d8731f460d61c] and merged upstream. 3.0 merge went without conflicts, but I did not verify if it works or not. If something is broken there, it's gonna be on Alex and Piotr. > Pig support for BulkOutputFormat as a parameter in url > -- > > Key: CASSANDRA-7410 > URL: https://issues.apache.org/jira/browse/CASSANDRA-7410 > Project: Cassandra > Issue Type: Improvement > Components: Hadoop >Reporter: Alex Liu >Assignee: Alex Liu >Priority: Minor > Fix For: 2.1.x, 2.2.x > > Attachments: 7410-2.0-branch.txt, 7410-2.1-branch.txt, > 7410-v2-2.0-branch.txt, 7410-v3-2.0-branch.txt, CASSANDRA-7410-v1-2.2.txt, > CASSANDRA-7410-v2-2.1-branch.txt, CASSANDRA-7410-v3-2.1-branch.txt, > CASSANDRA-7410-v4-2.0-branch.txt, CASSANDRA-7410-v4-2.1-branch.txt, > CASSANDRA-7410-v5-2.0-branch.txt > > > Add BulkOutputFormat support in Pig url -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-10323) Add more MaterializedView metrics
[ https://issues.apache.org/jira/browse/CASSANDRA-10323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aleksey Yeschenko updated CASSANDRA-10323: -- Fix Version/s: (was: 3.0.0 rc1) 3.0.x > Add more MaterializedView metrics > - > > Key: CASSANDRA-10323 > URL: https://issues.apache.org/jira/browse/CASSANDRA-10323 > Project: Cassandra > Issue Type: Improvement >Reporter: T Jake Luciani >Assignee: T Jake Luciani > Fix For: 3.0.x > > > We need to add more metrics to help understand where time is spent in > materialized view writes. We currently track the ratio of async base -> view > mutations that fail. > We should also add > * The amount of time spent waiting for the partition lock (contention) > * The amount of time spent reading data > Any others? > [~carlyeks] [~jkni] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9748) Can't see other nodes when using multiple network interfaces
[
https://issues.apache.org/jira/browse/CASSANDRA-9748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14745614#comment-14745614
]
Paulo Motta commented on CASSANDRA-9748:
Could you try using the {{GossipingPropertyFileSnitch}} option
{{prefer_local=true}}, if you haven't already, and check if that fixes the
problem?
> Can't see other nodes when using multiple network interfaces
>
>
> Key: CASSANDRA-9748
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9748
> Project: Cassandra
> Issue Type: Bug
> Environment: Cassandra 2.0.16; multi-DC configuration
>Reporter: Roman Bielik
>
> The idea is to setup a multi-DC environment across 2 different networks based
> on the following configuration recommendations:
> http://docs.datastax.com/en/cassandra/2.0/cassandra/configuration/configMultiNetworks.html
> Each node has 2 network interfaces. One used as a private network (DC1:
> 10.0.1.x and DC2: 10.0.2.x). The second one a "public" network where all
> nodes can see each other (this one has a higher latency).
> Using the following settings in cassandra.yaml:
> *seeds:* public IP (same as used in broadcast_address)
> *listen_address:* private IP
> *broadcast_address:* public IP
> *rpc_address:* 0.0.0.0
> *endpoint_snitch:* GossipingPropertyFileSnitch
> _(tried different combinations with no luck)_
> No firewall and no SSL/encryption used.
> The problem is that nodes do not see each other (a gossip problem I guess).
> The nodetool ring/status shows only the local node but not the other ones
> (even from the same DC).
> When I set listen_address to public IP, then everything works fine, but that
> is not the required configuration.
> _Note: Not using EC2 cloud!_
> netstat -anp | grep -E "(7199|9160|9042|7000)"
> tcp0 0 0.0.0.0:71990.0.0.0:*
> LISTEN 3587/java
> tcp0 0 10.0.1.1:9160 0.0.0.0:*
> LISTEN 3587/java
> tcp0 0 10.0.1.1:9042 0.0.0.0:*
> LISTEN 3587/java
> tcp0 0 10.0.1.1:7000 0.0.0.0:*
> LISTEN 3587/java
> tcp0 0 127.0.0.1:7199 127.0.0.1:52874
> ESTABLISHED 3587/java
> tcp0 0 10.0.1.1:7199 10.0.1.1:39650
> ESTABLISHED 3587/java
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (CASSANDRA-10338) Secondary index large values dtest failing on 3.0
[
https://issues.apache.org/jira/browse/CASSANDRA-10338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Benjamin Lerer updated CASSANDRA-10338:
---
Reviewer: Benjamin Lerer
> Secondary index large values dtest failing on 3.0
> -
>
> Key: CASSANDRA-10338
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10338
> Project: Cassandra
> Issue Type: Sub-task
>Reporter: Sam Tunnicliffe
>Assignee: Sam Tunnicliffe
> Fix For: 3.0.0 rc1
>
>
> {{secondary_index_test:TestSecondaryIndex.test_8280_validate_indexed_values}}
> has been failing since build
> [#147|http://cassci.datastax.com/view/cassandra-3.0/job/cassandra-3.0_dtest/147/].
> The most likely cause is CASSANDRA-6237 (though I haven't confirmed that).
> The problem is that if an oversized value is provided, the validation for LWT
> statements (both regular and in batches) is not performed up front, but when
> the {{Cql3CasRequest}} is executed via {{StorageProxy}}. This causes a
> timeout, rather than an immediate validation error & hence the test fails.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (CASSANDRA-10343) Erroneous partition deletion events are delivered to indexers
[
https://issues.apache.org/jira/browse/CASSANDRA-10343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sam Tunnicliffe updated CASSANDRA-10343:
Reviewer: Sylvain Lebresne
> Erroneous partition deletion events are delivered to indexers
> -
>
> Key: CASSANDRA-10343
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10343
> Project: Cassandra
> Issue Type: Bug
>Reporter: Sam Tunnicliffe
>Assignee: Sam Tunnicliffe
> Fix For: 3.0.0 rc1
>
>
> In {{AtomicBTreePartition#addAllWithSizeDelta}} we check
> {{update.deletionInfo().isLive()}} to determine whether the
> {{PartitionDeletion}} should be passed to the supplied {{IndexTransaction}}.
> This is incorrect and it results in the {{PartitionDeletion}} being passed on
> if *either* a partition delete or any range tombstones are present in the
> update.
> We need to change:
> {code}
> if (!update.deletionInfo().isLive())
> {code}
> to
> {code}
> if (!update.deletionInfo().getPartitionDeletion().isLive())
> {code}
> Note that the built-in indexes don't currently do anything with the
> {{deletionTime}} and so aren't affected, but any custom implementation which
> didn't examine it properly might remove data from its index where it shouldn't
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-7918) Provide graphing tool along with cassandra-stress
[ https://issues.apache.org/jira/browse/CASSANDRA-7918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14745621#comment-14745621 ] Joshua McKenzie commented on CASSANDRA-7918: [~enigmacurry]: Is this ticket superseded by CASSANDRA-9870? >From [~shawn.kumar]'s comment over there: bq. I have built off what Ryan had already written out, but changes were quite significant since the code was previously pretty much limited to displaying raw metrics and organized for that purpose I'm fine with reviewing this ticket if you want to rebase it or with us waiting and just doing the entire thing w/9870. > Provide graphing tool along with cassandra-stress > - > > Key: CASSANDRA-7918 > URL: https://issues.apache.org/jira/browse/CASSANDRA-7918 > Project: Cassandra > Issue Type: Improvement > Components: Tools >Reporter: Benedict >Assignee: Ryan McGuire >Priority: Minor > Attachments: 7918.patch, reads.svg > > > Whilst cstar makes some pretty graphs, they're a little limited and also > require you to run your tests through it. It would be useful to be able to > graph results from any stress run easily. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9451) Startup message response for unsupported protocol versions is incorrect
[
https://issues.apache.org/jira/browse/CASSANDRA-9451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14745625#comment-14745625
]
Zachary St Lawrence commented on CASSANDRA-9451:
Sorry was on vacation. When I got back to the office I reinstalled latest and
tested. The problem is in using a newer version of cassandra to connect to an
older server example ~ 2.1.4.
First let me demonstrate that configuration works in general for older versions.
{code}> apt-get install cassandra=2.1.9
Setting up cassandra (2.1.9)
> SSL_CERTFILE=~/qa_ssl_ca/qa_ca.pem cqlsh -k example -C -u sampleuser -p
> samplepassword cassandra.sqanet.com 9042 --ssl
Connected to cassandra.sqanet.com at sqa-cassandra.sqanet.com:9042.
[cqlsh 5.0.1 | Cassandra 2.1.4 | CQL spec 3.2.0 | Native protocol v3]
{code}
compare this to latest stable bundle.
{code}> apt-get install cassandra
Setting up cassandra (2.2.1)
> cqlsh --version
cqlsh 5.0.1
> SSL_CERTFILE=~/qa_ssl_ca/qa_ca.pem cqlsh --cqlversion=3.2.0 -k example -C -u
> sampleuser -p samplepassword cassandra.sqanet.com 9042 --ssl
Connection error: ('Unable to connect to any servers', {'cassandra.sqanet.com':
ConnectionException(u'Did not get expected SupportedMessage response; instead,
got: ',)})
{code}
> Startup message response for unsupported protocol versions is incorrect
> ---
>
> Key: CASSANDRA-9451
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9451
> Project: Cassandra
> Issue Type: Bug
> Environment: OS X 10.9
> Cassandra 2.1.5
>Reporter: Jorge Bay
>Assignee: Stefania
> Labels: client-impacting
> Fix For: 2.1.6, 2.2.0 rc1
>
> Attachments: 9451.txt
>
>
> The response to a STARTUP request with protocol v4 on a C* 2.1 host is an
> error with an incorrect error code (0).
> Instead of the error code being "Protocol error" ({{0x000A}}) it has error
> code 0 and message (wrapped by netty):
> {{"io.netty.handler.codec.DecoderException:
> org.apache.cassandra.transport.ProtocolException: Invalid or unsupported
> protocol version: 4"}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-7392) Abort in-progress queries that time out
[ https://issues.apache.org/jira/browse/CASSANDRA-7392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14745641#comment-14745641 ] Ariel Weisberg commented on CASSANDRA-7392: --- bq. aside from logging at DEBUG level is there any other blocker? Do we still retain logging of the top N entries or shall we log all timed out queries? The question about logging the top N entries is how is there a top N if the CQL statements aren't normalized? They are usually going to be unique so there is nothing to aggregate on. I think that is a real problem if we want to print a subset and still be informative. bq. I'm not sure if there is anything we can do about this? You can monitor the amount of memory pinned and bound it. For instance normalize the CQL statements and drop or truncate the parameters if too much is pinned. If the monitoring thread wakes up every 50 milliseconds it's pretty reasonable for it to do that. Let's be glad we don't have to do that now. > Abort in-progress queries that time out > --- > > Key: CASSANDRA-7392 > URL: https://issues.apache.org/jira/browse/CASSANDRA-7392 > Project: Cassandra > Issue Type: New Feature > Components: Core >Reporter: Jonathan Ellis >Assignee: Stefania >Priority: Critical > Fix For: 3.x > > > Currently we drop queries that time out before we get to them (because node > is overloaded) but not queries that time out while being processed. > (Particularly common for index queries on data that shouldn't be indexed.) > Adding the latter and logging when we have to interrupt one gets us a poor > man's "slow query log" for free. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-10251) JVM_OPTS repetition when started from init script
[
https://issues.apache.org/jira/browse/CASSANDRA-10251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Evans updated CASSANDRA-10251:
---
Reviewer: Brandon Williams
> JVM_OPTS repetition when started from init script
> -
>
> Key: CASSANDRA-10251
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10251
> Project: Cassandra
> Issue Type: Bug
> Components: Packaging
> Environment: Debian
>Reporter: Eric Evans
>Assignee: Eric Evans
> Attachments: cassandra.init.patch
>
>
> The Debian package init script sources {{cassandra-env.sh}}, and exports
> {{JVM_OPTS}}, a throw back to when we used jsvc, and constructed the full
> command with args. Now that we are using {{/usr/sbin/cassandra}}, which
> sources {{cassandra-env.sh}} itself, this results in the contents of
> {{JVM_OPTS}} appearing twice.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (CASSANDRA-10344) Optimize ReadResponse
Sylvain Lebresne created CASSANDRA-10344:
Summary: Optimize ReadResponse
Key: CASSANDRA-10344
URL: https://issues.apache.org/jira/browse/CASSANDRA-10344
Project: Cassandra
Issue Type: Improvement
Reporter: Sylvain Lebresne
Assignee: Sylvain Lebresne
Fix For: 3.0.0 rc1
The handling of {{ReadResponse}} has quite a bit of inefficiencies. The way it
works is based on constraints from early version of CASSANDRA-8099, but this
doesn't make sense anymore. This is particularly true for local response where
we fully serialize the response in memory to deserialize it a short time later.
But
# serialization/deserialization takes times, more than necessary in that case
# we serialize in a {{DataInputBuffer}} with a default initial size, which for
largish response might require a few somewhat costly resizing.
So, since we're materializing the full result in memory anyway, it should quite
a lot more efficient to materialize it in a simple list of
{{ImmutableBTreePartition}} in that case.
To a lesser extend, the serialization of {{ReadResponse}} that go over the wire
is probably not ideal either. Due to current assumptions of
{{MessagingService}}, we need to know the full serialized size of every
response upfront, which means we do have to materialize results in memory in
this case too. Currently, we do so by serialializing the full response in
memory first, and then writing that result. Here again, the serialization in
memory might require some resizing/copying, and we're fundamentally copying
things twice (this could be especially costly with largish user values). So
here too I suggest to materialize the result in a list of
{{ImmutableBTreePartition}}, compute the serialized size from it and then
serialize it. This also allow to do better sizing of our data structures on the
receiving side.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-10344) Optimize ReadResponse
[
https://issues.apache.org/jira/browse/CASSANDRA-10344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14745654#comment-14745654
]
Sylvain Lebresne commented on CASSANDRA-10344:
--
I've pushed patches for this
[here|https://github.com/pcmanus/cassandra/commits/10344]. I'm gonna run some
benchmark on this but that will be my first use of cstar so I'll update for
that once I have the results.
> Optimize ReadResponse
> -
>
> Key: CASSANDRA-10344
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10344
> Project: Cassandra
> Issue Type: Improvement
>Reporter: Sylvain Lebresne
>Assignee: Sylvain Lebresne
> Fix For: 3.0.0 rc1
>
>
> The handling of {{ReadResponse}} has quite a bit of inefficiencies. The way
> it works is based on constraints from early version of CASSANDRA-8099, but
> this doesn't make sense anymore. This is particularly true for local response
> where we fully serialize the response in memory to deserialize it a short
> time later. But
> # serialization/deserialization takes times, more than necessary in that case
> # we serialize in a {{DataInputBuffer}} with a default initial size, which
> for largish response might require a few somewhat costly resizing.
> So, since we're materializing the full result in memory anyway, it should
> quite a lot more efficient to materialize it in a simple list of
> {{ImmutableBTreePartition}} in that case.
> To a lesser extend, the serialization of {{ReadResponse}} that go over the
> wire is probably not ideal either. Due to current assumptions of
> {{MessagingService}}, we need to know the full serialized size of every
> response upfront, which means we do have to materialize results in memory in
> this case too. Currently, we do so by serialializing the full response in
> memory first, and then writing that result. Here again, the serialization in
> memory might require some resizing/copying, and we're fundamentally copying
> things twice (this could be especially costly with largish user values). So
> here too I suggest to materialize the result in a list of
> {{ImmutableBTreePartition}}, compute the serialized size from it and then
> serialize it. This also allow to do better sizing of our data structures on
> the receiving side.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-10326) Performance is worse in 3.0
[ https://issues.apache.org/jira/browse/CASSANDRA-10326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14745656#comment-14745656 ] Sylvain Lebresne commented on CASSANDRA-10326: -- bq. will you have time to look into this before GA? I'll do my best. Starting by seeing if CASSANDRA-10344 makes a difference. That said, it would help to get a more complete picture of what has been benchmarked so far and the results. For instance, it seems those tests all use reversed queries. I'm fine with reversed query, but does that mean forward queries didn't exhibit the same differences? Or is it just that for some reason you've started testing with reversed query and haven't results for forward ones? Or is it the same results but you randomly picked those examples? Basically, the more complete picture we have, the more likely we are to know where we should start looking for inefficiencies. > Performance is worse in 3.0 > --- > > Key: CASSANDRA-10326 > URL: https://issues.apache.org/jira/browse/CASSANDRA-10326 > Project: Cassandra > Issue Type: Bug >Reporter: Benedict >Priority: Critical > Fix For: 3.0.x > > > Performance is generally turning out to be worse after 8099, despite a number > of unrelated performance enhancements being delivered. This isn't entirely > unexpected, given a great deal of time was spent optimising the old code, > however things appear worse than we had hoped. > My expectation was that workloads making extensive use of CQL constructs > would be faster post-8099, however the latest tests performed with very large > CQL rows, including use of collections, still exhibit performance below that > of 2.1 and 2.2. > Eventually, as the dataset size grows large enough and the locality of access > is just right, the reduction in size of our dataset will yield a window > during which some users will perform better due simply to improved page cache > hit rates. We seem to see this in some of the tests. However we should be at > least as fast (and really faster) off the bat. > The following are some large partition benchmark results, with as many as 40K > rows per partition, running LCS. There are a number of parameters we can > modify to see how behaviour changes and under what scenarios we might still > be faster, but the picture painted isn't brilliant, and is consistent, so we > should really try and figure out what's up before GA. > [trades-with-flags (collections), > blade11b|http://cstar.datastax.com/graph?stats=f0a17292-5a13-11e5-847a-42010af0688f&metric=op_rate&operation=1_user&smoothing=1&show_aggregates=true&xmin=0&xmax=4387.02&ymin=0&ymax=122951.4] > [trades-with-flags (collections), > blade11|http://cstar.datastax.com/graph?stats=e250-5a13-11e5-ae0d-42010af0688f&metric=op_rate&operation=1_user&smoothing=1&show_aggregates=true&xmin=0&xmax=4424.75&ymin=0&ymax=130158.6] > [trades (no collections), > blade11|http://cstar.datastax.com/graph?stats=9b7da48e-570c-11e5-90fe-42010af0688f&metric=op_rate&operation=1_user&smoothing=1&show_aggregates=true&xmin=0&xmax=2682.46&ymin=0&ymax=142547.9] > [~slebresne]: will you have time to look into this before GA? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-10264) Unable to use conditions on static columns for DELETE
[
https://issues.apache.org/jira/browse/CASSANDRA-10264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14745662#comment-14745662
]
Benjamin Lerer commented on CASSANDRA-10264:
In fact there is 2 problems: one in cqlsh which trigger the invalid syntax and
one in {{ModificationStatement}} that prevent the query of being executed and
return an error.
Both are already there in 2.1.
I have some patch running on CI.
> Unable to use conditions on static columns for DELETE
> -
>
> Key: CASSANDRA-10264
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10264
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: Cassandra 2.2.0
>Reporter: DOAN DuyHai
>Assignee: Benjamin Lerer
>
> {noformat}
> cqlsh:test> create table static_table(id int, stat int static, ord int, val
> text, primary key(id,ord));
> cqlsh:test> insert into static_table (id,stat,ord,val) VALUES ( 1, 1, 1, '1');
> cqlsh:test> delete from static_table where id=1 and ord=1 if stat != 1;
> Invalid syntax at line 1, char 55
> delete from static_table where id=1 and ord=1 if stat != 1;
> ^
> {noformat}
> Same error if using =, <, <=, >= or > condition
> According to [~thobbs] the syntax should work. Plus, the error message is
> wrong
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-10326) Performance is worse in 3.0
[
https://issues.apache.org/jira/browse/CASSANDRA-10326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14745669#comment-14745669
]
Benedict commented on CASSANDRA-10326:
--
bq. Basically, the more complete picture we have, the more likely we are to
know where we should start looking for inefficiencies.
Agreed. I picked reverse queries quite arbitrarily, since we haven't (to my
knowledge) tested this previously, and this is the complete picture we have
currently, and we should certainly consider kicking off a batch of similar
tests exploring the option space (I had a lengthier blurb explaining this, but
I clearly trimmed it when I got too long).
It's also worth noting I just ran a comparison for more efficient {{skipBytes}}
[here|http://cstar.datastax.com/graph?stats=e31e6c18-5baf-11e5-b176-42010af0688f&metric=op_rate&operation=1_user&smoothing=1&show_aggregates=true&xmin=0&xmax=3423.09&ymin=0&ymax=143052.8]
and the results were significantly worse, which makes no sense at all (at
best, I'd expect no difference). So there may also be issues with consistency
of results at play. However the results I filed against this ticket were
remarkably consistent across runs.
> Performance is worse in 3.0
> ---
>
> Key: CASSANDRA-10326
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10326
> Project: Cassandra
> Issue Type: Bug
>Reporter: Benedict
>Priority: Critical
> Fix For: 3.0.x
>
>
> Performance is generally turning out to be worse after 8099, despite a number
> of unrelated performance enhancements being delivered. This isn't entirely
> unexpected, given a great deal of time was spent optimising the old code,
> however things appear worse than we had hoped.
> My expectation was that workloads making extensive use of CQL constructs
> would be faster post-8099, however the latest tests performed with very large
> CQL rows, including use of collections, still exhibit performance below that
> of 2.1 and 2.2.
> Eventually, as the dataset size grows large enough and the locality of access
> is just right, the reduction in size of our dataset will yield a window
> during which some users will perform better due simply to improved page cache
> hit rates. We seem to see this in some of the tests. However we should be at
> least as fast (and really faster) off the bat.
> The following are some large partition benchmark results, with as many as 40K
> rows per partition, running LCS. There are a number of parameters we can
> modify to see how behaviour changes and under what scenarios we might still
> be faster, but the picture painted isn't brilliant, and is consistent, so we
> should really try and figure out what's up before GA.
> [trades-with-flags (collections),
> blade11b|http://cstar.datastax.com/graph?stats=f0a17292-5a13-11e5-847a-42010af0688f&metric=op_rate&operation=1_user&smoothing=1&show_aggregates=true&xmin=0&xmax=4387.02&ymin=0&ymax=122951.4]
> [trades-with-flags (collections),
> blade11|http://cstar.datastax.com/graph?stats=e250-5a13-11e5-ae0d-42010af0688f&metric=op_rate&operation=1_user&smoothing=1&show_aggregates=true&xmin=0&xmax=4424.75&ymin=0&ymax=130158.6]
> [trades (no collections),
> blade11|http://cstar.datastax.com/graph?stats=9b7da48e-570c-11e5-90fe-42010af0688f&metric=op_rate&operation=1_user&smoothing=1&show_aggregates=true&xmin=0&xmax=2682.46&ymin=0&ymax=142547.9]
> [~slebresne]: will you have time to look into this before GA?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-10007) Repeated rows in paged result
[ https://issues.apache.org/jira/browse/CASSANDRA-10007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14745676#comment-14745676 ] Benjamin Lerer commented on CASSANDRA-10007: @Adam Holmberg could you check if the problem is still there? > Repeated rows in paged result > - > > Key: CASSANDRA-10007 > URL: https://issues.apache.org/jira/browse/CASSANDRA-10007 > Project: Cassandra > Issue Type: Bug > Components: Core >Reporter: Adam Holmberg >Assignee: Benjamin Lerer > Labels: client-impacting > Fix For: 3.x > > Attachments: paging-test.py > > > We noticed an anomaly in paged results while testing against 3.0.0-alpha1. It > seems that unbounded selects can return rows repeated at page boundaries. > Furthermore, the number of repeated rows seems to dither in count across > consecutive runs of the same query. > Does not reproduce on 2.2.0 and earlier. > I also noted that this behavior only manifests on multi-node clusters. > The attached script shows this behavior when run against 3.0.0-alpha1. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[2/3] cassandra git commit: Improve efficieny of ColumnDefinition comparison
Improve efficieny of ColumnDefinition comparison
patch by benedict; reviewed by sylvain for CASSANDRA-10316
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/d68325bf
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/d68325bf
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/d68325bf
Branch: refs/heads/trunk
Commit: d68325bfd3eec7c84515c81b417524c6e60b98d2
Parents: 7ca4db0
Author: Benedict Elliott Smith
Authored: Mon Sep 14 16:00:55 2015 +0100
Committer: Benedict Elliott Smith
Committed: Tue Sep 15 17:40:38 2015 +0100
--
.../apache/cassandra/config/ColumnDefinition.java | 17 +++--
.../apache/cassandra/cql3/ColumnIdentifier.java| 2 +-
2 files changed, 12 insertions(+), 7 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/d68325bf/src/java/org/apache/cassandra/config/ColumnDefinition.java
--
diff --git a/src/java/org/apache/cassandra/config/ColumnDefinition.java
b/src/java/org/apache/cassandra/config/ColumnDefinition.java
index 17276bc..7dc425f 100644
--- a/src/java/org/apache/cassandra/config/ColumnDefinition.java
+++ b/src/java/org/apache/cassandra/config/ColumnDefinition.java
@@ -48,6 +48,7 @@ public class ColumnDefinition extends ColumnSpecification
implements Comparable<
*/
public enum Kind
{
+// NOTE: if adding a new type, must modify comparisonOrder
PARTITION_KEY,
CLUSTERING,
REGULAR,
@@ -75,13 +76,17 @@ public class ColumnDefinition extends ColumnSpecification
implements Comparable<
/**
* These objects are compared frequently, so we encode several of their
comparison components
- * into a single int value so that this can be done efficiently
+ * into a single long value so that this can be done efficiently
*/
-private final int comparisonOrder;
+private final long comparisonOrder;
-private static int comparisonOrder(Kind kind, boolean isComplex, int
position)
+private static long comparisonOrder(Kind kind, boolean isComplex, long
position, ColumnIdentifier name)
{
-return (kind.ordinal() << 28) | (isComplex ? 1 << 27 : 0) | position;
+assert position >= 0 && position < 1 << 12;
+return (((long) kind.ordinal()) << 61)
+ | (isComplex ? 1L << 60 : 0)
+ | (position << 48)
+ | (name.prefixComparison >>> 16);
}
public static ColumnDefinition partitionKeyDef(CFMetaData cfm, ByteBuffer
name, AbstractType type, int position)
@@ -147,7 +152,7 @@ public class ColumnDefinition extends ColumnSpecification
implements Comparable<
this.cellPathComparator = makeCellPathComparator(kind, type);
this.cellComparator = cellPathComparator == null ?
ColumnData.comparator : (a, b) -> cellPathComparator.compare(a.path(),
b.path());
this.asymmetricCellPathComparator = cellPathComparator == null ? null
: (a, b) -> cellPathComparator.compare(((Cell)a).path(), (CellPath) b);
-this.comparisonOrder = comparisonOrder(kind, isComplex(), position());
+this.comparisonOrder = comparisonOrder(kind, isComplex(), position(),
name);
}
private static Comparator makeCellPathComparator(Kind kind,
AbstractType type)
@@ -308,7 +313,7 @@ public class ColumnDefinition extends ColumnSpecification
implements Comparable<
return 0;
if (comparisonOrder != other.comparisonOrder)
-return comparisonOrder - other.comparisonOrder;
+return Long.compare(comparisonOrder, other.comparisonOrder);
return this.name.compareTo(other.name);
}
http://git-wip-us.apache.org/repos/asf/cassandra/blob/d68325bf/src/java/org/apache/cassandra/cql3/ColumnIdentifier.java
--
diff --git a/src/java/org/apache/cassandra/cql3/ColumnIdentifier.java
b/src/java/org/apache/cassandra/cql3/ColumnIdentifier.java
index 6102bb9..4880c60 100644
--- a/src/java/org/apache/cassandra/cql3/ColumnIdentifier.java
+++ b/src/java/org/apache/cassandra/cql3/ColumnIdentifier.java
@@ -52,7 +52,7 @@ public class ColumnIdentifier extends
org.apache.cassandra.cql3.selection.Select
* since these objects are compared frequently, we stash an efficiently
compared prefix of the bytes, in the expectation
* that the majority of comparisons can be answered by this value only
*/
-private final long prefixComparison;
+public final long prefixComparison;
private final boolean interned;
private static final long EMPTY_SIZE = ObjectSizes.measure(new
ColumnIdentifier(ByteBufferUtil.EMPTY_BYTE_BUFFER, "", false));
[jira] [Commented] (CASSANDRA-10007) Repeated rows in paged result
[ https://issues.apache.org/jira/browse/CASSANDRA-10007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14745703#comment-14745703 ] Adam Holmberg commented on CASSANDRA-10007: --- I can confirm using my repro script that it does not surface on cassandra-3.0 @7ca4db0a. I would be more comfortable if the root cause was understood so we know it's not lurking, but I understand resource constraints. > Repeated rows in paged result > - > > Key: CASSANDRA-10007 > URL: https://issues.apache.org/jira/browse/CASSANDRA-10007 > Project: Cassandra > Issue Type: Bug > Components: Core >Reporter: Adam Holmberg >Assignee: Benjamin Lerer > Labels: client-impacting > Fix For: 3.x > > Attachments: paging-test.py > > > We noticed an anomaly in paged results while testing against 3.0.0-alpha1. It > seems that unbounded selects can return rows repeated at page boundaries. > Furthermore, the number of repeated rows seems to dither in count across > consecutive runs of the same query. > Does not reproduce on 2.2.0 and earlier. > I also noted that this behavior only manifests on multi-node clusters. > The attached script shows this behavior when run against 3.0.0-alpha1. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[3/3] cassandra git commit: Merge branch 'cassandra-3.0' into trunk
Merge branch 'cassandra-3.0' into trunk Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/e90ec26b Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/e90ec26b Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/e90ec26b Branch: refs/heads/trunk Commit: e90ec26b67eeca60a70c4c91fdb37ea9d09b7660 Parents: 1944e40 d68325b Author: Benedict Elliott Smith Authored: Tue Sep 15 17:41:28 2015 +0100 Committer: Benedict Elliott Smith Committed: Tue Sep 15 17:41:28 2015 +0100 -- .../apache/cassandra/config/ColumnDefinition.java | 17 +++-- .../apache/cassandra/cql3/ColumnIdentifier.java| 2 +- 2 files changed, 12 insertions(+), 7 deletions(-) --
[1/3] cassandra git commit: Improve efficieny of ColumnDefinition comparison
Repository: cassandra
Updated Branches:
refs/heads/cassandra-3.0 7ca4db0a1 -> d68325bfd
refs/heads/trunk 1944e402c -> e90ec26b6
Improve efficieny of ColumnDefinition comparison
patch by benedict; reviewed by sylvain for CASSANDRA-10316
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/d68325bf
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/d68325bf
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/d68325bf
Branch: refs/heads/cassandra-3.0
Commit: d68325bfd3eec7c84515c81b417524c6e60b98d2
Parents: 7ca4db0
Author: Benedict Elliott Smith
Authored: Mon Sep 14 16:00:55 2015 +0100
Committer: Benedict Elliott Smith
Committed: Tue Sep 15 17:40:38 2015 +0100
--
.../apache/cassandra/config/ColumnDefinition.java | 17 +++--
.../apache/cassandra/cql3/ColumnIdentifier.java| 2 +-
2 files changed, 12 insertions(+), 7 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/d68325bf/src/java/org/apache/cassandra/config/ColumnDefinition.java
--
diff --git a/src/java/org/apache/cassandra/config/ColumnDefinition.java
b/src/java/org/apache/cassandra/config/ColumnDefinition.java
index 17276bc..7dc425f 100644
--- a/src/java/org/apache/cassandra/config/ColumnDefinition.java
+++ b/src/java/org/apache/cassandra/config/ColumnDefinition.java
@@ -48,6 +48,7 @@ public class ColumnDefinition extends ColumnSpecification
implements Comparable<
*/
public enum Kind
{
+// NOTE: if adding a new type, must modify comparisonOrder
PARTITION_KEY,
CLUSTERING,
REGULAR,
@@ -75,13 +76,17 @@ public class ColumnDefinition extends ColumnSpecification
implements Comparable<
/**
* These objects are compared frequently, so we encode several of their
comparison components
- * into a single int value so that this can be done efficiently
+ * into a single long value so that this can be done efficiently
*/
-private final int comparisonOrder;
+private final long comparisonOrder;
-private static int comparisonOrder(Kind kind, boolean isComplex, int
position)
+private static long comparisonOrder(Kind kind, boolean isComplex, long
position, ColumnIdentifier name)
{
-return (kind.ordinal() << 28) | (isComplex ? 1 << 27 : 0) | position;
+assert position >= 0 && position < 1 << 12;
+return (((long) kind.ordinal()) << 61)
+ | (isComplex ? 1L << 60 : 0)
+ | (position << 48)
+ | (name.prefixComparison >>> 16);
}
public static ColumnDefinition partitionKeyDef(CFMetaData cfm, ByteBuffer
name, AbstractType type, int position)
@@ -147,7 +152,7 @@ public class ColumnDefinition extends ColumnSpecification
implements Comparable<
this.cellPathComparator = makeCellPathComparator(kind, type);
this.cellComparator = cellPathComparator == null ?
ColumnData.comparator : (a, b) -> cellPathComparator.compare(a.path(),
b.path());
this.asymmetricCellPathComparator = cellPathComparator == null ? null
: (a, b) -> cellPathComparator.compare(((Cell)a).path(), (CellPath) b);
-this.comparisonOrder = comparisonOrder(kind, isComplex(), position());
+this.comparisonOrder = comparisonOrder(kind, isComplex(), position(),
name);
}
private static Comparator makeCellPathComparator(Kind kind,
AbstractType type)
@@ -308,7 +313,7 @@ public class ColumnDefinition extends ColumnSpecification
implements Comparable<
return 0;
if (comparisonOrder != other.comparisonOrder)
-return comparisonOrder - other.comparisonOrder;
+return Long.compare(comparisonOrder, other.comparisonOrder);
return this.name.compareTo(other.name);
}
http://git-wip-us.apache.org/repos/asf/cassandra/blob/d68325bf/src/java/org/apache/cassandra/cql3/ColumnIdentifier.java
--
diff --git a/src/java/org/apache/cassandra/cql3/ColumnIdentifier.java
b/src/java/org/apache/cassandra/cql3/ColumnIdentifier.java
index 6102bb9..4880c60 100644
--- a/src/java/org/apache/cassandra/cql3/ColumnIdentifier.java
+++ b/src/java/org/apache/cassandra/cql3/ColumnIdentifier.java
@@ -52,7 +52,7 @@ public class ColumnIdentifier extends
org.apache.cassandra.cql3.selection.Select
* since these objects are compared frequently, we stash an efficiently
compared prefix of the bytes, in the expectation
* that the majority of comparisons can be answered by this value only
*/
-private final long prefixComparison;
+public final long prefixComparison;
private final boolean interned;
private static final long EMPTY_SIZ
[jira] [Commented] (CASSANDRA-7918) Provide graphing tool along with cassandra-stress
[ https://issues.apache.org/jira/browse/CASSANDRA-7918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14745705#comment-14745705 ] Ryan McGuire commented on CASSANDRA-7918: - We don't have any bandwidth to work on #9870 right now. I'm fine commiting this in stages, or waiting, but #9870 likely won't happen until after 3.0 ships. 9870 is more focussed on the visual aspect than the results capturing that this handles. > Provide graphing tool along with cassandra-stress > - > > Key: CASSANDRA-7918 > URL: https://issues.apache.org/jira/browse/CASSANDRA-7918 > Project: Cassandra > Issue Type: Improvement > Components: Tools >Reporter: Benedict >Assignee: Ryan McGuire >Priority: Minor > Attachments: 7918.patch, reads.svg > > > Whilst cstar makes some pretty graphs, they're a little limited and also > require you to run your tests through it. It would be useful to be able to > graph results from any stress run easily. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-10300) Add new options to repair documentation
[ https://issues.apache.org/jira/browse/CASSANDRA-10300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14745708#comment-14745708 ] Jim Witschey commented on CASSANDRA-10300: -- Thanks for reporting this. Since those docs are managed by DataStax, this has been re-filed on a DataStax-internal issue tracker. I'm closing as "Won't Fix", but it's been forwarded to the docs team. > Add new options to repair documentation > --- > > Key: CASSANDRA-10300 > URL: https://issues.apache.org/jira/browse/CASSANDRA-10300 > Project: Cassandra > Issue Type: Bug > Components: Documentation & website >Reporter: Marcus Olsson >Priority: Minor > > The repair documentation for C* 2.2 in > http://docs.datastax.com/en/cassandra/2.2/cassandra/tools/toolsRepair.html > seems to be a bit outdated when it comes to the different options that can be > used. > Based on the repair options found in the class > [Repair.java|https://github.com/apache/cassandra/blob/cassandra-2.2/src/java/org/apache/cassandra/tools/nodetool/Repair.java] > Remove: > * -par > * -inc > Add: > * -seq > * -full > * -j > * -tr > I also raised a > [question|http://www.mail-archive.com/user@cassandra.apache.org/msg43722.html] > on the mailing list regarding the repair documentation for C* 2.1. Thought > I'd just bring it up here since it's also regarding the repair documentation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (CASSANDRA-10300) Add new options to repair documentation
[ https://issues.apache.org/jira/browse/CASSANDRA-10300?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jim Witschey resolved CASSANDRA-10300. -- Resolution: Won't Fix > Add new options to repair documentation > --- > > Key: CASSANDRA-10300 > URL: https://issues.apache.org/jira/browse/CASSANDRA-10300 > Project: Cassandra > Issue Type: Bug > Components: Documentation & website >Reporter: Marcus Olsson >Priority: Minor > > The repair documentation for C* 2.2 in > http://docs.datastax.com/en/cassandra/2.2/cassandra/tools/toolsRepair.html > seems to be a bit outdated when it comes to the different options that can be > used. > Based on the repair options found in the class > [Repair.java|https://github.com/apache/cassandra/blob/cassandra-2.2/src/java/org/apache/cassandra/tools/nodetool/Repair.java] > Remove: > * -par > * -inc > Add: > * -seq > * -full > * -j > * -tr > I also raised a > [question|http://www.mail-archive.com/user@cassandra.apache.org/msg43722.html] > on the mailing list regarding the repair documentation for C* 2.1. Thought > I'd just bring it up here since it's also regarding the repair documentation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-10286) Windows utest 3.0: org.apache.cassandra.io.sstable.SSTableLoaderTest.testLoadingIncompleteSSTable
[
https://issues.apache.org/jira/browse/CASSANDRA-10286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14745712#comment-14745712
]
Benedict commented on CASSANDRA-10286:
--
Again. There are some suspicious "QueryMessage' object has no attribute" lines,
so I wonder if this needs a rebase to match driver updates
> Windows utest 3.0:
> org.apache.cassandra.io.sstable.SSTableLoaderTest.testLoadingIncompleteSSTable
> -
>
> Key: CASSANDRA-10286
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10286
> Project: Cassandra
> Issue Type: Sub-task
>Reporter: Philip Thompson
>Assignee: Stefania
> Fix For: 3.x
>
>
> Distinct error message from CASSANDRA-10210.
> {code}
> junit.framework.AssertionFailedError:
> at
> org.apache.cassandra.io.sstable.SSTableLoaderTest.testLoadingIncompleteSSTable(SSTableLoaderTest.java:186)
> {code}
> which I believe is from
> {code}
> ERROR 22:05:57 Unable to delete
> D:\temp\1441663555277-0\SSTableLoaderTest\Standard2\ma-1-big-Data.db
> java.nio.file.AccessDeniedException:
> D:\temp\1441663555277-0\SSTableLoaderTest\Standard2\ma-1-big-Data.db
> at
> sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:83)
> ~[na:1.8.0_51]
> at
> sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97)
> ~[na:1.8.0_51]
> at
> sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:102)
> ~[na:1.8.0_51]
> at
> sun.nio.fs.WindowsFileSystemProvider.implDelete(WindowsFileSystemProvider.java:269)
> ~[na:1.8.0_51]
> at
> sun.nio.fs.AbstractFileSystemProvider.delete(AbstractFileSystemProvider.java:103)
> ~[na:1.8.0_51]
> at java.nio.file.Files.delete(Files.java:1126) ~[na:1.8.0_51]
> at
> org.apache.cassandra.db.lifecycle.TransactionLog.delete(TransactionLog.java:792)
> [main/:na]
> at
> org.apache.cassandra.db.lifecycle.TransactionLog.access$100(TransactionLog.java:97)
> [main/:na]
> at
> org.apache.cassandra.db.lifecycle.TransactionLog$TransactionFile.lambda$deleteRecord$108(TransactionLog.java:504)
> [main/:na]
> at
> org.apache.cassandra.db.lifecycle.TransactionLog$TransactionFile$$Lambda$108/1799732569.accept(Unknown
> Source) [main/:na]
> at java.util.Arrays$ArrayList.forEach(Arrays.java:3880) [na:1.8.0_51]
> at
> org.apache.cassandra.db.lifecycle.TransactionLog$TransactionFile.deleteRecord(TransactionLog.java:504)
> [main/:na]
> at
> org.apache.cassandra.db.lifecycle.TransactionLog$TransactionFile$$Lambda$88/744169296.accept(Unknown
> Source) [main/:na]
> at
> java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184)
> [na:1.8.0_51]
> at
> java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)
> [na:1.8.0_51]
> at java.util.HashMap$KeySpliterator.forEachRemaining(HashMap.java:1540)
> [na:1.8.0_51]
> at
> java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:512)
> [na:1.8.0_51]
> at
> java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:502)
> [na:1.8.0_51]
> at
> java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151)
> [na:1.8.0_51]
> at
> java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174)
> [na:1.8.0_51]
> at
> java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
> [na:1.8.0_51]
> at
> java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418)
> [na:1.8.0_51]
> at
> org.apache.cassandra.db.lifecycle.TransactionLog$TransactionFile.deleteRecords(TransactionLog.java:490)
> [main/:na]
> at
> org.apache.cassandra.db.lifecycle.TransactionLog$TransactionData.removeUnfinishedLeftovers(TransactionLog.java:622)
> [main/:na]
> at
> org.apache.cassandra.db.lifecycle.TransactionLog$TransactionTidier.run(TransactionLog.java:837)
> [main/:na]
> at
> org.apache.cassandra.db.lifecycle.TransactionLog$TransactionTidier.tidy(TransactionLog.java:822)
> [main/:na]
> at
> org.apache.cassandra.utils.concurrent.Ref$GlobalState.release(Ref.java:294)
> [main/:na]
> at
> org.apache.cassandra.utils.concurrent.Ref$State.ensureReleased(Ref.java:172)
> [main/:na]
> at
> org.apache.cassandra.utils.concurrent.Ref.ensureReleased(Ref.java:92)
> [main/:na]
> at
> org.apache.cassandra.db.lifecycle.TransactionLog.complete(TransactionLog.java:950)
> [main/:na]
> at
> org.apache.cassandra.db.lifecycle.TransactionLog.doAbort(TransactionLog.java:969)
> [main/:na]
> at
> org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.abort(Transactional.java:144)
> [main/:na]
> at
> org.apache.cassa
[jira] [Comment Edited] (CASSANDRA-10286) Windows utest 3.0: org.apache.cassandra.io.sstable.SSTableLoaderTest.testLoadingIncompleteSSTable
[
https://issues.apache.org/jira/browse/CASSANDRA-10286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14745712#comment-14745712
]
Benedict edited comment on CASSANDRA-10286 at 9/15/15 4:48 PM:
---
Again. There are some suspicious "QueryMessage' object has no attribute" lines,
so I wonder if this needs a rebase to match driver updates
Or possible the wndows dtests that have been setup specially need updating?
[~philipthompson]?
was (Author: benedict):
Again. There are some suspicious "QueryMessage' object has no attribute" lines,
so I wonder if this needs a rebase to match driver updates
> Windows utest 3.0:
> org.apache.cassandra.io.sstable.SSTableLoaderTest.testLoadingIncompleteSSTable
> -
>
> Key: CASSANDRA-10286
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10286
> Project: Cassandra
> Issue Type: Sub-task
>Reporter: Philip Thompson
>Assignee: Stefania
> Fix For: 3.x
>
>
> Distinct error message from CASSANDRA-10210.
> {code}
> junit.framework.AssertionFailedError:
> at
> org.apache.cassandra.io.sstable.SSTableLoaderTest.testLoadingIncompleteSSTable(SSTableLoaderTest.java:186)
> {code}
> which I believe is from
> {code}
> ERROR 22:05:57 Unable to delete
> D:\temp\1441663555277-0\SSTableLoaderTest\Standard2\ma-1-big-Data.db
> java.nio.file.AccessDeniedException:
> D:\temp\1441663555277-0\SSTableLoaderTest\Standard2\ma-1-big-Data.db
> at
> sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:83)
> ~[na:1.8.0_51]
> at
> sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97)
> ~[na:1.8.0_51]
> at
> sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:102)
> ~[na:1.8.0_51]
> at
> sun.nio.fs.WindowsFileSystemProvider.implDelete(WindowsFileSystemProvider.java:269)
> ~[na:1.8.0_51]
> at
> sun.nio.fs.AbstractFileSystemProvider.delete(AbstractFileSystemProvider.java:103)
> ~[na:1.8.0_51]
> at java.nio.file.Files.delete(Files.java:1126) ~[na:1.8.0_51]
> at
> org.apache.cassandra.db.lifecycle.TransactionLog.delete(TransactionLog.java:792)
> [main/:na]
> at
> org.apache.cassandra.db.lifecycle.TransactionLog.access$100(TransactionLog.java:97)
> [main/:na]
> at
> org.apache.cassandra.db.lifecycle.TransactionLog$TransactionFile.lambda$deleteRecord$108(TransactionLog.java:504)
> [main/:na]
> at
> org.apache.cassandra.db.lifecycle.TransactionLog$TransactionFile$$Lambda$108/1799732569.accept(Unknown
> Source) [main/:na]
> at java.util.Arrays$ArrayList.forEach(Arrays.java:3880) [na:1.8.0_51]
> at
> org.apache.cassandra.db.lifecycle.TransactionLog$TransactionFile.deleteRecord(TransactionLog.java:504)
> [main/:na]
> at
> org.apache.cassandra.db.lifecycle.TransactionLog$TransactionFile$$Lambda$88/744169296.accept(Unknown
> Source) [main/:na]
> at
> java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184)
> [na:1.8.0_51]
> at
> java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)
> [na:1.8.0_51]
> at java.util.HashMap$KeySpliterator.forEachRemaining(HashMap.java:1540)
> [na:1.8.0_51]
> at
> java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:512)
> [na:1.8.0_51]
> at
> java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:502)
> [na:1.8.0_51]
> at
> java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151)
> [na:1.8.0_51]
> at
> java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174)
> [na:1.8.0_51]
> at
> java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
> [na:1.8.0_51]
> at
> java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418)
> [na:1.8.0_51]
> at
> org.apache.cassandra.db.lifecycle.TransactionLog$TransactionFile.deleteRecords(TransactionLog.java:490)
> [main/:na]
> at
> org.apache.cassandra.db.lifecycle.TransactionLog$TransactionData.removeUnfinishedLeftovers(TransactionLog.java:622)
> [main/:na]
> at
> org.apache.cassandra.db.lifecycle.TransactionLog$TransactionTidier.run(TransactionLog.java:837)
> [main/:na]
> at
> org.apache.cassandra.db.lifecycle.TransactionLog$TransactionTidier.tidy(TransactionLog.java:822)
> [main/:na]
> at
> org.apache.cassandra.utils.concurrent.Ref$GlobalState.release(Ref.java:294)
> [main/:na]
> at
> org.apache.cassandra.utils.concurrent.Ref$State.ensureReleased(Ref.java:172)
> [main/:na]
> at
> org.apache.cassandra.utils.concurrent.Ref.ensureReleased(Ref.java:92)
> [main/:na]
> at
> org.apache.cassandra.db.lifecycle.TransactionLog.comp
[1/3] cassandra git commit: Introduce unit tests for Rows, Cells, and DataResolver Fix Rows.diff complex deletion resolution
Repository: cassandra
Updated Branches:
refs/heads/cassandra-3.0 d68325bfd -> b263af93a
refs/heads/trunk e90ec26b6 -> ac3b1cc2c
Introduce unit tests for Rows, Cells, and DataResolver
Fix Rows.diff complex deletion resolution
patch by blake; reviewed by benedict for CASSANDRA-10266
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/b263af93
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/b263af93
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/b263af93
Branch: refs/heads/cassandra-3.0
Commit: b263af93a2899cf12ee5f35f0518460683fdac18
Parents: d68325b
Author: Blake Eggleston
Authored: Fri Sep 11 08:32:02 2015 -0700
Committer: Benedict Elliott Smith
Committed: Tue Sep 15 17:49:45 2015 +0100
--
src/java/org/apache/cassandra/db/rows/Rows.java | 32 +-
test/unit/org/apache/cassandra/db/CellTest.java | 48 +-
.../apache/cassandra/db/rows/RowBuilder.java| 85 +++
.../org/apache/cassandra/db/rows/RowsTest.java | 548 +++
.../cassandra/service/DataResolverTest.java | 225 +++-
5 files changed, 927 insertions(+), 11 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/b263af93/src/java/org/apache/cassandra/db/rows/Rows.java
--
diff --git a/src/java/org/apache/cassandra/db/rows/Rows.java
b/src/java/org/apache/cassandra/db/rows/Rows.java
index c3b4a92..ea2ca06 100644
--- a/src/java/org/apache/cassandra/db/rows/Rows.java
+++ b/src/java/org/apache/cassandra/db/rows/Rows.java
@@ -80,7 +80,7 @@ public abstract class Rows
{
++columnCount;
++cellCount;
-Cells.collectStats((Cell)cd, collector);
+Cells.collectStats((Cell) cd, collector);
}
else
{
@@ -105,11 +105,13 @@ public abstract class Rows
/**
* Given the result ({@code merged}) of merging multiple {@code inputs},
signals the difference between
* each input and {@code merged} to {@code diffListener}.
+ *
+ * Note that this method doesn't only emit cells etc where there's a
difference. The listener is informed
+ * of every corresponding entity between the merged and input rows,
including those that are equal.
*
+ * @param diffListener the listener to which to signal the differences
between the inputs and the merged result.
* @param merged the result of merging {@code inputs}.
* @param inputs the inputs whose merge yielded {@code merged}.
- * @param diffListener the listener to which to signal the differences
between the inputs and the merged
- * result.
*/
public static void diff(RowDiffListener diffListener, Row merged,
Row...inputs)
{
@@ -179,6 +181,10 @@ public abstract class Rows
}
else
{
+
+if (!mergedData.complexDeletion().isLive() ||
!inputData.complexDeletion().isLive())
+diffListener.onComplexDeletion(i,
clustering, column, mergedData.complexDeletion(), inputData.complexDeletion());
+
PeekingIterator mergedCells =
Iterators.peekingIterator(mergedData.iterator());
PeekingIterator inputCells =
Iterators.peekingIterator(inputData.iterator());
while (mergedCells.hasNext() &&
inputCells.hasNext())
@@ -221,8 +227,24 @@ public abstract class Rows
return builder.build();
}
-// Merge rows in memtable
-// Return the minimum timestamp delta between existing and update
+/**
+ * Merges two rows into the given builder, mainly for merging memtable
rows. In addition to reconciling the cells
+ * in each row, the liveness info, and deletion times for the row and
complex columns are also merged.
+ *
+ * Note that this method assumes that the provided rows can meaningfully
be reconciled together. That is,
+ * that the rows share the same clustering value, and belong to the same
partition.
+ *
+ * @param existing
+ * @param update
+ * @param builder the row build to which the result of the reconciliation
is written.
+ * @param nowInSec the current time in seconds (which plays a role during
reconciliation
+ * because deleted cells always have precedence on timestamp equality and
deciding if a
+ * cell is a live or not depends on the current time due to expiring
cells).
+ *
+ * @return the smallest timestamp delta between corresponding rows from
existing and update. A
+ * timestamp delta being computed as the difference between the cells and
DeletionTimes from {@code existing}
[3/3] cassandra git commit: Merge branch 'cassandra-3.0' into trunk
Merge branch 'cassandra-3.0' into trunk Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/ac3b1cc2 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/ac3b1cc2 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/ac3b1cc2 Branch: refs/heads/trunk Commit: ac3b1cc2c7eef99f1fb12e42202194f40447e544 Parents: e90ec26 b263af9 Author: Benedict Elliott Smith Authored: Tue Sep 15 17:50:43 2015 +0100 Committer: Benedict Elliott Smith Committed: Tue Sep 15 17:50:43 2015 +0100 -- src/java/org/apache/cassandra/db/rows/Rows.java | 32 +- test/unit/org/apache/cassandra/db/CellTest.java | 48 +- .../apache/cassandra/db/rows/RowBuilder.java| 85 +++ .../org/apache/cassandra/db/rows/RowsTest.java | 548 +++ .../cassandra/service/DataResolverTest.java | 225 +++- 5 files changed, 927 insertions(+), 11 deletions(-) --
[2/3] cassandra git commit: Introduce unit tests for Rows, Cells, and DataResolver Fix Rows.diff complex deletion resolution
Introduce unit tests for Rows, Cells, and DataResolver
Fix Rows.diff complex deletion resolution
patch by blake; reviewed by benedict for CASSANDRA-10266
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/b263af93
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/b263af93
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/b263af93
Branch: refs/heads/trunk
Commit: b263af93a2899cf12ee5f35f0518460683fdac18
Parents: d68325b
Author: Blake Eggleston
Authored: Fri Sep 11 08:32:02 2015 -0700
Committer: Benedict Elliott Smith
Committed: Tue Sep 15 17:49:45 2015 +0100
--
src/java/org/apache/cassandra/db/rows/Rows.java | 32 +-
test/unit/org/apache/cassandra/db/CellTest.java | 48 +-
.../apache/cassandra/db/rows/RowBuilder.java| 85 +++
.../org/apache/cassandra/db/rows/RowsTest.java | 548 +++
.../cassandra/service/DataResolverTest.java | 225 +++-
5 files changed, 927 insertions(+), 11 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/b263af93/src/java/org/apache/cassandra/db/rows/Rows.java
--
diff --git a/src/java/org/apache/cassandra/db/rows/Rows.java
b/src/java/org/apache/cassandra/db/rows/Rows.java
index c3b4a92..ea2ca06 100644
--- a/src/java/org/apache/cassandra/db/rows/Rows.java
+++ b/src/java/org/apache/cassandra/db/rows/Rows.java
@@ -80,7 +80,7 @@ public abstract class Rows
{
++columnCount;
++cellCount;
-Cells.collectStats((Cell)cd, collector);
+Cells.collectStats((Cell) cd, collector);
}
else
{
@@ -105,11 +105,13 @@ public abstract class Rows
/**
* Given the result ({@code merged}) of merging multiple {@code inputs},
signals the difference between
* each input and {@code merged} to {@code diffListener}.
+ *
+ * Note that this method doesn't only emit cells etc where there's a
difference. The listener is informed
+ * of every corresponding entity between the merged and input rows,
including those that are equal.
*
+ * @param diffListener the listener to which to signal the differences
between the inputs and the merged result.
* @param merged the result of merging {@code inputs}.
* @param inputs the inputs whose merge yielded {@code merged}.
- * @param diffListener the listener to which to signal the differences
between the inputs and the merged
- * result.
*/
public static void diff(RowDiffListener diffListener, Row merged,
Row...inputs)
{
@@ -179,6 +181,10 @@ public abstract class Rows
}
else
{
+
+if (!mergedData.complexDeletion().isLive() ||
!inputData.complexDeletion().isLive())
+diffListener.onComplexDeletion(i,
clustering, column, mergedData.complexDeletion(), inputData.complexDeletion());
+
PeekingIterator mergedCells =
Iterators.peekingIterator(mergedData.iterator());
PeekingIterator inputCells =
Iterators.peekingIterator(inputData.iterator());
while (mergedCells.hasNext() &&
inputCells.hasNext())
@@ -221,8 +227,24 @@ public abstract class Rows
return builder.build();
}
-// Merge rows in memtable
-// Return the minimum timestamp delta between existing and update
+/**
+ * Merges two rows into the given builder, mainly for merging memtable
rows. In addition to reconciling the cells
+ * in each row, the liveness info, and deletion times for the row and
complex columns are also merged.
+ *
+ * Note that this method assumes that the provided rows can meaningfully
be reconciled together. That is,
+ * that the rows share the same clustering value, and belong to the same
partition.
+ *
+ * @param existing
+ * @param update
+ * @param builder the row build to which the result of the reconciliation
is written.
+ * @param nowInSec the current time in seconds (which plays a role during
reconciliation
+ * because deleted cells always have precedence on timestamp equality and
deciding if a
+ * cell is a live or not depends on the current time due to expiring
cells).
+ *
+ * @return the smallest timestamp delta between corresponding rows from
existing and update. A
+ * timestamp delta being computed as the difference between the cells and
DeletionTimes from {@code existing}
+ * and those in {@code existing}.
+ */
public static long merge(Row existing,
Row update,
[jira] [Commented] (CASSANDRA-10320) Build off-heap key-cache compatible API
[ https://issues.apache.org/jira/browse/CASSANDRA-10320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14745719#comment-14745719 ] Robert Stupp commented on CASSANDRA-10320: -- Pushed a first series of commits to the branch. It's just some very basic refactoring without any functional change that encapsulates the internals of RowIndexEntry, makes IndexInfo a top-level class and removes IndexHelper as a _preparation step_ to change the internal representation of RowIndexEntry.IndexedEntry from an object graph to a ByteBuffer. > Build off-heap key-cache compatible API > --- > > Key: CASSANDRA-10320 > URL: https://issues.apache.org/jira/browse/CASSANDRA-10320 > Project: Cassandra > Issue Type: Improvement >Reporter: Robert Stupp >Assignee: Robert Stupp > Fix For: 3.x > > > (Follow-up to CASSANDRA-10314 and prerequisite for the final goal > CASSANDRA-9738.) > Ticket is about to extract the API code changes from CASSANDRA-9738 as a > sub-task - so everything from CASSANDRA-9738 without the actual off-heap > implementation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-10216) Remove target type from internal index metadata
[
https://issues.apache.org/jira/browse/CASSANDRA-10216?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14745721#comment-14745721
]
Sam Tunnicliffe commented on CASSANDRA-10216:
-
Thanks [~slebresne], I've pushed a commit which (hopefully) handles it properly.
> Remove target type from internal index metadata
> ---
>
> Key: CASSANDRA-10216
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10216
> Project: Cassandra
> Issue Type: Improvement
>Reporter: Sam Tunnicliffe
>Assignee: Sam Tunnicliffe
> Labels: client-impacting
> Fix For: 3.0.0 rc1
>
>
> As part of CASSANDRA-6716 & in anticipation of CASSANDRA-10124, a distinction
> was introduced between secondary indexes which target a fixed set of 1 or
> more columns in the base data, and those which are agnostic to the structure
> of the underlying rows. This distinction is manifested in
> {{IndexMetadata.targetType}} and {{system_schema.indexes}}, in the
> {{target_type}} column. It could be argued that this distinction complicates
> the codebase without providing any tangible benefit, given that the target
> type is not actually used anywhere.
> It's only the impact on {{system_schema.indexes}} that makes puts this on the
> critical path for 3.0, any code changes are just implementation details.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (CASSANDRA-10316) Improve ColumnDefinition comparison performance
[ https://issues.apache.org/jira/browse/CASSANDRA-10316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benedict updated CASSANDRA-10316: - Reviewer: 601126228506 Fix Version/s: (was: 3.0.x) 3.0.0 rc1 > Improve ColumnDefinition comparison performance > --- > > Key: CASSANDRA-10316 > URL: https://issues.apache.org/jira/browse/CASSANDRA-10316 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: Benedict >Assignee: Benedict > Fix For: 3.0.0 rc1 > > > We have already improved the performance here for the basic comparison, > however since these happen exceedingly frequently we may as well go a little > (easy step) further. This is a really tiny patch, and we should aim to > include before GA, but not RC. > Right now we use all of an int for the metadata presorting, but in fact we > only need 2-3 bytes for this. We can upcast the int to a long, and use the > remaining bits to store the prefix of the _name_. This way, a majority of > comparisons become a single long comparison. Which will be considerably > cheaper. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-10316) Improve ColumnDefinition comparison performance
[ https://issues.apache.org/jira/browse/CASSANDRA-10316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benedict updated CASSANDRA-10316: - Reviewer: Sylvain Lebresne (was: 601126228506) > Improve ColumnDefinition comparison performance > --- > > Key: CASSANDRA-10316 > URL: https://issues.apache.org/jira/browse/CASSANDRA-10316 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: Benedict >Assignee: Benedict > Fix For: 3.0.0 rc1 > > > We have already improved the performance here for the basic comparison, > however since these happen exceedingly frequently we may as well go a little > (easy step) further. This is a really tiny patch, and we should aim to > include before GA, but not RC. > Right now we use all of an int for the metadata presorting, but in fact we > only need 2-3 bytes for this. We can upcast the int to a long, and use the > remaining bits to store the prefix of the _name_. This way, a majority of > comparisons become a single long comparison. Which will be considerably > cheaper. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-10251) JVM_OPTS repetition when started from init script
[
https://issues.apache.org/jira/browse/CASSANDRA-10251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Evans updated CASSANDRA-10251:
---
Reviewer: Michael Shuler (was: Brandon Williams)
> JVM_OPTS repetition when started from init script
> -
>
> Key: CASSANDRA-10251
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10251
> Project: Cassandra
> Issue Type: Bug
> Components: Packaging
> Environment: Debian
>Reporter: Eric Evans
>Assignee: Eric Evans
> Attachments: cassandra.init.patch
>
>
> The Debian package init script sources {{cassandra-env.sh}}, and exports
> {{JVM_OPTS}}, a throw back to when we used jsvc, and constructed the full
> command with args. Now that we are using {{/usr/sbin/cassandra}}, which
> sources {{cassandra-env.sh}} itself, this results in the contents of
> {{JVM_OPTS}} appearing twice.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-7042) Disk space growth until restart
[
https://issues.apache.org/jira/browse/CASSANDRA-7042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14745738#comment-14745738
]
Jim Witschey commented on CASSANDRA-7042:
-
Is this still a concern? There's been no activity on this ticket in over a year.
> Disk space growth until restart
> ---
>
> Key: CASSANDRA-7042
> URL: https://issues.apache.org/jira/browse/CASSANDRA-7042
> Project: Cassandra
> Issue Type: Bug
> Environment: Ubuntu 12.04
> Sun Java 7
> Cassandra 2.0.6
>Reporter: Zach Aller
> Attachments: Screen Shot 2014-04-17 at 11.07.24 AM.png, Screen Shot
> 2014-04-18 at 11.47.30 AM.png, Screen Shot 2014-04-22 at 1.40.41 PM.png,
> after.log, before.log, system.log.9.gz, tabledump_after_restart.txt,
> tabledump_before_restart.txt
>
>
> Cassandra will constantly eat disk space not sure whats causing it the only
> thing that seems to fix it is a restart of cassandra this happens about every
> 3-5 hrs we will grow from about 350GB to 650GB with no end in site. Once we
> restart cassandra it usually all clears itself up and disks return to normal
> for a while then something triggers its and starts climbing again. Sometimes
> when we restart compactions pending skyrocket and if we restart a second time
> the compactions pending drop off back to a normal level. One other thing to
> note is the space is not free'd until cassandra starts back up and not when
> shutdown.
> I will get a clean log of before and after restarting next time it happens
> and post it.
> Here is a common ERROR in our logs that might be related
> {noformat}
> ERROR [CompactionExecutor:46] 2014-04-15 09:12:51,040 CassandraDaemon.java
> (line 196) Exception in thread Thread[CompactionExecutor:46,1,main]
> java.lang.RuntimeException: java.io.FileNotFoundException:
> /local-project/cassandra_data/data/wxgrid/grid/wxgrid-grid-jb-468677-Data.db
> (No such file or directory)
> at
> org.apache.cassandra.io.util.ThrottledReader.open(ThrottledReader.java:53)
> at
> org.apache.cassandra.io.sstable.SSTableReader.openDataReader(SSTableReader.java:1355)
> at
> org.apache.cassandra.io.sstable.SSTableScanner.(SSTableScanner.java:67)
> at
> org.apache.cassandra.io.sstable.SSTableReader.getScanner(SSTableReader.java:1161)
> at
> org.apache.cassandra.io.sstable.SSTableReader.getScanner(SSTableReader.java:1173)
> at
> org.apache.cassandra.db.compaction.LeveledCompactionStrategy.getScanners(LeveledCompactionStrategy.java:194)
> at
> org.apache.cassandra.db.compaction.AbstractCompactionStrategy.getScanners(AbstractCompactionStrategy.java:258)
> at
> org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:126)
> at
> org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48)
> at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> at
> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:60)
> at
> org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59)
> at
> org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:197)
> at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
> at java.util.concurrent.FutureTask.run(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
> Caused by: java.io.FileNotFoundException:
> /local-project/cassandra_data/data/wxgrid/grid/wxgrid-grid-jb-468677-Data.db
> (No such file or directory)
> at java.io.RandomAccessFile.open(Native Method)
> at java.io.RandomAccessFile.(Unknown Source)
> at
> org.apache.cassandra.io.util.RandomAccessReader.(RandomAccessReader.java:58)
> at
> org.apache.cassandra.io.util.ThrottledReader.(ThrottledReader.java:35)
> at
> org.apache.cassandra.io.util.ThrottledReader.open(ThrottledReader.java:49)
> ... 17 more
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (CASSANDRA-10345) incremental_repair_test.TestIncRepair.sstable_marking_test fails on cassandra-3.0
Yuki Morishita created CASSANDRA-10345: -- Summary: incremental_repair_test.TestIncRepair.sstable_marking_test fails on cassandra-3.0 Key: CASSANDRA-10345 URL: https://issues.apache.org/jira/browse/CASSANDRA-10345 Project: Cassandra Issue Type: Bug Reporter: Yuki Morishita Assignee: Yuki Morishita Priority: Minor Fix For: 3.0.x incremental_repair_test.TestIncRepair.sstable_marking_test is failing often on cassandra-3.0. http://cassci.datastax.com/view/cassandra-3.0/job/cassandra-3.0_dtest/lastCompletedBuild/testReport/incremental_repair_test/TestIncRepair/sstable_marking_test/history/ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9664) Allow MV's select statements to be more complex
[
https://issues.apache.org/jira/browse/CASSANDRA-9664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14745745#comment-14745745
]
Tyler Hobbs commented on CASSANDRA-9664:
Alright, I've rebased again on the latest from 9921 in another [new
branch|https://github.com/thobbs/cassandra/tree/9664-rebase-2]. The problem
with {{ALTER TABLE}} still exists, so there are still a couple of commented out
unit tests. However, this should be good enough for [~slebresne] to start code
review.
> Allow MV's select statements to be more complex
> ---
>
> Key: CASSANDRA-9664
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9664
> Project: Cassandra
> Issue Type: New Feature
>Reporter: Carl Yeksigian
>Assignee: Tyler Hobbs
> Labels: client-impacting, doc-impacting
> Fix For: 3.0.0 rc1
>
>
> [Materialized Views|https://issues.apache.org/jira/browse/CASSANDRA-6477] add
> support for a syntax which includes a {{SELECT}} statement, but only allows
> selection of direct columns, and does not allow any filtering to take place.
> We should add support to the MV {{SELECT}} statement to bring better parity
> with the normal CQL {{SELECT}} statement, specifically simple functions in
> the selected columns, as well as specifying a {{WHERE}} clause.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-10328) Inconsistent Schema Change Events Between Table and View
[
https://issues.apache.org/jira/browse/CASSANDRA-10328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14745767#comment-14745767
]
Tyler Hobbs commented on CASSANDRA-10328:
-
CASSANDRA-9996 is already open to remove the extra "keyspace updated" events.
It looks like we just need to fix the event when a view is dropped here? I'm
assigning this to [~Stefania] since she's already done some work around pushed
notifications.
> Inconsistent Schema Change Events Between Table and View
>
>
> Key: CASSANDRA-10328
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10328
> Project: Cassandra
> Issue Type: Bug
> Components: Core
>Reporter: Adam Holmberg
>
> I'm seeing inconsistent event delivery when it comes to dropping materialized
> views (when compared to tables or indexes).
> For example, create/drop/alter for a table:
> {code}
> cassandra@cqlsh:test> create TABLE t (k int PRIMARY KEY , v int);
> event_args={'keyspace': u'test', 'change_type': u'UPDATED', 'target_type':
> u'KEYSPACE'}, stream_id=-1)>
> event_args={'keyspace': u'test', 'change_type': u'CREATED', 'target_type':
> u'TABLE', u'table': u't'}, stream_id=-1)>
> cassandra@cqlsh:test> alter TABLE t add v1 int;
> event_args={'keyspace': u'test', 'change_type': u'UPDATED', 'target_type':
> u'KEYSPACE'}, stream_id=-1)>
> event_args={'keyspace': u'test', 'change_type': u'UPDATED', 'target_type':
> u'TABLE', u'table': u't'}, stream_id=-1)>
> cassandra@cqlsh:test> drop TABLE t;
> event_args={'keyspace': u'test', 'change_type': u'UPDATED', 'target_type':
> u'KEYSPACE'}, stream_id=-1)>
> event_args={'keyspace': u'test', 'change_type': u'DROPPED', 'target_type':
> u'TABLE', u'table': u't'}, stream_id=-1)>
> {code}
> And for a view:
> {code}
> cassandra@cqlsh:test> create MATERIALIZED VIEW mv as select * from scores
> WHERE game IS NOT NULL AND score IS NOT NULL AND user IS NOT NULL AND year IS
> NOT NULL AND month IS NOT NULL AND day IS NOT NULLPRIMARY KEY (game,
> user, year, month, day, score)WITH CLUSTERING ORDER BY (score desc);
> event_args={'keyspace': u'test', 'change_type': u'UPDATED', 'target_type':
> u'KEYSPACE'}, stream_id=-1)>
> event_args={'keyspace': u'test', 'change_type': u'CREATED', 'target_type':
> u'TABLE', u'table': u'mv'}, stream_id=-1)>
> cassandra@cqlsh:test> alter materialized view mv with min_index_interval =
> 100;
> event_args={'keyspace': u'test', 'change_type': u'UPDATED', 'target_type':
> u'KEYSPACE'}, stream_id=-1)>
> event_args={'keyspace': u'test', 'change_type': u'UPDATED', 'target_type':
> u'TABLE', u'table': u'mv'}, stream_id=-1)>
> cassandra@cqlsh:test> drop MATERIALIZED VIEW mv;
> event_args={'keyspace': u'test', 'change_type': u'UPDATED', 'target_type':
> u'KEYSPACE'}, stream_id=-1)>
> {code}
> The latter sequence is missing a table update event, meaning clients cannot
> tell that a view was dropped.
> This is on a [branch
> in-progress|https://github.com/iamaleksey/cassandra/commits/9921-3.0] for
> CASSANDRA-9921
> As a side note, I also believe they keyspace update events are unnecessary in
> both scenarios. To my knowledge, drivers only use these events to refresh
> meta on the keyspace definition itself, not the entities it contains. Please
> let me know if this is worthy of discussion or a distinct ticket.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (CASSANDRA-10328) Inconsistent Schema Change Events Between Table and View
[
https://issues.apache.org/jira/browse/CASSANDRA-10328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tyler Hobbs updated CASSANDRA-10328:
Assignee: Stefania
> Inconsistent Schema Change Events Between Table and View
>
>
> Key: CASSANDRA-10328
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10328
> Project: Cassandra
> Issue Type: Bug
> Components: Core
>Reporter: Adam Holmberg
>Assignee: Stefania
>
> I'm seeing inconsistent event delivery when it comes to dropping materialized
> views (when compared to tables or indexes).
> For example, create/drop/alter for a table:
> {code}
> cassandra@cqlsh:test> create TABLE t (k int PRIMARY KEY , v int);
> event_args={'keyspace': u'test', 'change_type': u'UPDATED', 'target_type':
> u'KEYSPACE'}, stream_id=-1)>
> event_args={'keyspace': u'test', 'change_type': u'CREATED', 'target_type':
> u'TABLE', u'table': u't'}, stream_id=-1)>
> cassandra@cqlsh:test> alter TABLE t add v1 int;
> event_args={'keyspace': u'test', 'change_type': u'UPDATED', 'target_type':
> u'KEYSPACE'}, stream_id=-1)>
> event_args={'keyspace': u'test', 'change_type': u'UPDATED', 'target_type':
> u'TABLE', u'table': u't'}, stream_id=-1)>
> cassandra@cqlsh:test> drop TABLE t;
> event_args={'keyspace': u'test', 'change_type': u'UPDATED', 'target_type':
> u'KEYSPACE'}, stream_id=-1)>
> event_args={'keyspace': u'test', 'change_type': u'DROPPED', 'target_type':
> u'TABLE', u'table': u't'}, stream_id=-1)>
> {code}
> And for a view:
> {code}
> cassandra@cqlsh:test> create MATERIALIZED VIEW mv as select * from scores
> WHERE game IS NOT NULL AND score IS NOT NULL AND user IS NOT NULL AND year IS
> NOT NULL AND month IS NOT NULL AND day IS NOT NULLPRIMARY KEY (game,
> user, year, month, day, score)WITH CLUSTERING ORDER BY (score desc);
> event_args={'keyspace': u'test', 'change_type': u'UPDATED', 'target_type':
> u'KEYSPACE'}, stream_id=-1)>
> event_args={'keyspace': u'test', 'change_type': u'CREATED', 'target_type':
> u'TABLE', u'table': u'mv'}, stream_id=-1)>
> cassandra@cqlsh:test> alter materialized view mv with min_index_interval =
> 100;
> event_args={'keyspace': u'test', 'change_type': u'UPDATED', 'target_type':
> u'KEYSPACE'}, stream_id=-1)>
> event_args={'keyspace': u'test', 'change_type': u'UPDATED', 'target_type':
> u'TABLE', u'table': u'mv'}, stream_id=-1)>
> cassandra@cqlsh:test> drop MATERIALIZED VIEW mv;
> event_args={'keyspace': u'test', 'change_type': u'UPDATED', 'target_type':
> u'KEYSPACE'}, stream_id=-1)>
> {code}
> The latter sequence is missing a table update event, meaning clients cannot
> tell that a view was dropped.
> This is on a [branch
> in-progress|https://github.com/iamaleksey/cassandra/commits/9921-3.0] for
> CASSANDRA-9921
> As a side note, I also believe they keyspace update events are unnecessary in
> both scenarios. To my knowledge, drivers only use these events to refresh
> meta on the keyspace definition itself, not the entities it contains. Please
> let me know if this is worthy of discussion or a distinct ticket.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-10007) Repeated rows in paged result
[ https://issues.apache.org/jira/browse/CASSANDRA-10007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14745772#comment-14745772 ] Benjamin Lerer commented on CASSANDRA-10007: Thanks for looking at it. > Repeated rows in paged result > - > > Key: CASSANDRA-10007 > URL: https://issues.apache.org/jira/browse/CASSANDRA-10007 > Project: Cassandra > Issue Type: Bug > Components: Core >Reporter: Adam Holmberg >Assignee: Benjamin Lerer > Labels: client-impacting > Fix For: 3.x > > Attachments: paging-test.py > > > We noticed an anomaly in paged results while testing against 3.0.0-alpha1. It > seems that unbounded selects can return rows repeated at page boundaries. > Furthermore, the number of repeated rows seems to dither in count across > consecutive runs of the same query. > Does not reproduce on 2.2.0 and earlier. > I also noted that this behavior only manifests on multi-node clusters. > The attached script shows this behavior when run against 3.0.0-alpha1. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9632) Preserve the Names of Query Parameters in QueryOptions
[
https://issues.apache.org/jira/browse/CASSANDRA-9632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14745773#comment-14745773
]
Sam Tunnicliffe commented on CASSANDRA-9632:
+1 the new patch LGTM (sorry it took so long to review).
The only comment I'd make would be that you should probably rebase & push
2.2/3.0/trunk versions for CI before committing.
> Preserve the Names of Query Parameters in QueryOptions
> --
>
> Key: CASSANDRA-9632
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9632
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
>Reporter: stephen mallette
>Assignee: Benjamin Lerer
>Priority: Minor
> Fix For: 2.2.x
>
> Attachments: 9632-2.2-V2.txt, 9632-2.2.txt
>
>
> When writing a custom {{QueryHandler}} that processes named parameters, the
> {{QueryOptions}} arrive to the various processing methods with the values
> converted to positional arguments and the names unavailable. A custom
> {{QueryHandler}} might want to make use of the names themselves so it would
> be helpful if they were preserved and exposed with their respective
> {{ByteBuffer}} values.
> https://github.com/apache/cassandra/blob/cassandra-2.1/src/java/org/apache/cassandra/cql3/QueryOptions.java#L205
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (CASSANDRA-10007) Repeated rows in paged result
[ https://issues.apache.org/jira/browse/CASSANDRA-10007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benjamin Lerer resolved CASSANDRA-10007. Resolution: Cannot Reproduce Fix Version/s: (was: 3.x) > Repeated rows in paged result > - > > Key: CASSANDRA-10007 > URL: https://issues.apache.org/jira/browse/CASSANDRA-10007 > Project: Cassandra > Issue Type: Bug > Components: Core >Reporter: Adam Holmberg >Assignee: Benjamin Lerer > Labels: client-impacting > Attachments: paging-test.py > > > We noticed an anomaly in paged results while testing against 3.0.0-alpha1. It > seems that unbounded selects can return rows repeated at page boundaries. > Furthermore, the number of repeated rows seems to dither in count across > consecutive runs of the same query. > Does not reproduce on 2.2.0 and earlier. > I also noted that this behavior only manifests on multi-node clusters. > The attached script shows this behavior when run against 3.0.0-alpha1. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[2/3] cassandra git commit: fix bad rebase
fix bad rebase
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/06336377
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/06336377
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/06336377
Branch: refs/heads/trunk
Commit: 06336377f7f0dc064a9882424c82a9f47586a3da
Parents: b263af9
Author: Benedict Elliott Smith
Authored: Tue Sep 15 18:05:29 2015 +0100
Committer: Benedict Elliott Smith
Committed: Tue Sep 15 18:05:29 2015 +0100
--
.../apache/cassandra/db/rows/RowBuilder.java| 5 ++---
.../org/apache/cassandra/db/rows/RowsTest.java | 22 ++--
2 files changed, 13 insertions(+), 14 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/06336377/test/unit/org/apache/cassandra/db/rows/RowBuilder.java
--
diff --git a/test/unit/org/apache/cassandra/db/rows/RowBuilder.java
b/test/unit/org/apache/cassandra/db/rows/RowBuilder.java
index caa5c40..b1223f1 100644
--- a/test/unit/org/apache/cassandra/db/rows/RowBuilder.java
+++ b/test/unit/org/apache/cassandra/db/rows/RowBuilder.java
@@ -36,7 +36,7 @@ public class RowBuilder implements Row.Builder
public List cells = new LinkedList<>();
public Clustering clustering = null;
public LivenessInfo livenessInfo = null;
-public DeletionTime deletionTime = null;
+public Row.Deletion deletionTime = null;
public List> complexDeletions = new
LinkedList<>();
public void addCell(Cell cell)
@@ -66,13 +66,12 @@ public class RowBuilder implements Row.Builder
livenessInfo = info;
}
-public void addRowDeletion(DeletionTime deletion)
+public void addRowDeletion(Row.Deletion deletion)
{
assert deletionTime == null;
deletionTime = deletion;
}
-
public void addComplexDeletion(ColumnDefinition column, DeletionTime
complexDeletion)
{
complexDeletions.add(Pair.create(column, complexDeletion));
http://git-wip-us.apache.org/repos/asf/cassandra/blob/06336377/test/unit/org/apache/cassandra/db/rows/RowsTest.java
--
diff --git a/test/unit/org/apache/cassandra/db/rows/RowsTest.java
b/test/unit/org/apache/cassandra/db/rows/RowsTest.java
index 306d687..7ee2d0a 100644
--- a/test/unit/org/apache/cassandra/db/rows/RowsTest.java
+++ b/test/unit/org/apache/cassandra/db/rows/RowsTest.java
@@ -149,8 +149,8 @@ public class RowsTest
updates++;
}
-List> deletions = new LinkedList<>();
-public void onDeletion(int i, Clustering clustering, DeletionTime
merged, DeletionTime original)
+List> deletions = new LinkedList<>();
+public void onDeletion(int i, Clustering clustering, Row.Deletion
merged, Row.Deletion original)
{
updateClustering(clustering);
deletions.add(MergedPair.create(i, merged, original));
@@ -240,7 +240,7 @@ public class RowsTest
BufferCell.live(kcvm, m,
secondToTs(now), BB1, CellPath.create(BB1)),
BufferCell.live(kcvm, m,
secondToTs(now), BB2, CellPath.create(BB2)));
expectedCells.forEach(originalBuilder::addCell);
-DeletionTime rowDeletion = new DeletionTime(ts, now);
+Row.Deletion rowDeletion = new Row.Deletion(new DeletionTime(ts, now),
false);
originalBuilder.addRowDeletion(rowDeletion);
RowBuilder builder = new RowBuilder();
@@ -268,14 +268,14 @@ public class RowsTest
BufferCell.live(kcvm, m,
ts, BB1, CellPath.create(BB1)),
BufferCell.live(kcvm, m,
ts, BB2, CellPath.create(BB2)));
expectedCells.forEach(builder::addCell);
-DeletionTime rowDeletion = new DeletionTime(ts, now);
+Row.Deletion rowDeletion = new Row.Deletion(new DeletionTime(ts, now),
false);
builder.addRowDeletion(rowDeletion);
StatsCollector collector = new StatsCollector();
Rows.collectStats(builder.build(), collector);
Assert.assertEquals(Lists.newArrayList(liveness), collector.liveness);
-Assert.assertEquals(Sets.newHashSet(rowDeletion, complexDeletion),
Sets.newHashSet(collector.deletions));
+Assert.assertEquals(Sets.newHashSet(rowDeletion.time(),
complexDeletion), Sets.newHashSet(collector.deletions));
Assert.assertEquals(Sets.newHashSet(expectedCells),
Sets.newHashSet(collector.cells));
Assert.assertEquals(2, collector.columnCount);
Assert.assertFalse(collector.hasLegacyCounterShards);
@@ -322,7 +322,7 @@ public class RowsTest
List r2E
[3/3] cassandra git commit: Merge branch 'cassandra-3.0' into trunk
Merge branch 'cassandra-3.0' into trunk Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/7a0698ee Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/7a0698ee Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/7a0698ee Branch: refs/heads/trunk Commit: 7a0698ee1bbcef4694a80e98a5cbcefbd59ed5f7 Parents: ac3b1cc 0633637 Author: Benedict Elliott Smith Authored: Tue Sep 15 18:05:35 2015 +0100 Committer: Benedict Elliott Smith Committed: Tue Sep 15 18:05:35 2015 +0100 -- .../apache/cassandra/db/rows/RowBuilder.java| 5 ++--- .../org/apache/cassandra/db/rows/RowsTest.java | 22 ++-- 2 files changed, 13 insertions(+), 14 deletions(-) --
[1/3] cassandra git commit: fix bad rebase
Repository: cassandra
Updated Branches:
refs/heads/cassandra-3.0 b263af93a -> 06336377f
refs/heads/trunk ac3b1cc2c -> 7a0698ee1
fix bad rebase
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/06336377
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/06336377
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/06336377
Branch: refs/heads/cassandra-3.0
Commit: 06336377f7f0dc064a9882424c82a9f47586a3da
Parents: b263af9
Author: Benedict Elliott Smith
Authored: Tue Sep 15 18:05:29 2015 +0100
Committer: Benedict Elliott Smith
Committed: Tue Sep 15 18:05:29 2015 +0100
--
.../apache/cassandra/db/rows/RowBuilder.java| 5 ++---
.../org/apache/cassandra/db/rows/RowsTest.java | 22 ++--
2 files changed, 13 insertions(+), 14 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/06336377/test/unit/org/apache/cassandra/db/rows/RowBuilder.java
--
diff --git a/test/unit/org/apache/cassandra/db/rows/RowBuilder.java
b/test/unit/org/apache/cassandra/db/rows/RowBuilder.java
index caa5c40..b1223f1 100644
--- a/test/unit/org/apache/cassandra/db/rows/RowBuilder.java
+++ b/test/unit/org/apache/cassandra/db/rows/RowBuilder.java
@@ -36,7 +36,7 @@ public class RowBuilder implements Row.Builder
public List cells = new LinkedList<>();
public Clustering clustering = null;
public LivenessInfo livenessInfo = null;
-public DeletionTime deletionTime = null;
+public Row.Deletion deletionTime = null;
public List> complexDeletions = new
LinkedList<>();
public void addCell(Cell cell)
@@ -66,13 +66,12 @@ public class RowBuilder implements Row.Builder
livenessInfo = info;
}
-public void addRowDeletion(DeletionTime deletion)
+public void addRowDeletion(Row.Deletion deletion)
{
assert deletionTime == null;
deletionTime = deletion;
}
-
public void addComplexDeletion(ColumnDefinition column, DeletionTime
complexDeletion)
{
complexDeletions.add(Pair.create(column, complexDeletion));
http://git-wip-us.apache.org/repos/asf/cassandra/blob/06336377/test/unit/org/apache/cassandra/db/rows/RowsTest.java
--
diff --git a/test/unit/org/apache/cassandra/db/rows/RowsTest.java
b/test/unit/org/apache/cassandra/db/rows/RowsTest.java
index 306d687..7ee2d0a 100644
--- a/test/unit/org/apache/cassandra/db/rows/RowsTest.java
+++ b/test/unit/org/apache/cassandra/db/rows/RowsTest.java
@@ -149,8 +149,8 @@ public class RowsTest
updates++;
}
-List> deletions = new LinkedList<>();
-public void onDeletion(int i, Clustering clustering, DeletionTime
merged, DeletionTime original)
+List> deletions = new LinkedList<>();
+public void onDeletion(int i, Clustering clustering, Row.Deletion
merged, Row.Deletion original)
{
updateClustering(clustering);
deletions.add(MergedPair.create(i, merged, original));
@@ -240,7 +240,7 @@ public class RowsTest
BufferCell.live(kcvm, m,
secondToTs(now), BB1, CellPath.create(BB1)),
BufferCell.live(kcvm, m,
secondToTs(now), BB2, CellPath.create(BB2)));
expectedCells.forEach(originalBuilder::addCell);
-DeletionTime rowDeletion = new DeletionTime(ts, now);
+Row.Deletion rowDeletion = new Row.Deletion(new DeletionTime(ts, now),
false);
originalBuilder.addRowDeletion(rowDeletion);
RowBuilder builder = new RowBuilder();
@@ -268,14 +268,14 @@ public class RowsTest
BufferCell.live(kcvm, m,
ts, BB1, CellPath.create(BB1)),
BufferCell.live(kcvm, m,
ts, BB2, CellPath.create(BB2)));
expectedCells.forEach(builder::addCell);
-DeletionTime rowDeletion = new DeletionTime(ts, now);
+Row.Deletion rowDeletion = new Row.Deletion(new DeletionTime(ts, now),
false);
builder.addRowDeletion(rowDeletion);
StatsCollector collector = new StatsCollector();
Rows.collectStats(builder.build(), collector);
Assert.assertEquals(Lists.newArrayList(liveness), collector.liveness);
-Assert.assertEquals(Sets.newHashSet(rowDeletion, complexDeletion),
Sets.newHashSet(collector.deletions));
+Assert.assertEquals(Sets.newHashSet(rowDeletion.time(),
complexDeletion), Sets.newHashSet(collector.deletions));
Assert.assertEquals(Sets.newHashSet(expectedCells),
Sets.newHashSet(collector.cells));
Assert.assertEquals(2, coll
cassandra git commit: Update cqlsh COPY for new internal driver serialization interface
Repository: cassandra Updated Branches: refs/heads/cassandra-3.0 06336377f -> b4544846d Update cqlsh COPY for new internal driver serialization interface Patch by Stefania Alborghetti; reviewed by Adam Holmberg for CASSANDRA-10318 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/b4544846 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/b4544846 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/b4544846 Branch: refs/heads/cassandra-3.0 Commit: b4544846def2bdd00ff841c7e3d9f2559620827b Parents: 0633637 Author: Stefania Alborghetti Authored: Tue Sep 15 12:11:09 2015 +0800 Committer: T Jake Luciani Committed: Tue Sep 15 13:26:47 2015 -0400 -- CHANGES.txt | 1 + bin/cqlsh.py | 7 +-- 2 files changed, 2 insertions(+), 6 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/b4544846/CHANGES.txt -- diff --git a/CHANGES.txt b/CHANGES.txt index 54d4ddf..36ae689 100644 --- a/CHANGES.txt +++ b/CHANGES.txt @@ -1,4 +1,5 @@ 3.0.0-rc1 + * Update cqlsh COPY for new internal driver serialization interface (CASSANDRA-10318) * Give index implementations more control over rebuild operations (CASSANDRA-10312) * Update index file format (CASSANDRA-10314) * Add "shadowable" row tombstones to deal with mv timestamp issues (CASSANDRA-10261) http://git-wip-us.apache.org/repos/asf/cassandra/blob/b4544846/bin/cqlsh.py -- diff --git a/bin/cqlsh.py b/bin/cqlsh.py index 2636a59..df16371 100644 --- a/bin/cqlsh.py +++ b/bin/cqlsh.py @@ -2388,16 +2388,11 @@ class ImportProcess(multiprocessing.Process): fetch_size=None, paging_state=None, timestamp=insert_timestamp) request_id = conn.get_request_id() -binary_message = query_message.to_binary( -stream_id=request_id, protocol_version=DEFAULT_PROTOCOL_VERSION, compression=None) +conn.send_msg(query_message, request_id=request_id, cb=partial(callback, current_record)) -# add the message directly to the connection's queue with conn.lock: conn.in_flight += 1 -conn._callbacks[request_id] = partial(callback, current_record) -conn.deque.append(binary_message) - # every 50 records, clear the pending writes queue and read # any responses we have if insert_num % 50 == 0:
