[jira] [Updated] (CASSANDRA-8072) Exception during startup: Unable to gossip with any seeds

[Sam Tunnicliffe (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-8072?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sam Tunnicliffe updated CASSANDRA-8072:
---
Reproduced In: 2.1.3, 2.0.11, 2.0.10  (was: 2.0.10, 2.0.11, 2.1.3)
 Reviewer: Sam Tunnicliffe

> Exception during startup: Unable to gossip with any seeds
> -
>
> Key: CASSANDRA-8072
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8072
> Project: Cassandra
>  Issue Type: Bug
>  Components: Lifecycle
>Reporter: Ryan Springer
>Assignee: Stefania
> Fix For: 2.1.x
>
> Attachments: cas-dev-dt-01-uw1-cassandra-seed01_logs.tar.bz2, 
> cas-dev-dt-01-uw1-cassandra-seed02_logs.tar.bz2, 
> cas-dev-dt-01-uw1-cassandra02_logs.tar.bz2, 
> casandra-system-log-with-assert-patch.log, screenshot-1.png, 
> trace_logs.tar.bz2
>
>
> When Opscenter 4.1.4 or 5.0.1 tries to provision a 2-node DSC 2.0.10 cluster 
> in either ec2 or locally, an error occurs sometimes with one of the nodes 
> refusing to start C*.  The error in the /var/log/cassandra/system.log is:
> ERROR [main] 2014-10-06 15:54:52,292 CassandraDaemon.java (line 513) 
> Exception encountered during startup
> java.lang.RuntimeException: Unable to gossip with any seeds
> at org.apache.cassandra.gms.Gossiper.doShadowRound(Gossiper.java:1200)
> at 
> org.apache.cassandra.service.StorageService.checkForEndpointCollision(StorageService.java:444)
> at 
> org.apache.cassandra.service.StorageService.prepareToJoin(StorageService.java:655)
> at 
> org.apache.cassandra.service.StorageService.initServer(StorageService.java:609)
> at 
> org.apache.cassandra.service.StorageService.initServer(StorageService.java:502)
> at 
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:378)
> at 
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:496)
> at 
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:585)
>  INFO [StorageServiceShutdownHook] 2014-10-06 15:54:52,326 Gossiper.java 
> (line 1279) Announcing shutdown
>  INFO [StorageServiceShutdownHook] 2014-10-06 15:54:54,326 
> MessagingService.java (line 701) Waiting for messaging service to quiesce
>  INFO [ACCEPT-localhost/127.0.0.1] 2014-10-06 15:54:54,327 
> MessagingService.java (line 941) MessagingService has terminated the accept() 
> thread
> This errors does not always occur when provisioning a 2-node cluster, but 
> probably around half of the time on only one of the nodes.  I haven't been 
> able to reproduce this error with DSC 2.0.9, and there have been no code or 
> definition file changes in Opscenter.
> I can reproduce locally with the above steps.  I'm happy to test any proposed 
> fixes since I'm the only person able to reproduce reliably so far.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-9303) Match cassandra-loader options in COPY FROM

[Stefania (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-9303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15085424#comment-15085424
 ] 

Stefania commented on CASSANDRA-9303:
-

dtests are fine now, after rebasing the dtest branch as well.

However 2.2+ branches changed overnight, so I've rebased again and restarted CI 
for 2.2+.

In the end to make life easier, I converted the merge commits into simple 
commits so the rebase can be done without having to re-resolve the original 
merge conflicts.

> Match cassandra-loader options in COPY FROM
> ---
>
> Key: CASSANDRA-9303
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9303
> Project: Cassandra
>  Issue Type: New Feature
>  Components: Tools
>Reporter: Jonathan Ellis
>Assignee: Stefania
>Priority: Critical
> Fix For: 2.1.x, 2.2.x, 3.0.x, 3.x
>
> Attachments: dtest.out
>
>
> https://github.com/brianmhess/cassandra-loader added a bunch of options to 
> handle real world requirements, we should match those.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[4/6] cassandra git commit: Merge branch 'cassandra-3.0' into cassandra-3.3

[slebresne]

Merge branch 'cassandra-3.0' into cassandra-3.3


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/10c1e65d
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/10c1e65d
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/10c1e65d

Branch: refs/heads/trunk
Commit: 10c1e65db81a0c24081562c25d07f758ae6e3045
Parents: bd2785e 12b6c0a
Author: Sylvain Lebresne 
Authored: Wed Jan 6 10:44:36 2016 +0100
Committer: Sylvain Lebresne 
Committed: Wed Jan 6 10:44:36 2016 +0100

--
 src/java/org/apache/cassandra/db/rows/Row.java | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
--

[5/6] cassandra git commit: Merge branch 'cassandra-3.0' into cassandra-3.3

[slebresne]

Merge branch 'cassandra-3.0' into cassandra-3.3


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/10c1e65d
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/10c1e65d
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/10c1e65d

Branch: refs/heads/cassandra-3.3
Commit: 10c1e65db81a0c24081562c25d07f758ae6e3045
Parents: bd2785e 12b6c0a
Author: Sylvain Lebresne 
Authored: Wed Jan 6 10:44:36 2016 +0100
Committer: Sylvain Lebresne 
Committed: Wed Jan 6 10:44:36 2016 +0100

--
 src/java/org/apache/cassandra/db/rows/Row.java | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
--

[2/6] cassandra git commit: Fix typo in javadoc

[slebresne]

Fix typo in javadoc


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/12b6c0a5
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/12b6c0a5
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/12b6c0a5

Branch: refs/heads/cassandra-3.3
Commit: 12b6c0a5264138a6f569e5e957362ddaf1a4a0be
Parents: 3e45fa1
Author: Sylvain Lebresne 
Authored: Wed Jan 6 10:44:22 2016 +0100
Committer: Sylvain Lebresne 
Committed: Wed Jan 6 10:44:22 2016 +0100

--
 src/java/org/apache/cassandra/db/rows/Row.java | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/12b6c0a5/src/java/org/apache/cassandra/db/rows/Row.java
--
diff --git a/src/java/org/apache/cassandra/db/rows/Row.java 
b/src/java/org/apache/cassandra/db/rows/Row.java
index 8a67e9b..5f79a66 100644
--- a/src/java/org/apache/cassandra/db/rows/Row.java
+++ b/src/java/org/apache/cassandra/db/rows/Row.java
@@ -394,11 +394,11 @@ public interface Row extends Unfiltered, 
Collection
 public Clustering clustering();
 
 /**
- * Adds the liveness information for the partition key columns of this 
row.
+ * Adds the liveness information for the primary key columns of this 
row.
  *
  * This call is optional (skipping it is equivalent to calling {@code 
addPartitionKeyLivenessInfo(LivenessInfo.NONE)}).
  *
- * @param info the liveness information for the partition key columns 
of the built row.
+ * @param info the liveness information for the primary key columns of 
the built row.
  */
 public void addPrimaryKeyLivenessInfo(LivenessInfo info);
 

[1/6] cassandra git commit: Fix typo in javadoc

[slebresne]

Repository: cassandra
Updated Branches:
  refs/heads/cassandra-3.0 3e45fa1ab -> 12b6c0a52
  refs/heads/cassandra-3.3 bd2785e9c -> 10c1e65db
  refs/heads/trunk 650d76d71 -> cfe31c3b9


Fix typo in javadoc


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/12b6c0a5
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/12b6c0a5
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/12b6c0a5

Branch: refs/heads/cassandra-3.0
Commit: 12b6c0a5264138a6f569e5e957362ddaf1a4a0be
Parents: 3e45fa1
Author: Sylvain Lebresne 
Authored: Wed Jan 6 10:44:22 2016 +0100
Committer: Sylvain Lebresne 
Committed: Wed Jan 6 10:44:22 2016 +0100

--
 src/java/org/apache/cassandra/db/rows/Row.java | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/12b6c0a5/src/java/org/apache/cassandra/db/rows/Row.java
--
diff --git a/src/java/org/apache/cassandra/db/rows/Row.java 
b/src/java/org/apache/cassandra/db/rows/Row.java
index 8a67e9b..5f79a66 100644
--- a/src/java/org/apache/cassandra/db/rows/Row.java
+++ b/src/java/org/apache/cassandra/db/rows/Row.java
@@ -394,11 +394,11 @@ public interface Row extends Unfiltered, 
Collection
 public Clustering clustering();
 
 /**
- * Adds the liveness information for the partition key columns of this 
row.
+ * Adds the liveness information for the primary key columns of this 
row.
  *
  * This call is optional (skipping it is equivalent to calling {@code 
addPartitionKeyLivenessInfo(LivenessInfo.NONE)}).
  *
- * @param info the liveness information for the partition key columns 
of the built row.
+ * @param info the liveness information for the primary key columns of 
the built row.
  */
 public void addPrimaryKeyLivenessInfo(LivenessInfo info);
 

[3/6] cassandra git commit: Fix typo in javadoc

[slebresne]

Fix typo in javadoc


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/12b6c0a5
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/12b6c0a5
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/12b6c0a5

Branch: refs/heads/trunk
Commit: 12b6c0a5264138a6f569e5e957362ddaf1a4a0be
Parents: 3e45fa1
Author: Sylvain Lebresne 
Authored: Wed Jan 6 10:44:22 2016 +0100
Committer: Sylvain Lebresne 
Committed: Wed Jan 6 10:44:22 2016 +0100

--
 src/java/org/apache/cassandra/db/rows/Row.java | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/12b6c0a5/src/java/org/apache/cassandra/db/rows/Row.java
--
diff --git a/src/java/org/apache/cassandra/db/rows/Row.java 
b/src/java/org/apache/cassandra/db/rows/Row.java
index 8a67e9b..5f79a66 100644
--- a/src/java/org/apache/cassandra/db/rows/Row.java
+++ b/src/java/org/apache/cassandra/db/rows/Row.java
@@ -394,11 +394,11 @@ public interface Row extends Unfiltered, 
Collection
 public Clustering clustering();
 
 /**
- * Adds the liveness information for the partition key columns of this 
row.
+ * Adds the liveness information for the primary key columns of this 
row.
  *
  * This call is optional (skipping it is equivalent to calling {@code 
addPartitionKeyLivenessInfo(LivenessInfo.NONE)}).
  *
- * @param info the liveness information for the partition key columns 
of the built row.
+ * @param info the liveness information for the primary key columns of 
the built row.
  */
 public void addPrimaryKeyLivenessInfo(LivenessInfo info);
 

[6/6] cassandra git commit: Merge branch 'cassandra-3.3' into trunk

[slebresne]

Merge branch 'cassandra-3.3' into trunk


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/cfe31c3b
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/cfe31c3b
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/cfe31c3b

Branch: refs/heads/trunk
Commit: cfe31c3b9128f2959dc9dd827de6f29fe0da1010
Parents: 650d76d 10c1e65
Author: Sylvain Lebresne 
Authored: Wed Jan 6 10:44:45 2016 +0100
Committer: Sylvain Lebresne 
Committed: Wed Jan 6 10:44:45 2016 +0100

--
 src/java/org/apache/cassandra/db/rows/Row.java | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
--

[jira] [Commented] (CASSANDRA-10887) Pending range calculator gives wrong pending ranges for moves

[Branimir Lambov (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-10887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15085308#comment-15085308
 ] 

Branimir Lambov commented on CASSANDRA-10887:
-

Patch looks good, 
[utests|http://cassci.datastax.com/job/blambov-kohlisankalp-10887-testall/3/] 
look good. I couldn't make sense of the 
[dtests|http://cassci.datastax.com/job/blambov-kohlisankalp-10887-dtest/3/] 
though.
I pushed a 2.1 version 
[here|https://github.com/blambov/cassandra/tree/kohlisankalp/10887-2.1] to run 
more stable dtests, and the results there 
([utest|http://cassci.datastax.com/job/blambov-kohlisankalp-10887-testall/3/], 
[dtest|http://cassci.datastax.com/job/blambov-kohlisankalp-10887-2.1-dtest/1/]) 
appear fine.

The patch is ready to commit, but does not apply cleanly to later versions, 
could you make patches for 2.2 (and 3.0 if necessary)?

> Pending range calculator gives wrong pending ranges for moves
> -
>
> Key: CASSANDRA-10887
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10887
> Project: Cassandra
>  Issue Type: Bug
>  Components: Coordination
>Reporter: Richard Low
>Assignee: sankalp kohli
>Priority: Critical
> Fix For: 2.1.x, 2.2.x, 3.0.x, 3.x
>
> Attachments: CASSANDRA-10887.diff, CASSANDRA_10887_v2.diff, 
> CASSANDRA_10887_v3.diff
>
>
> My understanding is the PendingRangeCalculator is meant to calculate who 
> should receive extra writes during range movements. However, it adds the 
> wrong ranges for moves. An extreme example of this can be seen in the 
> following reproduction. Create a 5 node cluster (I did this on 2.0.16 and 
> 2.2.4) and a keyspace RF=3 and a simple table. Then start moving a node and 
> immediately kill -9 it. Now you see a node as down and moving in the ring. 
> Try a quorum write for a partition that is stored on that node - it will fail 
> with a timeout. Further, all CAS reads or writes fail immediately with 
> unavailable exception because they attempt to include the moving node twice. 
> This is likely to be the cause of CASSANDRA-10423.
> In my example I had this ring:
> 127.0.0.1  rack1   Up Normal  170.97 KB   20.00%  
> -9223372036854775808
> 127.0.0.2  rack1   Up Normal  124.06 KB   20.00%  
> -5534023222112865485
> 127.0.0.3  rack1   Down   Moving  108.7 KB40.00%  
> 1844674407370955160
> 127.0.0.4  rack1   Up Normal  142.58 KB   0.00%   
> 1844674407370955161
> 127.0.0.5  rack1   Up Normal  118.64 KB   20.00%  
> 5534023222112865484
> Node 3 was moving to -1844674407370955160. I added logging to print the 
> pending and natural endpoints. For ranges owned by node 3, node 3 appeared in 
> pending and natural endpoints. The blockFor is increased to 3 so we’re 
> effectively doing CL.ALL operations. This manifests as write timeouts and CAS 
> unavailables when the node is down.
> The correct pending range for this scenario is node 1 is gaining the range 
> (-1844674407370955160, 1844674407370955160). So node 1 should be added as a 
> destination for writes and CAS for this range, not node 3.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (CASSANDRA-10970) SSL/TLS: Certificate Domain is ignored

[Matthias Brandt (JIRA)]

Matthias Brandt created CASSANDRA-10970:
---

 Summary: SSL/TLS: Certificate Domain is ignored
 Key: CASSANDRA-10970
 URL: https://issues.apache.org/jira/browse/CASSANDRA-10970
 Project: Cassandra
  Issue Type: Bug
Reporter: Matthias Brandt


I've set up server_encryption_options as well as client_encryption_options. In 
both settings, I use the same keystore with an wild-card SSL certificate in it. 
It is signed by our own CA, which root certificate is in the configured 
truststore:
{code}
server_encryption_options:
internode_encryption: all
keystore: /etc/cassandra/conf/wildcard-cert.keystore
keystore_password: ""
truststore: /etc/cassandra/conf/hpo-cacerts
truststore_password: changeit
require_client_auth: true

client_encryption_options:
enabled: true
keystore: /etc/cassandra/conf/wildcard-cert.keystore
keystore_password: ""
require_client_auth: false
{code}

The certifcate's subject is:
{code}CN=*.my.domain.com,OU=my unit,O=my org{code}

When I deploy this setting on a server which domain is 
node1.my.*other-domain*.com a connection via cqlsh wrongly works. Additionally, 
the inter-node connection between other nodes in this wrong domain also works.

I would expect that the connection would fail with a meaningful error message.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-9303) Match cassandra-loader options in COPY FROM

[Stefania (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-9303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15085537#comment-15085537
 ] 

Stefania commented on CASSANDRA-9303:
-

CI still OK, ready to commit.

> Match cassandra-loader options in COPY FROM
> ---
>
> Key: CASSANDRA-9303
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9303
> Project: Cassandra
>  Issue Type: New Feature
>  Components: Tools
>Reporter: Jonathan Ellis
>Assignee: Stefania
>Priority: Critical
> Fix For: 2.1.x, 2.2.x, 3.0.x, 3.x
>
> Attachments: dtest.out
>
>
> https://github.com/brianmhess/cassandra-loader added a bunch of options to 
> handle real world requirements, we should match those.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[5/6] cassandra git commit: Merge branch 'cassandra-3.0' into cassandra-3.3

[slebresne]

Merge branch 'cassandra-3.0' into cassandra-3.3


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/7fbc39b7
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/7fbc39b7
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/7fbc39b7

Branch: refs/heads/cassandra-3.3
Commit: 7fbc39b743f7a08cb03f6b83bebe34c1bc342b9a
Parents: 10c1e65 f937c8b
Author: Sylvain Lebresne 
Authored: Wed Jan 6 14:44:45 2016 +0100
Committer: Sylvain Lebresne 
Committed: Wed Jan 6 14:44:45 2016 +0100

--
 conf/cassandra-env.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/7fbc39b7/conf/cassandra-env.sh
--

[6/6] cassandra git commit: Merge branch 'cassandra-3.3' into trunk

[slebresne]

Merge branch 'cassandra-3.3' into trunk


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/786d6767
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/786d6767
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/786d6767

Branch: refs/heads/trunk
Commit: 786d67675cf749c175cd1e6c6b2ef0d1ac620234
Parents: cfe31c3 7fbc39b
Author: Sylvain Lebresne 
Authored: Wed Jan 6 14:44:55 2016 +0100
Committer: Sylvain Lebresne 
Committed: Wed Jan 6 14:44:55 2016 +0100

--
 conf/cassandra-env.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--

[1/6] cassandra git commit: Revert wrong line removal from bad merge

[slebresne]

Repository: cassandra
Updated Branches:
  refs/heads/cassandra-3.0 12b6c0a52 -> f937c8bee
  refs/heads/cassandra-3.3 10c1e65db -> 7fbc39b74
  refs/heads/trunk cfe31c3b9 -> 786d67675


Revert wrong line removal from bad merge


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/f937c8be
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/f937c8be
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/f937c8be

Branch: refs/heads/cassandra-3.0
Commit: f937c8bee87666ac1ba1ac96e0c7803629cc3508
Parents: 12b6c0a
Author: Sylvain Lebresne 
Authored: Wed Jan 6 14:44:29 2016 +0100
Committer: Sylvain Lebresne 
Committed: Wed Jan 6 14:44:29 2016 +0100

--
 conf/cassandra-env.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/f937c8be/conf/cassandra-env.sh
--
diff --git a/conf/cassandra-env.sh b/conf/cassandra-env.sh
index f322803..83fe4c5 100644
--- a/conf/cassandra-env.sh
+++ b/conf/cassandra-env.sh
@@ -33,7 +33,7 @@ calculate_heap_sizes()
 Darwin)
 system_memory_in_bytes=`sysctl hw.memsize | awk '{print $2}'`
 system_memory_in_mb=`expr $system_memory_in_bytes / 1024 / 1024`
-
+system_cpu_cores=`sysctl hw.ncpu | awk '{print $2}'`
 ;;
 *)
 # assume reasonable defaults for e.g. a modern desktop or

[jira] [Comment Edited] (CASSANDRA-9303) Match cassandra-loader options in COPY FROM

[Stefania (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-9303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15085537#comment-15085537
 ] 

Stefania edited comment on CASSANDRA-9303 at 1/6/16 2:08 PM:
-

CI still OK (ignore build #16 for 3.0 which was run with the incorrect dtest 
branch).

Ticket ready to commit.

Note that you will get conflicts up-merging all the way up to 3.2.

The dtest pull request to be merged is 
[here|https://github.com/riptano/cassandra-dtest/pull/724].


was (Author: stefania):
CI still OK, ready to commit.

Note that you will get conflicts up-merging all the way up to 3.2.

The dtest pull request to be merged is 
[here|https://github.com/riptano/cassandra-dtest/pull/724].

> Match cassandra-loader options in COPY FROM
> ---
>
> Key: CASSANDRA-9303
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9303
> Project: Cassandra
>  Issue Type: New Feature
>  Components: Tools
>Reporter: Jonathan Ellis
>Assignee: Stefania
>Priority: Critical
> Fix For: 2.1.x, 2.2.x, 3.0.x, 3.x
>
> Attachments: dtest.out
>
>
> https://github.com/brianmhess/cassandra-loader added a bunch of options to 
> handle real world requirements, we should match those.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-10743) Failed upgradesstables (upgrade from 2.2.2 to 3.0.0)

[Elijah Epifanov (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-10743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15085523#comment-15085523
 ] 

Elijah Epifanov commented on CASSANDRA-10743:
-

for me it's system_distributed keyspace which errs out.

It's DDL:

CREATE KEYSPACE system_distributed WITH replication = {'class': 
'NetworkTopologyStrategy', 'leaseweb': '3'}  AND durable_writes = true;

CREATE TABLE system_distributed.repair_history (
keyspace_name text,
columnfamily_name text,
id timeuuid,
coordinator inet,
exception_message text,
exception_stacktrace text,
finished_at timestamp,
parent_id timeuuid,
participants set,
range_begin text,
range_end text,
started_at timestamp,
status text,
PRIMARY KEY ((keyspace_name, columnfamily_name), id)
) WITH CLUSTERING ORDER BY (id ASC)
AND bloom_filter_fp_chance = 0.01
AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
AND comment = 'Repair history'
AND compaction = {'class': 
'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy', 
'max_threshold': '32', 'min_threshold': '4'}
AND compression = {'chunk_length_in_kb': '64', 'class': 
'org.apache.cassandra.io.compress.LZ4Compressor'}
AND crc_check_chance = 1.0
AND dclocal_read_repair_chance = 0.0
AND default_time_to_live = 0
AND gc_grace_seconds = 0
AND max_index_interval = 2048
AND memtable_flush_period_in_ms = 360
AND min_index_interval = 128
AND read_repair_chance = 0.0
AND speculative_retry = '99PERCENTILE';

CREATE TABLE system_distributed.parent_repair_history (
parent_id timeuuid PRIMARY KEY,
columnfamily_names set,
exception_message text,
exception_stacktrace text,
finished_at timestamp,
keyspace_name text,
requested_ranges set,
started_at timestamp,
successful_ranges set
) WITH bloom_filter_fp_chance = 0.01
AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
AND comment = 'Repair history'
AND compaction = {'class': 
'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy', 
'max_threshold': '32', 'min_threshold': '4'}
AND compression = {'chunk_length_in_kb': '64', 'class': 
'org.apache.cassandra.io.compress.LZ4Compressor'}
AND crc_check_chance = 1.0
AND dclocal_read_repair_chance = 0.0
AND default_time_to_live = 0
AND gc_grace_seconds = 0
AND max_index_interval = 2048
AND memtable_flush_period_in_ms = 360
AND min_index_interval = 128
AND read_repair_chance = 0.0
AND speculative_retry = '99PERCENTILE';


> Failed upgradesstables (upgrade from 2.2.2 to 3.0.0)
> 
>
> Key: CASSANDRA-10743
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10743
> Project: Cassandra
>  Issue Type: Bug
> Environment: CentOS Linux release 7.1.1503, OpenJDK Runtime 
> Environment (build 1.8.0_65-b17), DSC Cassandra 3.0.0 (tar.gz)
>Reporter: Gábor Auth
> Attachments: faulty-tables.tar.gz, schema.ddl
>
>
> {code}
> [cassandra@dc01-rack01-cass01 ~]$ 
> /home/cassandra/dsc-cassandra-3.0.0/bin/nodetool upgradesstables
> error: null
> -- StackTrace --
> java.lang.UnsupportedOperationException
> at 
> org.apache.cassandra.db.rows.CellPath$EmptyCellPath.get(CellPath.java:143)
> at 
> org.apache.cassandra.db.marshal.CollectionType$CollectionPathSerializer.serializedSize(CollectionType.java:226)
> at 
> org.apache.cassandra.db.rows.BufferCell$Serializer.serializedSize(BufferCell.java:325)
> at 
> org.apache.cassandra.db.rows.UnfilteredSerializer.sizeOfComplexColumn(UnfilteredSerializer.java:297)
> at 
> org.apache.cassandra.db.rows.UnfilteredSerializer.serializedRowBodySize(UnfilteredSerializer.java:282)
> at 
> org.apache.cassandra.db.rows.UnfilteredSerializer.serialize(UnfilteredSerializer.java:163)
> at 
> org.apache.cassandra.db.rows.UnfilteredSerializer.serialize(UnfilteredSerializer.java:108)
> at 
> org.apache.cassandra.db.ColumnIndex$Builder.add(ColumnIndex.java:144)
> at 
> org.apache.cassandra.db.ColumnIndex$Builder.build(ColumnIndex.java:112)
> at 
> org.apache.cassandra.db.ColumnIndex.writeAndBuildIndex(ColumnIndex.java:52)
> at 
> org.apache.cassandra.io.sstable.format.big.BigTableWriter.append(BigTableWriter.java:149)
> at 
> org.apache.cassandra.io.sstable.SSTableRewriter.append(SSTableRewriter.java:121)
> at 
> org.apache.cassandra.db.compaction.writers.DefaultCompactionWriter.realAppend(DefaultCompactionWriter.java:57)
> at 
> org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.append(CompactionAwareWriter.java:110)
> at 
> org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:182)
> at 
> 

[3/6] cassandra git commit: Revert wrong line removal from bad merge

[slebresne]

Revert wrong line removal from bad merge


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/f937c8be
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/f937c8be
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/f937c8be

Branch: refs/heads/trunk
Commit: f937c8bee87666ac1ba1ac96e0c7803629cc3508
Parents: 12b6c0a
Author: Sylvain Lebresne 
Authored: Wed Jan 6 14:44:29 2016 +0100
Committer: Sylvain Lebresne 
Committed: Wed Jan 6 14:44:29 2016 +0100

--
 conf/cassandra-env.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/f937c8be/conf/cassandra-env.sh
--
diff --git a/conf/cassandra-env.sh b/conf/cassandra-env.sh
index f322803..83fe4c5 100644
--- a/conf/cassandra-env.sh
+++ b/conf/cassandra-env.sh
@@ -33,7 +33,7 @@ calculate_heap_sizes()
 Darwin)
 system_memory_in_bytes=`sysctl hw.memsize | awk '{print $2}'`
 system_memory_in_mb=`expr $system_memory_in_bytes / 1024 / 1024`
-
+system_cpu_cores=`sysctl hw.ncpu | awk '{print $2}'`
 ;;
 *)
 # assume reasonable defaults for e.g. a modern desktop or

[jira] [Comment Edited] (CASSANDRA-9303) Match cassandra-loader options in COPY FROM

[Stefania (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-9303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15083372#comment-15083372
 ] 

Stefania edited comment on CASSANDRA-9303 at 1/6/16 1:44 PM:
-

bq. Can't you squash it? I mean, that's what the committer would have to do 
anyway so on top of giving accurate test results, it'll also make the committer 
job easier.

I thought a committer could do a squashed merge without necessarily having to 
rebase as long as the patch applies or is a rebase always necessary?

In any case, here are the branches squashed and rebased. I've also updated 
_CHANGES.txt_ and _NEWS.txt_. 

I've restarted CI again to rule out any mistakes up-merging. 

||2.1||2.2||3.0||3.2||trunk||
|[patch|https://github.com/stef1927/cassandra/commits/9303-2.1]|[patch|https://github.com/stef1927/cassandra/commits/9303-2.2]|[patch|https://github.com/stef1927/cassandra/commits/9303-3.0]|[patch|https://github.com/stef1927/cassandra/commits/9303-3.2]|[patch|https://github.com/stef1927/cassandra/commits/9303]|
|[testall|http://cassci.datastax.com/view/Dev/view/stef1927/job/stef1927-9303-2.1-testall/]|[testall|http://cassci.datastax.com/view/Dev/view/stef1927/job/stef1927-9303-2.2-testall/]|[testall|http://cassci.datastax.com/view/Dev/view/stef1927/job/stef1927-9303-3.0-testall/]|[testall|http://cassci.datastax.com/view/Dev/view/stef1927/job/stef1927-9303-3.2-testall/]|[testall|http://cassci.datastax.com/view/Dev/view/stef1927/job/stef1927-9303-testall/]|
|[dtest|http://cassci.datastax.com/view/Dev/view/stef1927/job/stef1927-9303-2.1-dtest/]|[dtest|http://cassci.datastax.com/view/Dev/view/stef1927/job/stef1927-9303-2.2-dtest/]|[dtest|http://cassci.datastax.com/view/Dev/view/stef1927/job/stef1927-9303-3.0-dtest/]|[dtest|http://cassci.datastax.com/view/Dev/view/stef1927/job/stef1927-9303-3.2-dtest/]|[dtest|http://cassci.datastax.com/view/Dev/view/stef1927/job/stef1927-9303-dtest/]|

-Old branches still exist with an {{-old}} suffix.-


was (Author: stefania):
bq. Can't you squash it? I mean, that's what the committer would have to do 
anyway so on top of giving accurate test results, it'll also make the committer 
job easier.

I thought a committer could do a squashed merge without necessarily having to 
rebase as long as the patch applies or is a rebase always necessary?

In any case, here are the branches squashed and rebased. I've also updated 
_CHANGES.txt_ and _NEWS.txt_. 

I've restarted CI again to rule out any mistakes up-merging. 

||2.1||2.2||3.0||3.2||trunk||
|[patch|https://github.com/stef1927/cassandra/commits/9303-2.1]|[patch|https://github.com/stef1927/cassandra/commits/9303-2.2]|[patch|https://github.com/stef1927/cassandra/commits/9303-3.0]|[patch|https://github.com/stef1927/cassandra/commits/9303-3.2]|[patch|https://github.com/stef1927/cassandra/commits/9303]|
|[testall|http://cassci.datastax.com/view/Dev/view/stef1927/job/stef1927-9303-2.1-testall/]|[testall|http://cassci.datastax.com/view/Dev/view/stef1927/job/stef1927-9303-2.2-testall/]|[testall|http://cassci.datastax.com/view/Dev/view/stef1927/job/stef1927-9303-3.0-testall/]|[testall|http://cassci.datastax.com/view/Dev/view/stef1927/job/stef1927-9303-3.2-testall/]|[testall|http://cassci.datastax.com/view/Dev/view/stef1927/job/stef1927-9303-testall/]|
|[dtest|http://cassci.datastax.com/view/Dev/view/stef1927/job/stef1927-9303-2.1-dtest/]|[dtest|http://cassci.datastax.com/view/Dev/view/stef1927/job/stef1927-9303-2.2-dtest/]|[dtest|http://cassci.datastax.com/view/Dev/view/stef1927/job/stef1927-9303-3.0-dtest/]|[dtest|http://cassci.datastax.com/view/Dev/view/stef1927/job/stef1927-9303-3.2-dtest/]|[dtest|http://cassci.datastax.com/view/Dev/view/stef1927/job/stef1927-9303-dtest/]|

Old branches still exist with an {{-old}} suffix.

> Match cassandra-loader options in COPY FROM
> ---
>
> Key: CASSANDRA-9303
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9303
> Project: Cassandra
>  Issue Type: New Feature
>  Components: Tools
>Reporter: Jonathan Ellis
>Assignee: Stefania
>Priority: Critical
> Fix For: 2.1.x, 2.2.x, 3.0.x, 3.x
>
> Attachments: dtest.out
>
>
> https://github.com/brianmhess/cassandra-loader added a bunch of options to 
> handle real world requirements, we should match those.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[4/6] cassandra git commit: Merge branch 'cassandra-3.0' into cassandra-3.3

[slebresne]

Merge branch 'cassandra-3.0' into cassandra-3.3


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/7fbc39b7
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/7fbc39b7
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/7fbc39b7

Branch: refs/heads/trunk
Commit: 7fbc39b743f7a08cb03f6b83bebe34c1bc342b9a
Parents: 10c1e65 f937c8b
Author: Sylvain Lebresne 
Authored: Wed Jan 6 14:44:45 2016 +0100
Committer: Sylvain Lebresne 
Committed: Wed Jan 6 14:44:45 2016 +0100

--
 conf/cassandra-env.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/7fbc39b7/conf/cassandra-env.sh
--

[2/6] cassandra git commit: Revert wrong line removal from bad merge

[slebresne]

Revert wrong line removal from bad merge


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/f937c8be
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/f937c8be
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/f937c8be

Branch: refs/heads/cassandra-3.3
Commit: f937c8bee87666ac1ba1ac96e0c7803629cc3508
Parents: 12b6c0a
Author: Sylvain Lebresne 
Authored: Wed Jan 6 14:44:29 2016 +0100
Committer: Sylvain Lebresne 
Committed: Wed Jan 6 14:44:29 2016 +0100

--
 conf/cassandra-env.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/f937c8be/conf/cassandra-env.sh
--
diff --git a/conf/cassandra-env.sh b/conf/cassandra-env.sh
index f322803..83fe4c5 100644
--- a/conf/cassandra-env.sh
+++ b/conf/cassandra-env.sh
@@ -33,7 +33,7 @@ calculate_heap_sizes()
 Darwin)
 system_memory_in_bytes=`sysctl hw.memsize | awk '{print $2}'`
 system_memory_in_mb=`expr $system_memory_in_bytes / 1024 / 1024`
-
+system_cpu_cores=`sysctl hw.ncpu | awk '{print $2}'`
 ;;
 *)
 # assume reasonable defaults for e.g. a modern desktop or

[jira] [Comment Edited] (CASSANDRA-9303) Match cassandra-loader options in COPY FROM

[Stefania (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-9303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15085537#comment-15085537
 ] 

Stefania edited comment on CASSANDRA-9303 at 1/6/16 1:46 PM:
-

CI still OK, ready to commit.

Note that you will get conflicts up-merging all the way up to 3.2.


was (Author: stefania):
CI still OK, ready to commit.

> Match cassandra-loader options in COPY FROM
> ---
>
> Key: CASSANDRA-9303
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9303
> Project: Cassandra
>  Issue Type: New Feature
>  Components: Tools
>Reporter: Jonathan Ellis
>Assignee: Stefania
>Priority: Critical
> Fix For: 2.1.x, 2.2.x, 3.0.x, 3.x
>
> Attachments: dtest.out
>
>
> https://github.com/brianmhess/cassandra-loader added a bunch of options to 
> handle real world requirements, we should match those.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (CASSANDRA-9303) Match cassandra-loader options in COPY FROM

[Stefania (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-9303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15085537#comment-15085537
 ] 

Stefania edited comment on CASSANDRA-9303 at 1/6/16 1:47 PM:
-

CI still OK, ready to commit.

Note that you will get conflicts up-merging all the way up to 3.2.

The dtest pull request to be merged is 
[here|https://github.com/riptano/cassandra-dtest/pull/724].


was (Author: stefania):
CI still OK, ready to commit.

Note that you will get conflicts up-merging all the way up to 3.2.

> Match cassandra-loader options in COPY FROM
> ---
>
> Key: CASSANDRA-9303
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9303
> Project: Cassandra
>  Issue Type: New Feature
>  Components: Tools
>Reporter: Jonathan Ellis
>Assignee: Stefania
>Priority: Critical
> Fix For: 2.1.x, 2.2.x, 3.0.x, 3.x
>
> Attachments: dtest.out
>
>
> https://github.com/brianmhess/cassandra-loader added a bunch of options to 
> handle real world requirements, we should match those.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-10951) Fix ReadCommandTest

[Branimir Lambov (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-10951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15085733#comment-15085733
 ] 

Branimir Lambov commented on CASSANDRA-10951:
-

Patch uploaded here:
|[code|https://github.com/blambov/cassandra/tree/10951]|[utests|http://cassci.datastax.com/view/Dev/view/blambov/job/blambov-10951-testall/]|[dtests|http://cassci.datastax.com/view/Dev/view/blambov/job/blambov-10951-dtest/]|

{{AbortedQueriesTester}} still times out; 
{{RecoveryManagerTest.testRecoverPIT}} passes locally but timed out twice on 
cassci. Will look into it a bit more.

> Fix ReadCommandTest
> ---
>
> Key: CASSANDRA-10951
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10951
> Project: Cassandra
>  Issue Type: Bug
>  Components: Local Write-Read Paths, Testing
>Reporter: Yuki Morishita
>Assignee: Branimir Lambov
>Priority: Blocker
> Fix For: 3.2
>
>
> [ReadCommandTest is 
> failing|http://cassci.datastax.com/view/trunk/job/trunk_testall/641/testReport/org.apache.cassandra.db/ReadCommandTest/history/]
>  since CASSANDRA-9975 was merged to trunk.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-10951) Fix ReadCommandTest

[Yuki Morishita (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-10951?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yuki Morishita updated CASSANDRA-10951:
---
Priority: Blocker  (was: Major)

> Fix ReadCommandTest
> ---
>
> Key: CASSANDRA-10951
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10951
> Project: Cassandra
>  Issue Type: Bug
>  Components: Testing
>Reporter: Yuki Morishita
>Assignee: Branimir Lambov
>Priority: Blocker
> Fix For: 3.x
>
>
> [ReadCommandTest is 
> failing|http://cassci.datastax.com/view/trunk/job/trunk_testall/641/testReport/org.apache.cassandra.db/ReadCommandTest/history/]
>  since CASSANDRA_9975 was merged to trunk.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-10368) Support Restricting non-PK Cols in Materialized View Select Statements

[Jeremiah Jordan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-10368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jeremiah Jordan updated CASSANDRA-10368:

Fix Version/s: (was: 3.2)
   3.x

> Support Restricting non-PK Cols in Materialized View Select Statements
> --
>
> Key: CASSANDRA-10368
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10368
> Project: Cassandra
>  Issue Type: Improvement
>  Components: CQL
>Reporter: Tyler Hobbs
>Priority: Minor
> Fix For: 3.x
>
>
> CASSANDRA-9664 allows materialized views to restrict primary key columns in 
> the select statement.  Due to CASSANDRA-10261, the patch did not include 
> support for restricting non-PK columns.  Now that the timestamp issue has 
> been resolved, we can add support for this.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-10938) test_bulk_round_trip_blogposts is failing occasionally

[Stefania (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-10938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15085721#comment-15085721
 ] 

Stefania commented on CASSANDRA-10938:
--

[~slebresne]: as discussed, I confirm that I cannot reproduce this locally on 
Linux (although we have very similar failures on cassci Linux jobs). I suggest 
switching to a standard CHM and testing that it works well on Windows and 
cassci too.

> test_bulk_round_trip_blogposts is failing occasionally
> --
>
> Key: CASSANDRA-10938
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10938
> Project: Cassandra
>  Issue Type: Sub-task
>  Components: Tools
>Reporter: Stefania
>Assignee: Stefania
> Fix For: 2.1.x
>
> Attachments: 6452.nps, 6452.png, 7300.nps, 7300a.png, 7300b.png, 
> node1_debug.log, node2_debug.log, node3_debug.log, recording_127.0.0.1.jfr
>
>
> We get timeouts occasionally that cause the number of records to be incorrect:
> http://cassci.datastax.com/job/trunk_dtest/858/testReport/cqlsh_tests.cqlsh_copy_tests/CqlshCopyTest/test_bulk_round_trip_blogposts/



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-10910) Materialized view remained rows

[Carl Yeksigian (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-10910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15085720#comment-15085720
 ] 

Carl Yeksigian commented on CASSANDRA-10910:


I've rerun the tests with the latest changes, and the runs look OK, with just 
some CI flapping.

> Materialized view remained rows
> ---
>
> Key: CASSANDRA-10910
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10910
> Project: Cassandra
>  Issue Type: Bug
> Environment: Cassandra 3.0.0
>Reporter: Gábor Auth
>Assignee: Carl Yeksigian
> Fix For: 3.0.x, 3.x
>
>
> I've created a table and a materialized view.
> {code}
> > CREATE TABLE test (id text PRIMARY KEY, key text, value int);
> > CREATE MATERIALIZED VIEW test_view AS SELECT * FROM test WHERE key IS NOT 
> > NULL PRIMARY KEY(key, id);
> {code}
> I've put a value into the table:
> {code}
> > update test set key='key', value=1 where id='id';
> > select * from test; select * from test_view ;
>  id | key | value
> +-+---
>  id | key | 1
> (1 rows)
>  key | id | value
> -++---
>  key | id | 1
> (1 rows)
> {code}
> I've updated the value without specified the key of the materialized view:
> {code}
> > update test set value=2 where id='id';
> > select * from test; select * from test_view ;
>  id | key | value
> +-+---
>  id | key | 2
> (1 rows)
>  key | id | value
> -++---
>  key | id | 2
> (1 rows)
> {code}
> It works as I think...
> ...but I've updated the key of the materialized view:
> {code}
> > update test set key='newKey' where id='id';
> > select * from test; select * from test_view ;
>  id | key| value
> ++---
>  id | newKey | 2
> (1 rows)
>  key| id | value
> ++---
> key | id | 2
>  newKey | id | 2
> (2 rows)
> {code}
> ...I've updated the value of the row:
> {code}
> > update test set key='newKey', value=3 where id='id';
> > select * from test; select * from test_view ;
>  id | key| value
> ++---
>  id | newKey | 3
> (1 rows)
>  key| id | value
> ++---
> key | id | 2
>  newKey | id | 3
> (2 rows)
> {code}
> ...I've deleted the row by the id key:
> {code}
> > delete from test where id='id';
> > select * from test; select * from test_view ;
>  id | key | value
> +-+---
> (0 rows)
>  key | id | value
> -++---
>  key | id | 2
> (1 rows)
> {code}
> Is it a bug?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-10839) cqlsh failed to format value bytearray

[Stefania (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-10839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15085727#comment-15085727
 ] 

Stefania commented on CASSANDRA-10839:
--

Thanks for the review. Now the question is whether we actually commit this to 
2.1 since it should be now for critical fixes only. cc [~iamaleksey]

For 2.2+ onwards we don't need this patch since we no longer support python 2.7.

> cqlsh failed to format value bytearray
> --
>
> Key: CASSANDRA-10839
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10839
> Project: Cassandra
>  Issue Type: Bug
>  Components: Tools
>Reporter: Severin Leonhardt
>Assignee: Stefania
>Priority: Minor
> Fix For: 2.1.x
>
>
> Execute the following in cqlsh (5.0.1):
> {noformat}
> > create table test(column blob, primary key(column));
> > insert into test (column) VALUES(0x00);
> > select * from test;
>  column
> 
>  bytearray(b'\x00')
> (1 rows)
> Failed to format value bytearray(b'\x00') : b2a_hex() argument 1 must be 
> string or read-only buffer, not bytearray
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-10951) Fix ReadCommandTest

[T Jake Luciani (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-10951?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

T Jake Luciani updated CASSANDRA-10951:
---
Fix Version/s: (was: 3.x)
   3.2

> Fix ReadCommandTest
> ---
>
> Key: CASSANDRA-10951
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10951
> Project: Cassandra
>  Issue Type: Bug
>  Components: Local Write-Read Paths, Testing
>Reporter: Yuki Morishita
>Assignee: Branimir Lambov
>Priority: Blocker
> Fix For: 3.2
>
>
> [ReadCommandTest is 
> failing|http://cassci.datastax.com/view/trunk/job/trunk_testall/641/testReport/org.apache.cassandra.db/ReadCommandTest/history/]
>  since CASSANDRA-9975 was merged to trunk.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-10951) Fix ReadCommandTest

[Yuki Morishita (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-10951?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yuki Morishita updated CASSANDRA-10951:
---
Description: [ReadCommandTest is 
failing|http://cassci.datastax.com/view/trunk/job/trunk_testall/641/testReport/org.apache.cassandra.db/ReadCommandTest/history/]
 since CASSANDRA-9975 was merged to trunk.  (was: [ReadCommandTest is 
failing|http://cassci.datastax.com/view/trunk/job/trunk_testall/641/testReport/org.apache.cassandra.db/ReadCommandTest/history/]
 since CASSANDRA_9975 was merged to trunk.)

> Fix ReadCommandTest
> ---
>
> Key: CASSANDRA-10951
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10951
> Project: Cassandra
>  Issue Type: Bug
>  Components: Testing
>Reporter: Yuki Morishita
>Assignee: Branimir Lambov
>Priority: Blocker
> Fix For: 3.x
>
>
> [ReadCommandTest is 
> failing|http://cassci.datastax.com/view/trunk/job/trunk_testall/641/testReport/org.apache.cassandra.db/ReadCommandTest/history/]
>  since CASSANDRA-9975 was merged to trunk.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-10938) test_bulk_round_trip_blogposts is failing occasionally

[Sylvain Lebresne (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-10938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15085739#comment-15085739
 ] 

Sylvain Lebresne commented on CASSANDRA-10938:
--

I agree with changing to CHM here if that fixes Windows and cassci as 
performance is not terribly critical in that case (important but not critical).

It does very much look like we're hitting either a very bad case or a bug in 
NBHM. From a very quick look at the source, the methods that takes all the CPU 
have some CAS loops so if lots of threads are trying to access the same slot 
(which sounds to be the case if I understand the failing test correctly) while 
a resizing is going on, then I suppose crappy scheduling could lead us there. 
That bad behavior is a bit worrisome in the sense that we use NBHM in other 
places, but it's also the first time we see this behavior so it's unclear the 
ROI of investigating too much is worth it right now.

> test_bulk_round_trip_blogposts is failing occasionally
> --
>
> Key: CASSANDRA-10938
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10938
> Project: Cassandra
>  Issue Type: Sub-task
>  Components: Tools
>Reporter: Stefania
>Assignee: Stefania
> Fix For: 2.1.x
>
> Attachments: 6452.nps, 6452.png, 7300.nps, 7300a.png, 7300b.png, 
> node1_debug.log, node2_debug.log, node3_debug.log, recording_127.0.0.1.jfr
>
>
> We get timeouts occasionally that cause the number of records to be incorrect:
> http://cassci.datastax.com/job/trunk_dtest/858/testReport/cqlsh_tests.cqlsh_copy_tests/CqlshCopyTest/test_bulk_round_trip_blogposts/



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-10951) Fix ReadCommandTest

[Yuki Morishita (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-10951?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yuki Morishita updated CASSANDRA-10951:
---
Component/s: Local Write-Read Paths

> Fix ReadCommandTest
> ---
>
> Key: CASSANDRA-10951
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10951
> Project: Cassandra
>  Issue Type: Bug
>  Components: Local Write-Read Paths, Testing
>Reporter: Yuki Morishita
>Assignee: Branimir Lambov
>Priority: Blocker
> Fix For: 3.x
>
>
> [ReadCommandTest is 
> failing|http://cassci.datastax.com/view/trunk/job/trunk_testall/641/testReport/org.apache.cassandra.db/ReadCommandTest/history/]
>  since CASSANDRA-9975 was merged to trunk.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-10951) Fix ReadCommandTest

[Yuki Morishita (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-10951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15085625#comment-15085625
 ] 

Yuki Morishita commented on CASSANDRA-10951:


Bumping to Blocker since CASSANDRA-7392 is not working right now.
dtest cql_tests.AbortedQueriesTester is also not passing.

> Fix ReadCommandTest
> ---
>
> Key: CASSANDRA-10951
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10951
> Project: Cassandra
>  Issue Type: Bug
>  Components: Testing
>Reporter: Yuki Morishita
>Assignee: Branimir Lambov
>Priority: Blocker
> Fix For: 3.x
>
>
> [ReadCommandTest is 
> failing|http://cassci.datastax.com/view/trunk/job/trunk_testall/641/testReport/org.apache.cassandra.db/ReadCommandTest/history/]
>  since CASSANDRA_9975 was merged to trunk.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-10951) Fix ReadCommandTest

[Sylvain Lebresne (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-10951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15085701#comment-15085701
 ] 

Sylvain Lebresne commented on CASSANDRA-10951:
--

I'll note for what it's worth that CASSANDRA-7392 is a brand new feature in 3.2 
so it would certainly be nice to have it work, but _if_ that issue turns out to 
be harder to fix than we think, I wouldn't actually block 3.2 on it as 
postponing a functioning CASSANDRA-7392 to 3.2 (or even 3.3) is not a huge huge 
deal. That said, I believe [~blambov] has a patch and is finishing testing it 
so hopefully we'll have an update very soon and happy to wait on that.

> Fix ReadCommandTest
> ---
>
> Key: CASSANDRA-10951
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10951
> Project: Cassandra
>  Issue Type: Bug
>  Components: Local Write-Read Paths, Testing
>Reporter: Yuki Morishita
>Assignee: Branimir Lambov
>Priority: Blocker
> Fix For: 3.2
>
>
> [ReadCommandTest is 
> failing|http://cassci.datastax.com/view/trunk/job/trunk_testall/641/testReport/org.apache.cassandra.db/ReadCommandTest/history/]
>  since CASSANDRA-9975 was merged to trunk.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-10839) cqlsh failed to format value bytearray

[Jim Witschey (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-10839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15085706#comment-15085706
 ] 

Jim Witschey commented on CASSANDRA-10839:
--

Sorry for the delay. The change looks fine to me, and the failures are all 
reasonable:

http://cassci.datastax.com/userContent/cstar_report/index.html?jobs=stef1927-10839-2.1-dtest_known=true_passing_known=true

> cqlsh failed to format value bytearray
> --
>
> Key: CASSANDRA-10839
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10839
> Project: Cassandra
>  Issue Type: Bug
>  Components: Tools
>Reporter: Severin Leonhardt
>Assignee: Stefania
>Priority: Minor
> Fix For: 2.1.x
>
>
> Execute the following in cqlsh (5.0.1):
> {noformat}
> > create table test(column blob, primary key(column));
> > insert into test (column) VALUES(0x00);
> > select * from test;
>  column
> 
>  bytearray(b'\x00')
> (1 rows)
> Failed to format value bytearray(b'\x00') : b2a_hex() argument 1 must be 
> string or read-only buffer, not bytearray
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-10863) HSHA test_closing_connections test still flaps on 3.0

[Carl Yeksigian (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-10863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15085712#comment-15085712
 ] 

Carl Yeksigian commented on CASSANDRA-10863:


I've run this a couple more times and it doesn't look like it is failing. I've 
opened a dtest PR.

> HSHA test_closing_connections test still flaps on 3.0
> -
>
> Key: CASSANDRA-10863
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10863
> Project: Cassandra
>  Issue Type: Sub-task
>Reporter: Jim Witschey
>Assignee: Carl Yeksigian
> Fix For: 3.0.x
>
>
> The problem reported in CASSANDRA-10570 still seems to be happening on 
> CassCI, as recently as a couple days ago:
> http://cassci.datastax.com/job/cassandra-3.0_dtest/433/
> [~carlyeks] The fix in #10570 was to increase how long we'd sleep in the 
> tests. Should we just bump it up further, or does this make you suspect a 
> larger problem here?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-10970) SSL/TLS: Certificate Domain is ignored

[Matthias Brandt (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-10970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15085945#comment-15085945
 ] 

Matthias Brandt commented on CASSANDRA-10970:
-

{quote}Are you saying the system hostname of the server(s) is some other 
domain?{quote}
Not only the hostname is different, it's a complete different server. In my 
current setup, I'm connecting against node1.my.other-domain.com, which responds 
with a certificate for *.my.domain.com. 

Just forgot to say: This issue belongs to cqlsh.
Just found out that this also was in issue in the java driver from datastax in 
versions < 3.x: https://datastax-oss.atlassian.net/browse/JAVA-716

> SSL/TLS: Certificate Domain is ignored
> --
>
> Key: CASSANDRA-10970
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10970
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Matthias Brandt
>
> I've set up server_encryption_options as well as client_encryption_options. 
> In both settings, I use the same keystore with an wild-card SSL certificate 
> in it. It is signed by our own CA, which root certificate is in the 
> configured truststore:
> {code}
> server_encryption_options:
> internode_encryption: all
> keystore: /etc/cassandra/conf/wildcard-cert.keystore
> keystore_password: ""
> truststore: /etc/cassandra/conf/my-cacerts
> truststore_password: changeit
> require_client_auth: true
> client_encryption_options:
> enabled: true
> keystore: /etc/cassandra/conf/wildcard-cert.keystore
> keystore_password: ""
> require_client_auth: false
> {code}
> The certifcate's subject is:
> {code}CN=*.my.domain.com,OU=my unit,O=my org{code}
> When I deploy this setting on a server which domain is 
> node1.my.*other-domain*.com a connection via cqlsh wrongly works. 
> Additionally, the inter-node connection between other nodes in this wrong 
> domain also works.
> I would expect that the connection would fail with a meaningful error message.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-10661) Integrate SASI to Cassandra

[Sam Tunnicliffe (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-10661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15085957#comment-15085957
 ] 

Sam Tunnicliffe commented on CASSANDRA-10661:
-

This is looking pretty good. 

A problem (which isn't caught by any of the unit tests btw) is that due to the 
fact that under the hood 3.x considers all compact storage columns as static. 
This breaks interactions with sasi-indexed tables via CQL - for example, try 
running through the examples in the original [SASI 
readme|https://github.com/xedin/sasi/blob/master/README.md] and you'll find 
querying mostly broken. 

{code}
cqlsh:demo> select first_name, last_name, age, height, created_at from sasi 
where first_name = 'M';
InvalidRequest: code=2200 [Invalid query] message="Queries using 2ndary indexes 
don't support selecting only static columns"
cqlsh:demo>
cqlsh:demo>
cqlsh:demo> select * from sasi where first_name = 'M';

 id | age | created_at | first_name | height | last_name
+-++++---

(0 rows)
{code}

Fortunately, I believe we can simply drop the use of COMPACT STORAGE. My 
(limited) testing suggests that when tables are created without it, everything 
that's currently implemented works as expected.

The new SASI specific tests look good and are all green, but we obviously need 
to run this through CI before it's committed. On a related note, are there any 
dtests that may be worth adding? The utest coverage is pretty comprehensive 
(modulo the CQL issues) so I wouldn't say it was absolutely critical, but some 
multi-node & CQL based tests would be nice to have.

Otherwise, this first phase of integration looks good to me. On initial review 
I found one bug and a handful of nits. I have a few scenarios I want to run 
through, mostly to verify how sasi interacts with some of the parts of the 
index subsystem that were changed in 3.0.

Initial review comments:

* The regex matching in o.a.c.io.sstable.Component.Type::fromRepresentation 
throws an NPE when it encounters an unknown name and tries to match it to a 
CUSTOM component.
* In SASIIndex, getMetadataReloadTask & getBlockingFlushTask should be able to 
just return null (like getInitializationTask does). In the case of the former, 
that is true right now as the only call site is in SIM where nulls are properly 
handled. getBlockingFlushTask is also called from KeyCacheCqlTest which doesn't 
check for nulls so would need tweaking slightly. (This is totally minor, the 
irregularity in SASIIndex just bugged me).
* I couldn't see why a PeekingIterator is used in OnDiskIndex::search
* The use of "a" and "b" in the o.a.c.i.sasi.plan.Expression ctor seems like it 
could have the potential for pain when debugging. I'm sure that it isn't very 
likely we'll ever care too much & I don't have any particularly better 
suggestion but if you do, could these be changed to something more greppable 
(or extracted to constants)?
* The anonymous extension of Expression in Operation::analyzeGroup can be 
replaced with {{perColumn.add(new Expression(controller, 
columnIndex).add(e.operator(), token));}}
* MemIndex::estimateSize is unused
* It doesn't really affect anything, but just for clarity I would rename 
MemoryUtil.DIRECT_BYTE_BUFFER_R_CLASS to RO_DIRECT_BYTE_BUFFER_CLASS
* Most trivial of nits: brace placement in SchemaLoader (ln 255)


> Integrate SASI to Cassandra
> ---
>
> Key: CASSANDRA-10661
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10661
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Local Write-Read Paths
>Reporter: Pavel Yaskevich
>Assignee: Pavel Yaskevich
>  Labels: sasi
> Fix For: 3.x
>
>
> We have recently released new secondary index engine 
> (https://github.com/xedin/sasi) build using SecondaryIndex API, there are 
> still couple of things to work out regarding 3.x since it's currently 
> targeted on 2.0 released. I want to make this an umbrella issue to all of the 
> things related to integration of SASI, which are also tracked in 
> [sasi_issues|https://github.com/xedin/sasi/issues], into mainline Cassandra 
> 3.x release.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-10973) disk_balance_decommission_test is failing on trunk

[Alan Boudreault (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-10973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alan Boudreault updated CASSANDRA-10973:

Description: 
http://cassci.datastax.com/job/trunk_dtest/891/testReport/junit/disk_balance_test/TestDiskBalance/disk_balance_decommission_test/

{code}
==
FAIL: disk_balance_decommission_test (disk_balance_test.TestDiskBalance)
--
Traceback (most recent call last):
  File "/home/aboudreault/git/cstar/cassandra-dtest/disk_balance_test.py", line 
74, in disk_balance_decommission_test
self.assert_balanced(node)
  File "/home/aboudreault/git/cstar/cassandra-dtest/disk_balance_test.py", line 
127, in assert_balanced
assert_almost_equal(*sums, error=0.2, error_message=node.name)
  File "/home/aboudreault/git/cstar/cassandra-dtest/assertions.py", line 65, in 
assert_almost_equal
assert vmin > vmax * (1.0 - error) or vmin == vmax, "values not within 
%.2f%% of the max: %s (%s)" % (error * 100, args, kwargs['error_message'])
AssertionError: values not within 20.00% of the max: (482095, 477840, 612940) 
(node2)
 >> begin captured logging << 
dtest: DEBUG: cluster ccm directory: /tmp/dtest-SLbi3e
- >> end captured logging << -

--
Ran 1 test in 121.295s

FAILED (failures=1)
{code}

  was:
http://cassci.datastax.com/job/trunk_dtest/891/testReport/junit/disk_balance_test/TestDiskBalance/disk_balance_bootstrap_test/

{code}
==
FAIL: disk_balance_decommission_test (disk_balance_test.TestDiskBalance)
--
Traceback (most recent call last):
  File "/home/aboudreault/git/cstar/cassandra-dtest/disk_balance_test.py", line 
74, in disk_balance_decommission_test
self.assert_balanced(node)
  File "/home/aboudreault/git/cstar/cassandra-dtest/disk_balance_test.py", line 
127, in assert_balanced
assert_almost_equal(*sums, error=0.2, error_message=node.name)
  File "/home/aboudreault/git/cstar/cassandra-dtest/assertions.py", line 65, in 
assert_almost_equal
assert vmin > vmax * (1.0 - error) or vmin == vmax, "values not within 
%.2f%% of the max: %s (%s)" % (error * 100, args, kwargs['error_message'])
AssertionError: values not within 20.00% of the max: (482095, 477840, 612940) 
(node2)
 >> begin captured logging << 
dtest: DEBUG: cluster ccm directory: /tmp/dtest-SLbi3e
- >> end captured logging << -

--
Ran 1 test in 121.295s

FAILED (failures=1)
{code}


> disk_balance_decommission_test is failing on trunk
> --
>
> Key: CASSANDRA-10973
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10973
> Project: Cassandra
>  Issue Type: Bug
>  Components: Tools
>Reporter: Alan Boudreault
> Fix For: 3.2
>
>
> http://cassci.datastax.com/job/trunk_dtest/891/testReport/junit/disk_balance_test/TestDiskBalance/disk_balance_decommission_test/
> {code}
> ==
> FAIL: disk_balance_decommission_test (disk_balance_test.TestDiskBalance)
> --
> Traceback (most recent call last):
>   File "/home/aboudreault/git/cstar/cassandra-dtest/disk_balance_test.py", 
> line 74, in disk_balance_decommission_test
> self.assert_balanced(node)
>   File "/home/aboudreault/git/cstar/cassandra-dtest/disk_balance_test.py", 
> line 127, in assert_balanced
> assert_almost_equal(*sums, error=0.2, error_message=node.name)
>   File "/home/aboudreault/git/cstar/cassandra-dtest/assertions.py", line 65, 
> in assert_almost_equal
> assert vmin > vmax * (1.0 - error) or vmin == vmax, "values not within 
> %.2f%% of the max: %s (%s)" % (error * 100, args, kwargs['error_message'])
> AssertionError: values not within 20.00% of the max: (482095, 477840, 612940) 
> (node2)
>  >> begin captured logging << 
> dtest: DEBUG: cluster ccm directory: /tmp/dtest-SLbi3e
> - >> end captured logging << -
> --
> Ran 1 test in 121.295s
> FAILED (failures=1)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (CASSANDRA-10972) File based hints don't implement backpressure and can OOM

[Ariel Weisberg (JIRA)]

Ariel Weisberg created CASSANDRA-10972:
--

 Summary: File based hints don't implement backpressure and can OOM
 Key: CASSANDRA-10972
 URL: https://issues.apache.org/jira/browse/CASSANDRA-10972
 Project: Cassandra
  Issue Type: Bug
Reporter: Ariel Weisberg
Assignee: Ariel Weisberg
 Fix For: 3.0.x, 3.x


This is something I reproduced in practice. I have what I think is a reasonable 
implementation of backpressure, but still need to put together a unit test.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-10970) SSL/TLS: Certificate Domain is ignored

[Michael Shuler (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-10970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15085916#comment-15085916
 ] 

Michael Shuler commented on CASSANDRA-10970:


bq. When I deploy this setting on a server which domain is 
node1.my.other-domain.com a connection via cqlsh wrongly works. Additionally, 
the inter-node connection between other nodes in this wrong domain also works.

Are you saying the system hostname of the server(s) is some other domain? That 
should be fine. An SSL example where this is expected to work:

{noformat}
mshuler@hana:~$ host www.google.com | head -1
www.google.com has address 74.125.22.105
mshuler@hana:~$ host 74.125.22.105
105.22.125.74.in-addr.arpa domain name pointer qh-in-f105.1e100.net.
{noformat}

An SSL/TLS connection to https://www.google.com/ is really coming from the host 
{{qh-in-f105.1e100.net}}, but the httpd service at that host describes itself 
as {{www.google.com}} and provides a certificate for the same, even though the 
host is really from the {{1e100.net}} domain.

Am I following your comment correctly? I think your desire to have this 
connection fail is possibly incorrect. You provided a certificate, it validated 
from your local CA, so the connections succeed, as expected.

> SSL/TLS: Certificate Domain is ignored
> --
>
> Key: CASSANDRA-10970
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10970
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Matthias Brandt
>
> I've set up server_encryption_options as well as client_encryption_options. 
> In both settings, I use the same keystore with an wild-card SSL certificate 
> in it. It is signed by our own CA, which root certificate is in the 
> configured truststore:
> {code}
> server_encryption_options:
> internode_encryption: all
> keystore: /etc/cassandra/conf/wildcard-cert.keystore
> keystore_password: ""
> truststore: /etc/cassandra/conf/hpo-cacerts
> truststore_password: changeit
> require_client_auth: true
> client_encryption_options:
> enabled: true
> keystore: /etc/cassandra/conf/wildcard-cert.keystore
> keystore_password: ""
> require_client_auth: false
> {code}
> The certifcate's subject is:
> {code}CN=*.my.domain.com,OU=my unit,O=my org{code}
> When I deploy this setting on a server which domain is 
> node1.my.*other-domain*.com a connection via cqlsh wrongly works. 
> Additionally, the inter-node connection between other nodes in this wrong 
> domain also works.
> I would expect that the connection would fail with a meaningful error message.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (CASSANDRA-10863) HSHA test_closing_connections test still flaps on 3.0

[Carl Yeksigian (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-10863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Carl Yeksigian resolved CASSANDRA-10863.

Resolution: Fixed

Committed fix to dtest as 
[{{40b03e6}}|https://github.com/riptano/cassandra-dtest/commit/40b03e6f9a488a19c5580be97278b8b502448cc2].

> HSHA test_closing_connections test still flaps on 3.0
> -
>
> Key: CASSANDRA-10863
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10863
> Project: Cassandra
>  Issue Type: Sub-task
>Reporter: Jim Witschey
>Assignee: Carl Yeksigian
> Fix For: 3.0.x
>
>
> The problem reported in CASSANDRA-10570 still seems to be happening on 
> CassCI, as recently as a couple days ago:
> http://cassci.datastax.com/job/cassandra-3.0_dtest/433/
> [~carlyeks] The fix in #10570 was to increase how long we'd sleep in the 
> tests. Should we just bump it up further, or does this make you suspect a 
> larger problem here?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-10887) Pending range calculator gives wrong pending ranges for moves

[sankalp kohli (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-10887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15085951#comment-15085951
 ] 

sankalp kohli commented on CASSANDRA-10887:
---

Let me get the patches for 2.2,3.0 and trunk

> Pending range calculator gives wrong pending ranges for moves
> -
>
> Key: CASSANDRA-10887
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10887
> Project: Cassandra
>  Issue Type: Bug
>  Components: Coordination
>Reporter: Richard Low
>Assignee: sankalp kohli
>Priority: Critical
> Fix For: 2.1.x, 2.2.x, 3.0.x, 3.x
>
> Attachments: CASSANDRA-10887.diff, CASSANDRA_10887_v2.diff, 
> CASSANDRA_10887_v3.diff
>
>
> My understanding is the PendingRangeCalculator is meant to calculate who 
> should receive extra writes during range movements. However, it adds the 
> wrong ranges for moves. An extreme example of this can be seen in the 
> following reproduction. Create a 5 node cluster (I did this on 2.0.16 and 
> 2.2.4) and a keyspace RF=3 and a simple table. Then start moving a node and 
> immediately kill -9 it. Now you see a node as down and moving in the ring. 
> Try a quorum write for a partition that is stored on that node - it will fail 
> with a timeout. Further, all CAS reads or writes fail immediately with 
> unavailable exception because they attempt to include the moving node twice. 
> This is likely to be the cause of CASSANDRA-10423.
> In my example I had this ring:
> 127.0.0.1  rack1   Up Normal  170.97 KB   20.00%  
> -9223372036854775808
> 127.0.0.2  rack1   Up Normal  124.06 KB   20.00%  
> -5534023222112865485
> 127.0.0.3  rack1   Down   Moving  108.7 KB40.00%  
> 1844674407370955160
> 127.0.0.4  rack1   Up Normal  142.58 KB   0.00%   
> 1844674407370955161
> 127.0.0.5  rack1   Up Normal  118.64 KB   20.00%  
> 5534023222112865484
> Node 3 was moving to -1844674407370955160. I added logging to print the 
> pending and natural endpoints. For ranges owned by node 3, node 3 appeared in 
> pending and natural endpoints. The blockFor is increased to 3 so we’re 
> effectively doing CL.ALL operations. This manifests as write timeouts and CAS 
> unavailables when the node is down.
> The correct pending range for this scenario is node 1 is gaining the range 
> (-1844674407370955160, 1844674407370955160). So node 1 should be added as a 
> destination for writes and CAS for this range, not node 3.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (CASSANDRA-10973) disk_balance_decommission_test is failing on trunk

[Alan Boudreault (JIRA)]

Alan Boudreault created CASSANDRA-10973:
---

 Summary: disk_balance_decommission_test is failing on trunk
 Key: CASSANDRA-10973
 URL: https://issues.apache.org/jira/browse/CASSANDRA-10973
 Project: Cassandra
  Issue Type: Bug
  Components: Tools
Reporter: Alan Boudreault
 Fix For: 3.2


{code}
==
FAIL: disk_balance_decommission_test (disk_balance_test.TestDiskBalance)
--
Traceback (most recent call last):
  File "/home/aboudreault/git/cstar/cassandra-dtest/disk_balance_test.py", line 
74, in disk_balance_decommission_test
self.assert_balanced(node)
  File "/home/aboudreault/git/cstar/cassandra-dtest/disk_balance_test.py", line 
127, in assert_balanced
assert_almost_equal(*sums, error=0.2, error_message=node.name)
  File "/home/aboudreault/git/cstar/cassandra-dtest/assertions.py", line 65, in 
assert_almost_equal
assert vmin > vmax * (1.0 - error) or vmin == vmax, "values not within 
%.2f%% of the max: %s (%s)" % (error * 100, args, kwargs['error_message'])
AssertionError: values not within 20.00% of the max: (482095, 477840, 612940) 
(node2)
 >> begin captured logging << 
dtest: DEBUG: cluster ccm directory: /tmp/dtest-SLbi3e
- >> end captured logging << -

--
Ran 1 test in 121.295s

FAILED (failures=1)
{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-10974) disk_balance_bootstrap_test is failing on trunk

[Alan Boudreault (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-10974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alan Boudreault updated CASSANDRA-10974:

Description: 
http://cassci.datastax.com/job/trunk_dtest/891/testReport/junit/disk_balance_test/TestDiskBalance/disk_balance_bootstrap_test/

{code}
==
FAIL: disk_balance_bootstrap_test (disk_balance_test.TestDiskBalance)
--
Traceback (most recent call last):
  File "/home/aboudreault/git/cstar/cassandra-dtest/disk_balance_test.py", line 
51, in disk_balance_bootstrap_test
self.assert_balanced(node)
  File "/home/aboudreault/git/cstar/cassandra-dtest/disk_balance_test.py", line 
127, in assert_balanced
assert_almost_equal(*sums, error=0.2, error_message=node.name)
  File "/home/aboudreault/git/cstar/cassandra-dtest/assertions.py", line 65, in 
assert_almost_equal
assert vmin > vmax * (1.0 - error) or vmin == vmax, "values not within 
%.2f%% of the max: %s (%s)" % (error * 100, args, kwargs['error_message'])
AssertionError: values not within 20.00% of the max: (529955, 386060, 473640) 
(node4)
 >> begin captured logging << 
dtest: DEBUG: cluster ccm directory: /tmp/dtest-nNoQzp
- >> end captured logging << -

--
Ran 1 test in 114.862s

FAILED (failures=1)

{code}

  was:
{code}
==
FAIL: disk_balance_bootstrap_test (disk_balance_test.TestDiskBalance)
--
Traceback (most recent call last):
  File "/home/aboudreault/git/cstar/cassandra-dtest/disk_balance_test.py", line 
51, in disk_balance_bootstrap_test
self.assert_balanced(node)
  File "/home/aboudreault/git/cstar/cassandra-dtest/disk_balance_test.py", line 
127, in assert_balanced
assert_almost_equal(*sums, error=0.2, error_message=node.name)
  File "/home/aboudreault/git/cstar/cassandra-dtest/assertions.py", line 65, in 
assert_almost_equal
assert vmin > vmax * (1.0 - error) or vmin == vmax, "values not within 
%.2f%% of the max: %s (%s)" % (error * 100, args, kwargs['error_message'])
AssertionError: values not within 20.00% of the max: (529955, 386060, 473640) 
(node4)
 >> begin captured logging << 
dtest: DEBUG: cluster ccm directory: /tmp/dtest-nNoQzp
- >> end captured logging << -

--
Ran 1 test in 114.862s

FAILED (failures=1)

{code}


> disk_balance_bootstrap_test is failing on trunk
> ---
>
> Key: CASSANDRA-10974
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10974
> Project: Cassandra
>  Issue Type: Bug
>  Components: Tools
>Reporter: Alan Boudreault
> Fix For: 3.2
>
>
> http://cassci.datastax.com/job/trunk_dtest/891/testReport/junit/disk_balance_test/TestDiskBalance/disk_balance_bootstrap_test/
> {code}
> ==
> FAIL: disk_balance_bootstrap_test (disk_balance_test.TestDiskBalance)
> --
> Traceback (most recent call last):
>   File "/home/aboudreault/git/cstar/cassandra-dtest/disk_balance_test.py", 
> line 51, in disk_balance_bootstrap_test
> self.assert_balanced(node)
>   File "/home/aboudreault/git/cstar/cassandra-dtest/disk_balance_test.py", 
> line 127, in assert_balanced
> assert_almost_equal(*sums, error=0.2, error_message=node.name)
>   File "/home/aboudreault/git/cstar/cassandra-dtest/assertions.py", line 65, 
> in assert_almost_equal
> assert vmin > vmax * (1.0 - error) or vmin == vmax, "values not within 
> %.2f%% of the max: %s (%s)" % (error * 100, args, kwargs['error_message'])
> AssertionError: values not within 20.00% of the max: (529955, 386060, 473640) 
> (node4)
>  >> begin captured logging << 
> dtest: DEBUG: cluster ccm directory: /tmp/dtest-nNoQzp
> - >> end captured logging << -
> --
> Ran 1 test in 114.862s
> FAILED (failures=1)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-10973) disk_balance_decommission_test is failing on trunk

[Alan Boudreault (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-10973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alan Boudreault updated CASSANDRA-10973:

Description: 
http://cassci.datastax.com/job/trunk_dtest/891/testReport/junit/disk_balance_test/TestDiskBalance/disk_balance_bootstrap_test/

{code}
==
FAIL: disk_balance_decommission_test (disk_balance_test.TestDiskBalance)
--
Traceback (most recent call last):
  File "/home/aboudreault/git/cstar/cassandra-dtest/disk_balance_test.py", line 
74, in disk_balance_decommission_test
self.assert_balanced(node)
  File "/home/aboudreault/git/cstar/cassandra-dtest/disk_balance_test.py", line 
127, in assert_balanced
assert_almost_equal(*sums, error=0.2, error_message=node.name)
  File "/home/aboudreault/git/cstar/cassandra-dtest/assertions.py", line 65, in 
assert_almost_equal
assert vmin > vmax * (1.0 - error) or vmin == vmax, "values not within 
%.2f%% of the max: %s (%s)" % (error * 100, args, kwargs['error_message'])
AssertionError: values not within 20.00% of the max: (482095, 477840, 612940) 
(node2)
 >> begin captured logging << 
dtest: DEBUG: cluster ccm directory: /tmp/dtest-SLbi3e
- >> end captured logging << -

--
Ran 1 test in 121.295s

FAILED (failures=1)
{code}

  was:
{code}
==
FAIL: disk_balance_decommission_test (disk_balance_test.TestDiskBalance)
--
Traceback (most recent call last):
  File "/home/aboudreault/git/cstar/cassandra-dtest/disk_balance_test.py", line 
74, in disk_balance_decommission_test
self.assert_balanced(node)
  File "/home/aboudreault/git/cstar/cassandra-dtest/disk_balance_test.py", line 
127, in assert_balanced
assert_almost_equal(*sums, error=0.2, error_message=node.name)
  File "/home/aboudreault/git/cstar/cassandra-dtest/assertions.py", line 65, in 
assert_almost_equal
assert vmin > vmax * (1.0 - error) or vmin == vmax, "values not within 
%.2f%% of the max: %s (%s)" % (error * 100, args, kwargs['error_message'])
AssertionError: values not within 20.00% of the max: (482095, 477840, 612940) 
(node2)
 >> begin captured logging << 
dtest: DEBUG: cluster ccm directory: /tmp/dtest-SLbi3e
- >> end captured logging << -

--
Ran 1 test in 121.295s

FAILED (failures=1)
{code}


> disk_balance_decommission_test is failing on trunk
> --
>
> Key: CASSANDRA-10973
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10973
> Project: Cassandra
>  Issue Type: Bug
>  Components: Tools
>Reporter: Alan Boudreault
> Fix For: 3.2
>
>
> http://cassci.datastax.com/job/trunk_dtest/891/testReport/junit/disk_balance_test/TestDiskBalance/disk_balance_bootstrap_test/
> {code}
> ==
> FAIL: disk_balance_decommission_test (disk_balance_test.TestDiskBalance)
> --
> Traceback (most recent call last):
>   File "/home/aboudreault/git/cstar/cassandra-dtest/disk_balance_test.py", 
> line 74, in disk_balance_decommission_test
> self.assert_balanced(node)
>   File "/home/aboudreault/git/cstar/cassandra-dtest/disk_balance_test.py", 
> line 127, in assert_balanced
> assert_almost_equal(*sums, error=0.2, error_message=node.name)
>   File "/home/aboudreault/git/cstar/cassandra-dtest/assertions.py", line 65, 
> in assert_almost_equal
> assert vmin > vmax * (1.0 - error) or vmin == vmax, "values not within 
> %.2f%% of the max: %s (%s)" % (error * 100, args, kwargs['error_message'])
> AssertionError: values not within 20.00% of the max: (482095, 477840, 612940) 
> (node2)
>  >> begin captured logging << 
> dtest: DEBUG: cluster ccm directory: /tmp/dtest-SLbi3e
> - >> end captured logging << -
> --
> Ran 1 test in 121.295s
> FAILED (failures=1)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (CASSANDRA-10974) disk_balance_bootstrap_test is failing on trunk

[Alan Boudreault (JIRA)]

Alan Boudreault created CASSANDRA-10974:
---

 Summary: disk_balance_bootstrap_test is failing on trunk
 Key: CASSANDRA-10974
 URL: https://issues.apache.org/jira/browse/CASSANDRA-10974
 Project: Cassandra
  Issue Type: Bug
  Components: Tools
Reporter: Alan Boudreault
 Fix For: 3.2


{code}
==
FAIL: disk_balance_bootstrap_test (disk_balance_test.TestDiskBalance)
--
Traceback (most recent call last):
  File "/home/aboudreault/git/cstar/cassandra-dtest/disk_balance_test.py", line 
51, in disk_balance_bootstrap_test
self.assert_balanced(node)
  File "/home/aboudreault/git/cstar/cassandra-dtest/disk_balance_test.py", line 
127, in assert_balanced
assert_almost_equal(*sums, error=0.2, error_message=node.name)
  File "/home/aboudreault/git/cstar/cassandra-dtest/assertions.py", line 65, in 
assert_almost_equal
assert vmin > vmax * (1.0 - error) or vmin == vmax, "values not within 
%.2f%% of the max: %s (%s)" % (error * 100, args, kwargs['error_message'])
AssertionError: values not within 20.00% of the max: (529955, 386060, 473640) 
(node4)
 >> begin captured logging << 
dtest: DEBUG: cluster ccm directory: /tmp/dtest-nNoQzp
- >> end captured logging << -

--
Ran 1 test in 114.862s

FAILED (failures=1)

{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-10502) Cassandra query degradation with high frequency updated tables

[Jean-Francois Gosselin (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-10502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15086025#comment-15086025
 ] 

Jean-Francois Gosselin commented on CASSANDRA-10502:


[~thobbs] Have you tried to dump the data for this key with sstable2json ? 

> Cassandra query degradation with high frequency updated tables
> --
>
> Key: CASSANDRA-10502
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10502
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Dodong Juan
>Priority: Minor
>  Labels: perfomance, query, triage
> Fix For: 2.2.x
>
>
> Hi,
> So we are developing a system that computes profile of things that it 
> observes.  The observation comes in form of events. Each thing that it 
> observe has an id and each thing has a set of subthings in it which has 
> measurement of some kind. Roughly there are about 500 subthings within each 
> thing. We receive events containing measurements of these 500 subthings every 
> 10 seconds or so.
> So as we receive events, we  read the old profile value, calculate the new 
> profile based on the new value and save it back. 
> One of the things we observe are the processes running on the server.
> We use the following schema to hold the profile. 
> {noformat}
> CREATE TABLE processinfometric_profile (
> profilecontext text,
> id text,
> month text,
> day text,
> hour text,
> minute text,
> command text,
> cpu map,
> majorfaults map,
> minorfaults map,
> nice map,
> pagefaults map,
> pid map,
> ppid map,
> priority map,
> resident map,
> rss map,
> sharesize map,
> size map,
> starttime map,
> state map,
> threads map,
> user map,
> vsize map,
> PRIMARY KEY ((profilecontext, id, month, day, hour, minute), command)
> ) WITH CLUSTERING ORDER BY (command ASC)
> AND bloom_filter_fp_chance = 0.1
> AND caching = '{"keys":"ALL", "rows_per_partition":"NONE"}'
> AND comment = ''
> AND compaction = {'class': 
> 'org.apache.cassandra.db.compaction.LeveledCompactionStrategy'}
> AND compression = {'sstable_compression': 
> 'org.apache.cassandra.io.compress.LZ4Compressor'}
> AND dclocal_read_repair_chance = 0.1
> AND default_time_to_live = 0
> AND gc_grace_seconds = 864000
> AND max_index_interval = 2048
> AND memtable_flush_period_in_ms = 0
> AND min_index_interval = 128
> AND read_repair_chance = 0.0
> AND speculative_retry = '99.0PERCENTILE';
> {noformat}
> This profile will then be use for certain analytics that can use in the 
> context of the ‘thing’ or in the context of specific thing and subthing. 
> A profile can be defined as monthly, daily, hourly. So in case of monthly the 
> month will be set to the current month (i.e. ‘Oct’) and the day and hour will 
> be set to empty ‘’ string.
> The problem that we have observed is that over time (actually in just a 
> matter of hours) we will see a huge degradation of query response  for the 
> monthly profile. At the start it will be respinding in 10-100 ms and after a 
> couple of hours it will go to 2000-3000 ms . If you leave it for a couple of 
> days you will start experiencing readtimeouts . The query is basically just :
> {noformat}
> select * from myprofile where id=‘1’ and month=‘Oct’ and day=‘’ and hour=‘' 
> and minute=''
> {noformat}
> This will have only about 500 rows or so.
> We were using Cassandra 2.2.1 , but upgraded to 2.2.2 to see if it fixed the 
> issue to no avail. And since this is a test, we are running on a single node.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-10973) disk_balance_decommission_test is failing on trunk

[Philip Thompson (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-10973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Philip Thompson updated CASSANDRA-10973:

Labels: dtest  (was: )

> disk_balance_decommission_test is failing on trunk
> --
>
> Key: CASSANDRA-10973
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10973
> Project: Cassandra
>  Issue Type: Bug
>  Components: Tools
>Reporter: Alan Boudreault
>  Labels: dtest
> Fix For: 3.2
>
>
> http://cassci.datastax.com/job/trunk_dtest/891/testReport/junit/disk_balance_test/TestDiskBalance/disk_balance_decommission_test/
> {code}
> ==
> FAIL: disk_balance_decommission_test (disk_balance_test.TestDiskBalance)
> --
> Traceback (most recent call last):
>   File "/home/aboudreault/git/cstar/cassandra-dtest/disk_balance_test.py", 
> line 74, in disk_balance_decommission_test
> self.assert_balanced(node)
>   File "/home/aboudreault/git/cstar/cassandra-dtest/disk_balance_test.py", 
> line 127, in assert_balanced
> assert_almost_equal(*sums, error=0.2, error_message=node.name)
>   File "/home/aboudreault/git/cstar/cassandra-dtest/assertions.py", line 65, 
> in assert_almost_equal
> assert vmin > vmax * (1.0 - error) or vmin == vmax, "values not within 
> %.2f%% of the max: %s (%s)" % (error * 100, args, kwargs['error_message'])
> AssertionError: values not within 20.00% of the max: (482095, 477840, 612940) 
> (node2)
>  >> begin captured logging << 
> dtest: DEBUG: cluster ccm directory: /tmp/dtest-SLbi3e
> - >> end captured logging << -
> --
> Ran 1 test in 121.295s
> FAILED (failures=1)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-10974) disk_balance_bootstrap_test is failing on trunk

[Philip Thompson (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-10974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Philip Thompson updated CASSANDRA-10974:

Labels: dtest  (was: )

> disk_balance_bootstrap_test is failing on trunk
> ---
>
> Key: CASSANDRA-10974
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10974
> Project: Cassandra
>  Issue Type: Bug
>  Components: Tools
>Reporter: Alan Boudreault
>  Labels: dtest
> Fix For: 3.2
>
>
> http://cassci.datastax.com/job/trunk_dtest/891/testReport/junit/disk_balance_test/TestDiskBalance/disk_balance_bootstrap_test/
> {code}
> ==
> FAIL: disk_balance_bootstrap_test (disk_balance_test.TestDiskBalance)
> --
> Traceback (most recent call last):
>   File "/home/aboudreault/git/cstar/cassandra-dtest/disk_balance_test.py", 
> line 51, in disk_balance_bootstrap_test
> self.assert_balanced(node)
>   File "/home/aboudreault/git/cstar/cassandra-dtest/disk_balance_test.py", 
> line 127, in assert_balanced
> assert_almost_equal(*sums, error=0.2, error_message=node.name)
>   File "/home/aboudreault/git/cstar/cassandra-dtest/assertions.py", line 65, 
> in assert_almost_equal
> assert vmin > vmax * (1.0 - error) or vmin == vmax, "values not within 
> %.2f%% of the max: %s (%s)" % (error * 100, args, kwargs['error_message'])
> AssertionError: values not within 20.00% of the max: (529955, 386060, 473640) 
> (node4)
>  >> begin captured logging << 
> dtest: DEBUG: cluster ccm directory: /tmp/dtest-nNoQzp
> - >> end captured logging << -
> --
> Ran 1 test in 114.862s
> FAILED (failures=1)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[Cassandra Wiki] Update of "Committers" by CarlYeksigian

[Apache Wiki]

Dear Wiki user,

You have subscribed to a wiki page or wiki category on "Cassandra Wiki" for 
change notification.

The "Committers" page has been changed by CarlYeksigian:
https://wiki.apache.org/cassandra/Committers?action=diff=55=56

  ||Robert Stupp ||Jan 2015 ||Datastax || ||
  ||Sam Tunnicliffe ||May 2015 ||Datastax || ||
  ||Benjamin Lerer ||Jul 2015 ||Datastax || ||
+ ||Carl Yeksigian || Jan 2016 || Datastax || Also a 
[[http://thrift.apache.org|Thrift]] committer ||
  
  
  {{https://c.statcounter.com/9397521/0/fe557aad/1/|stats}}

[jira] [Commented] (CASSANDRA-10971) Compressed commit log has no backpressure and can OOM

[Ariel Weisberg (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-10971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15086097#comment-15086097
 ] 

Ariel Weisberg commented on CASSANDRA-10971:


|[trunk 
code|https://github.com/apache/cassandra/compare/trunk...aweisberg:CASSANDRA-10971-trunk?expand=1]|[utest|http://cassci.datastax.com/view/Dev/view/aweisberg/job/aweisberg-CASSANDRA-10971-trunk-testall/]|[dtest|http://cassci.datastax.com/view/Dev/view/aweisberg/job/aweisberg-CASSANDRA-10971-trunk-dtest/]|
|[3.0 
code|https://github.com/apache/cassandra/compare/cassandra-3.0...aweisberg:CASSANDRA-10971-3.0?expand=1]|[utest|http://cassci.datastax.com/view/Dev/view/aweisberg/job/aweisberg-CASSANDRA-10971-3.0-testall/]|[dtest|http://cassci.datastax.com/view/Dev/view/aweisberg/job/aweisberg-CASSANDRA-10971-3.0-dtest/]|
|[2.2 
code|https://github.com/apache/cassandra/compare/cassandra-2.2...aweisberg:CASSANDRA-10971-2.2?expand=1]|[utest|http://cassci.datastax.com/view/Dev/view/aweisberg/job/aweisberg-CASSANDRA-10971-2.2-testall/]|[dtest|http://cassci.datastax.com/view/Dev/view/aweisberg/job/aweisberg-CASSANDRA-10971-2.2-dtest/]|

> Compressed commit log has no backpressure and can OOM
> -
>
> Key: CASSANDRA-10971
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10971
> Project: Cassandra
>  Issue Type: Bug
>  Components: Local Write-Read Paths
>Reporter: Ariel Weisberg
>Assignee: Ariel Weisberg
> Fix For: 2.2.x, 3.0.x, 3.x
>
>
> I validated this via a unit test that slowed the ability of the log to drain 
> to the filesystem. The compressed commit log will keep allocating buffers 
> pending compression until it OOMs.
> I have a fix that am not very happy with because the whole signal a thread to 
> allocate a segment that depends on a resource that may not be available 
> results in some obtuse usage of {{CompleatableFuture}} to rendezvous 
> available buffers with {{CommitLogSegmentManager}} thread waiting to finish 
> constructing a new segment. The {{CLSM}} thread is in turn signaled by the 
> thread(s) that actually wants to write to the next segment, but aren't able 
> to do it themselves.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (CASSANDRA-10975) Histogram buckets exposed in jmx are sorted by count

[Chris Lohfink (JIRA)]

Chris Lohfink created CASSANDRA-10975:
-

 Summary: Histogram buckets exposed in jmx are sorted by count
 Key: CASSANDRA-10975
 URL: https://issues.apache.org/jira/browse/CASSANDRA-10975
 Project: Cassandra
  Issue Type: Bug
  Components: Observability
Reporter: Chris Lohfink


The estimated histogram snapshot lets its parent provide the getValues 
implementation which sorts the bucket array:
https://github.com/dropwizard/metrics/blob/3.1-maintenance/metrics-core/src/main/java/com/codahale/metrics/UniformSnapshot.java#L25

making it hard to determine what count belonged to what bucket. Along with 
removal of the pre 2.2 deprecated metrics this makes it nearly impossible to 
track latencies over time.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-10975) Histogram buckets exposed in jmx are sorted by count

[Chris Lohfink (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-10975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chris Lohfink updated CASSANDRA-10975:
--
Attachment: 10975-2.2.patch

> Histogram buckets exposed in jmx are sorted by count
> 
>
> Key: CASSANDRA-10975
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10975
> Project: Cassandra
>  Issue Type: Bug
>  Components: Observability
>Reporter: Chris Lohfink
> Attachments: 10975-2.2.patch
>
>
> The estimated histogram snapshot lets its parent provide the getValues 
> implementation which sorts the bucket array:
> https://github.com/dropwizard/metrics/blob/3.1-maintenance/metrics-core/src/main/java/com/codahale/metrics/UniformSnapshot.java#L25
> making it hard to determine what count belonged to what bucket. Along with 
> removal of the pre 2.2 deprecated metrics this makes it nearly impossible to 
> track latencies over time.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (CASSANDRA-10879) HSHA dtest for closing connections almost always fails on CASSCI developer branches

[Carl Yeksigian (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-10879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Carl Yeksigian resolved CASSANDRA-10879.

Resolution: Fixed

The fix for CASSANDRA-10863 should resolve this issue as well; CI runs on my 
dev branches have completed successfully.

> HSHA dtest for closing connections almost always fails on CASSCI developer 
> branches
> ---
>
> Key: CASSANDRA-10879
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10879
> Project: Cassandra
>  Issue Type: Bug
>  Components: Testing
>Reporter: Branimir Lambov
>Assignee: Carl Yeksigian
>Priority: Critical
>
> The test consistently succeeds on 2.1/2.2 branches, but has been failing 
> almost every time on developer ones for quite a while, for example [9258 
> dtest|http://cassci.datastax.com/view/Dev/view/blambov/job/blambov-dikang85-9258-dtest/lastCompletedBuild/testReport/],
>  [10817 
> dtest|http://cassci.datastax.com/view/Dev/view/krummas/job/krummas-marcuse-10817-dtest/],
>  [10059 (committed) 
> dtest|http://cassci.datastax.com/view/Dev/view/blambov/job/blambov-10059-dtest/lastCompletedBuild/testReport/],
>  [unchanged 2.2 
> dtest|http://cassci.datastax.com/view/Dev/view/blambov/job/blambov-cassandra-2.2-dtest/lastCompletedBuild/testReport/]
>  and many others. The failures cannot be genuine if the same code committed 
> to the main branch no longer fails the test.
> This is making it hard to identify genuine failures and blocking 
> CASSANDRA-9669 in particular.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-10970) SSL/TLS: Certificate Domain is ignored

[Matthias Brandt (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-10970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matthias Brandt updated CASSANDRA-10970:

Description: 
I've set up server_encryption_options as well as client_encryption_options. In 
both settings, I use the same keystore with an wild-card SSL certificate in it. 
It is signed by our own CA, which root certificate is in the configured 
truststore:
{code}
server_encryption_options:
internode_encryption: all
keystore: /etc/cassandra/conf/wildcard-cert.keystore
keystore_password: ""
truststore: /etc/cassandra/conf/my-cacerts
truststore_password: changeit
require_client_auth: true

client_encryption_options:
enabled: true
keystore: /etc/cassandra/conf/wildcard-cert.keystore
keystore_password: ""
require_client_auth: false
{code}

The certifcate's subject is:
{code}CN=*.my.domain.com,OU=my unit,O=my org{code}

When I deploy this setting on a server which domain is 
node1.my.*other-domain*.com a connection via cqlsh wrongly works. Additionally, 
the inter-node connection between other nodes in this wrong domain also works.

I would expect that the connection would fail with a meaningful error message.

  was:
I've set up server_encryption_options as well as client_encryption_options. In 
both settings, I use the same keystore with an wild-card SSL certificate in it. 
It is signed by our own CA, which root certificate is in the configured 
truststore:
{code}
server_encryption_options:
internode_encryption: all
keystore: /etc/cassandra/conf/wildcard-cert.keystore
keystore_password: ""
truststore: /etc/cassandra/conf/hpo-cacerts
truststore_password: changeit
require_client_auth: true

client_encryption_options:
enabled: true
keystore: /etc/cassandra/conf/wildcard-cert.keystore
keystore_password: ""
require_client_auth: false
{code}

The certifcate's subject is:
{code}CN=*.my.domain.com,OU=my unit,O=my org{code}

When I deploy this setting on a server which domain is 
node1.my.*other-domain*.com a connection via cqlsh wrongly works. Additionally, 
the inter-node connection between other nodes in this wrong domain also works.

I would expect that the connection would fail with a meaningful error message.


> SSL/TLS: Certificate Domain is ignored
> --
>
> Key: CASSANDRA-10970
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10970
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Matthias Brandt
>
> I've set up server_encryption_options as well as client_encryption_options. 
> In both settings, I use the same keystore with an wild-card SSL certificate 
> in it. It is signed by our own CA, which root certificate is in the 
> configured truststore:
> {code}
> server_encryption_options:
> internode_encryption: all
> keystore: /etc/cassandra/conf/wildcard-cert.keystore
> keystore_password: ""
> truststore: /etc/cassandra/conf/my-cacerts
> truststore_password: changeit
> require_client_auth: true
> client_encryption_options:
> enabled: true
> keystore: /etc/cassandra/conf/wildcard-cert.keystore
> keystore_password: ""
> require_client_auth: false
> {code}
> The certifcate's subject is:
> {code}CN=*.my.domain.com,OU=my unit,O=my org{code}
> When I deploy this setting on a server which domain is 
> node1.my.*other-domain*.com a connection via cqlsh wrongly works. 
> Additionally, the inter-node connection between other nodes in this wrong 
> domain also works.
> I would expect that the connection would fail with a meaningful error message.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (CASSANDRA-10951) Fix ReadCommandTest

[Branimir Lambov (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-10951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15085733#comment-15085733
 ] 

Branimir Lambov edited comment on CASSANDRA-10951 at 1/6/16 4:23 PM:
-

Patch uploaded here:
|[code|https://github.com/blambov/cassandra/tree/10951]|[utests|http://cassci.datastax.com/view/Dev/view/blambov/job/blambov-10951-testall/]|[dtests|http://cassci.datastax.com/view/Dev/view/blambov/job/blambov-10951-dtest/]|

{{AbortedQueriesTester}} still doesn't time out; 
{{RecoveryManagerTest.testRecoverPIT}} passes locally but timed out twice on 
cassci. Will look into it a bit more.


was (Author: blambov):
Patch uploaded here:
|[code|https://github.com/blambov/cassandra/tree/10951]|[utests|http://cassci.datastax.com/view/Dev/view/blambov/job/blambov-10951-testall/]|[dtests|http://cassci.datastax.com/view/Dev/view/blambov/job/blambov-10951-dtest/]|

{{AbortedQueriesTester}} still times out; 
{{RecoveryManagerTest.testRecoverPIT}} passes locally but timed out twice on 
cassci. Will look into it a bit more.

> Fix ReadCommandTest
> ---
>
> Key: CASSANDRA-10951
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10951
> Project: Cassandra
>  Issue Type: Bug
>  Components: Local Write-Read Paths, Testing
>Reporter: Yuki Morishita
>Assignee: Branimir Lambov
>Priority: Blocker
> Fix For: 3.2
>
>
> [ReadCommandTest is 
> failing|http://cassci.datastax.com/view/trunk/job/trunk_testall/641/testReport/org.apache.cassandra.db/ReadCommandTest/history/]
>  since CASSANDRA-9975 was merged to trunk.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[3/6] cassandra git commit: MV timestamp should be the maximum of the values, not the minimum

[carl]

MV timestamp should be the maximum of the values, not the minimum

patch by Carl Yeksigian; reviewed by Jake Luciani for CASSANDRA-10910


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/70c08ece
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/70c08ece
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/70c08ece

Branch: refs/heads/trunk
Commit: 70c08ece563731cd546d24541f225c888f4d02f5
Parents: f937c8b
Author: Carl Yeksigian 
Authored: Wed Jan 6 10:41:47 2016 -0500
Committer: Carl Yeksigian 
Committed: Wed Jan 6 10:42:36 2016 -0500

--
 .../apache/cassandra/db/view/TemporalRow.java   | 65 ++--
 .../org/apache/cassandra/cql3/ViewTest.java | 30 +
 2 files changed, 76 insertions(+), 19 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/70c08ece/src/java/org/apache/cassandra/db/view/TemporalRow.java
--
diff --git a/src/java/org/apache/cassandra/db/view/TemporalRow.java 
b/src/java/org/apache/cassandra/db/view/TemporalRow.java
index 8898857..8ee310d 100644
--- a/src/java/org/apache/cassandra/db/view/TemporalRow.java
+++ b/src/java/org/apache/cassandra/db/view/TemporalRow.java
@@ -22,6 +22,7 @@ import java.nio.ByteBuffer;
 import java.util.ArrayList;
 import java.util.Collection;
 import java.util.Collections;
+import java.util.Comparator;
 import java.util.HashMap;
 import java.util.Iterator;
 import java.util.List;
@@ -279,9 +280,7 @@ public class TemporalRow
 this.nowInSec = nowInSec;
 
 LivenessInfo liveness = row.primaryKeyLivenessInfo();
-this.viewClusteringLocalDeletionTime = 
minValueIfSet(viewClusteringLocalDeletionTime, 
row.deletion().time().localDeletionTime(), NO_DELETION_TIME);
-this.viewClusteringTimestamp = minValueIfSet(viewClusteringTimestamp, 
liveness.timestamp(), NO_TIMESTAMP);
-this.viewClusteringTtl = minValueIfSet(viewClusteringTtl, 
liveness.ttl(), NO_TTL);
+updateLiveness(liveness.ttl(), liveness.timestamp(), 
row.deletion().time().localDeletionTime());
 
 List clusteringDefs = 
baseCfs.metadata.clusteringColumns();
 clusteringColumns = new HashMap<>();
@@ -295,6 +294,31 @@ public class TemporalRow
 }
 }
 
+/*
+ * PK ts:5, ttl:1, deletion: 2
+ * Col ts:4, ttl:2, deletion: 3
+ *
+ * TTL use min, since it expires at the lowest time which we are expiring. 
If we have the above values, we
+ * would want to return 1, since the base row expires in 1 second.
+ *
+ * Timestamp uses max, as this is the time that the row has been written 
to the view. See CASSANDRA-10910.
+ *
+ * Local Deletion Time should use max, as this deletion will cover all 
previous values written.
+ */
+@SuppressWarnings("unchecked")
+private void updateLiveness(int ttl, long timestamp, int localDeletionTime)
+{
+// We are returning whichever is higher from valueIfSet
+// Natural order will return the max: 1.compareTo(2) < 0, so 2 is 
returned
+// Reverse order will return the min: 1.compareTo(2) > 0, so 1 is 
returned
+final Comparator max = Comparator.naturalOrder();
+final Comparator min = Comparator.reverseOrder();
+
+this.viewClusteringTtl = valueIfSet(viewClusteringTtl, ttl, NO_TTL, 
min);
+this.viewClusteringTimestamp = valueIfSet(viewClusteringTimestamp, 
timestamp, NO_TIMESTAMP, max);
+this.viewClusteringLocalDeletionTime = 
valueIfSet(viewClusteringLocalDeletionTime, localDeletionTime, 
NO_DELETION_TIME, max);
+}
+
 @Override
 public String toString()
 {
@@ -351,30 +375,33 @@ public class TemporalRow
 // If this column is part of the view's primary keys
 if (viewPrimaryKey.contains(identifier))
 {
-this.viewClusteringTtl = minValueIfSet(this.viewClusteringTtl, 
ttl, NO_TTL);
-this.viewClusteringTimestamp = 
minValueIfSet(this.viewClusteringTimestamp, timestamp, NO_TIMESTAMP);
-this.viewClusteringLocalDeletionTime = 
minValueIfSet(this.viewClusteringLocalDeletionTime, localDeletionTime, 
NO_DELETION_TIME);
+updateLiveness(ttl, timestamp, localDeletionTime);
 }
 
 innerMap.get(cellPath).setVersion(new TemporalCell(value, timestamp, 
ttl, localDeletionTime, isNew));
 }
 
-private static int minValueIfSet(int existing, int update, int 
defaultValue)
-{
-if (existing == defaultValue)
-return update;
-if (update == defaultValue)
-return existing;
-return Math.min(existing, update);
-}
-
-private static long minValueIfSet(long existing, long update, long 
defaultValue)
+/**

[04/50] [abbrv] cassandra git commit: Merge commit '11716547f87f4c88a2790323744b0ed97175854d' into cassandra-2.2

[slebresne]

Merge commit '11716547f87f4c88a2790323744b0ed97175854d' into cassandra-2.2


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/078aabe3
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/078aabe3
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/078aabe3

Branch: refs/heads/trunk
Commit: 078aabe3d40bfa511642918ea6529016fe7c0620
Parents: 9ca7e16 1171654
Author: Sylvain Lebresne 
Authored: Wed Jan 6 17:56:01 2016 +0100
Committer: Sylvain Lebresne 
Committed: Wed Jan 6 17:56:01 2016 +0100

--

--

[02/50] [abbrv] cassandra git commit: Merge commit '11716547f87f4c88a2790323744b0ed97175854d' into cassandra-2.2

[slebresne]

Merge commit '11716547f87f4c88a2790323744b0ed97175854d' into cassandra-2.2


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/078aabe3
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/078aabe3
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/078aabe3

Branch: refs/heads/cassandra-3.0
Commit: 078aabe3d40bfa511642918ea6529016fe7c0620
Parents: 9ca7e16 1171654
Author: Sylvain Lebresne 
Authored: Wed Jan 6 17:56:01 2016 +0100
Committer: Sylvain Lebresne 
Committed: Wed Jan 6 17:56:01 2016 +0100

--

--

[08/50] [abbrv] cassandra git commit: Match cassandra-loader options in COPY FROM (2.2 version)

[slebresne]

http://git-wip-us.apache.org/repos/asf/cassandra/blob/202cf9b0/pylib/cqlshlib/formatting.py
--
diff --git a/pylib/cqlshlib/formatting.py b/pylib/cqlshlib/formatting.py
index 62ecd10..2e219c8 100644
--- a/pylib/cqlshlib/formatting.py
+++ b/pylib/cqlshlib/formatting.py
@@ -60,7 +60,8 @@ empty_colormap = defaultdict(lambda: '')
 
 
 def format_by_type(cqltype, val, encoding, colormap=None, addcolor=False,
-   nullval=None, date_time_format=None, float_precision=None):
+   nullval=None, date_time_format=None, float_precision=None,
+   decimal_sep=None, thousands_sep=None, boolean_styles=None):
 if nullval is None:
 nullval = default_null_placeholder
 if val is None:
@@ -75,7 +76,8 @@ def format_by_type(cqltype, val, encoding, colormap=None, 
addcolor=False,
 float_precision = default_float_precision
 return format_value(cqltype, val, encoding=encoding, colormap=colormap,
 date_time_format=date_time_format, 
float_precision=float_precision,
-nullval=nullval)
+nullval=nullval, decimal_sep=decimal_sep, 
thousands_sep=thousands_sep,
+boolean_styles=boolean_styles)
 
 
 def color_text(bval, colormap, displaywidth=None):
@@ -155,7 +157,9 @@ def format_python_formatted_type(val, colormap, color, 
quote=False):
 
 
 @formatter_for('Decimal')
-def format_value_decimal(val, colormap, **_):
+def format_value_decimal(val, float_precision, colormap, decimal_sep=None, 
thousands_sep=None, **_):
+if (decimal_sep and decimal_sep != '.') or thousands_sep:
+return format_floating_point_type(val, colormap, float_precision, 
decimal_sep, thousands_sep)
 return format_python_formatted_type(val, colormap, 'decimal')
 
 
@@ -170,34 +174,60 @@ def formatter_value_inet(val, colormap, quote=False, **_):
 
 
 @formatter_for('bool')
-def format_value_boolean(val, colormap, **_):
+def format_value_boolean(val, colormap, boolean_styles=None, **_):
+if boolean_styles:
+val = boolean_styles[0] if val else boolean_styles[1]
 return format_python_formatted_type(val, colormap, 'boolean')
 
 
-def format_floating_point_type(val, colormap, float_precision, **_):
+def format_floating_point_type(val, colormap, float_precision, 
decimal_sep=None, thousands_sep=None, **_):
 if math.isnan(val):
 bval = 'NaN'
 elif math.isinf(val):
 bval = 'Infinity' if val > 0 else '-Infinity'
 else:
-exponent = int(math.log10(abs(val))) if abs(val) > 
sys.float_info.epsilon else -sys.maxsize - 1
-if -4 <= exponent < float_precision:
-# when this is true %g will not use scientific notation,
-# increasing precision should not change this decision
-# so we increase the precision to take into account the
-# digits to the left of the decimal point
-float_precision = float_precision + exponent + 1
-bval = '%.*g' % (float_precision, val)
+if thousands_sep:
+dpart, ipart = math.modf(val)
+bval = format_integer_with_thousands_sep(ipart, thousands_sep)
+dpart_str = ('%.*f' % (float_precision, 
math.fabs(dpart)))[2:].rstrip('0')
+if dpart_str:
+bval += '%s%s' % ('.' if not decimal_sep else decimal_sep, 
dpart_str)
+else:
+exponent = int(math.log10(abs(val))) if abs(val) > 
sys.float_info.epsilon else -sys.maxsize - 1
+if -4 <= exponent < float_precision:
+# when this is true %g will not use scientific notation,
+# increasing precision should not change this decision
+# so we increase the precision to take into account the
+# digits to the left of the decimal point
+float_precision = float_precision + exponent + 1
+bval = '%.*g' % (float_precision, val)
+if decimal_sep:
+bval = bval.replace('.', decimal_sep)
+
 return colorme(bval, colormap, 'float')
 
 formatter_for('float')(format_floating_point_type)
 
 
-def format_integer_type(val, colormap, **_):
+def format_integer_type(val, colormap, thousands_sep=None, **_):
 # base-10 only for now; support others?
-bval = str(val)
+bval = format_integer_with_thousands_sep(val, thousands_sep) if 
thousands_sep else str(val)
 return colorme(bval, colormap, 'int')
 
+# We can get rid of this in cassandra-2.2
+if sys.version_info >= (2, 7):
+def format_integer_with_thousands_sep(val, thousands_sep=','):
+return "{:,.0f}".format(val).replace(',', thousands_sep)
+else:
+def format_integer_with_thousands_sep(val, thousands_sep=','):
+if val < 0:
+return '-' + format_integer_with_thousands_sep(-val, thousands_sep)
+result = ''
+while val >= 1000:
+val, r = 

[06/50] [abbrv] cassandra git commit: Match cassandra-loader options in COPY FROM (2.2 version)

[slebresne]

http://git-wip-us.apache.org/repos/asf/cassandra/blob/202cf9b0/pylib/cqlshlib/copyutil.py
--
diff --git a/pylib/cqlshlib/copyutil.py b/pylib/cqlshlib/copyutil.py
index 6d9a455..a154363 100644
--- a/pylib/cqlshlib/copyutil.py
+++ b/pylib/cqlshlib/copyutil.py
@@ -14,12 +14,14 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
+import ConfigParser
 import csv
+import datetime
 import json
+import glob
 import multiprocessing as mp
 import os
 import Queue
-import random
 import re
 import struct
 import sys
@@ -46,81 +48,202 @@ from displaying import NO_COLOR_MAP
 from formatting import format_value_default, DateTimeFormat, EMPTY, 
get_formatter
 from sslhandling import ssl_settings
 
+CopyOptions = namedtuple('CopyOptions', 'copy dialect unrecognized')
 
-def parse_options(shell, opts):
-"""
-Parse options for import (COPY FROM) and export (COPY TO) operations.
-Extract from opts csv and dialect options.
 
-:return: 3 dictionaries: the csv options, the dialect options, any 
unrecognized options.
-"""
-dialect_options = shell.csv_dialect_defaults.copy()
-if 'quote' in opts:
-dialect_options['quotechar'] = opts.pop('quote')
-if 'escape' in opts:
-dialect_options['escapechar'] = opts.pop('escape')
-if 'delimiter' in opts:
-dialect_options['delimiter'] = opts.pop('delimiter')
-if dialect_options['quotechar'] == dialect_options['escapechar']:
-dialect_options['doublequote'] = True
-del dialect_options['escapechar']
-
-csv_options = dict()
-csv_options['nullval'] = opts.pop('null', '')
-csv_options['header'] = bool(opts.pop('header', '').lower() == 'true')
-csv_options['encoding'] = opts.pop('encoding', 'utf8')
-csv_options['maxrequests'] = int(opts.pop('maxrequests', 6))
-csv_options['pagesize'] = int(opts.pop('pagesize', 1000))
-# by default the page timeout is 10 seconds per 1000 entries in the page 
size or 10 seconds if pagesize is smaller
-csv_options['pagetimeout'] = int(opts.pop('pagetimeout', max(10, 10 * 
(csv_options['pagesize'] / 1000
-csv_options['maxattempts'] = int(opts.pop('maxattempts', 5))
-csv_options['dtformats'] = DateTimeFormat(opts.pop('timeformat', 
shell.display_timestamp_format),
-  shell.display_date_format,
-  shell.display_nanotime_format)
-csv_options['float_precision'] = shell.display_float_precision
-csv_options['chunksize'] = int(opts.pop('chunksize', 1000))
-csv_options['ingestrate'] = int(opts.pop('ingestrate', 10))
-csv_options['maxbatchsize'] = int(opts.pop('maxbatchsize', 20))
-csv_options['minbatchsize'] = int(opts.pop('minbatchsize', 2))
-csv_options['reportfrequency'] = float(opts.pop('reportfrequency', 0.25))
-
-return csv_options, dialect_options, opts
-
-
-def get_num_processes(cap):
+def safe_normpath(fname):
 """
-Pick a reasonable number of child processes. We need to leave at
-least one core for the parent process.  This doesn't necessarily
-need to be capped, but 4 is currently enough to keep
-a single local Cassandra node busy so we use this for import, whilst
-for export we use 16 since we can connect to multiple Cassandra nodes.
-Eventually this parameter will become an option.
+:return the normalized path but only if there is a filename, we don't want 
to convert
+an empty string (which means no file name) to a dot. Also expand any user 
variables such as ~ to the full path
 """
-try:
-return max(1, min(cap, mp.cpu_count() - 1))
-except NotImplementedError:
-return 1
+return os.path.normpath(os.path.expanduser(fname)) if fname else fname
 
 
 class CopyTask(object):
 """
 A base class for ImportTask and ExportTask
 """
-def __init__(self, shell, ks, cf, columns, fname, csv_options, 
dialect_options, protocol_version, config_file):
+def __init__(self, shell, ks, table, columns, fname, opts, 
protocol_version, config_file, direction):
 self.shell = shell
-self.csv_options = csv_options
-self.dialect_options = dialect_options
 self.ks = ks
-self.cf = cf
-self.columns = shell.get_column_names(ks, cf) if columns is None else 
columns
-self.fname = fname
+self.table = table
+self.local_dc = shell.conn.metadata.get_host(shell.hostname).datacenter
+self.fname = safe_normpath(fname)
 self.protocol_version = protocol_version
 self.config_file = config_file
+# do not display messages when exporting to STDOUT
+self.printmsg = self._printmsg if self.fname is not None or direction 
== 'in' else lambda _, eol='\n': None
+self.options = self.parse_options(opts, direction)
+
+self.num_processes = 

[07/50] [abbrv] cassandra git commit: Match cassandra-loader options in COPY FROM (2.2 version)

[slebresne]

Match cassandra-loader options in COPY FROM (2.2 version)

patch by Stefania; reviewed by pauloricardomg for CASSANDRA-9303


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/202cf9b0
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/202cf9b0
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/202cf9b0

Branch: refs/heads/cassandra-3.2
Commit: 202cf9b0bed8bbff41318f1f10043aabf3a7cd4d
Parents: 078aabe
Author: Stefania Alborghetti 
Authored: Wed Jan 6 12:10:13 2016 +0100
Committer: Sylvain Lebresne 
Committed: Wed Jan 6 17:56:30 2016 +0100

--
 CHANGES.txt |1 +
 NEWS.txt|7 +
 bin/cqlsh.py|  135 +-
 conf/cqlshrc.sample |   17 +-
 pylib/cqlshlib/copyutil.py  | 1260 +-
 pylib/cqlshlib/formatting.py|   96 +-
 .../cql3/statements/BatchStatement.java |   28 +-
 .../cassandra/service/ClientWarningsTest.java   |5 +-
 tools/bin/cassandra-stress.bat  |2 +-
 9 files changed, 1134 insertions(+), 417 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/202cf9b0/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index fc87c7d..b12f593 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -13,6 +13,7 @@
  * Disable reloading of GossipingPropertyFileSnitch (CASSANDRA-9474)
  * Verify tables in pseudo-system keyspaces at startup (CASSANDRA-10761)
 Merged from 2.1:
+ * Match cassandra-loader options in COPY FROM (CASSANDRA-9303)
  * Fix binding to any address in CqlBulkRecordWriter (CASSANDRA-9309)
  * cqlsh fails to decode utf-8 characters for text typed columns 
(CASSANDRA-10875)
  * Log error when stream session fails (CASSANDRA-9294)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/202cf9b0/NEWS.txt
--
diff --git a/NEWS.txt b/NEWS.txt
index 57e321e..8cbe4f7 100644
--- a/NEWS.txt
+++ b/NEWS.txt
@@ -269,6 +269,13 @@ Upgrading
  to exclude data centers when the global status is enabled, see 
CASSANDRA-9035 for details.
 
 
+2.1.13
+==
+
+New features
+
+- New options for cqlsh COPY FROM and COPY TO, see CASSANDRA-9303 for 
details.
+
 2.1.10
 =
 

http://git-wip-us.apache.org/repos/asf/cassandra/blob/202cf9b0/bin/cqlsh.py
--
diff --git a/bin/cqlsh.py b/bin/cqlsh.py
index 8469836..c38bc2e 100644
--- a/bin/cqlsh.py
+++ b/bin/cqlsh.py
@@ -41,7 +41,6 @@ import optparse
 import os
 import platform
 import sys
-import time
 import traceback
 import warnings
 import webbrowser
@@ -151,7 +150,8 @@ cqlshlibdir = os.path.join(CASSANDRA_PATH, 'pylib')
 if os.path.isdir(cqlshlibdir):
 sys.path.insert(0, cqlshlibdir)
 
-from cqlshlib import cql3handling, cqlhandling, copyutil, pylexotron, 
sslhandling
+from cqlshlib import cql3handling, cqlhandling, pylexotron, sslhandling
+from cqlshlib.copyutil import ExportTask, ImportTask
 from cqlshlib.displaying import (ANSI_RESET, BLUE, COLUMN_NAME_COLORS, CYAN,
  RED, FormattedValue, colorme)
 from cqlshlib.formatting import (DEFAULT_DATE_FORMAT, DEFAULT_NANOTIME_FORMAT,
@@ -456,10 +456,12 @@ def complete_copy_column_names(ctxt, cqlsh):
 return set(colnames[1:]) - set(existcols)
 
 
-COPY_COMMON_OPTIONS = ['DELIMITER', 'QUOTE', 'ESCAPE', 'HEADER', 'NULL',
-   'MAXATTEMPTS', 'REPORTFREQUENCY']
-COPY_FROM_OPTIONS = ['CHUNKSIZE', 'INGESTRATE', 'MAXBATCHSIZE', 'MINBATCHSIZE']
-COPY_TO_OPTIONS = ['ENCODING', 'TIMEFORMAT', 'PAGESIZE', 'PAGETIMEOUT', 
'MAXREQUESTS']
+COPY_COMMON_OPTIONS = ['DELIMITER', 'QUOTE', 'ESCAPE', 'HEADER', 'NULL', 
'DATETIMEFORMAT',
+   'MAXATTEMPTS', 'REPORTFREQUENCY', 'DECIMALSEP', 
'THOUSANDSSEP', 'BOOLSTYLE',
+   'NUMPROCESSES', 'CONFIGFILE', 'RATEFILE']
+COPY_FROM_OPTIONS = ['CHUNKSIZE', 'INGESTRATE', 'MAXBATCHSIZE', 
'MINBATCHSIZE', 'MAXROWS',
+ 'SKIPROWS', 'SKIPCOLS', 'MAXPARSEERRORS', 
'MAXINSERTERRORS', 'ERRFILE']
+COPY_TO_OPTIONS = ['ENCODING', 'PAGESIZE', 'PAGETIMEOUT', 'BEGINTOKEN', 
'ENDTOKEN', 'MAXOUTPUTSIZE', 'MAXREQUESTS']
 
 
 @cqlsh_syntax_completer('copyOption', 'optnames')
@@ -575,23 +577,6 @@ warnings.showwarning = show_warning_without_quoting_line
 warnings.filterwarnings('always', 
category=cql3handling.UnexpectedTableStructure)
 
 
-def describe_interval(seconds):
-desc = []
-for length, unit in ((86400, 'day'), (3600, 'hour'), (60, 'minute')):
-num = 

[jira] [Commented] (CASSANDRA-10910) Materialized view remained rows

[T Jake Luciani (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-10910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15085792#comment-15085792
 ] 

T Jake Luciani commented on CASSANDRA-10910:


+1

> Materialized view remained rows
> ---
>
> Key: CASSANDRA-10910
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10910
> Project: Cassandra
>  Issue Type: Bug
> Environment: Cassandra 3.0.0
>Reporter: Gábor Auth
>Assignee: Carl Yeksigian
> Fix For: 3.0.x, 3.x
>
>
> I've created a table and a materialized view.
> {code}
> > CREATE TABLE test (id text PRIMARY KEY, key text, value int);
> > CREATE MATERIALIZED VIEW test_view AS SELECT * FROM test WHERE key IS NOT 
> > NULL PRIMARY KEY(key, id);
> {code}
> I've put a value into the table:
> {code}
> > update test set key='key', value=1 where id='id';
> > select * from test; select * from test_view ;
>  id | key | value
> +-+---
>  id | key | 1
> (1 rows)
>  key | id | value
> -++---
>  key | id | 1
> (1 rows)
> {code}
> I've updated the value without specified the key of the materialized view:
> {code}
> > update test set value=2 where id='id';
> > select * from test; select * from test_view ;
>  id | key | value
> +-+---
>  id | key | 2
> (1 rows)
>  key | id | value
> -++---
>  key | id | 2
> (1 rows)
> {code}
> It works as I think...
> ...but I've updated the key of the materialized view:
> {code}
> > update test set key='newKey' where id='id';
> > select * from test; select * from test_view ;
>  id | key| value
> ++---
>  id | newKey | 2
> (1 rows)
>  key| id | value
> ++---
> key | id | 2
>  newKey | id | 2
> (2 rows)
> {code}
> ...I've updated the value of the row:
> {code}
> > update test set key='newKey', value=3 where id='id';
> > select * from test; select * from test_view ;
>  id | key| value
> ++---
>  id | newKey | 3
> (1 rows)
>  key| id | value
> ++---
> key | id | 2
>  newKey | id | 3
> (2 rows)
> {code}
> ...I've deleted the row by the id key:
> {code}
> > delete from test where id='id';
> > select * from test; select * from test_view ;
>  id | key | value
> +-+---
> (0 rows)
>  key | id | value
> -++---
>  key | id | 2
> (1 rows)
> {code}
> Is it a bug?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[6/6] cassandra git commit: Merge branch 'cassandra-3.3' into trunk

[carl]

Merge branch 'cassandra-3.3' into trunk


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/fa1707fa
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/fa1707fa
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/fa1707fa

Branch: refs/heads/trunk
Commit: fa1707fa6859d6391ae7a33ca69c634ba12d83e5
Parents: 786d676 91aeb26
Author: Carl Yeksigian 
Authored: Wed Jan 6 11:49:33 2016 -0500
Committer: Carl Yeksigian 
Committed: Wed Jan 6 11:49:33 2016 -0500

--
 .../apache/cassandra/db/view/TemporalRow.java   | 65 ++--
 .../org/apache/cassandra/cql3/ViewTest.java | 30 +
 2 files changed, 76 insertions(+), 19 deletions(-)
--

[1/6] cassandra git commit: MV timestamp should be the maximum of the values, not the minimum

[carl]

Repository: cassandra
Updated Branches:
  refs/heads/cassandra-3.0 f937c8bee -> 70c08ece5
  refs/heads/cassandra-3.3 7fbc39b74 -> 91aeb2637
  refs/heads/trunk 786d67675 -> fa1707fa6


MV timestamp should be the maximum of the values, not the minimum

patch by Carl Yeksigian; reviewed by Jake Luciani for CASSANDRA-10910


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/70c08ece
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/70c08ece
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/70c08ece

Branch: refs/heads/cassandra-3.0
Commit: 70c08ece563731cd546d24541f225c888f4d02f5
Parents: f937c8b
Author: Carl Yeksigian 
Authored: Wed Jan 6 10:41:47 2016 -0500
Committer: Carl Yeksigian 
Committed: Wed Jan 6 10:42:36 2016 -0500

--
 .../apache/cassandra/db/view/TemporalRow.java   | 65 ++--
 .../org/apache/cassandra/cql3/ViewTest.java | 30 +
 2 files changed, 76 insertions(+), 19 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/70c08ece/src/java/org/apache/cassandra/db/view/TemporalRow.java
--
diff --git a/src/java/org/apache/cassandra/db/view/TemporalRow.java 
b/src/java/org/apache/cassandra/db/view/TemporalRow.java
index 8898857..8ee310d 100644
--- a/src/java/org/apache/cassandra/db/view/TemporalRow.java
+++ b/src/java/org/apache/cassandra/db/view/TemporalRow.java
@@ -22,6 +22,7 @@ import java.nio.ByteBuffer;
 import java.util.ArrayList;
 import java.util.Collection;
 import java.util.Collections;
+import java.util.Comparator;
 import java.util.HashMap;
 import java.util.Iterator;
 import java.util.List;
@@ -279,9 +280,7 @@ public class TemporalRow
 this.nowInSec = nowInSec;
 
 LivenessInfo liveness = row.primaryKeyLivenessInfo();
-this.viewClusteringLocalDeletionTime = 
minValueIfSet(viewClusteringLocalDeletionTime, 
row.deletion().time().localDeletionTime(), NO_DELETION_TIME);
-this.viewClusteringTimestamp = minValueIfSet(viewClusteringTimestamp, 
liveness.timestamp(), NO_TIMESTAMP);
-this.viewClusteringTtl = minValueIfSet(viewClusteringTtl, 
liveness.ttl(), NO_TTL);
+updateLiveness(liveness.ttl(), liveness.timestamp(), 
row.deletion().time().localDeletionTime());
 
 List clusteringDefs = 
baseCfs.metadata.clusteringColumns();
 clusteringColumns = new HashMap<>();
@@ -295,6 +294,31 @@ public class TemporalRow
 }
 }
 
+/*
+ * PK ts:5, ttl:1, deletion: 2
+ * Col ts:4, ttl:2, deletion: 3
+ *
+ * TTL use min, since it expires at the lowest time which we are expiring. 
If we have the above values, we
+ * would want to return 1, since the base row expires in 1 second.
+ *
+ * Timestamp uses max, as this is the time that the row has been written 
to the view. See CASSANDRA-10910.
+ *
+ * Local Deletion Time should use max, as this deletion will cover all 
previous values written.
+ */
+@SuppressWarnings("unchecked")
+private void updateLiveness(int ttl, long timestamp, int localDeletionTime)
+{
+// We are returning whichever is higher from valueIfSet
+// Natural order will return the max: 1.compareTo(2) < 0, so 2 is 
returned
+// Reverse order will return the min: 1.compareTo(2) > 0, so 1 is 
returned
+final Comparator max = Comparator.naturalOrder();
+final Comparator min = Comparator.reverseOrder();
+
+this.viewClusteringTtl = valueIfSet(viewClusteringTtl, ttl, NO_TTL, 
min);
+this.viewClusteringTimestamp = valueIfSet(viewClusteringTimestamp, 
timestamp, NO_TIMESTAMP, max);
+this.viewClusteringLocalDeletionTime = 
valueIfSet(viewClusteringLocalDeletionTime, localDeletionTime, 
NO_DELETION_TIME, max);
+}
+
 @Override
 public String toString()
 {
@@ -351,30 +375,33 @@ public class TemporalRow
 // If this column is part of the view's primary keys
 if (viewPrimaryKey.contains(identifier))
 {
-this.viewClusteringTtl = minValueIfSet(this.viewClusteringTtl, 
ttl, NO_TTL);
-this.viewClusteringTimestamp = 
minValueIfSet(this.viewClusteringTimestamp, timestamp, NO_TIMESTAMP);
-this.viewClusteringLocalDeletionTime = 
minValueIfSet(this.viewClusteringLocalDeletionTime, localDeletionTime, 
NO_DELETION_TIME);
+updateLiveness(ttl, timestamp, localDeletionTime);
 }
 
 innerMap.get(cellPath).setVersion(new TemporalCell(value, timestamp, 
ttl, localDeletionTime, isNew));
 }
 
-private static int minValueIfSet(int existing, int update, int 
defaultValue)
-{
-if (existing == defaultValue)
-return update;
-if (update == 

[5/6] cassandra git commit: Merge branch 'cassandra-3.0' into cassandra-3.3

[carl]

Merge branch 'cassandra-3.0' into cassandra-3.3


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/91aeb263
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/91aeb263
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/91aeb263

Branch: refs/heads/cassandra-3.3
Commit: 91aeb2637ebec72992de67e02f48d3597b15219f
Parents: 7fbc39b 70c08ec
Author: Carl Yeksigian 
Authored: Wed Jan 6 10:43:35 2016 -0500
Committer: Carl Yeksigian 
Committed: Wed Jan 6 10:43:35 2016 -0500

--
 .../apache/cassandra/db/view/TemporalRow.java   | 65 ++--
 .../org/apache/cassandra/cql3/ViewTest.java | 30 +
 2 files changed, 76 insertions(+), 19 deletions(-)
--

[2/6] cassandra git commit: MV timestamp should be the maximum of the values, not the minimum

[carl]

MV timestamp should be the maximum of the values, not the minimum

patch by Carl Yeksigian; reviewed by Jake Luciani for CASSANDRA-10910


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/70c08ece
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/70c08ece
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/70c08ece

Branch: refs/heads/cassandra-3.3
Commit: 70c08ece563731cd546d24541f225c888f4d02f5
Parents: f937c8b
Author: Carl Yeksigian 
Authored: Wed Jan 6 10:41:47 2016 -0500
Committer: Carl Yeksigian 
Committed: Wed Jan 6 10:42:36 2016 -0500

--
 .../apache/cassandra/db/view/TemporalRow.java   | 65 ++--
 .../org/apache/cassandra/cql3/ViewTest.java | 30 +
 2 files changed, 76 insertions(+), 19 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/70c08ece/src/java/org/apache/cassandra/db/view/TemporalRow.java
--
diff --git a/src/java/org/apache/cassandra/db/view/TemporalRow.java 
b/src/java/org/apache/cassandra/db/view/TemporalRow.java
index 8898857..8ee310d 100644
--- a/src/java/org/apache/cassandra/db/view/TemporalRow.java
+++ b/src/java/org/apache/cassandra/db/view/TemporalRow.java
@@ -22,6 +22,7 @@ import java.nio.ByteBuffer;
 import java.util.ArrayList;
 import java.util.Collection;
 import java.util.Collections;
+import java.util.Comparator;
 import java.util.HashMap;
 import java.util.Iterator;
 import java.util.List;
@@ -279,9 +280,7 @@ public class TemporalRow
 this.nowInSec = nowInSec;
 
 LivenessInfo liveness = row.primaryKeyLivenessInfo();
-this.viewClusteringLocalDeletionTime = 
minValueIfSet(viewClusteringLocalDeletionTime, 
row.deletion().time().localDeletionTime(), NO_DELETION_TIME);
-this.viewClusteringTimestamp = minValueIfSet(viewClusteringTimestamp, 
liveness.timestamp(), NO_TIMESTAMP);
-this.viewClusteringTtl = minValueIfSet(viewClusteringTtl, 
liveness.ttl(), NO_TTL);
+updateLiveness(liveness.ttl(), liveness.timestamp(), 
row.deletion().time().localDeletionTime());
 
 List clusteringDefs = 
baseCfs.metadata.clusteringColumns();
 clusteringColumns = new HashMap<>();
@@ -295,6 +294,31 @@ public class TemporalRow
 }
 }
 
+/*
+ * PK ts:5, ttl:1, deletion: 2
+ * Col ts:4, ttl:2, deletion: 3
+ *
+ * TTL use min, since it expires at the lowest time which we are expiring. 
If we have the above values, we
+ * would want to return 1, since the base row expires in 1 second.
+ *
+ * Timestamp uses max, as this is the time that the row has been written 
to the view. See CASSANDRA-10910.
+ *
+ * Local Deletion Time should use max, as this deletion will cover all 
previous values written.
+ */
+@SuppressWarnings("unchecked")
+private void updateLiveness(int ttl, long timestamp, int localDeletionTime)
+{
+// We are returning whichever is higher from valueIfSet
+// Natural order will return the max: 1.compareTo(2) < 0, so 2 is 
returned
+// Reverse order will return the min: 1.compareTo(2) > 0, so 1 is 
returned
+final Comparator max = Comparator.naturalOrder();
+final Comparator min = Comparator.reverseOrder();
+
+this.viewClusteringTtl = valueIfSet(viewClusteringTtl, ttl, NO_TTL, 
min);
+this.viewClusteringTimestamp = valueIfSet(viewClusteringTimestamp, 
timestamp, NO_TIMESTAMP, max);
+this.viewClusteringLocalDeletionTime = 
valueIfSet(viewClusteringLocalDeletionTime, localDeletionTime, 
NO_DELETION_TIME, max);
+}
+
 @Override
 public String toString()
 {
@@ -351,30 +375,33 @@ public class TemporalRow
 // If this column is part of the view's primary keys
 if (viewPrimaryKey.contains(identifier))
 {
-this.viewClusteringTtl = minValueIfSet(this.viewClusteringTtl, 
ttl, NO_TTL);
-this.viewClusteringTimestamp = 
minValueIfSet(this.viewClusteringTimestamp, timestamp, NO_TIMESTAMP);
-this.viewClusteringLocalDeletionTime = 
minValueIfSet(this.viewClusteringLocalDeletionTime, localDeletionTime, 
NO_DELETION_TIME);
+updateLiveness(ttl, timestamp, localDeletionTime);
 }
 
 innerMap.get(cellPath).setVersion(new TemporalCell(value, timestamp, 
ttl, localDeletionTime, isNew));
 }
 
-private static int minValueIfSet(int existing, int update, int 
defaultValue)
-{
-if (existing == defaultValue)
-return update;
-if (update == defaultValue)
-return existing;
-return Math.min(existing, update);
-}
-
-private static long minValueIfSet(long existing, long update, long 
defaultValue)
+ 

[03/50] [abbrv] cassandra git commit: Merge commit '11716547f87f4c88a2790323744b0ed97175854d' into cassandra-2.2

[slebresne]

Merge commit '11716547f87f4c88a2790323744b0ed97175854d' into cassandra-2.2


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/078aabe3
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/078aabe3
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/078aabe3

Branch: refs/heads/cassandra-3.2
Commit: 078aabe3d40bfa511642918ea6529016fe7c0620
Parents: 9ca7e16 1171654
Author: Sylvain Lebresne 
Authored: Wed Jan 6 17:56:01 2016 +0100
Committer: Sylvain Lebresne 
Committed: Wed Jan 6 17:56:01 2016 +0100

--

--

[22/50] [abbrv] cassandra git commit: Merge commit '202cf9b0bed8bbff41318f1f10043aabf3a7cd4d' into cassandra-3.0

[slebresne]

Merge commit '202cf9b0bed8bbff41318f1f10043aabf3a7cd4d' into cassandra-3.0


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/3dbc56f1
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/3dbc56f1
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/3dbc56f1

Branch: refs/heads/cassandra-3.0
Commit: 3dbc56f163690c824ffc0aac580a884f34536d62
Parents: e9e127a 202cf9b
Author: Sylvain Lebresne 
Authored: Wed Jan 6 17:57:37 2016 +0100
Committer: Sylvain Lebresne 
Committed: Wed Jan 6 17:57:37 2016 +0100

--

--

[42/50] [abbrv] cassandra git commit: Match cassandra-loader options in COPY FROM (3.2 version)

[slebresne]

http://git-wip-us.apache.org/repos/asf/cassandra/blob/85b8d02a/pylib/cqlshlib/formatting.py
--
diff --git a/pylib/cqlshlib/formatting.py b/pylib/cqlshlib/formatting.py
index 62ecd10..2e219c8 100644
--- a/pylib/cqlshlib/formatting.py
+++ b/pylib/cqlshlib/formatting.py
@@ -60,7 +60,8 @@ empty_colormap = defaultdict(lambda: '')
 
 
 def format_by_type(cqltype, val, encoding, colormap=None, addcolor=False,
-   nullval=None, date_time_format=None, float_precision=None):
+   nullval=None, date_time_format=None, float_precision=None,
+   decimal_sep=None, thousands_sep=None, boolean_styles=None):
 if nullval is None:
 nullval = default_null_placeholder
 if val is None:
@@ -75,7 +76,8 @@ def format_by_type(cqltype, val, encoding, colormap=None, 
addcolor=False,
 float_precision = default_float_precision
 return format_value(cqltype, val, encoding=encoding, colormap=colormap,
 date_time_format=date_time_format, 
float_precision=float_precision,
-nullval=nullval)
+nullval=nullval, decimal_sep=decimal_sep, 
thousands_sep=thousands_sep,
+boolean_styles=boolean_styles)
 
 
 def color_text(bval, colormap, displaywidth=None):
@@ -155,7 +157,9 @@ def format_python_formatted_type(val, colormap, color, 
quote=False):
 
 
 @formatter_for('Decimal')
-def format_value_decimal(val, colormap, **_):
+def format_value_decimal(val, float_precision, colormap, decimal_sep=None, 
thousands_sep=None, **_):
+if (decimal_sep and decimal_sep != '.') or thousands_sep:
+return format_floating_point_type(val, colormap, float_precision, 
decimal_sep, thousands_sep)
 return format_python_formatted_type(val, colormap, 'decimal')
 
 
@@ -170,34 +174,60 @@ def formatter_value_inet(val, colormap, quote=False, **_):
 
 
 @formatter_for('bool')
-def format_value_boolean(val, colormap, **_):
+def format_value_boolean(val, colormap, boolean_styles=None, **_):
+if boolean_styles:
+val = boolean_styles[0] if val else boolean_styles[1]
 return format_python_formatted_type(val, colormap, 'boolean')
 
 
-def format_floating_point_type(val, colormap, float_precision, **_):
+def format_floating_point_type(val, colormap, float_precision, 
decimal_sep=None, thousands_sep=None, **_):
 if math.isnan(val):
 bval = 'NaN'
 elif math.isinf(val):
 bval = 'Infinity' if val > 0 else '-Infinity'
 else:
-exponent = int(math.log10(abs(val))) if abs(val) > 
sys.float_info.epsilon else -sys.maxsize - 1
-if -4 <= exponent < float_precision:
-# when this is true %g will not use scientific notation,
-# increasing precision should not change this decision
-# so we increase the precision to take into account the
-# digits to the left of the decimal point
-float_precision = float_precision + exponent + 1
-bval = '%.*g' % (float_precision, val)
+if thousands_sep:
+dpart, ipart = math.modf(val)
+bval = format_integer_with_thousands_sep(ipart, thousands_sep)
+dpart_str = ('%.*f' % (float_precision, 
math.fabs(dpart)))[2:].rstrip('0')
+if dpart_str:
+bval += '%s%s' % ('.' if not decimal_sep else decimal_sep, 
dpart_str)
+else:
+exponent = int(math.log10(abs(val))) if abs(val) > 
sys.float_info.epsilon else -sys.maxsize - 1
+if -4 <= exponent < float_precision:
+# when this is true %g will not use scientific notation,
+# increasing precision should not change this decision
+# so we increase the precision to take into account the
+# digits to the left of the decimal point
+float_precision = float_precision + exponent + 1
+bval = '%.*g' % (float_precision, val)
+if decimal_sep:
+bval = bval.replace('.', decimal_sep)
+
 return colorme(bval, colormap, 'float')
 
 formatter_for('float')(format_floating_point_type)
 
 
-def format_integer_type(val, colormap, **_):
+def format_integer_type(val, colormap, thousands_sep=None, **_):
 # base-10 only for now; support others?
-bval = str(val)
+bval = format_integer_with_thousands_sep(val, thousands_sep) if 
thousands_sep else str(val)
 return colorme(bval, colormap, 'int')
 
+# We can get rid of this in cassandra-2.2
+if sys.version_info >= (2, 7):
+def format_integer_with_thousands_sep(val, thousands_sep=','):
+return "{:,.0f}".format(val).replace(',', thousands_sep)
+else:
+def format_integer_with_thousands_sep(val, thousands_sep=','):
+if val < 0:
+return '-' + format_integer_with_thousands_sep(-val, thousands_sep)
+result = ''
+while val >= 1000:
+val, r = 

[35/50] [abbrv] cassandra git commit: Match cassandra-loader options in COPY FROM (3.0 version)

[slebresne]

Match cassandra-loader options in COPY FROM (3.0 version)

patch by Stefania; reviewed by pauloricardomg for CASSANDRA-9303


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/f2883879
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/f2883879
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/f2883879

Branch: refs/heads/cassandra-3.2
Commit: f288387917fd787f8476c2bca7dfb9d7892c24e8
Parents: 3dbc56f
Author: Stefania Alborghetti 
Authored: Wed Jan 6 12:12:12 2016 +0100
Committer: Sylvain Lebresne 
Committed: Wed Jan 6 17:58:20 2016 +0100

--
 CHANGES.txt |1 +
 NEWS.txt|7 +
 bin/cqlsh.py|  135 +-
 conf/cqlshrc.sample |   17 +-
 pylib/cqlshlib/copyutil.py  | 1260 +-
 pylib/cqlshlib/formatting.py|   96 +-
 .../cql3/statements/BatchStatement.java |   26 +-
 .../cassandra/service/ClientWarningsTest.java   |6 +-
 tools/bin/cassandra-stress.bat  |2 +-
 9 files changed, 1134 insertions(+), 416 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/f2883879/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index cf872d9..3cb0b1e 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -25,6 +25,7 @@ Merged from 2.2:
  * Disable reloading of GossipingPropertyFileSnitch (CASSANDRA-9474)
  * Verify tables in pseudo-system keyspaces at startup (CASSANDRA-10761)
 Merged from 2.1:
+ * Match cassandra-loader options in COPY FROM (CASSANDRA-9303)
  * Fix binding to any address in CqlBulkRecordWriter (CASSANDRA-9309)
  * cqlsh fails to decode utf-8 characters for text typed columns 
(CASSANDRA-10875)
  * Log error when stream session fails (CASSANDRA-9294)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/f2883879/NEWS.txt
--
diff --git a/NEWS.txt b/NEWS.txt
index 26a83a9..64837e1 100644
--- a/NEWS.txt
+++ b/NEWS.txt
@@ -383,6 +383,13 @@ Upgrading
  to exclude data centers when the global status is enabled, see 
CASSANDRA-9035 for details.
 
 
+2.1.13
+==
+
+New features
+
+- New options for cqlsh COPY FROM and COPY TO, see CASSANDRA-9303 for 
details.
+
 2.1.10
 =
 

http://git-wip-us.apache.org/repos/asf/cassandra/blob/f2883879/bin/cqlsh.py
--
diff --git a/bin/cqlsh.py b/bin/cqlsh.py
index d8e8db5..2f53cc1 100644
--- a/bin/cqlsh.py
+++ b/bin/cqlsh.py
@@ -41,7 +41,6 @@ import optparse
 import os
 import platform
 import sys
-import time
 import traceback
 import warnings
 import webbrowser
@@ -152,7 +151,8 @@ cqlshlibdir = os.path.join(CASSANDRA_PATH, 'pylib')
 if os.path.isdir(cqlshlibdir):
 sys.path.insert(0, cqlshlibdir)
 
-from cqlshlib import cql3handling, cqlhandling, copyutil, pylexotron, 
sslhandling
+from cqlshlib import cql3handling, cqlhandling, pylexotron, sslhandling
+from cqlshlib.copyutil import ExportTask, ImportTask
 from cqlshlib.displaying import (ANSI_RESET, BLUE, COLUMN_NAME_COLORS, CYAN,
  RED, FormattedValue, colorme)
 from cqlshlib.formatting import (DEFAULT_DATE_FORMAT, DEFAULT_NANOTIME_FORMAT,
@@ -458,10 +458,12 @@ def complete_copy_column_names(ctxt, cqlsh):
 return set(colnames[1:]) - set(existcols)
 
 
-COPY_COMMON_OPTIONS = ['DELIMITER', 'QUOTE', 'ESCAPE', 'HEADER', 'NULL',
-   'MAXATTEMPTS', 'REPORTFREQUENCY']
-COPY_FROM_OPTIONS = ['CHUNKSIZE', 'INGESTRATE', 'MAXBATCHSIZE', 'MINBATCHSIZE']
-COPY_TO_OPTIONS = ['ENCODING', 'TIMEFORMAT', 'PAGESIZE', 'PAGETIMEOUT', 
'MAXREQUESTS']
+COPY_COMMON_OPTIONS = ['DELIMITER', 'QUOTE', 'ESCAPE', 'HEADER', 'NULL', 
'DATETIMEFORMAT',
+   'MAXATTEMPTS', 'REPORTFREQUENCY', 'DECIMALSEP', 
'THOUSANDSSEP', 'BOOLSTYLE',
+   'NUMPROCESSES', 'CONFIGFILE', 'RATEFILE']
+COPY_FROM_OPTIONS = ['CHUNKSIZE', 'INGESTRATE', 'MAXBATCHSIZE', 
'MINBATCHSIZE', 'MAXROWS',
+ 'SKIPROWS', 'SKIPCOLS', 'MAXPARSEERRORS', 
'MAXINSERTERRORS', 'ERRFILE']
+COPY_TO_OPTIONS = ['ENCODING', 'PAGESIZE', 'PAGETIMEOUT', 'BEGINTOKEN', 
'ENDTOKEN', 'MAXOUTPUTSIZE', 'MAXREQUESTS']
 
 
 @cqlsh_syntax_completer('copyOption', 'optnames')
@@ -581,23 +583,6 @@ warnings.showwarning = show_warning_without_quoting_line
 warnings.filterwarnings('always', 
category=cql3handling.UnexpectedTableStructure)
 
 
-def describe_interval(seconds):
-desc = []
-for length, unit in ((86400, 'day'), (3600, 'hour'), (60, 'minute')):
- 

[28/50] [abbrv] cassandra git commit: Match cassandra-loader options in COPY FROM (3.0 version)

[slebresne]

http://git-wip-us.apache.org/repos/asf/cassandra/blob/f2883879/pylib/cqlshlib/copyutil.py
--
diff --git a/pylib/cqlshlib/copyutil.py b/pylib/cqlshlib/copyutil.py
index 6d9a455..a154363 100644
--- a/pylib/cqlshlib/copyutil.py
+++ b/pylib/cqlshlib/copyutil.py
@@ -14,12 +14,14 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
+import ConfigParser
 import csv
+import datetime
 import json
+import glob
 import multiprocessing as mp
 import os
 import Queue
-import random
 import re
 import struct
 import sys
@@ -46,81 +48,202 @@ from displaying import NO_COLOR_MAP
 from formatting import format_value_default, DateTimeFormat, EMPTY, 
get_formatter
 from sslhandling import ssl_settings
 
+CopyOptions = namedtuple('CopyOptions', 'copy dialect unrecognized')
 
-def parse_options(shell, opts):
-"""
-Parse options for import (COPY FROM) and export (COPY TO) operations.
-Extract from opts csv and dialect options.
 
-:return: 3 dictionaries: the csv options, the dialect options, any 
unrecognized options.
-"""
-dialect_options = shell.csv_dialect_defaults.copy()
-if 'quote' in opts:
-dialect_options['quotechar'] = opts.pop('quote')
-if 'escape' in opts:
-dialect_options['escapechar'] = opts.pop('escape')
-if 'delimiter' in opts:
-dialect_options['delimiter'] = opts.pop('delimiter')
-if dialect_options['quotechar'] == dialect_options['escapechar']:
-dialect_options['doublequote'] = True
-del dialect_options['escapechar']
-
-csv_options = dict()
-csv_options['nullval'] = opts.pop('null', '')
-csv_options['header'] = bool(opts.pop('header', '').lower() == 'true')
-csv_options['encoding'] = opts.pop('encoding', 'utf8')
-csv_options['maxrequests'] = int(opts.pop('maxrequests', 6))
-csv_options['pagesize'] = int(opts.pop('pagesize', 1000))
-# by default the page timeout is 10 seconds per 1000 entries in the page 
size or 10 seconds if pagesize is smaller
-csv_options['pagetimeout'] = int(opts.pop('pagetimeout', max(10, 10 * 
(csv_options['pagesize'] / 1000
-csv_options['maxattempts'] = int(opts.pop('maxattempts', 5))
-csv_options['dtformats'] = DateTimeFormat(opts.pop('timeformat', 
shell.display_timestamp_format),
-  shell.display_date_format,
-  shell.display_nanotime_format)
-csv_options['float_precision'] = shell.display_float_precision
-csv_options['chunksize'] = int(opts.pop('chunksize', 1000))
-csv_options['ingestrate'] = int(opts.pop('ingestrate', 10))
-csv_options['maxbatchsize'] = int(opts.pop('maxbatchsize', 20))
-csv_options['minbatchsize'] = int(opts.pop('minbatchsize', 2))
-csv_options['reportfrequency'] = float(opts.pop('reportfrequency', 0.25))
-
-return csv_options, dialect_options, opts
-
-
-def get_num_processes(cap):
+def safe_normpath(fname):
 """
-Pick a reasonable number of child processes. We need to leave at
-least one core for the parent process.  This doesn't necessarily
-need to be capped, but 4 is currently enough to keep
-a single local Cassandra node busy so we use this for import, whilst
-for export we use 16 since we can connect to multiple Cassandra nodes.
-Eventually this parameter will become an option.
+:return the normalized path but only if there is a filename, we don't want 
to convert
+an empty string (which means no file name) to a dot. Also expand any user 
variables such as ~ to the full path
 """
-try:
-return max(1, min(cap, mp.cpu_count() - 1))
-except NotImplementedError:
-return 1
+return os.path.normpath(os.path.expanduser(fname)) if fname else fname
 
 
 class CopyTask(object):
 """
 A base class for ImportTask and ExportTask
 """
-def __init__(self, shell, ks, cf, columns, fname, csv_options, 
dialect_options, protocol_version, config_file):
+def __init__(self, shell, ks, table, columns, fname, opts, 
protocol_version, config_file, direction):
 self.shell = shell
-self.csv_options = csv_options
-self.dialect_options = dialect_options
 self.ks = ks
-self.cf = cf
-self.columns = shell.get_column_names(ks, cf) if columns is None else 
columns
-self.fname = fname
+self.table = table
+self.local_dc = shell.conn.metadata.get_host(shell.hostname).datacenter
+self.fname = safe_normpath(fname)
 self.protocol_version = protocol_version
 self.config_file = config_file
+# do not display messages when exporting to STDOUT
+self.printmsg = self._printmsg if self.fname is not None or direction 
== 'in' else lambda _, eol='\n': None
+self.options = self.parse_options(opts, direction)
+
+self.num_processes = 

[18/50] [abbrv] cassandra git commit: Match cassandra-loader options in COPY FROM (2.2 version)

[slebresne]

http://git-wip-us.apache.org/repos/asf/cassandra/blob/202cf9b0/pylib/cqlshlib/copyutil.py
--
diff --git a/pylib/cqlshlib/copyutil.py b/pylib/cqlshlib/copyutil.py
index 6d9a455..a154363 100644
--- a/pylib/cqlshlib/copyutil.py
+++ b/pylib/cqlshlib/copyutil.py
@@ -14,12 +14,14 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
+import ConfigParser
 import csv
+import datetime
 import json
+import glob
 import multiprocessing as mp
 import os
 import Queue
-import random
 import re
 import struct
 import sys
@@ -46,81 +48,202 @@ from displaying import NO_COLOR_MAP
 from formatting import format_value_default, DateTimeFormat, EMPTY, 
get_formatter
 from sslhandling import ssl_settings
 
+CopyOptions = namedtuple('CopyOptions', 'copy dialect unrecognized')
 
-def parse_options(shell, opts):
-"""
-Parse options for import (COPY FROM) and export (COPY TO) operations.
-Extract from opts csv and dialect options.
 
-:return: 3 dictionaries: the csv options, the dialect options, any 
unrecognized options.
-"""
-dialect_options = shell.csv_dialect_defaults.copy()
-if 'quote' in opts:
-dialect_options['quotechar'] = opts.pop('quote')
-if 'escape' in opts:
-dialect_options['escapechar'] = opts.pop('escape')
-if 'delimiter' in opts:
-dialect_options['delimiter'] = opts.pop('delimiter')
-if dialect_options['quotechar'] == dialect_options['escapechar']:
-dialect_options['doublequote'] = True
-del dialect_options['escapechar']
-
-csv_options = dict()
-csv_options['nullval'] = opts.pop('null', '')
-csv_options['header'] = bool(opts.pop('header', '').lower() == 'true')
-csv_options['encoding'] = opts.pop('encoding', 'utf8')
-csv_options['maxrequests'] = int(opts.pop('maxrequests', 6))
-csv_options['pagesize'] = int(opts.pop('pagesize', 1000))
-# by default the page timeout is 10 seconds per 1000 entries in the page 
size or 10 seconds if pagesize is smaller
-csv_options['pagetimeout'] = int(opts.pop('pagetimeout', max(10, 10 * 
(csv_options['pagesize'] / 1000
-csv_options['maxattempts'] = int(opts.pop('maxattempts', 5))
-csv_options['dtformats'] = DateTimeFormat(opts.pop('timeformat', 
shell.display_timestamp_format),
-  shell.display_date_format,
-  shell.display_nanotime_format)
-csv_options['float_precision'] = shell.display_float_precision
-csv_options['chunksize'] = int(opts.pop('chunksize', 1000))
-csv_options['ingestrate'] = int(opts.pop('ingestrate', 10))
-csv_options['maxbatchsize'] = int(opts.pop('maxbatchsize', 20))
-csv_options['minbatchsize'] = int(opts.pop('minbatchsize', 2))
-csv_options['reportfrequency'] = float(opts.pop('reportfrequency', 0.25))
-
-return csv_options, dialect_options, opts
-
-
-def get_num_processes(cap):
+def safe_normpath(fname):
 """
-Pick a reasonable number of child processes. We need to leave at
-least one core for the parent process.  This doesn't necessarily
-need to be capped, but 4 is currently enough to keep
-a single local Cassandra node busy so we use this for import, whilst
-for export we use 16 since we can connect to multiple Cassandra nodes.
-Eventually this parameter will become an option.
+:return the normalized path but only if there is a filename, we don't want 
to convert
+an empty string (which means no file name) to a dot. Also expand any user 
variables such as ~ to the full path
 """
-try:
-return max(1, min(cap, mp.cpu_count() - 1))
-except NotImplementedError:
-return 1
+return os.path.normpath(os.path.expanduser(fname)) if fname else fname
 
 
 class CopyTask(object):
 """
 A base class for ImportTask and ExportTask
 """
-def __init__(self, shell, ks, cf, columns, fname, csv_options, 
dialect_options, protocol_version, config_file):
+def __init__(self, shell, ks, table, columns, fname, opts, 
protocol_version, config_file, direction):
 self.shell = shell
-self.csv_options = csv_options
-self.dialect_options = dialect_options
 self.ks = ks
-self.cf = cf
-self.columns = shell.get_column_names(ks, cf) if columns is None else 
columns
-self.fname = fname
+self.table = table
+self.local_dc = shell.conn.metadata.get_host(shell.hostname).datacenter
+self.fname = safe_normpath(fname)
 self.protocol_version = protocol_version
 self.config_file = config_file
+# do not display messages when exporting to STDOUT
+self.printmsg = self._printmsg if self.fname is not None or direction 
== 'in' else lambda _, eol='\n': None
+self.options = self.parse_options(opts, direction)
+
+self.num_processes = 

[13/50] [abbrv] cassandra git commit: Match cassandra-loader options in COPY FROM (2.2 version)

[slebresne]

Match cassandra-loader options in COPY FROM (2.2 version)

patch by Stefania; reviewed by pauloricardomg for CASSANDRA-9303


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/202cf9b0
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/202cf9b0
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/202cf9b0

Branch: refs/heads/cassandra-3.3
Commit: 202cf9b0bed8bbff41318f1f10043aabf3a7cd4d
Parents: 078aabe
Author: Stefania Alborghetti 
Authored: Wed Jan 6 12:10:13 2016 +0100
Committer: Sylvain Lebresne 
Committed: Wed Jan 6 17:56:30 2016 +0100

--
 CHANGES.txt |1 +
 NEWS.txt|7 +
 bin/cqlsh.py|  135 +-
 conf/cqlshrc.sample |   17 +-
 pylib/cqlshlib/copyutil.py  | 1260 +-
 pylib/cqlshlib/formatting.py|   96 +-
 .../cql3/statements/BatchStatement.java |   28 +-
 .../cassandra/service/ClientWarningsTest.java   |5 +-
 tools/bin/cassandra-stress.bat  |2 +-
 9 files changed, 1134 insertions(+), 417 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/202cf9b0/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index fc87c7d..b12f593 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -13,6 +13,7 @@
  * Disable reloading of GossipingPropertyFileSnitch (CASSANDRA-9474)
  * Verify tables in pseudo-system keyspaces at startup (CASSANDRA-10761)
 Merged from 2.1:
+ * Match cassandra-loader options in COPY FROM (CASSANDRA-9303)
  * Fix binding to any address in CqlBulkRecordWriter (CASSANDRA-9309)
  * cqlsh fails to decode utf-8 characters for text typed columns 
(CASSANDRA-10875)
  * Log error when stream session fails (CASSANDRA-9294)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/202cf9b0/NEWS.txt
--
diff --git a/NEWS.txt b/NEWS.txt
index 57e321e..8cbe4f7 100644
--- a/NEWS.txt
+++ b/NEWS.txt
@@ -269,6 +269,13 @@ Upgrading
  to exclude data centers when the global status is enabled, see 
CASSANDRA-9035 for details.
 
 
+2.1.13
+==
+
+New features
+
+- New options for cqlsh COPY FROM and COPY TO, see CASSANDRA-9303 for 
details.
+
 2.1.10
 =
 

http://git-wip-us.apache.org/repos/asf/cassandra/blob/202cf9b0/bin/cqlsh.py
--
diff --git a/bin/cqlsh.py b/bin/cqlsh.py
index 8469836..c38bc2e 100644
--- a/bin/cqlsh.py
+++ b/bin/cqlsh.py
@@ -41,7 +41,6 @@ import optparse
 import os
 import platform
 import sys
-import time
 import traceback
 import warnings
 import webbrowser
@@ -151,7 +150,8 @@ cqlshlibdir = os.path.join(CASSANDRA_PATH, 'pylib')
 if os.path.isdir(cqlshlibdir):
 sys.path.insert(0, cqlshlibdir)
 
-from cqlshlib import cql3handling, cqlhandling, copyutil, pylexotron, 
sslhandling
+from cqlshlib import cql3handling, cqlhandling, pylexotron, sslhandling
+from cqlshlib.copyutil import ExportTask, ImportTask
 from cqlshlib.displaying import (ANSI_RESET, BLUE, COLUMN_NAME_COLORS, CYAN,
  RED, FormattedValue, colorme)
 from cqlshlib.formatting import (DEFAULT_DATE_FORMAT, DEFAULT_NANOTIME_FORMAT,
@@ -456,10 +456,12 @@ def complete_copy_column_names(ctxt, cqlsh):
 return set(colnames[1:]) - set(existcols)
 
 
-COPY_COMMON_OPTIONS = ['DELIMITER', 'QUOTE', 'ESCAPE', 'HEADER', 'NULL',
-   'MAXATTEMPTS', 'REPORTFREQUENCY']
-COPY_FROM_OPTIONS = ['CHUNKSIZE', 'INGESTRATE', 'MAXBATCHSIZE', 'MINBATCHSIZE']
-COPY_TO_OPTIONS = ['ENCODING', 'TIMEFORMAT', 'PAGESIZE', 'PAGETIMEOUT', 
'MAXREQUESTS']
+COPY_COMMON_OPTIONS = ['DELIMITER', 'QUOTE', 'ESCAPE', 'HEADER', 'NULL', 
'DATETIMEFORMAT',
+   'MAXATTEMPTS', 'REPORTFREQUENCY', 'DECIMALSEP', 
'THOUSANDSSEP', 'BOOLSTYLE',
+   'NUMPROCESSES', 'CONFIGFILE', 'RATEFILE']
+COPY_FROM_OPTIONS = ['CHUNKSIZE', 'INGESTRATE', 'MAXBATCHSIZE', 
'MINBATCHSIZE', 'MAXROWS',
+ 'SKIPROWS', 'SKIPCOLS', 'MAXPARSEERRORS', 
'MAXINSERTERRORS', 'ERRFILE']
+COPY_TO_OPTIONS = ['ENCODING', 'PAGESIZE', 'PAGETIMEOUT', 'BEGINTOKEN', 
'ENDTOKEN', 'MAXOUTPUTSIZE', 'MAXREQUESTS']
 
 
 @cqlsh_syntax_completer('copyOption', 'optnames')
@@ -575,23 +577,6 @@ warnings.showwarning = show_warning_without_quoting_line
 warnings.filterwarnings('always', 
category=cql3handling.UnexpectedTableStructure)
 
 
-def describe_interval(seconds):
-desc = []
-for length, unit in ((86400, 'day'), (3600, 'hour'), (60, 'minute')):
-num = 

[09/50] [abbrv] cassandra git commit: Match cassandra-loader options in COPY FROM (2.2 version)

[slebresne]

http://git-wip-us.apache.org/repos/asf/cassandra/blob/202cf9b0/pylib/cqlshlib/copyutil.py
--
diff --git a/pylib/cqlshlib/copyutil.py b/pylib/cqlshlib/copyutil.py
index 6d9a455..a154363 100644
--- a/pylib/cqlshlib/copyutil.py
+++ b/pylib/cqlshlib/copyutil.py
@@ -14,12 +14,14 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
+import ConfigParser
 import csv
+import datetime
 import json
+import glob
 import multiprocessing as mp
 import os
 import Queue
-import random
 import re
 import struct
 import sys
@@ -46,81 +48,202 @@ from displaying import NO_COLOR_MAP
 from formatting import format_value_default, DateTimeFormat, EMPTY, 
get_formatter
 from sslhandling import ssl_settings
 
+CopyOptions = namedtuple('CopyOptions', 'copy dialect unrecognized')
 
-def parse_options(shell, opts):
-"""
-Parse options for import (COPY FROM) and export (COPY TO) operations.
-Extract from opts csv and dialect options.
 
-:return: 3 dictionaries: the csv options, the dialect options, any 
unrecognized options.
-"""
-dialect_options = shell.csv_dialect_defaults.copy()
-if 'quote' in opts:
-dialect_options['quotechar'] = opts.pop('quote')
-if 'escape' in opts:
-dialect_options['escapechar'] = opts.pop('escape')
-if 'delimiter' in opts:
-dialect_options['delimiter'] = opts.pop('delimiter')
-if dialect_options['quotechar'] == dialect_options['escapechar']:
-dialect_options['doublequote'] = True
-del dialect_options['escapechar']
-
-csv_options = dict()
-csv_options['nullval'] = opts.pop('null', '')
-csv_options['header'] = bool(opts.pop('header', '').lower() == 'true')
-csv_options['encoding'] = opts.pop('encoding', 'utf8')
-csv_options['maxrequests'] = int(opts.pop('maxrequests', 6))
-csv_options['pagesize'] = int(opts.pop('pagesize', 1000))
-# by default the page timeout is 10 seconds per 1000 entries in the page 
size or 10 seconds if pagesize is smaller
-csv_options['pagetimeout'] = int(opts.pop('pagetimeout', max(10, 10 * 
(csv_options['pagesize'] / 1000
-csv_options['maxattempts'] = int(opts.pop('maxattempts', 5))
-csv_options['dtformats'] = DateTimeFormat(opts.pop('timeformat', 
shell.display_timestamp_format),
-  shell.display_date_format,
-  shell.display_nanotime_format)
-csv_options['float_precision'] = shell.display_float_precision
-csv_options['chunksize'] = int(opts.pop('chunksize', 1000))
-csv_options['ingestrate'] = int(opts.pop('ingestrate', 10))
-csv_options['maxbatchsize'] = int(opts.pop('maxbatchsize', 20))
-csv_options['minbatchsize'] = int(opts.pop('minbatchsize', 2))
-csv_options['reportfrequency'] = float(opts.pop('reportfrequency', 0.25))
-
-return csv_options, dialect_options, opts
-
-
-def get_num_processes(cap):
+def safe_normpath(fname):
 """
-Pick a reasonable number of child processes. We need to leave at
-least one core for the parent process.  This doesn't necessarily
-need to be capped, but 4 is currently enough to keep
-a single local Cassandra node busy so we use this for import, whilst
-for export we use 16 since we can connect to multiple Cassandra nodes.
-Eventually this parameter will become an option.
+:return the normalized path but only if there is a filename, we don't want 
to convert
+an empty string (which means no file name) to a dot. Also expand any user 
variables such as ~ to the full path
 """
-try:
-return max(1, min(cap, mp.cpu_count() - 1))
-except NotImplementedError:
-return 1
+return os.path.normpath(os.path.expanduser(fname)) if fname else fname
 
 
 class CopyTask(object):
 """
 A base class for ImportTask and ExportTask
 """
-def __init__(self, shell, ks, cf, columns, fname, csv_options, 
dialect_options, protocol_version, config_file):
+def __init__(self, shell, ks, table, columns, fname, opts, 
protocol_version, config_file, direction):
 self.shell = shell
-self.csv_options = csv_options
-self.dialect_options = dialect_options
 self.ks = ks
-self.cf = cf
-self.columns = shell.get_column_names(ks, cf) if columns is None else 
columns
-self.fname = fname
+self.table = table
+self.local_dc = shell.conn.metadata.get_host(shell.hostname).datacenter
+self.fname = safe_normpath(fname)
 self.protocol_version = protocol_version
 self.config_file = config_file
+# do not display messages when exporting to STDOUT
+self.printmsg = self._printmsg if self.fname is not None or direction 
== 'in' else lambda _, eol='\n': None
+self.options = self.parse_options(opts, direction)
+
+self.num_processes = 

[27/50] [abbrv] cassandra git commit: Match cassandra-loader options in COPY FROM (3.0 version)

[slebresne]

http://git-wip-us.apache.org/repos/asf/cassandra/blob/f2883879/pylib/cqlshlib/formatting.py
--
diff --git a/pylib/cqlshlib/formatting.py b/pylib/cqlshlib/formatting.py
index 62ecd10..2e219c8 100644
--- a/pylib/cqlshlib/formatting.py
+++ b/pylib/cqlshlib/formatting.py
@@ -60,7 +60,8 @@ empty_colormap = defaultdict(lambda: '')
 
 
 def format_by_type(cqltype, val, encoding, colormap=None, addcolor=False,
-   nullval=None, date_time_format=None, float_precision=None):
+   nullval=None, date_time_format=None, float_precision=None,
+   decimal_sep=None, thousands_sep=None, boolean_styles=None):
 if nullval is None:
 nullval = default_null_placeholder
 if val is None:
@@ -75,7 +76,8 @@ def format_by_type(cqltype, val, encoding, colormap=None, 
addcolor=False,
 float_precision = default_float_precision
 return format_value(cqltype, val, encoding=encoding, colormap=colormap,
 date_time_format=date_time_format, 
float_precision=float_precision,
-nullval=nullval)
+nullval=nullval, decimal_sep=decimal_sep, 
thousands_sep=thousands_sep,
+boolean_styles=boolean_styles)
 
 
 def color_text(bval, colormap, displaywidth=None):
@@ -155,7 +157,9 @@ def format_python_formatted_type(val, colormap, color, 
quote=False):
 
 
 @formatter_for('Decimal')
-def format_value_decimal(val, colormap, **_):
+def format_value_decimal(val, float_precision, colormap, decimal_sep=None, 
thousands_sep=None, **_):
+if (decimal_sep and decimal_sep != '.') or thousands_sep:
+return format_floating_point_type(val, colormap, float_precision, 
decimal_sep, thousands_sep)
 return format_python_formatted_type(val, colormap, 'decimal')
 
 
@@ -170,34 +174,60 @@ def formatter_value_inet(val, colormap, quote=False, **_):
 
 
 @formatter_for('bool')
-def format_value_boolean(val, colormap, **_):
+def format_value_boolean(val, colormap, boolean_styles=None, **_):
+if boolean_styles:
+val = boolean_styles[0] if val else boolean_styles[1]
 return format_python_formatted_type(val, colormap, 'boolean')
 
 
-def format_floating_point_type(val, colormap, float_precision, **_):
+def format_floating_point_type(val, colormap, float_precision, 
decimal_sep=None, thousands_sep=None, **_):
 if math.isnan(val):
 bval = 'NaN'
 elif math.isinf(val):
 bval = 'Infinity' if val > 0 else '-Infinity'
 else:
-exponent = int(math.log10(abs(val))) if abs(val) > 
sys.float_info.epsilon else -sys.maxsize - 1
-if -4 <= exponent < float_precision:
-# when this is true %g will not use scientific notation,
-# increasing precision should not change this decision
-# so we increase the precision to take into account the
-# digits to the left of the decimal point
-float_precision = float_precision + exponent + 1
-bval = '%.*g' % (float_precision, val)
+if thousands_sep:
+dpart, ipart = math.modf(val)
+bval = format_integer_with_thousands_sep(ipart, thousands_sep)
+dpart_str = ('%.*f' % (float_precision, 
math.fabs(dpart)))[2:].rstrip('0')
+if dpart_str:
+bval += '%s%s' % ('.' if not decimal_sep else decimal_sep, 
dpart_str)
+else:
+exponent = int(math.log10(abs(val))) if abs(val) > 
sys.float_info.epsilon else -sys.maxsize - 1
+if -4 <= exponent < float_precision:
+# when this is true %g will not use scientific notation,
+# increasing precision should not change this decision
+# so we increase the precision to take into account the
+# digits to the left of the decimal point
+float_precision = float_precision + exponent + 1
+bval = '%.*g' % (float_precision, val)
+if decimal_sep:
+bval = bval.replace('.', decimal_sep)
+
 return colorme(bval, colormap, 'float')
 
 formatter_for('float')(format_floating_point_type)
 
 
-def format_integer_type(val, colormap, **_):
+def format_integer_type(val, colormap, thousands_sep=None, **_):
 # base-10 only for now; support others?
-bval = str(val)
+bval = format_integer_with_thousands_sep(val, thousands_sep) if 
thousands_sep else str(val)
 return colorme(bval, colormap, 'int')
 
+# We can get rid of this in cassandra-2.2
+if sys.version_info >= (2, 7):
+def format_integer_with_thousands_sep(val, thousands_sep=','):
+return "{:,.0f}".format(val).replace(',', thousands_sep)
+else:
+def format_integer_with_thousands_sep(val, thousands_sep=','):
+if val < 0:
+return '-' + format_integer_with_thousands_sep(-val, thousands_sep)
+result = ''
+while val >= 1000:
+val, r = 

[17/50] [abbrv] cassandra git commit: Match cassandra-loader options in COPY FROM (2.2 version)

[slebresne]

http://git-wip-us.apache.org/repos/asf/cassandra/blob/202cf9b0/pylib/cqlshlib/formatting.py
--
diff --git a/pylib/cqlshlib/formatting.py b/pylib/cqlshlib/formatting.py
index 62ecd10..2e219c8 100644
--- a/pylib/cqlshlib/formatting.py
+++ b/pylib/cqlshlib/formatting.py
@@ -60,7 +60,8 @@ empty_colormap = defaultdict(lambda: '')
 
 
 def format_by_type(cqltype, val, encoding, colormap=None, addcolor=False,
-   nullval=None, date_time_format=None, float_precision=None):
+   nullval=None, date_time_format=None, float_precision=None,
+   decimal_sep=None, thousands_sep=None, boolean_styles=None):
 if nullval is None:
 nullval = default_null_placeholder
 if val is None:
@@ -75,7 +76,8 @@ def format_by_type(cqltype, val, encoding, colormap=None, 
addcolor=False,
 float_precision = default_float_precision
 return format_value(cqltype, val, encoding=encoding, colormap=colormap,
 date_time_format=date_time_format, 
float_precision=float_precision,
-nullval=nullval)
+nullval=nullval, decimal_sep=decimal_sep, 
thousands_sep=thousands_sep,
+boolean_styles=boolean_styles)
 
 
 def color_text(bval, colormap, displaywidth=None):
@@ -155,7 +157,9 @@ def format_python_formatted_type(val, colormap, color, 
quote=False):
 
 
 @formatter_for('Decimal')
-def format_value_decimal(val, colormap, **_):
+def format_value_decimal(val, float_precision, colormap, decimal_sep=None, 
thousands_sep=None, **_):
+if (decimal_sep and decimal_sep != '.') or thousands_sep:
+return format_floating_point_type(val, colormap, float_precision, 
decimal_sep, thousands_sep)
 return format_python_formatted_type(val, colormap, 'decimal')
 
 
@@ -170,34 +174,60 @@ def formatter_value_inet(val, colormap, quote=False, **_):
 
 
 @formatter_for('bool')
-def format_value_boolean(val, colormap, **_):
+def format_value_boolean(val, colormap, boolean_styles=None, **_):
+if boolean_styles:
+val = boolean_styles[0] if val else boolean_styles[1]
 return format_python_formatted_type(val, colormap, 'boolean')
 
 
-def format_floating_point_type(val, colormap, float_precision, **_):
+def format_floating_point_type(val, colormap, float_precision, 
decimal_sep=None, thousands_sep=None, **_):
 if math.isnan(val):
 bval = 'NaN'
 elif math.isinf(val):
 bval = 'Infinity' if val > 0 else '-Infinity'
 else:
-exponent = int(math.log10(abs(val))) if abs(val) > 
sys.float_info.epsilon else -sys.maxsize - 1
-if -4 <= exponent < float_precision:
-# when this is true %g will not use scientific notation,
-# increasing precision should not change this decision
-# so we increase the precision to take into account the
-# digits to the left of the decimal point
-float_precision = float_precision + exponent + 1
-bval = '%.*g' % (float_precision, val)
+if thousands_sep:
+dpart, ipart = math.modf(val)
+bval = format_integer_with_thousands_sep(ipart, thousands_sep)
+dpart_str = ('%.*f' % (float_precision, 
math.fabs(dpart)))[2:].rstrip('0')
+if dpart_str:
+bval += '%s%s' % ('.' if not decimal_sep else decimal_sep, 
dpart_str)
+else:
+exponent = int(math.log10(abs(val))) if abs(val) > 
sys.float_info.epsilon else -sys.maxsize - 1
+if -4 <= exponent < float_precision:
+# when this is true %g will not use scientific notation,
+# increasing precision should not change this decision
+# so we increase the precision to take into account the
+# digits to the left of the decimal point
+float_precision = float_precision + exponent + 1
+bval = '%.*g' % (float_precision, val)
+if decimal_sep:
+bval = bval.replace('.', decimal_sep)
+
 return colorme(bval, colormap, 'float')
 
 formatter_for('float')(format_floating_point_type)
 
 
-def format_integer_type(val, colormap, **_):
+def format_integer_type(val, colormap, thousands_sep=None, **_):
 # base-10 only for now; support others?
-bval = str(val)
+bval = format_integer_with_thousands_sep(val, thousands_sep) if 
thousands_sep else str(val)
 return colorme(bval, colormap, 'int')
 
+# We can get rid of this in cassandra-2.2
+if sys.version_info >= (2, 7):
+def format_integer_with_thousands_sep(val, thousands_sep=','):
+return "{:,.0f}".format(val).replace(',', thousands_sep)
+else:
+def format_integer_with_thousands_sep(val, thousands_sep=','):
+if val < 0:
+return '-' + format_integer_with_thousands_sep(-val, thousands_sep)
+result = ''
+while val >= 1000:
+val, r = 

[12/50] [abbrv] cassandra git commit: Match cassandra-loader options in COPY FROM (2.2 version)

[slebresne]

http://git-wip-us.apache.org/repos/asf/cassandra/blob/202cf9b0/pylib/cqlshlib/copyutil.py
--
diff --git a/pylib/cqlshlib/copyutil.py b/pylib/cqlshlib/copyutil.py
index 6d9a455..a154363 100644
--- a/pylib/cqlshlib/copyutil.py
+++ b/pylib/cqlshlib/copyutil.py
@@ -14,12 +14,14 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
+import ConfigParser
 import csv
+import datetime
 import json
+import glob
 import multiprocessing as mp
 import os
 import Queue
-import random
 import re
 import struct
 import sys
@@ -46,81 +48,202 @@ from displaying import NO_COLOR_MAP
 from formatting import format_value_default, DateTimeFormat, EMPTY, 
get_formatter
 from sslhandling import ssl_settings
 
+CopyOptions = namedtuple('CopyOptions', 'copy dialect unrecognized')
 
-def parse_options(shell, opts):
-"""
-Parse options for import (COPY FROM) and export (COPY TO) operations.
-Extract from opts csv and dialect options.
 
-:return: 3 dictionaries: the csv options, the dialect options, any 
unrecognized options.
-"""
-dialect_options = shell.csv_dialect_defaults.copy()
-if 'quote' in opts:
-dialect_options['quotechar'] = opts.pop('quote')
-if 'escape' in opts:
-dialect_options['escapechar'] = opts.pop('escape')
-if 'delimiter' in opts:
-dialect_options['delimiter'] = opts.pop('delimiter')
-if dialect_options['quotechar'] == dialect_options['escapechar']:
-dialect_options['doublequote'] = True
-del dialect_options['escapechar']
-
-csv_options = dict()
-csv_options['nullval'] = opts.pop('null', '')
-csv_options['header'] = bool(opts.pop('header', '').lower() == 'true')
-csv_options['encoding'] = opts.pop('encoding', 'utf8')
-csv_options['maxrequests'] = int(opts.pop('maxrequests', 6))
-csv_options['pagesize'] = int(opts.pop('pagesize', 1000))
-# by default the page timeout is 10 seconds per 1000 entries in the page 
size or 10 seconds if pagesize is smaller
-csv_options['pagetimeout'] = int(opts.pop('pagetimeout', max(10, 10 * 
(csv_options['pagesize'] / 1000
-csv_options['maxattempts'] = int(opts.pop('maxattempts', 5))
-csv_options['dtformats'] = DateTimeFormat(opts.pop('timeformat', 
shell.display_timestamp_format),
-  shell.display_date_format,
-  shell.display_nanotime_format)
-csv_options['float_precision'] = shell.display_float_precision
-csv_options['chunksize'] = int(opts.pop('chunksize', 1000))
-csv_options['ingestrate'] = int(opts.pop('ingestrate', 10))
-csv_options['maxbatchsize'] = int(opts.pop('maxbatchsize', 20))
-csv_options['minbatchsize'] = int(opts.pop('minbatchsize', 2))
-csv_options['reportfrequency'] = float(opts.pop('reportfrequency', 0.25))
-
-return csv_options, dialect_options, opts
-
-
-def get_num_processes(cap):
+def safe_normpath(fname):
 """
-Pick a reasonable number of child processes. We need to leave at
-least one core for the parent process.  This doesn't necessarily
-need to be capped, but 4 is currently enough to keep
-a single local Cassandra node busy so we use this for import, whilst
-for export we use 16 since we can connect to multiple Cassandra nodes.
-Eventually this parameter will become an option.
+:return the normalized path but only if there is a filename, we don't want 
to convert
+an empty string (which means no file name) to a dot. Also expand any user 
variables such as ~ to the full path
 """
-try:
-return max(1, min(cap, mp.cpu_count() - 1))
-except NotImplementedError:
-return 1
+return os.path.normpath(os.path.expanduser(fname)) if fname else fname
 
 
 class CopyTask(object):
 """
 A base class for ImportTask and ExportTask
 """
-def __init__(self, shell, ks, cf, columns, fname, csv_options, 
dialect_options, protocol_version, config_file):
+def __init__(self, shell, ks, table, columns, fname, opts, 
protocol_version, config_file, direction):
 self.shell = shell
-self.csv_options = csv_options
-self.dialect_options = dialect_options
 self.ks = ks
-self.cf = cf
-self.columns = shell.get_column_names(ks, cf) if columns is None else 
columns
-self.fname = fname
+self.table = table
+self.local_dc = shell.conn.metadata.get_host(shell.hostname).datacenter
+self.fname = safe_normpath(fname)
 self.protocol_version = protocol_version
 self.config_file = config_file
+# do not display messages when exporting to STDOUT
+self.printmsg = self._printmsg if self.fname is not None or direction 
== 'in' else lambda _, eol='\n': None
+self.options = self.parse_options(opts, direction)
+
+self.num_processes = 

[01/50] [abbrv] cassandra git commit: Merge commit '11716547f87f4c88a2790323744b0ed97175854d' into cassandra-2.2

[slebresne]

Repository: cassandra
Updated Branches:
  refs/heads/cassandra-2.1 b551b8e1e -> 11716547f
  refs/heads/cassandra-2.2 9ca7e1626 -> 202cf9b0b
  refs/heads/cassandra-3.0 e9e127abf -> f28838791
  refs/heads/cassandra-3.2 e2c634189 -> 85b8d02aa
  refs/heads/cassandra-3.3 a7854fdd3 -> 65b1f2dac
  refs/heads/trunk 8383e89b8 -> 9a88f8550


Merge commit '11716547f87f4c88a2790323744b0ed97175854d' into cassandra-2.2


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/078aabe3
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/078aabe3
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/078aabe3

Branch: refs/heads/cassandra-3.3
Commit: 078aabe3d40bfa511642918ea6529016fe7c0620
Parents: 9ca7e16 1171654
Author: Sylvain Lebresne 
Authored: Wed Jan 6 17:56:01 2016 +0100
Committer: Sylvain Lebresne 
Committed: Wed Jan 6 17:56:01 2016 +0100

--

--

[40/50] [abbrv] cassandra git commit: Match cassandra-loader options in COPY FROM (3.2 version)

[slebresne]

http://git-wip-us.apache.org/repos/asf/cassandra/blob/85b8d02a/pylib/cqlshlib/copyutil.py
--
diff --git a/pylib/cqlshlib/copyutil.py b/pylib/cqlshlib/copyutil.py
index d7eff9a..0e973cc 100644
--- a/pylib/cqlshlib/copyutil.py
+++ b/pylib/cqlshlib/copyutil.py
@@ -14,12 +14,14 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
+import ConfigParser
 import csv
+import datetime
 import json
+import glob
 import multiprocessing as mp
 import os
 import Queue
-import random
 import re
 import struct
 import sys
@@ -46,82 +48,203 @@ from displaying import NO_COLOR_MAP
 from formatting import format_value_default, DateTimeFormat, EMPTY, 
get_formatter
 from sslhandling import ssl_settings
 
+CopyOptions = namedtuple('CopyOptions', 'copy dialect unrecognized')
 
-def parse_options(shell, opts):
-"""
-Parse options for import (COPY FROM) and export (COPY TO) operations.
-Extract from opts csv and dialect options.
 
-:return: 3 dictionaries: the csv options, the dialect options, any 
unrecognized options.
-"""
-dialect_options = shell.csv_dialect_defaults.copy()
-if 'quote' in opts:
-dialect_options['quotechar'] = opts.pop('quote')
-if 'escape' in opts:
-dialect_options['escapechar'] = opts.pop('escape')
-if 'delimiter' in opts:
-dialect_options['delimiter'] = opts.pop('delimiter')
-if dialect_options['quotechar'] == dialect_options['escapechar']:
-dialect_options['doublequote'] = True
-del dialect_options['escapechar']
-
-csv_options = dict()
-csv_options['nullval'] = opts.pop('null', '')
-csv_options['header'] = bool(opts.pop('header', '').lower() == 'true')
-csv_options['encoding'] = opts.pop('encoding', 'utf8')
-csv_options['maxrequests'] = int(opts.pop('maxrequests', 6))
-csv_options['pagesize'] = int(opts.pop('pagesize', 1000))
-# by default the page timeout is 10 seconds per 1000 entries in the page 
size or 10 seconds if pagesize is smaller
-csv_options['pagetimeout'] = int(opts.pop('pagetimeout', max(10, 10 * 
(csv_options['pagesize'] / 1000
-csv_options['maxattempts'] = int(opts.pop('maxattempts', 5))
-csv_options['dtformats'] = DateTimeFormat(opts.pop('timeformat', 
shell.display_timestamp_format),
-  shell.display_date_format,
-  shell.display_nanotime_format)
-csv_options['float_precision'] = shell.display_float_precision
-csv_options['chunksize'] = int(opts.pop('chunksize', 1000))
-csv_options['ingestrate'] = int(opts.pop('ingestrate', 10))
-csv_options['maxbatchsize'] = int(opts.pop('maxbatchsize', 20))
-csv_options['minbatchsize'] = int(opts.pop('minbatchsize', 2))
-csv_options['reportfrequency'] = float(opts.pop('reportfrequency', 0.25))
-csv_options['ttl'] = int(opts.pop('ttl', -1))
-
-return csv_options, dialect_options, opts
-
-
-def get_num_processes(cap):
+def safe_normpath(fname):
 """
-Pick a reasonable number of child processes. We need to leave at
-least one core for the parent process.  This doesn't necessarily
-need to be capped, but 4 is currently enough to keep
-a single local Cassandra node busy so we use this for import, whilst
-for export we use 16 since we can connect to multiple Cassandra nodes.
-Eventually this parameter will become an option.
+:return the normalized path but only if there is a filename, we don't want 
to convert
+an empty string (which means no file name) to a dot. Also expand any user 
variables such as ~ to the full path
 """
-try:
-return max(1, min(cap, mp.cpu_count() - 1))
-except NotImplementedError:
-return 1
+return os.path.normpath(os.path.expanduser(fname)) if fname else fname
 
 
 class CopyTask(object):
 """
 A base class for ImportTask and ExportTask
 """
-def __init__(self, shell, ks, cf, columns, fname, csv_options, 
dialect_options, protocol_version, config_file):
+def __init__(self, shell, ks, table, columns, fname, opts, 
protocol_version, config_file, direction):
 self.shell = shell
-self.csv_options = csv_options
-self.dialect_options = dialect_options
 self.ks = ks
-self.cf = cf
-self.columns = shell.get_column_names(ks, cf) if columns is None else 
columns
-self.fname = fname
+self.table = table
+self.local_dc = shell.conn.metadata.get_host(shell.hostname).datacenter
+self.fname = safe_normpath(fname)
 self.protocol_version = protocol_version
 self.config_file = config_file
+# do not display messages when exporting to STDOUT
+self.printmsg = self._printmsg if self.fname is not None or direction 
== 'in' else lambda _, eol='\n': None
+self.options = 

[50/50] [abbrv] cassandra git commit: Merge branch 'cassandra-3.3' into trunk

[slebresne]

Merge branch 'cassandra-3.3' into trunk


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/9a88f855
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/9a88f855
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/9a88f855

Branch: refs/heads/trunk
Commit: 9a88f8550093f1f04040d8c502e5b642169cef10
Parents: 8383e89 65b1f2d
Author: Sylvain Lebresne 
Authored: Wed Jan 6 18:01:49 2016 +0100
Committer: Sylvain Lebresne 
Committed: Wed Jan 6 18:01:49 2016 +0100

--
 CHANGES.txt |5 +-
 NEWS.txt|9 +-
 bin/cqlsh.py|  137 +-
 conf/cassandra-env.ps1  |3 -
 conf/cassandra-env.sh   |3 -
 conf/cqlshrc.sample |   17 +-
 conf/jvm.options|   18 +-
 debian/patches/002cassandra_logdir_fix.dpatch   |   18 +-
 pylib/cqlshlib/copyutil.py  | 1263 +-
 pylib/cqlshlib/formatting.py|   96 +-
 src/java/org/apache/cassandra/cql3/Cql.g|4 +-
 .../cql3/statements/BatchStatement.java |   24 +
 .../org/apache/cassandra/db/ReadCommand.java|2 +-
 .../db/compaction/CompactionIterator.java   |6 +-
 .../cassandra/db/partitions/PurgeFunction.java  |   12 +-
 .../apache/cassandra/db/rows/BufferCell.java|2 +-
 src/java/org/apache/cassandra/db/rows/Row.java  |4 +-
 .../apache/cassandra/db/view/TemporalRow.java   |   65 +-
 .../cassandra/locator/PendingRangeMaps.java |  209 ---
 .../apache/cassandra/locator/TokenMetadata.java |   61 +-
 .../apache/cassandra/service/StartupChecks.java |9 +-
 .../cassandra/service/StorageService.java   |2 +-
 .../test/microbench/PendingRangesBench.java |   89 --
 .../org/apache/cassandra/cql3/ViewTest.java |   30 -
 .../cassandra/db/compaction/TTLExpiryTest.java  |   47 +-
 .../cassandra/locator/PendingRangeMapsTest.java |   78 --
 .../cassandra/service/ClientWarningsTest.java   |6 +-
 tools/bin/cassandra-stress.bat  |2 +-
 28 files changed, 1226 insertions(+), 995 deletions(-)
--

[05/50] [abbrv] cassandra git commit: Match cassandra-loader options in COPY FROM (2.2 version)

[slebresne]

http://git-wip-us.apache.org/repos/asf/cassandra/blob/202cf9b0/pylib/cqlshlib/formatting.py
--
diff --git a/pylib/cqlshlib/formatting.py b/pylib/cqlshlib/formatting.py
index 62ecd10..2e219c8 100644
--- a/pylib/cqlshlib/formatting.py
+++ b/pylib/cqlshlib/formatting.py
@@ -60,7 +60,8 @@ empty_colormap = defaultdict(lambda: '')
 
 
 def format_by_type(cqltype, val, encoding, colormap=None, addcolor=False,
-   nullval=None, date_time_format=None, float_precision=None):
+   nullval=None, date_time_format=None, float_precision=None,
+   decimal_sep=None, thousands_sep=None, boolean_styles=None):
 if nullval is None:
 nullval = default_null_placeholder
 if val is None:
@@ -75,7 +76,8 @@ def format_by_type(cqltype, val, encoding, colormap=None, 
addcolor=False,
 float_precision = default_float_precision
 return format_value(cqltype, val, encoding=encoding, colormap=colormap,
 date_time_format=date_time_format, 
float_precision=float_precision,
-nullval=nullval)
+nullval=nullval, decimal_sep=decimal_sep, 
thousands_sep=thousands_sep,
+boolean_styles=boolean_styles)
 
 
 def color_text(bval, colormap, displaywidth=None):
@@ -155,7 +157,9 @@ def format_python_formatted_type(val, colormap, color, 
quote=False):
 
 
 @formatter_for('Decimal')
-def format_value_decimal(val, colormap, **_):
+def format_value_decimal(val, float_precision, colormap, decimal_sep=None, 
thousands_sep=None, **_):
+if (decimal_sep and decimal_sep != '.') or thousands_sep:
+return format_floating_point_type(val, colormap, float_precision, 
decimal_sep, thousands_sep)
 return format_python_formatted_type(val, colormap, 'decimal')
 
 
@@ -170,34 +174,60 @@ def formatter_value_inet(val, colormap, quote=False, **_):
 
 
 @formatter_for('bool')
-def format_value_boolean(val, colormap, **_):
+def format_value_boolean(val, colormap, boolean_styles=None, **_):
+if boolean_styles:
+val = boolean_styles[0] if val else boolean_styles[1]
 return format_python_formatted_type(val, colormap, 'boolean')
 
 
-def format_floating_point_type(val, colormap, float_precision, **_):
+def format_floating_point_type(val, colormap, float_precision, 
decimal_sep=None, thousands_sep=None, **_):
 if math.isnan(val):
 bval = 'NaN'
 elif math.isinf(val):
 bval = 'Infinity' if val > 0 else '-Infinity'
 else:
-exponent = int(math.log10(abs(val))) if abs(val) > 
sys.float_info.epsilon else -sys.maxsize - 1
-if -4 <= exponent < float_precision:
-# when this is true %g will not use scientific notation,
-# increasing precision should not change this decision
-# so we increase the precision to take into account the
-# digits to the left of the decimal point
-float_precision = float_precision + exponent + 1
-bval = '%.*g' % (float_precision, val)
+if thousands_sep:
+dpart, ipart = math.modf(val)
+bval = format_integer_with_thousands_sep(ipart, thousands_sep)
+dpart_str = ('%.*f' % (float_precision, 
math.fabs(dpart)))[2:].rstrip('0')
+if dpart_str:
+bval += '%s%s' % ('.' if not decimal_sep else decimal_sep, 
dpart_str)
+else:
+exponent = int(math.log10(abs(val))) if abs(val) > 
sys.float_info.epsilon else -sys.maxsize - 1
+if -4 <= exponent < float_precision:
+# when this is true %g will not use scientific notation,
+# increasing precision should not change this decision
+# so we increase the precision to take into account the
+# digits to the left of the decimal point
+float_precision = float_precision + exponent + 1
+bval = '%.*g' % (float_precision, val)
+if decimal_sep:
+bval = bval.replace('.', decimal_sep)
+
 return colorme(bval, colormap, 'float')
 
 formatter_for('float')(format_floating_point_type)
 
 
-def format_integer_type(val, colormap, **_):
+def format_integer_type(val, colormap, thousands_sep=None, **_):
 # base-10 only for now; support others?
-bval = str(val)
+bval = format_integer_with_thousands_sep(val, thousands_sep) if 
thousands_sep else str(val)
 return colorme(bval, colormap, 'int')
 
+# We can get rid of this in cassandra-2.2
+if sys.version_info >= (2, 7):
+def format_integer_with_thousands_sep(val, thousands_sep=','):
+return "{:,.0f}".format(val).replace(',', thousands_sep)
+else:
+def format_integer_with_thousands_sep(val, thousands_sep=','):
+if val < 0:
+return '-' + format_integer_with_thousands_sep(-val, thousands_sep)
+result = ''
+while val >= 1000:
+val, r = 

[34/50] [abbrv] cassandra git commit: Match cassandra-loader options in COPY FROM (3.0 version)

[slebresne]

http://git-wip-us.apache.org/repos/asf/cassandra/blob/f2883879/pylib/cqlshlib/copyutil.py
--
diff --git a/pylib/cqlshlib/copyutil.py b/pylib/cqlshlib/copyutil.py
index 6d9a455..a154363 100644
--- a/pylib/cqlshlib/copyutil.py
+++ b/pylib/cqlshlib/copyutil.py
@@ -14,12 +14,14 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
+import ConfigParser
 import csv
+import datetime
 import json
+import glob
 import multiprocessing as mp
 import os
 import Queue
-import random
 import re
 import struct
 import sys
@@ -46,81 +48,202 @@ from displaying import NO_COLOR_MAP
 from formatting import format_value_default, DateTimeFormat, EMPTY, 
get_formatter
 from sslhandling import ssl_settings
 
+CopyOptions = namedtuple('CopyOptions', 'copy dialect unrecognized')
 
-def parse_options(shell, opts):
-"""
-Parse options for import (COPY FROM) and export (COPY TO) operations.
-Extract from opts csv and dialect options.
 
-:return: 3 dictionaries: the csv options, the dialect options, any 
unrecognized options.
-"""
-dialect_options = shell.csv_dialect_defaults.copy()
-if 'quote' in opts:
-dialect_options['quotechar'] = opts.pop('quote')
-if 'escape' in opts:
-dialect_options['escapechar'] = opts.pop('escape')
-if 'delimiter' in opts:
-dialect_options['delimiter'] = opts.pop('delimiter')
-if dialect_options['quotechar'] == dialect_options['escapechar']:
-dialect_options['doublequote'] = True
-del dialect_options['escapechar']
-
-csv_options = dict()
-csv_options['nullval'] = opts.pop('null', '')
-csv_options['header'] = bool(opts.pop('header', '').lower() == 'true')
-csv_options['encoding'] = opts.pop('encoding', 'utf8')
-csv_options['maxrequests'] = int(opts.pop('maxrequests', 6))
-csv_options['pagesize'] = int(opts.pop('pagesize', 1000))
-# by default the page timeout is 10 seconds per 1000 entries in the page 
size or 10 seconds if pagesize is smaller
-csv_options['pagetimeout'] = int(opts.pop('pagetimeout', max(10, 10 * 
(csv_options['pagesize'] / 1000
-csv_options['maxattempts'] = int(opts.pop('maxattempts', 5))
-csv_options['dtformats'] = DateTimeFormat(opts.pop('timeformat', 
shell.display_timestamp_format),
-  shell.display_date_format,
-  shell.display_nanotime_format)
-csv_options['float_precision'] = shell.display_float_precision
-csv_options['chunksize'] = int(opts.pop('chunksize', 1000))
-csv_options['ingestrate'] = int(opts.pop('ingestrate', 10))
-csv_options['maxbatchsize'] = int(opts.pop('maxbatchsize', 20))
-csv_options['minbatchsize'] = int(opts.pop('minbatchsize', 2))
-csv_options['reportfrequency'] = float(opts.pop('reportfrequency', 0.25))
-
-return csv_options, dialect_options, opts
-
-
-def get_num_processes(cap):
+def safe_normpath(fname):
 """
-Pick a reasonable number of child processes. We need to leave at
-least one core for the parent process.  This doesn't necessarily
-need to be capped, but 4 is currently enough to keep
-a single local Cassandra node busy so we use this for import, whilst
-for export we use 16 since we can connect to multiple Cassandra nodes.
-Eventually this parameter will become an option.
+:return the normalized path but only if there is a filename, we don't want 
to convert
+an empty string (which means no file name) to a dot. Also expand any user 
variables such as ~ to the full path
 """
-try:
-return max(1, min(cap, mp.cpu_count() - 1))
-except NotImplementedError:
-return 1
+return os.path.normpath(os.path.expanduser(fname)) if fname else fname
 
 
 class CopyTask(object):
 """
 A base class for ImportTask and ExportTask
 """
-def __init__(self, shell, ks, cf, columns, fname, csv_options, 
dialect_options, protocol_version, config_file):
+def __init__(self, shell, ks, table, columns, fname, opts, 
protocol_version, config_file, direction):
 self.shell = shell
-self.csv_options = csv_options
-self.dialect_options = dialect_options
 self.ks = ks
-self.cf = cf
-self.columns = shell.get_column_names(ks, cf) if columns is None else 
columns
-self.fname = fname
+self.table = table
+self.local_dc = shell.conn.metadata.get_host(shell.hostname).datacenter
+self.fname = safe_normpath(fname)
 self.protocol_version = protocol_version
 self.config_file = config_file
+# do not display messages when exporting to STDOUT
+self.printmsg = self._printmsg if self.fname is not None or direction 
== 'in' else lambda _, eol='\n': None
+self.options = self.parse_options(opts, direction)
+
+self.num_processes = 

[21/50] [abbrv] cassandra git commit: Merge commit '202cf9b0bed8bbff41318f1f10043aabf3a7cd4d' into cassandra-3.0

[slebresne]

Merge commit '202cf9b0bed8bbff41318f1f10043aabf3a7cd4d' into cassandra-3.0


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/3dbc56f1
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/3dbc56f1
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/3dbc56f1

Branch: refs/heads/cassandra-3.3
Commit: 3dbc56f163690c824ffc0aac580a884f34536d62
Parents: e9e127a 202cf9b
Author: Sylvain Lebresne 
Authored: Wed Jan 6 17:57:37 2016 +0100
Committer: Sylvain Lebresne 
Committed: Wed Jan 6 17:57:37 2016 +0100

--

--

[15/50] [abbrv] cassandra git commit: Match cassandra-loader options in COPY FROM (2.2 version)

[slebresne]

http://git-wip-us.apache.org/repos/asf/cassandra/blob/202cf9b0/pylib/cqlshlib/copyutil.py
--
diff --git a/pylib/cqlshlib/copyutil.py b/pylib/cqlshlib/copyutil.py
index 6d9a455..a154363 100644
--- a/pylib/cqlshlib/copyutil.py
+++ b/pylib/cqlshlib/copyutil.py
@@ -14,12 +14,14 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
+import ConfigParser
 import csv
+import datetime
 import json
+import glob
 import multiprocessing as mp
 import os
 import Queue
-import random
 import re
 import struct
 import sys
@@ -46,81 +48,202 @@ from displaying import NO_COLOR_MAP
 from formatting import format_value_default, DateTimeFormat, EMPTY, 
get_formatter
 from sslhandling import ssl_settings
 
+CopyOptions = namedtuple('CopyOptions', 'copy dialect unrecognized')
 
-def parse_options(shell, opts):
-"""
-Parse options for import (COPY FROM) and export (COPY TO) operations.
-Extract from opts csv and dialect options.
 
-:return: 3 dictionaries: the csv options, the dialect options, any 
unrecognized options.
-"""
-dialect_options = shell.csv_dialect_defaults.copy()
-if 'quote' in opts:
-dialect_options['quotechar'] = opts.pop('quote')
-if 'escape' in opts:
-dialect_options['escapechar'] = opts.pop('escape')
-if 'delimiter' in opts:
-dialect_options['delimiter'] = opts.pop('delimiter')
-if dialect_options['quotechar'] == dialect_options['escapechar']:
-dialect_options['doublequote'] = True
-del dialect_options['escapechar']
-
-csv_options = dict()
-csv_options['nullval'] = opts.pop('null', '')
-csv_options['header'] = bool(opts.pop('header', '').lower() == 'true')
-csv_options['encoding'] = opts.pop('encoding', 'utf8')
-csv_options['maxrequests'] = int(opts.pop('maxrequests', 6))
-csv_options['pagesize'] = int(opts.pop('pagesize', 1000))
-# by default the page timeout is 10 seconds per 1000 entries in the page 
size or 10 seconds if pagesize is smaller
-csv_options['pagetimeout'] = int(opts.pop('pagetimeout', max(10, 10 * 
(csv_options['pagesize'] / 1000
-csv_options['maxattempts'] = int(opts.pop('maxattempts', 5))
-csv_options['dtformats'] = DateTimeFormat(opts.pop('timeformat', 
shell.display_timestamp_format),
-  shell.display_date_format,
-  shell.display_nanotime_format)
-csv_options['float_precision'] = shell.display_float_precision
-csv_options['chunksize'] = int(opts.pop('chunksize', 1000))
-csv_options['ingestrate'] = int(opts.pop('ingestrate', 10))
-csv_options['maxbatchsize'] = int(opts.pop('maxbatchsize', 20))
-csv_options['minbatchsize'] = int(opts.pop('minbatchsize', 2))
-csv_options['reportfrequency'] = float(opts.pop('reportfrequency', 0.25))
-
-return csv_options, dialect_options, opts
-
-
-def get_num_processes(cap):
+def safe_normpath(fname):
 """
-Pick a reasonable number of child processes. We need to leave at
-least one core for the parent process.  This doesn't necessarily
-need to be capped, but 4 is currently enough to keep
-a single local Cassandra node busy so we use this for import, whilst
-for export we use 16 since we can connect to multiple Cassandra nodes.
-Eventually this parameter will become an option.
+:return the normalized path but only if there is a filename, we don't want 
to convert
+an empty string (which means no file name) to a dot. Also expand any user 
variables such as ~ to the full path
 """
-try:
-return max(1, min(cap, mp.cpu_count() - 1))
-except NotImplementedError:
-return 1
+return os.path.normpath(os.path.expanduser(fname)) if fname else fname
 
 
 class CopyTask(object):
 """
 A base class for ImportTask and ExportTask
 """
-def __init__(self, shell, ks, cf, columns, fname, csv_options, 
dialect_options, protocol_version, config_file):
+def __init__(self, shell, ks, table, columns, fname, opts, 
protocol_version, config_file, direction):
 self.shell = shell
-self.csv_options = csv_options
-self.dialect_options = dialect_options
 self.ks = ks
-self.cf = cf
-self.columns = shell.get_column_names(ks, cf) if columns is None else 
columns
-self.fname = fname
+self.table = table
+self.local_dc = shell.conn.metadata.get_host(shell.hostname).datacenter
+self.fname = safe_normpath(fname)
 self.protocol_version = protocol_version
 self.config_file = config_file
+# do not display messages when exporting to STDOUT
+self.printmsg = self._printmsg if self.fname is not None or direction 
== 'in' else lambda _, eol='\n': None
+self.options = self.parse_options(opts, direction)
+
+self.num_processes = 

[14/50] [abbrv] cassandra git commit: Match cassandra-loader options in COPY FROM (2.2 version)

[slebresne]

http://git-wip-us.apache.org/repos/asf/cassandra/blob/202cf9b0/pylib/cqlshlib/formatting.py
--
diff --git a/pylib/cqlshlib/formatting.py b/pylib/cqlshlib/formatting.py
index 62ecd10..2e219c8 100644
--- a/pylib/cqlshlib/formatting.py
+++ b/pylib/cqlshlib/formatting.py
@@ -60,7 +60,8 @@ empty_colormap = defaultdict(lambda: '')
 
 
 def format_by_type(cqltype, val, encoding, colormap=None, addcolor=False,
-   nullval=None, date_time_format=None, float_precision=None):
+   nullval=None, date_time_format=None, float_precision=None,
+   decimal_sep=None, thousands_sep=None, boolean_styles=None):
 if nullval is None:
 nullval = default_null_placeholder
 if val is None:
@@ -75,7 +76,8 @@ def format_by_type(cqltype, val, encoding, colormap=None, 
addcolor=False,
 float_precision = default_float_precision
 return format_value(cqltype, val, encoding=encoding, colormap=colormap,
 date_time_format=date_time_format, 
float_precision=float_precision,
-nullval=nullval)
+nullval=nullval, decimal_sep=decimal_sep, 
thousands_sep=thousands_sep,
+boolean_styles=boolean_styles)
 
 
 def color_text(bval, colormap, displaywidth=None):
@@ -155,7 +157,9 @@ def format_python_formatted_type(val, colormap, color, 
quote=False):
 
 
 @formatter_for('Decimal')
-def format_value_decimal(val, colormap, **_):
+def format_value_decimal(val, float_precision, colormap, decimal_sep=None, 
thousands_sep=None, **_):
+if (decimal_sep and decimal_sep != '.') or thousands_sep:
+return format_floating_point_type(val, colormap, float_precision, 
decimal_sep, thousands_sep)
 return format_python_formatted_type(val, colormap, 'decimal')
 
 
@@ -170,34 +174,60 @@ def formatter_value_inet(val, colormap, quote=False, **_):
 
 
 @formatter_for('bool')
-def format_value_boolean(val, colormap, **_):
+def format_value_boolean(val, colormap, boolean_styles=None, **_):
+if boolean_styles:
+val = boolean_styles[0] if val else boolean_styles[1]
 return format_python_formatted_type(val, colormap, 'boolean')
 
 
-def format_floating_point_type(val, colormap, float_precision, **_):
+def format_floating_point_type(val, colormap, float_precision, 
decimal_sep=None, thousands_sep=None, **_):
 if math.isnan(val):
 bval = 'NaN'
 elif math.isinf(val):
 bval = 'Infinity' if val > 0 else '-Infinity'
 else:
-exponent = int(math.log10(abs(val))) if abs(val) > 
sys.float_info.epsilon else -sys.maxsize - 1
-if -4 <= exponent < float_precision:
-# when this is true %g will not use scientific notation,
-# increasing precision should not change this decision
-# so we increase the precision to take into account the
-# digits to the left of the decimal point
-float_precision = float_precision + exponent + 1
-bval = '%.*g' % (float_precision, val)
+if thousands_sep:
+dpart, ipart = math.modf(val)
+bval = format_integer_with_thousands_sep(ipart, thousands_sep)
+dpart_str = ('%.*f' % (float_precision, 
math.fabs(dpart)))[2:].rstrip('0')
+if dpart_str:
+bval += '%s%s' % ('.' if not decimal_sep else decimal_sep, 
dpart_str)
+else:
+exponent = int(math.log10(abs(val))) if abs(val) > 
sys.float_info.epsilon else -sys.maxsize - 1
+if -4 <= exponent < float_precision:
+# when this is true %g will not use scientific notation,
+# increasing precision should not change this decision
+# so we increase the precision to take into account the
+# digits to the left of the decimal point
+float_precision = float_precision + exponent + 1
+bval = '%.*g' % (float_precision, val)
+if decimal_sep:
+bval = bval.replace('.', decimal_sep)
+
 return colorme(bval, colormap, 'float')
 
 formatter_for('float')(format_floating_point_type)
 
 
-def format_integer_type(val, colormap, **_):
+def format_integer_type(val, colormap, thousands_sep=None, **_):
 # base-10 only for now; support others?
-bval = str(val)
+bval = format_integer_with_thousands_sep(val, thousands_sep) if 
thousands_sep else str(val)
 return colorme(bval, colormap, 'int')
 
+# We can get rid of this in cassandra-2.2
+if sys.version_info >= (2, 7):
+def format_integer_with_thousands_sep(val, thousands_sep=','):
+return "{:,.0f}".format(val).replace(',', thousands_sep)
+else:
+def format_integer_with_thousands_sep(val, thousands_sep=','):
+if val < 0:
+return '-' + format_integer_with_thousands_sep(-val, thousands_sep)
+result = ''
+while val >= 1000:
+val, r = 

[31/50] [abbrv] cassandra git commit: Match cassandra-loader options in COPY FROM (3.0 version)

[slebresne]

http://git-wip-us.apache.org/repos/asf/cassandra/blob/f2883879/pylib/cqlshlib/copyutil.py
--
diff --git a/pylib/cqlshlib/copyutil.py b/pylib/cqlshlib/copyutil.py
index 6d9a455..a154363 100644
--- a/pylib/cqlshlib/copyutil.py
+++ b/pylib/cqlshlib/copyutil.py
@@ -14,12 +14,14 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
+import ConfigParser
 import csv
+import datetime
 import json
+import glob
 import multiprocessing as mp
 import os
 import Queue
-import random
 import re
 import struct
 import sys
@@ -46,81 +48,202 @@ from displaying import NO_COLOR_MAP
 from formatting import format_value_default, DateTimeFormat, EMPTY, 
get_formatter
 from sslhandling import ssl_settings
 
+CopyOptions = namedtuple('CopyOptions', 'copy dialect unrecognized')
 
-def parse_options(shell, opts):
-"""
-Parse options for import (COPY FROM) and export (COPY TO) operations.
-Extract from opts csv and dialect options.
 
-:return: 3 dictionaries: the csv options, the dialect options, any 
unrecognized options.
-"""
-dialect_options = shell.csv_dialect_defaults.copy()
-if 'quote' in opts:
-dialect_options['quotechar'] = opts.pop('quote')
-if 'escape' in opts:
-dialect_options['escapechar'] = opts.pop('escape')
-if 'delimiter' in opts:
-dialect_options['delimiter'] = opts.pop('delimiter')
-if dialect_options['quotechar'] == dialect_options['escapechar']:
-dialect_options['doublequote'] = True
-del dialect_options['escapechar']
-
-csv_options = dict()
-csv_options['nullval'] = opts.pop('null', '')
-csv_options['header'] = bool(opts.pop('header', '').lower() == 'true')
-csv_options['encoding'] = opts.pop('encoding', 'utf8')
-csv_options['maxrequests'] = int(opts.pop('maxrequests', 6))
-csv_options['pagesize'] = int(opts.pop('pagesize', 1000))
-# by default the page timeout is 10 seconds per 1000 entries in the page 
size or 10 seconds if pagesize is smaller
-csv_options['pagetimeout'] = int(opts.pop('pagetimeout', max(10, 10 * 
(csv_options['pagesize'] / 1000
-csv_options['maxattempts'] = int(opts.pop('maxattempts', 5))
-csv_options['dtformats'] = DateTimeFormat(opts.pop('timeformat', 
shell.display_timestamp_format),
-  shell.display_date_format,
-  shell.display_nanotime_format)
-csv_options['float_precision'] = shell.display_float_precision
-csv_options['chunksize'] = int(opts.pop('chunksize', 1000))
-csv_options['ingestrate'] = int(opts.pop('ingestrate', 10))
-csv_options['maxbatchsize'] = int(opts.pop('maxbatchsize', 20))
-csv_options['minbatchsize'] = int(opts.pop('minbatchsize', 2))
-csv_options['reportfrequency'] = float(opts.pop('reportfrequency', 0.25))
-
-return csv_options, dialect_options, opts
-
-
-def get_num_processes(cap):
+def safe_normpath(fname):
 """
-Pick a reasonable number of child processes. We need to leave at
-least one core for the parent process.  This doesn't necessarily
-need to be capped, but 4 is currently enough to keep
-a single local Cassandra node busy so we use this for import, whilst
-for export we use 16 since we can connect to multiple Cassandra nodes.
-Eventually this parameter will become an option.
+:return the normalized path but only if there is a filename, we don't want 
to convert
+an empty string (which means no file name) to a dot. Also expand any user 
variables such as ~ to the full path
 """
-try:
-return max(1, min(cap, mp.cpu_count() - 1))
-except NotImplementedError:
-return 1
+return os.path.normpath(os.path.expanduser(fname)) if fname else fname
 
 
 class CopyTask(object):
 """
 A base class for ImportTask and ExportTask
 """
-def __init__(self, shell, ks, cf, columns, fname, csv_options, 
dialect_options, protocol_version, config_file):
+def __init__(self, shell, ks, table, columns, fname, opts, 
protocol_version, config_file, direction):
 self.shell = shell
-self.csv_options = csv_options
-self.dialect_options = dialect_options
 self.ks = ks
-self.cf = cf
-self.columns = shell.get_column_names(ks, cf) if columns is None else 
columns
-self.fname = fname
+self.table = table
+self.local_dc = shell.conn.metadata.get_host(shell.hostname).datacenter
+self.fname = safe_normpath(fname)
 self.protocol_version = protocol_version
 self.config_file = config_file
+# do not display messages when exporting to STDOUT
+self.printmsg = self._printmsg if self.fname is not None or direction 
== 'in' else lambda _, eol='\n': None
+self.options = self.parse_options(opts, direction)
+
+self.num_processes = 

[39/50] [abbrv] cassandra git commit: Match cassandra-loader options in COPY FROM (3.2 version)

[slebresne]

http://git-wip-us.apache.org/repos/asf/cassandra/blob/85b8d02a/pylib/cqlshlib/formatting.py
--
diff --git a/pylib/cqlshlib/formatting.py b/pylib/cqlshlib/formatting.py
index 62ecd10..2e219c8 100644
--- a/pylib/cqlshlib/formatting.py
+++ b/pylib/cqlshlib/formatting.py
@@ -60,7 +60,8 @@ empty_colormap = defaultdict(lambda: '')
 
 
 def format_by_type(cqltype, val, encoding, colormap=None, addcolor=False,
-   nullval=None, date_time_format=None, float_precision=None):
+   nullval=None, date_time_format=None, float_precision=None,
+   decimal_sep=None, thousands_sep=None, boolean_styles=None):
 if nullval is None:
 nullval = default_null_placeholder
 if val is None:
@@ -75,7 +76,8 @@ def format_by_type(cqltype, val, encoding, colormap=None, 
addcolor=False,
 float_precision = default_float_precision
 return format_value(cqltype, val, encoding=encoding, colormap=colormap,
 date_time_format=date_time_format, 
float_precision=float_precision,
-nullval=nullval)
+nullval=nullval, decimal_sep=decimal_sep, 
thousands_sep=thousands_sep,
+boolean_styles=boolean_styles)
 
 
 def color_text(bval, colormap, displaywidth=None):
@@ -155,7 +157,9 @@ def format_python_formatted_type(val, colormap, color, 
quote=False):
 
 
 @formatter_for('Decimal')
-def format_value_decimal(val, colormap, **_):
+def format_value_decimal(val, float_precision, colormap, decimal_sep=None, 
thousands_sep=None, **_):
+if (decimal_sep and decimal_sep != '.') or thousands_sep:
+return format_floating_point_type(val, colormap, float_precision, 
decimal_sep, thousands_sep)
 return format_python_formatted_type(val, colormap, 'decimal')
 
 
@@ -170,34 +174,60 @@ def formatter_value_inet(val, colormap, quote=False, **_):
 
 
 @formatter_for('bool')
-def format_value_boolean(val, colormap, **_):
+def format_value_boolean(val, colormap, boolean_styles=None, **_):
+if boolean_styles:
+val = boolean_styles[0] if val else boolean_styles[1]
 return format_python_formatted_type(val, colormap, 'boolean')
 
 
-def format_floating_point_type(val, colormap, float_precision, **_):
+def format_floating_point_type(val, colormap, float_precision, 
decimal_sep=None, thousands_sep=None, **_):
 if math.isnan(val):
 bval = 'NaN'
 elif math.isinf(val):
 bval = 'Infinity' if val > 0 else '-Infinity'
 else:
-exponent = int(math.log10(abs(val))) if abs(val) > 
sys.float_info.epsilon else -sys.maxsize - 1
-if -4 <= exponent < float_precision:
-# when this is true %g will not use scientific notation,
-# increasing precision should not change this decision
-# so we increase the precision to take into account the
-# digits to the left of the decimal point
-float_precision = float_precision + exponent + 1
-bval = '%.*g' % (float_precision, val)
+if thousands_sep:
+dpart, ipart = math.modf(val)
+bval = format_integer_with_thousands_sep(ipart, thousands_sep)
+dpart_str = ('%.*f' % (float_precision, 
math.fabs(dpart)))[2:].rstrip('0')
+if dpart_str:
+bval += '%s%s' % ('.' if not decimal_sep else decimal_sep, 
dpart_str)
+else:
+exponent = int(math.log10(abs(val))) if abs(val) > 
sys.float_info.epsilon else -sys.maxsize - 1
+if -4 <= exponent < float_precision:
+# when this is true %g will not use scientific notation,
+# increasing precision should not change this decision
+# so we increase the precision to take into account the
+# digits to the left of the decimal point
+float_precision = float_precision + exponent + 1
+bval = '%.*g' % (float_precision, val)
+if decimal_sep:
+bval = bval.replace('.', decimal_sep)
+
 return colorme(bval, colormap, 'float')
 
 formatter_for('float')(format_floating_point_type)
 
 
-def format_integer_type(val, colormap, **_):
+def format_integer_type(val, colormap, thousands_sep=None, **_):
 # base-10 only for now; support others?
-bval = str(val)
+bval = format_integer_with_thousands_sep(val, thousands_sep) if 
thousands_sep else str(val)
 return colorme(bval, colormap, 'int')
 
+# We can get rid of this in cassandra-2.2
+if sys.version_info >= (2, 7):
+def format_integer_with_thousands_sep(val, thousands_sep=','):
+return "{:,.0f}".format(val).replace(',', thousands_sep)
+else:
+def format_integer_with_thousands_sep(val, thousands_sep=','):
+if val < 0:
+return '-' + format_integer_with_thousands_sep(-val, thousands_sep)
+result = ''
+while val >= 1000:
+val, r = 

[29/50] [abbrv] cassandra git commit: Match cassandra-loader options in COPY FROM (3.0 version)

[slebresne]

Match cassandra-loader options in COPY FROM (3.0 version)

patch by Stefania; reviewed by pauloricardomg for CASSANDRA-9303


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/f2883879
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/f2883879
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/f2883879

Branch: refs/heads/cassandra-3.3
Commit: f288387917fd787f8476c2bca7dfb9d7892c24e8
Parents: 3dbc56f
Author: Stefania Alborghetti 
Authored: Wed Jan 6 12:12:12 2016 +0100
Committer: Sylvain Lebresne 
Committed: Wed Jan 6 17:58:20 2016 +0100

--
 CHANGES.txt |1 +
 NEWS.txt|7 +
 bin/cqlsh.py|  135 +-
 conf/cqlshrc.sample |   17 +-
 pylib/cqlshlib/copyutil.py  | 1260 +-
 pylib/cqlshlib/formatting.py|   96 +-
 .../cql3/statements/BatchStatement.java |   26 +-
 .../cassandra/service/ClientWarningsTest.java   |6 +-
 tools/bin/cassandra-stress.bat  |2 +-
 9 files changed, 1134 insertions(+), 416 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/f2883879/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index cf872d9..3cb0b1e 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -25,6 +25,7 @@ Merged from 2.2:
  * Disable reloading of GossipingPropertyFileSnitch (CASSANDRA-9474)
  * Verify tables in pseudo-system keyspaces at startup (CASSANDRA-10761)
 Merged from 2.1:
+ * Match cassandra-loader options in COPY FROM (CASSANDRA-9303)
  * Fix binding to any address in CqlBulkRecordWriter (CASSANDRA-9309)
  * cqlsh fails to decode utf-8 characters for text typed columns 
(CASSANDRA-10875)
  * Log error when stream session fails (CASSANDRA-9294)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/f2883879/NEWS.txt
--
diff --git a/NEWS.txt b/NEWS.txt
index 26a83a9..64837e1 100644
--- a/NEWS.txt
+++ b/NEWS.txt
@@ -383,6 +383,13 @@ Upgrading
  to exclude data centers when the global status is enabled, see 
CASSANDRA-9035 for details.
 
 
+2.1.13
+==
+
+New features
+
+- New options for cqlsh COPY FROM and COPY TO, see CASSANDRA-9303 for 
details.
+
 2.1.10
 =
 

http://git-wip-us.apache.org/repos/asf/cassandra/blob/f2883879/bin/cqlsh.py
--
diff --git a/bin/cqlsh.py b/bin/cqlsh.py
index d8e8db5..2f53cc1 100644
--- a/bin/cqlsh.py
+++ b/bin/cqlsh.py
@@ -41,7 +41,6 @@ import optparse
 import os
 import platform
 import sys
-import time
 import traceback
 import warnings
 import webbrowser
@@ -152,7 +151,8 @@ cqlshlibdir = os.path.join(CASSANDRA_PATH, 'pylib')
 if os.path.isdir(cqlshlibdir):
 sys.path.insert(0, cqlshlibdir)
 
-from cqlshlib import cql3handling, cqlhandling, copyutil, pylexotron, 
sslhandling
+from cqlshlib import cql3handling, cqlhandling, pylexotron, sslhandling
+from cqlshlib.copyutil import ExportTask, ImportTask
 from cqlshlib.displaying import (ANSI_RESET, BLUE, COLUMN_NAME_COLORS, CYAN,
  RED, FormattedValue, colorme)
 from cqlshlib.formatting import (DEFAULT_DATE_FORMAT, DEFAULT_NANOTIME_FORMAT,
@@ -458,10 +458,12 @@ def complete_copy_column_names(ctxt, cqlsh):
 return set(colnames[1:]) - set(existcols)
 
 
-COPY_COMMON_OPTIONS = ['DELIMITER', 'QUOTE', 'ESCAPE', 'HEADER', 'NULL',
-   'MAXATTEMPTS', 'REPORTFREQUENCY']
-COPY_FROM_OPTIONS = ['CHUNKSIZE', 'INGESTRATE', 'MAXBATCHSIZE', 'MINBATCHSIZE']
-COPY_TO_OPTIONS = ['ENCODING', 'TIMEFORMAT', 'PAGESIZE', 'PAGETIMEOUT', 
'MAXREQUESTS']
+COPY_COMMON_OPTIONS = ['DELIMITER', 'QUOTE', 'ESCAPE', 'HEADER', 'NULL', 
'DATETIMEFORMAT',
+   'MAXATTEMPTS', 'REPORTFREQUENCY', 'DECIMALSEP', 
'THOUSANDSSEP', 'BOOLSTYLE',
+   'NUMPROCESSES', 'CONFIGFILE', 'RATEFILE']
+COPY_FROM_OPTIONS = ['CHUNKSIZE', 'INGESTRATE', 'MAXBATCHSIZE', 
'MINBATCHSIZE', 'MAXROWS',
+ 'SKIPROWS', 'SKIPCOLS', 'MAXPARSEERRORS', 
'MAXINSERTERRORS', 'ERRFILE']
+COPY_TO_OPTIONS = ['ENCODING', 'PAGESIZE', 'PAGETIMEOUT', 'BEGINTOKEN', 
'ENDTOKEN', 'MAXOUTPUTSIZE', 'MAXREQUESTS']
 
 
 @cqlsh_syntax_completer('copyOption', 'optnames')
@@ -581,23 +583,6 @@ warnings.showwarning = show_warning_without_quoting_line
 warnings.filterwarnings('always', 
category=cql3handling.UnexpectedTableStructure)
 
 
-def describe_interval(seconds):
-desc = []
-for length, unit in ((86400, 'day'), (3600, 'hour'), (60, 'minute')):
- 

[41/50] [abbrv] cassandra git commit: Match cassandra-loader options in COPY FROM (3.2 version)

[slebresne]

Match cassandra-loader options in COPY FROM (3.2 version)

patch by Stefania; reviewed by pauloricardomg for CASSANDRA-9303


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/85b8d02a
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/85b8d02a
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/85b8d02a

Branch: refs/heads/cassandra-3.3
Commit: 85b8d02aaebae25fa430b3d036058323a42c8e4e
Parents: 08acfe1
Author: Stefania Alborghetti 
Authored: Wed Jan 6 12:12:12 2016 +0100
Committer: Sylvain Lebresne 
Committed: Wed Jan 6 18:00:46 2016 +0100

--
 CHANGES.txt |1 +
 NEWS.txt|7 +
 bin/cqlsh.py|  137 +-
 conf/cqlshrc.sample |   17 +-
 pylib/cqlshlib/copyutil.py  | 1263 +-
 pylib/cqlshlib/formatting.py|   96 +-
 .../cql3/statements/BatchStatement.java |   24 +
 .../cassandra/service/ClientWarningsTest.java   |6 +-
 tools/bin/cassandra-stress.bat  |2 +-
 9 files changed, 1135 insertions(+), 418 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/85b8d02a/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 43acd43..339ac4a 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -43,6 +43,7 @@ Merged from 2.2:
  * Add new types to Stress (CASSANDRA-9556)
  * Add property to allow listening on broadcast interface (CASSANDRA-9748)
 Merged from 2.1:
+ * Match cassandra-loader options in COPY FROM (CASSANDRA-9303)
  * Fix binding to any address in CqlBulkRecordWriter (CASSANDRA-9309)
  * cqlsh fails to decode utf-8 characters for text typed columns 
(CASSANDRA-10875)
  * Log error when stream session fails (CASSANDRA-9294)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/85b8d02a/NEWS.txt
--
diff --git a/NEWS.txt b/NEWS.txt
index f5f50c1..3d468b6 100644
--- a/NEWS.txt
+++ b/NEWS.txt
@@ -400,6 +400,13 @@ Upgrading
  to exclude data centers when the global status is enabled, see 
CASSANDRA-9035 for details.
 
 
+2.1.13
+==
+
+New features
+
+- New options for cqlsh COPY FROM and COPY TO, see CASSANDRA-9303 for 
details.
+
 2.1.10
 =
 

http://git-wip-us.apache.org/repos/asf/cassandra/blob/85b8d02a/bin/cqlsh.py
--
diff --git a/bin/cqlsh.py b/bin/cqlsh.py
index 5497f39..65352c8 100644
--- a/bin/cqlsh.py
+++ b/bin/cqlsh.py
@@ -41,7 +41,6 @@ import optparse
 import os
 import platform
 import sys
-import time
 import traceback
 import warnings
 import webbrowser
@@ -152,7 +151,8 @@ cqlshlibdir = os.path.join(CASSANDRA_PATH, 'pylib')
 if os.path.isdir(cqlshlibdir):
 sys.path.insert(0, cqlshlibdir)
 
-from cqlshlib import cql3handling, cqlhandling, copyutil, pylexotron, 
sslhandling
+from cqlshlib import cql3handling, cqlhandling, pylexotron, sslhandling
+from cqlshlib.copyutil import ExportTask, ImportTask
 from cqlshlib.displaying import (ANSI_RESET, BLUE, COLUMN_NAME_COLORS, CYAN,
  RED, FormattedValue, colorme)
 from cqlshlib.formatting import (DEFAULT_DATE_FORMAT, DEFAULT_NANOTIME_FORMAT,
@@ -458,10 +458,12 @@ def complete_copy_column_names(ctxt, cqlsh):
 return set(colnames[1:]) - set(existcols)
 
 
-COPY_COMMON_OPTIONS = ['DELIMITER', 'QUOTE', 'ESCAPE', 'HEADER', 'NULL',
-   'MAXATTEMPTS', 'REPORTFREQUENCY']
-COPY_FROM_OPTIONS = ['CHUNKSIZE', 'INGESTRATE', 'MAXBATCHSIZE', 
'MINBATCHSIZE', 'TTL']
-COPY_TO_OPTIONS = ['ENCODING', 'TIMEFORMAT', 'PAGESIZE', 'PAGETIMEOUT', 
'MAXREQUESTS']
+COPY_COMMON_OPTIONS = ['DELIMITER', 'QUOTE', 'ESCAPE', 'HEADER', 'NULL', 
'DATETIMEFORMAT',
+   'MAXATTEMPTS', 'REPORTFREQUENCY', 'DECIMALSEP', 
'THOUSANDSSEP', 'BOOLSTYLE',
+   'NUMPROCESSES', 'CONFIGFILE', 'RATEFILE']
+COPY_FROM_OPTIONS = ['CHUNKSIZE', 'INGESTRATE', 'MAXBATCHSIZE', 
'MINBATCHSIZE', 'MAXROWS',
+ 'SKIPROWS', 'SKIPCOLS', 'MAXPARSEERRORS', 
'MAXINSERTERRORS', 'ERRFILE', 'TTL']
+COPY_TO_OPTIONS = ['ENCODING', 'PAGESIZE', 'PAGETIMEOUT', 'BEGINTOKEN', 
'ENDTOKEN', 'MAXOUTPUTSIZE', 'MAXREQUESTS']
 
 
 @cqlsh_syntax_completer('copyOption', 'optnames')
@@ -581,23 +583,6 @@ warnings.showwarning = show_warning_without_quoting_line
 warnings.filterwarnings('always', 
category=cql3handling.UnexpectedTableStructure)
 
 
-def describe_interval(seconds):
-desc = []
-for length, unit in ((86400, 'day'), (3600, 'hour'), (60, 'minute')):
-num = 

[20/50] [abbrv] cassandra git commit: Merge commit '202cf9b0bed8bbff41318f1f10043aabf3a7cd4d' into cassandra-3.0

[slebresne]

Merge commit '202cf9b0bed8bbff41318f1f10043aabf3a7cd4d' into cassandra-3.0


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/3dbc56f1
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/3dbc56f1
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/3dbc56f1

Branch: refs/heads/cassandra-3.2
Commit: 3dbc56f163690c824ffc0aac580a884f34536d62
Parents: e9e127a 202cf9b
Author: Sylvain Lebresne 
Authored: Wed Jan 6 17:57:37 2016 +0100
Committer: Sylvain Lebresne 
Committed: Wed Jan 6 17:57:37 2016 +0100

--

--

[32/50] [abbrv] cassandra git commit: Match cassandra-loader options in COPY FROM (3.0 version)

[slebresne]

Match cassandra-loader options in COPY FROM (3.0 version)

patch by Stefania; reviewed by pauloricardomg for CASSANDRA-9303


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/f2883879
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/f2883879
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/f2883879

Branch: refs/heads/cassandra-3.0
Commit: f288387917fd787f8476c2bca7dfb9d7892c24e8
Parents: 3dbc56f
Author: Stefania Alborghetti 
Authored: Wed Jan 6 12:12:12 2016 +0100
Committer: Sylvain Lebresne 
Committed: Wed Jan 6 17:58:20 2016 +0100

--
 CHANGES.txt |1 +
 NEWS.txt|7 +
 bin/cqlsh.py|  135 +-
 conf/cqlshrc.sample |   17 +-
 pylib/cqlshlib/copyutil.py  | 1260 +-
 pylib/cqlshlib/formatting.py|   96 +-
 .../cql3/statements/BatchStatement.java |   26 +-
 .../cassandra/service/ClientWarningsTest.java   |6 +-
 tools/bin/cassandra-stress.bat  |2 +-
 9 files changed, 1134 insertions(+), 416 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/f2883879/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index cf872d9..3cb0b1e 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -25,6 +25,7 @@ Merged from 2.2:
  * Disable reloading of GossipingPropertyFileSnitch (CASSANDRA-9474)
  * Verify tables in pseudo-system keyspaces at startup (CASSANDRA-10761)
 Merged from 2.1:
+ * Match cassandra-loader options in COPY FROM (CASSANDRA-9303)
  * Fix binding to any address in CqlBulkRecordWriter (CASSANDRA-9309)
  * cqlsh fails to decode utf-8 characters for text typed columns 
(CASSANDRA-10875)
  * Log error when stream session fails (CASSANDRA-9294)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/f2883879/NEWS.txt
--
diff --git a/NEWS.txt b/NEWS.txt
index 26a83a9..64837e1 100644
--- a/NEWS.txt
+++ b/NEWS.txt
@@ -383,6 +383,13 @@ Upgrading
  to exclude data centers when the global status is enabled, see 
CASSANDRA-9035 for details.
 
 
+2.1.13
+==
+
+New features
+
+- New options for cqlsh COPY FROM and COPY TO, see CASSANDRA-9303 for 
details.
+
 2.1.10
 =
 

http://git-wip-us.apache.org/repos/asf/cassandra/blob/f2883879/bin/cqlsh.py
--
diff --git a/bin/cqlsh.py b/bin/cqlsh.py
index d8e8db5..2f53cc1 100644
--- a/bin/cqlsh.py
+++ b/bin/cqlsh.py
@@ -41,7 +41,6 @@ import optparse
 import os
 import platform
 import sys
-import time
 import traceback
 import warnings
 import webbrowser
@@ -152,7 +151,8 @@ cqlshlibdir = os.path.join(CASSANDRA_PATH, 'pylib')
 if os.path.isdir(cqlshlibdir):
 sys.path.insert(0, cqlshlibdir)
 
-from cqlshlib import cql3handling, cqlhandling, copyutil, pylexotron, 
sslhandling
+from cqlshlib import cql3handling, cqlhandling, pylexotron, sslhandling
+from cqlshlib.copyutil import ExportTask, ImportTask
 from cqlshlib.displaying import (ANSI_RESET, BLUE, COLUMN_NAME_COLORS, CYAN,
  RED, FormattedValue, colorme)
 from cqlshlib.formatting import (DEFAULT_DATE_FORMAT, DEFAULT_NANOTIME_FORMAT,
@@ -458,10 +458,12 @@ def complete_copy_column_names(ctxt, cqlsh):
 return set(colnames[1:]) - set(existcols)
 
 
-COPY_COMMON_OPTIONS = ['DELIMITER', 'QUOTE', 'ESCAPE', 'HEADER', 'NULL',
-   'MAXATTEMPTS', 'REPORTFREQUENCY']
-COPY_FROM_OPTIONS = ['CHUNKSIZE', 'INGESTRATE', 'MAXBATCHSIZE', 'MINBATCHSIZE']
-COPY_TO_OPTIONS = ['ENCODING', 'TIMEFORMAT', 'PAGESIZE', 'PAGETIMEOUT', 
'MAXREQUESTS']
+COPY_COMMON_OPTIONS = ['DELIMITER', 'QUOTE', 'ESCAPE', 'HEADER', 'NULL', 
'DATETIMEFORMAT',
+   'MAXATTEMPTS', 'REPORTFREQUENCY', 'DECIMALSEP', 
'THOUSANDSSEP', 'BOOLSTYLE',
+   'NUMPROCESSES', 'CONFIGFILE', 'RATEFILE']
+COPY_FROM_OPTIONS = ['CHUNKSIZE', 'INGESTRATE', 'MAXBATCHSIZE', 
'MINBATCHSIZE', 'MAXROWS',
+ 'SKIPROWS', 'SKIPCOLS', 'MAXPARSEERRORS', 
'MAXINSERTERRORS', 'ERRFILE']
+COPY_TO_OPTIONS = ['ENCODING', 'PAGESIZE', 'PAGETIMEOUT', 'BEGINTOKEN', 
'ENDTOKEN', 'MAXOUTPUTSIZE', 'MAXREQUESTS']
 
 
 @cqlsh_syntax_completer('copyOption', 'optnames')
@@ -581,23 +583,6 @@ warnings.showwarning = show_warning_without_quoting_line
 warnings.filterwarnings('always', 
category=cql3handling.UnexpectedTableStructure)
 
 
-def describe_interval(seconds):
-desc = []
-for length, unit in ((86400, 'day'), (3600, 'hour'), (60, 'minute')):
- 

[48/50] [abbrv] cassandra git commit: Merge branch 'cassandra-3.2' into cassandra-3.3

[slebresne]

Merge branch 'cassandra-3.2' into cassandra-3.3


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/65b1f2da
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/65b1f2da
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/65b1f2da

Branch: refs/heads/cassandra-3.3
Commit: 65b1f2dac0701d02cd65c0526e98697c966e9560
Parents: a7854fd 85b8d02
Author: Sylvain Lebresne 
Authored: Wed Jan 6 18:01:37 2016 +0100
Committer: Sylvain Lebresne 
Committed: Wed Jan 6 18:01:37 2016 +0100

--
 CHANGES.txt |5 +-
 NEWS.txt|9 +-
 bin/cqlsh.py|  137 +-
 conf/cassandra-env.ps1  |3 -
 conf/cassandra-env.sh   |3 -
 conf/cqlshrc.sample |   17 +-
 conf/jvm.options|   18 +-
 debian/patches/002cassandra_logdir_fix.dpatch   |   18 +-
 pylib/cqlshlib/copyutil.py  | 1263 +-
 pylib/cqlshlib/formatting.py|   96 +-
 src/java/org/apache/cassandra/cql3/Cql.g|4 +-
 .../cql3/statements/BatchStatement.java |   24 +
 .../org/apache/cassandra/db/ReadCommand.java|2 +-
 .../db/compaction/CompactionIterator.java   |6 +-
 .../cassandra/db/partitions/PurgeFunction.java  |   12 +-
 .../apache/cassandra/db/rows/BufferCell.java|2 +-
 src/java/org/apache/cassandra/db/rows/Row.java  |4 +-
 .../apache/cassandra/db/view/TemporalRow.java   |   65 +-
 .../cassandra/locator/PendingRangeMaps.java |  209 ---
 .../apache/cassandra/locator/TokenMetadata.java |   61 +-
 .../apache/cassandra/service/StartupChecks.java |9 +-
 .../cassandra/service/StorageService.java   |2 +-
 .../test/microbench/PendingRangesBench.java |   89 --
 .../org/apache/cassandra/cql3/ViewTest.java |   30 -
 .../cassandra/db/compaction/TTLExpiryTest.java  |   47 +-
 .../cassandra/locator/PendingRangeMapsTest.java |   78 --
 .../cassandra/service/ClientWarningsTest.java   |6 +-
 tools/bin/cassandra-stress.bat  |2 +-
 28 files changed, 1226 insertions(+), 995 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/65b1f2da/CHANGES.txt
--

[49/50] [abbrv] cassandra git commit: Merge branch 'cassandra-3.2' into cassandra-3.3

[slebresne]

Merge branch 'cassandra-3.2' into cassandra-3.3


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/65b1f2da
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/65b1f2da
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/65b1f2da

Branch: refs/heads/trunk
Commit: 65b1f2dac0701d02cd65c0526e98697c966e9560
Parents: a7854fd 85b8d02
Author: Sylvain Lebresne 
Authored: Wed Jan 6 18:01:37 2016 +0100
Committer: Sylvain Lebresne 
Committed: Wed Jan 6 18:01:37 2016 +0100

--
 CHANGES.txt |5 +-
 NEWS.txt|9 +-
 bin/cqlsh.py|  137 +-
 conf/cassandra-env.ps1  |3 -
 conf/cassandra-env.sh   |3 -
 conf/cqlshrc.sample |   17 +-
 conf/jvm.options|   18 +-
 debian/patches/002cassandra_logdir_fix.dpatch   |   18 +-
 pylib/cqlshlib/copyutil.py  | 1263 +-
 pylib/cqlshlib/formatting.py|   96 +-
 src/java/org/apache/cassandra/cql3/Cql.g|4 +-
 .../cql3/statements/BatchStatement.java |   24 +
 .../org/apache/cassandra/db/ReadCommand.java|2 +-
 .../db/compaction/CompactionIterator.java   |6 +-
 .../cassandra/db/partitions/PurgeFunction.java  |   12 +-
 .../apache/cassandra/db/rows/BufferCell.java|2 +-
 src/java/org/apache/cassandra/db/rows/Row.java  |4 +-
 .../apache/cassandra/db/view/TemporalRow.java   |   65 +-
 .../cassandra/locator/PendingRangeMaps.java |  209 ---
 .../apache/cassandra/locator/TokenMetadata.java |   61 +-
 .../apache/cassandra/service/StartupChecks.java |9 +-
 .../cassandra/service/StorageService.java   |2 +-
 .../test/microbench/PendingRangesBench.java |   89 --
 .../org/apache/cassandra/cql3/ViewTest.java |   30 -
 .../cassandra/db/compaction/TTLExpiryTest.java  |   47 +-
 .../cassandra/locator/PendingRangeMapsTest.java |   78 --
 .../cassandra/service/ClientWarningsTest.java   |6 +-
 tools/bin/cassandra-stress.bat  |2 +-
 28 files changed, 1226 insertions(+), 995 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/65b1f2da/CHANGES.txt
--

[46/50] [abbrv] cassandra git commit: Match cassandra-loader options in COPY FROM (3.2 version)

[slebresne]

http://git-wip-us.apache.org/repos/asf/cassandra/blob/85b8d02a/pylib/cqlshlib/copyutil.py
--
diff --git a/pylib/cqlshlib/copyutil.py b/pylib/cqlshlib/copyutil.py
index d7eff9a..0e973cc 100644
--- a/pylib/cqlshlib/copyutil.py
+++ b/pylib/cqlshlib/copyutil.py
@@ -14,12 +14,14 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
+import ConfigParser
 import csv
+import datetime
 import json
+import glob
 import multiprocessing as mp
 import os
 import Queue
-import random
 import re
 import struct
 import sys
@@ -46,82 +48,203 @@ from displaying import NO_COLOR_MAP
 from formatting import format_value_default, DateTimeFormat, EMPTY, 
get_formatter
 from sslhandling import ssl_settings
 
+CopyOptions = namedtuple('CopyOptions', 'copy dialect unrecognized')
 
-def parse_options(shell, opts):
-"""
-Parse options for import (COPY FROM) and export (COPY TO) operations.
-Extract from opts csv and dialect options.
 
-:return: 3 dictionaries: the csv options, the dialect options, any 
unrecognized options.
-"""
-dialect_options = shell.csv_dialect_defaults.copy()
-if 'quote' in opts:
-dialect_options['quotechar'] = opts.pop('quote')
-if 'escape' in opts:
-dialect_options['escapechar'] = opts.pop('escape')
-if 'delimiter' in opts:
-dialect_options['delimiter'] = opts.pop('delimiter')
-if dialect_options['quotechar'] == dialect_options['escapechar']:
-dialect_options['doublequote'] = True
-del dialect_options['escapechar']
-
-csv_options = dict()
-csv_options['nullval'] = opts.pop('null', '')
-csv_options['header'] = bool(opts.pop('header', '').lower() == 'true')
-csv_options['encoding'] = opts.pop('encoding', 'utf8')
-csv_options['maxrequests'] = int(opts.pop('maxrequests', 6))
-csv_options['pagesize'] = int(opts.pop('pagesize', 1000))
-# by default the page timeout is 10 seconds per 1000 entries in the page 
size or 10 seconds if pagesize is smaller
-csv_options['pagetimeout'] = int(opts.pop('pagetimeout', max(10, 10 * 
(csv_options['pagesize'] / 1000
-csv_options['maxattempts'] = int(opts.pop('maxattempts', 5))
-csv_options['dtformats'] = DateTimeFormat(opts.pop('timeformat', 
shell.display_timestamp_format),
-  shell.display_date_format,
-  shell.display_nanotime_format)
-csv_options['float_precision'] = shell.display_float_precision
-csv_options['chunksize'] = int(opts.pop('chunksize', 1000))
-csv_options['ingestrate'] = int(opts.pop('ingestrate', 10))
-csv_options['maxbatchsize'] = int(opts.pop('maxbatchsize', 20))
-csv_options['minbatchsize'] = int(opts.pop('minbatchsize', 2))
-csv_options['reportfrequency'] = float(opts.pop('reportfrequency', 0.25))
-csv_options['ttl'] = int(opts.pop('ttl', -1))
-
-return csv_options, dialect_options, opts
-
-
-def get_num_processes(cap):
+def safe_normpath(fname):
 """
-Pick a reasonable number of child processes. We need to leave at
-least one core for the parent process.  This doesn't necessarily
-need to be capped, but 4 is currently enough to keep
-a single local Cassandra node busy so we use this for import, whilst
-for export we use 16 since we can connect to multiple Cassandra nodes.
-Eventually this parameter will become an option.
+:return the normalized path but only if there is a filename, we don't want 
to convert
+an empty string (which means no file name) to a dot. Also expand any user 
variables such as ~ to the full path
 """
-try:
-return max(1, min(cap, mp.cpu_count() - 1))
-except NotImplementedError:
-return 1
+return os.path.normpath(os.path.expanduser(fname)) if fname else fname
 
 
 class CopyTask(object):
 """
 A base class for ImportTask and ExportTask
 """
-def __init__(self, shell, ks, cf, columns, fname, csv_options, 
dialect_options, protocol_version, config_file):
+def __init__(self, shell, ks, table, columns, fname, opts, 
protocol_version, config_file, direction):
 self.shell = shell
-self.csv_options = csv_options
-self.dialect_options = dialect_options
 self.ks = ks
-self.cf = cf
-self.columns = shell.get_column_names(ks, cf) if columns is None else 
columns
-self.fname = fname
+self.table = table
+self.local_dc = shell.conn.metadata.get_host(shell.hostname).datacenter
+self.fname = safe_normpath(fname)
 self.protocol_version = protocol_version
 self.config_file = config_file
+# do not display messages when exporting to STDOUT
+self.printmsg = self._printmsg if self.fname is not None or direction 
== 'in' else lambda _, eol='\n': None
+self.options = 

[38/50] [abbrv] cassandra git commit: Merge commit 'f288387917fd787f8476c2bca7dfb9d7892c24e8' into cassandra-3.2

[slebresne]

Merge commit 'f288387917fd787f8476c2bca7dfb9d7892c24e8' into cassandra-3.2


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/08acfe11
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/08acfe11
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/08acfe11

Branch: refs/heads/trunk
Commit: 08acfe11725bf56aff5cf98d6ad3129367c9da6a
Parents: e2c6341 f288387
Author: Sylvain Lebresne 
Authored: Wed Jan 6 17:59:35 2016 +0100
Committer: Sylvain Lebresne 
Committed: Wed Jan 6 17:59:35 2016 +0100

--

--

[24/50] [abbrv] cassandra git commit: Match cassandra-loader options in COPY FROM (3.0 version)

[slebresne]

http://git-wip-us.apache.org/repos/asf/cassandra/blob/f2883879/pylib/cqlshlib/formatting.py
--
diff --git a/pylib/cqlshlib/formatting.py b/pylib/cqlshlib/formatting.py
index 62ecd10..2e219c8 100644
--- a/pylib/cqlshlib/formatting.py
+++ b/pylib/cqlshlib/formatting.py
@@ -60,7 +60,8 @@ empty_colormap = defaultdict(lambda: '')
 
 
 def format_by_type(cqltype, val, encoding, colormap=None, addcolor=False,
-   nullval=None, date_time_format=None, float_precision=None):
+   nullval=None, date_time_format=None, float_precision=None,
+   decimal_sep=None, thousands_sep=None, boolean_styles=None):
 if nullval is None:
 nullval = default_null_placeholder
 if val is None:
@@ -75,7 +76,8 @@ def format_by_type(cqltype, val, encoding, colormap=None, 
addcolor=False,
 float_precision = default_float_precision
 return format_value(cqltype, val, encoding=encoding, colormap=colormap,
 date_time_format=date_time_format, 
float_precision=float_precision,
-nullval=nullval)
+nullval=nullval, decimal_sep=decimal_sep, 
thousands_sep=thousands_sep,
+boolean_styles=boolean_styles)
 
 
 def color_text(bval, colormap, displaywidth=None):
@@ -155,7 +157,9 @@ def format_python_formatted_type(val, colormap, color, 
quote=False):
 
 
 @formatter_for('Decimal')
-def format_value_decimal(val, colormap, **_):
+def format_value_decimal(val, float_precision, colormap, decimal_sep=None, 
thousands_sep=None, **_):
+if (decimal_sep and decimal_sep != '.') or thousands_sep:
+return format_floating_point_type(val, colormap, float_precision, 
decimal_sep, thousands_sep)
 return format_python_formatted_type(val, colormap, 'decimal')
 
 
@@ -170,34 +174,60 @@ def formatter_value_inet(val, colormap, quote=False, **_):
 
 
 @formatter_for('bool')
-def format_value_boolean(val, colormap, **_):
+def format_value_boolean(val, colormap, boolean_styles=None, **_):
+if boolean_styles:
+val = boolean_styles[0] if val else boolean_styles[1]
 return format_python_formatted_type(val, colormap, 'boolean')
 
 
-def format_floating_point_type(val, colormap, float_precision, **_):
+def format_floating_point_type(val, colormap, float_precision, 
decimal_sep=None, thousands_sep=None, **_):
 if math.isnan(val):
 bval = 'NaN'
 elif math.isinf(val):
 bval = 'Infinity' if val > 0 else '-Infinity'
 else:
-exponent = int(math.log10(abs(val))) if abs(val) > 
sys.float_info.epsilon else -sys.maxsize - 1
-if -4 <= exponent < float_precision:
-# when this is true %g will not use scientific notation,
-# increasing precision should not change this decision
-# so we increase the precision to take into account the
-# digits to the left of the decimal point
-float_precision = float_precision + exponent + 1
-bval = '%.*g' % (float_precision, val)
+if thousands_sep:
+dpart, ipart = math.modf(val)
+bval = format_integer_with_thousands_sep(ipart, thousands_sep)
+dpart_str = ('%.*f' % (float_precision, 
math.fabs(dpart)))[2:].rstrip('0')
+if dpart_str:
+bval += '%s%s' % ('.' if not decimal_sep else decimal_sep, 
dpart_str)
+else:
+exponent = int(math.log10(abs(val))) if abs(val) > 
sys.float_info.epsilon else -sys.maxsize - 1
+if -4 <= exponent < float_precision:
+# when this is true %g will not use scientific notation,
+# increasing precision should not change this decision
+# so we increase the precision to take into account the
+# digits to the left of the decimal point
+float_precision = float_precision + exponent + 1
+bval = '%.*g' % (float_precision, val)
+if decimal_sep:
+bval = bval.replace('.', decimal_sep)
+
 return colorme(bval, colormap, 'float')
 
 formatter_for('float')(format_floating_point_type)
 
 
-def format_integer_type(val, colormap, **_):
+def format_integer_type(val, colormap, thousands_sep=None, **_):
 # base-10 only for now; support others?
-bval = str(val)
+bval = format_integer_with_thousands_sep(val, thousands_sep) if 
thousands_sep else str(val)
 return colorme(bval, colormap, 'int')
 
+# We can get rid of this in cassandra-2.2
+if sys.version_info >= (2, 7):
+def format_integer_with_thousands_sep(val, thousands_sep=','):
+return "{:,.0f}".format(val).replace(',', thousands_sep)
+else:
+def format_integer_with_thousands_sep(val, thousands_sep=','):
+if val < 0:
+return '-' + format_integer_with_thousands_sep(-val, thousands_sep)
+result = ''
+while val >= 1000:
+val, r = 

[36/50] [abbrv] cassandra git commit: Merge commit 'f288387917fd787f8476c2bca7dfb9d7892c24e8' into cassandra-3.2

[slebresne]

Merge commit 'f288387917fd787f8476c2bca7dfb9d7892c24e8' into cassandra-3.2


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/08acfe11
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/08acfe11
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/08acfe11

Branch: refs/heads/cassandra-3.2
Commit: 08acfe11725bf56aff5cf98d6ad3129367c9da6a
Parents: e2c6341 f288387
Author: Sylvain Lebresne 
Authored: Wed Jan 6 17:59:35 2016 +0100
Committer: Sylvain Lebresne 
Committed: Wed Jan 6 17:59:35 2016 +0100

--

--

[26/50] [abbrv] cassandra git commit: Match cassandra-loader options in COPY FROM (3.0 version)

[slebresne]

Match cassandra-loader options in COPY FROM (3.0 version)

patch by Stefania; reviewed by pauloricardomg for CASSANDRA-9303


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/f2883879
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/f2883879
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/f2883879

Branch: refs/heads/trunk
Commit: f288387917fd787f8476c2bca7dfb9d7892c24e8
Parents: 3dbc56f
Author: Stefania Alborghetti 
Authored: Wed Jan 6 12:12:12 2016 +0100
Committer: Sylvain Lebresne 
Committed: Wed Jan 6 17:58:20 2016 +0100

--
 CHANGES.txt |1 +
 NEWS.txt|7 +
 bin/cqlsh.py|  135 +-
 conf/cqlshrc.sample |   17 +-
 pylib/cqlshlib/copyutil.py  | 1260 +-
 pylib/cqlshlib/formatting.py|   96 +-
 .../cql3/statements/BatchStatement.java |   26 +-
 .../cassandra/service/ClientWarningsTest.java   |6 +-
 tools/bin/cassandra-stress.bat  |2 +-
 9 files changed, 1134 insertions(+), 416 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/f2883879/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index cf872d9..3cb0b1e 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -25,6 +25,7 @@ Merged from 2.2:
  * Disable reloading of GossipingPropertyFileSnitch (CASSANDRA-9474)
  * Verify tables in pseudo-system keyspaces at startup (CASSANDRA-10761)
 Merged from 2.1:
+ * Match cassandra-loader options in COPY FROM (CASSANDRA-9303)
  * Fix binding to any address in CqlBulkRecordWriter (CASSANDRA-9309)
  * cqlsh fails to decode utf-8 characters for text typed columns 
(CASSANDRA-10875)
  * Log error when stream session fails (CASSANDRA-9294)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/f2883879/NEWS.txt
--
diff --git a/NEWS.txt b/NEWS.txt
index 26a83a9..64837e1 100644
--- a/NEWS.txt
+++ b/NEWS.txt
@@ -383,6 +383,13 @@ Upgrading
  to exclude data centers when the global status is enabled, see 
CASSANDRA-9035 for details.
 
 
+2.1.13
+==
+
+New features
+
+- New options for cqlsh COPY FROM and COPY TO, see CASSANDRA-9303 for 
details.
+
 2.1.10
 =
 

http://git-wip-us.apache.org/repos/asf/cassandra/blob/f2883879/bin/cqlsh.py
--
diff --git a/bin/cqlsh.py b/bin/cqlsh.py
index d8e8db5..2f53cc1 100644
--- a/bin/cqlsh.py
+++ b/bin/cqlsh.py
@@ -41,7 +41,6 @@ import optparse
 import os
 import platform
 import sys
-import time
 import traceback
 import warnings
 import webbrowser
@@ -152,7 +151,8 @@ cqlshlibdir = os.path.join(CASSANDRA_PATH, 'pylib')
 if os.path.isdir(cqlshlibdir):
 sys.path.insert(0, cqlshlibdir)
 
-from cqlshlib import cql3handling, cqlhandling, copyutil, pylexotron, 
sslhandling
+from cqlshlib import cql3handling, cqlhandling, pylexotron, sslhandling
+from cqlshlib.copyutil import ExportTask, ImportTask
 from cqlshlib.displaying import (ANSI_RESET, BLUE, COLUMN_NAME_COLORS, CYAN,
  RED, FormattedValue, colorme)
 from cqlshlib.formatting import (DEFAULT_DATE_FORMAT, DEFAULT_NANOTIME_FORMAT,
@@ -458,10 +458,12 @@ def complete_copy_column_names(ctxt, cqlsh):
 return set(colnames[1:]) - set(existcols)
 
 
-COPY_COMMON_OPTIONS = ['DELIMITER', 'QUOTE', 'ESCAPE', 'HEADER', 'NULL',
-   'MAXATTEMPTS', 'REPORTFREQUENCY']
-COPY_FROM_OPTIONS = ['CHUNKSIZE', 'INGESTRATE', 'MAXBATCHSIZE', 'MINBATCHSIZE']
-COPY_TO_OPTIONS = ['ENCODING', 'TIMEFORMAT', 'PAGESIZE', 'PAGETIMEOUT', 
'MAXREQUESTS']
+COPY_COMMON_OPTIONS = ['DELIMITER', 'QUOTE', 'ESCAPE', 'HEADER', 'NULL', 
'DATETIMEFORMAT',
+   'MAXATTEMPTS', 'REPORTFREQUENCY', 'DECIMALSEP', 
'THOUSANDSSEP', 'BOOLSTYLE',
+   'NUMPROCESSES', 'CONFIGFILE', 'RATEFILE']
+COPY_FROM_OPTIONS = ['CHUNKSIZE', 'INGESTRATE', 'MAXBATCHSIZE', 
'MINBATCHSIZE', 'MAXROWS',
+ 'SKIPROWS', 'SKIPCOLS', 'MAXPARSEERRORS', 
'MAXINSERTERRORS', 'ERRFILE']
+COPY_TO_OPTIONS = ['ENCODING', 'PAGESIZE', 'PAGETIMEOUT', 'BEGINTOKEN', 
'ENDTOKEN', 'MAXOUTPUTSIZE', 'MAXREQUESTS']
 
 
 @cqlsh_syntax_completer('copyOption', 'optnames')
@@ -581,23 +583,6 @@ warnings.showwarning = show_warning_without_quoting_line
 warnings.filterwarnings('always', 
category=cql3handling.UnexpectedTableStructure)
 
 
-def describe_interval(seconds):
-desc = []
-for length, unit in ((86400, 'day'), (3600, 'hour'), (60, 'minute')):
-num